shutil error while copying( fifo_console_send, fifo_console_recv, fifo_commands) in py/tests/test_cleanup_build.py

[Adnan-Elhammoudi]

 ## test/py/tests/test_bind.py ss                                                                     [  0%]
│   │   │   │    ## test/py/tests/test_bootmenu.py s                                                                  [  1%]
│   │   │   │    ## test/py/tests/test_button.py ss                                                                   [  2%]
│   │   │   │    ## test/py/tests/test_cleanup_build.py EE                                                            [  3%]
│   │   │   │    ## test/py/tests/test_event_dump.py s                                                                [  3%]
│   │   │   │    ## test/py/tests/test_extension.py s                                                                 [  4%]
│   │   │   │    ## test/py/tests/test_fit.py s                                                                       [  4%]
│   │   │   │    ## test/py/tests/test_fit_auto_signed.py s                                                           [  5%]
│   │   │   │    ## test/py/tests/test_fit_ecdsa.py s                                                                 [  5%]
│   │   │   │    ## test/py/tests/test_fpga.py ssssssssssssssss                                                       [ 13%]
│   │   │   │    ## test/py/tests/test_handoff.py s                                                                   [ 14%]
│   │   │   │    ## test/py/tests/test_kconfig.py ss                                                                  [ 15%]
│   │   │   │    ## test/py/tests/test_log.py ss                                                                      [ 16%]
│   │   │   │    ## test/py/tests/test_lsblk.py s                                                                     [ 16%]
│   │   │   │    ## test/py/tests/test_of_migrate.py sss                                                              [ 17%]
│   │   │   │    ## test/py/tests/test_ofplatdata.py s                                                                [ 18%]
│   │   │   │    ## test/py/tests/test_pstore.py sss                                                                  [ 19%]
│   │   │   │    ## test/py/tests/test_qfw.py ss                                                                      [ 20%]
│   │   │   │    ## test/py/tests/test_sandbox_exit.py ssss                                                           [ 22%]
│   │   │   │    ## test/py/tests/test_scp03.py s                                                                     [ 23%]
│   │   │   │    ## test/py/tests/test_source.py s                                                                    [ 23%]
│   │   │   │    ## test/py/tests/test_spl.py ss                                                                      [ 24%]
│   │   │   │    ## test/py/tests/test_stackprotector.py s                                                            [ 25%]
│   │   │   │    ## test/py/tests/test_tpm2.py sssssssssss                                                            [ 30%]
│   │   │   │    ## test/py/tests/test_trace.py s                                                                     [ 31%]
│   │   │   │    ## test/py/tests/test_ut.py s                                                                        [ 31%]
│   │   │   │    ## test/py/tests/test_vbe.py s                                                                       [ 32%]
│   │   │   │    ## test/py/tests/test_vbe_vpl.py s                                                                   [ 32%]
│   │   │   │    ## test/py/tests/test_vboot.py ssssssssssssssssssssssssssssss                                        [ 47%]
│   │   │   │    ## test/py/tests/test_vpl.py s                                                                       [ 47%]
│   │   │   │    ## test/py/tests/test_android/test_ab.py s                                                           [ 48%]
│   │   │   │    ## test/py/tests/test_android/test_abootimg.py ss                                                    [ 49%]
│   │   │   │    ## test/py/tests/test_android/test_avb.py ssssss                                                     [ 51%]
│   │   │   │    ## test/py/tests/test_cat/test_cat.py s                                                              [ 52%]
│   │   │   │    ## test/py/tests/test_fs/test_basic.py sssssssssssssssssssssssssssssssssssssss                       [ 71%]
│   │   │   │    ## test/py/tests/test_fs/test_erofs.py s                                                             [ 71%]
│   │   │   │    ## test/py/tests/test_fs/test_ext.py ssssssssssssssssssssssss                                        [ 83%]
│   │   │   │    ## test/py/tests/test_fs/test_mkdir.py ssssssssssss                                                  [ 89%]
│   │   │   │    ## test/py/tests/test_fs/test_symlink.py ssss                                                        [ 91%]
│   │   │   │    ## test/py/tests/test_fs/test_unlink.py ssssssssssssss                                               [ 98%]
│   │   │   │    ## test/py/tests/test_fs/test_squashfs/test_sqfs_load.py s                                           [ 98%]
│   │   │   │    ## test/py/tests/test_fs/test_squashfs/test_sqfs_ls.py s                                             [ 99%]
│   │   │   │    ## test/py/tests/test_semihosting/test_hostfs.py s                                                   [ 99%]
│   │   │   │    ## test/py/tests/test_xxd/test_xxd.py s                                                              [100%]
│   │   │   │    ## 
│   │   │   │    ## ================================================ ERRORS =================================================
│   │   │   │    ## _____________________________________ ERROR at setup of test_clean ______________________________________
│   │   │   │    ## test/py/tests/test_cleanup_build.py:20: in tmp_copy_of_builddir
│   │   │   │    ##     shutil.copytree(
│   │   │   │    ## /usr/lib/python3.8/shutil.py:557: in copytree
│   │   │   │    ##     return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
│   │   │   │    ## /usr/lib/python3.8/shutil.py:513: in _copytree
│   │   │   │    ##     raise Error(errors)
│   │   │   │    ## E   shutil.Error: [('./uboot-testpy-tbot/fifo_commands', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_clean0/uboot-testpy-tbot/fifo_commands', '`./uboot-testpy-tbot/fifo_commands` is a named pipe'), ('./uboot-testpy-tbot/fifo_console_send', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_clean0/uboot-testpy-tbot/fifo_console_send', '`./uboot-testpy-tbot/fifo_console_send` is a named pipe'), ('./uboot-testpy-tbot/fifo_console_recv', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_clean0/uboot-testpy-tbot/fifo_console_recv', '`./uboot-testpy-tbot/fifo_console_recv` is a named pipe')]
│   │   │   │    ## ____________________________________ ERROR at setup of test_mrproper ____________________________________
│   │   │   │    ## test/py/tests/test_cleanup_build.py:20: in tmp_copy_of_builddir
│   │   │   │    ##     shutil.copytree(
│   │   │   │    ## /usr/lib/python3.8/shutil.py:557: in copytree
│   │   │   │    ##     return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
│   │   │   │    ## /usr/lib/python3.8/shutil.py:513: in _copytree
│   │   │   │    ##     raise Error(errors)
│   │   │   │    ## E   shutil.Error: [('./uboot-testpy-tbot/fifo_commands', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_mrproper0/uboot-testpy-tbot/fifo_commands', '`./uboot-testpy-tbot/fifo_commands` is a named pipe'), ('./uboot-testpy-tbot/fifo_console_send', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_mrproper0/uboot-testpy-tbot/fifo_console_send', '`./uboot-testpy-tbot/fifo_console_send` is a named pipe'), ('./uboot-testpy-tbot/fifo_console_recv', '/tmp/pytest-of-adnanelhammoudi/pytest-6/test_mrproper0/uboot-testpy-tbot/fifo_console_recv', '`./uboot-testpy-tbot/fifo_console_recv` is a named pipe')]
│   │   │   │    ## ======================================== short test summary info ========================================
│   │   │   │    ## ERROR test/py/tests/test_cleanup_build.py::test_clean - shutil.Error: [('./uboot-testpy-tbot/fifo_comm...
│   │   │   │    ## ERROR test/py/tests/test_cleanup_build.py::test_mrproper - shutil.Error: [('./uboot-testpy-tbot/fifo_c...
│   │   │   │    ## ============================ 204 skipped, 223 deselected, 2 errors in 2.78s =============================
│   │   │   └─Fail. (3.820s)
│   │   ├─POWEROFF (imx8q)
│   │   └─Fail. (5.321s)
FAILED

======================================================================= FAILURES ========================================================================
____________________________________________________________________ test_run_testpy ____________________________________________________________________

    @tbot.testcase
    def test_run_testpy():
        with tbot.ctx() as cx:
            lb = cx.request(tbot.role.LocalHost)
            b = cx.request(tbot.role.Board)
            ub = cx.request(tbot.role.BoardUBoot, exclusive=True)
            build_dir = lb.workdir
>           uboot.testpy(
            build_dir,
            boardenv=BOARDENV,
            board=b,
            uboot=ub,
            testpy_args=["-k", "not (gpio or version or dm or efi or env or hashes or gpt or help or hush or net or md or part or pinmux or shell or sleep)"],)

test_cases/test_uboot.py:14: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.local/lib/python3.8/site-packages/tbot/decorators.py:92: in wrapped
    return tc(*args, **kwargs)
../../.local/lib/python3.8/site-packages/tbot_contrib/uboot/_testpy.py:353: in testpy
    chan_testpy.terminate0()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tbot.machine.linux.util.RunCommandProxy object at 0x7fee57663dd0>

    def terminate0(self) -> str:
        """
        Wait for the command to end **successfully**.
    
        Asserts that the command returned with retcode 0.  If it did not, an
        exception is raised.
    
        :returns: Remaining output of the command until completion.
        :rtype: str
        """
        retcode, output = self.terminate()
        if retcode != 0:
>           raise tbot.error.CommandFailure(
                self._exc_host, self._exc_args, repr=self._cmd
            )
E           tbot.error.CommandFailure: command failed: [Local] ./test/py/test.py --build-dir . --board-type tbot-imx8q -k 'not (gpio or version or dm or efi or env or hashes or gpt or help or hush or net or md or part or pinmux or shell or sleep)'

../../.local/lib/python3.8/site-packages/tbot/machine/linux/util.py:184: CommandFailure

[haxtibal]

Hi, author of test/py/tests/test_cleanup_build.py here.

This portion of your test case

            build_dir = lb.workdir
# ...
            uboot.testpy(
            build_dir,

causes that tbots hook script directory "$workdir/uboot-testpy-tbot" becomes a subdirectory of build_dir. test_cleanup_build.py recursively copies build_dir, which now accidentally contains named pipes from the hooks scripts. Named pipes can't be copied with shutil.copytree, thus the exception.

The immediate solution is to keep hook scripts out of the U-Boot build directory, e.g. by using build_dir = lb.workdir / "u-boot".

I'm afraid this won't suffice though. I wrote test_cleanup_build.py under the assumption that U-Boots test/py/test.py always has an out-of-source U-Boot build. See commit 42c0e5b for reasoning. tbot on the other hand seems to expect an in-source build. See how it calls ./test/py/test.py --build-dir .. This is incompatible, test_cleanup_build.py will fail when running against in-source builds. As both expectations are hard coded, I don't see a solution other than changing either tbot or U-Boot code.

@Rahix What do you think? Is there a way to configure an out-of-source U-Boot build directory for tbot_contrib.uboot.testpy right now? If not, would it be feasible to implement it by extending the API like uboot.testpy(source_dir, build_dir, ...)?

[Rahix]

There are a number of things here to unpack. First of all, regarding your original question @Adnan-Elhammoudi: As @haxtibal correctly identified, this section of the testcase code is not right:

            build_dir = lb.workdir
# ...
            uboot.testpy(
            build_dir,

You really should not use your U-Boot source directory as the workdir for tbot. tbot expects that it can just dump a lot of files into the workdir and that is bound to lead to problems when the workdir is really a U-Boot source tree. The structure should look like this instead, as was suggested above as well:

+-workdir/
  |
  +-u-boot/

And then your code becomes

            build_dir = lb.workdir / "u-boot"
# ...
            uboot.testpy(
            build_dir,

This should solve the problem of trying to copy FIFO files.

---

Now regarding your message @haxtibal: I think both sides need to improve here:

1. I wrote test_cleanup_build.py under the assumption that U-Boots test/py/test.py always has an out-of-source U-Boot build.

   I think this must not be assumed. I think the easiest solution is to just skip your tests when an in-source build is performed. This will allow people to run tests against a board "the easy way" (where they probably don't care about your buildsystem tests) and in the U-Boot CI, your tests continue to run normally as part of an out-of-tree build.

2. If not, would it be feasible to implement it by extending the API like uboot.testpy(source_dir, build_dir, ...)?

   On the other hand I agree that this needs to happen just as much so people can use the test/py wrapper in environments where they build out-of-tree.

[Rahix]

I've opened #108 for the out-of-tree support in contrib.uboot.testpy.

[Adnan-Elhammoudi]

Thank you for your reply @haxtibal @Rahix

I have changed the working directory as suggested, and now I am getting an assert error for test_clean and test_mrproper below.

My problem has been extended because certain tests, located in the test/py/tests/ directory, are hanging and producing an error that says "channel is currently borrowed by another machine," which is described in issue #106 . I have excluded a list of the tests that are causing this issue, and I have excluded them from further testing. I noticed that most of these tests are utilizing the boardenv.

...boot/scene_internal.h, net/arp.h, net/wol.h, net/bootp.h, net/eth_internal.h, net/dhcpv6.h, net/cdp.h, net/link_local.h, net/net_rand.h, net/rarp.h, net/nfs.h, net/ping.h, net/dns.h
│   │   │   │    ## E   assert not [PosixPath('api/api.c'), PosixPath('api/api_net.c'), PosixPath('api/api_platform-powerpc.c'), PosixPath('api/api_platform-mips.c'), PosixPath('api/api_platform-arm.c'), PosixPath('api/api_storage.c'), ...]
│   │   │   │    ## ----------------------------------------- Captured stdout call ------------------------------------------
│   │   │   │    ## +make O=/tmp/pytest-of-adnanelhammoudi/pytest-33/test_mrproper0 mrproper
│   │   │   │    ## make[1]: Entering directory '/tmp/pytest-of-adnanelhammoudi/pytest-33/test_mrproper0'
│   │   │   │    ##   CLEAN   dts/../arch/arm/dts
│   │   │   │    ##   CLEAN   dts
│   │   │   │    ##   CLEAN   lib/efi_loader
│   │   │   │    ##   CLEAN   tools
│   │   │   │    ##   CLEAN   tools/generated
│   │   │   │    ##   CLEAN   include/autoconf.mk include/autoconf.mk.dep include/bmp_logo.h include/bmp_logo_data.h include/config.h include/generated/env.in include/generated/env.txt drivers/video/u_boot_logo.S u-boot-nodtb.bin u-boot.lds u-boot.map u-boot.cfg u-boot.bin u-boot.srec u-boot u-boot.sym System.map
│   │   │   │    ##   CLEAN   scripts/basic
│   │   │   │    ##   CLEAN   scripts/dtc
│   │   │   │    ##   CLEAN   scripts/kconfig
│   │   │   │    ##   CLEAN   include/config include/generated
│   │   │   │    ##   CLEAN   .config .config.old
│   │   │   │    ## make[1]: Leaving directory '/tmp/pytest-of-adnanelhammoudi/pytest-33/test_mrproper0'
│   │   │   │    ## ======================================== short test summary info ========================================
│   │   │   │    ## FAILED test/py/tests/test_cleanup_build.py::test_clean - AssertionError: leftovers: api/api.c, api/api...
│   │   │   │    ## FAILED test/py/tests/test_cleanup_build.py::test_mrproper - AssertionError: leftovers: api/api.c, api/...
│   │   │   │    ## ============================ 2 failed, 204 skipped, 223 deselected in 9.05s =============================
│   │   │   └─Fail. (9.976s)
│   │   ├─POWEROFF (imx8q)
│   │   └─Fail. (11.495s)
FAILED

[haxtibal]

@Rahix

I think this must not be assumed. I think the easiest solution is to just skip your tests when an in-source build is performed. This will allow people to run tests against a board "the easy way" (where they probably don't care about your buildsystem tests) and in the U-Boot CI, your tests continue to run normally as part of an out-of-tree build.

Agreed. Just submitted a patch to U-Boot.

@Adnan-Elhammoudi

now I am getting an assert error for test_clean and test_mrproper below.

Yes, that's what I've said above with "I'm afraid this won't suffice though [...]". Your options as of now are

• Use contrib: u-boot: testpy: Add support for out-of-tree U-Boot builds #109 and specify an out of tree build.
• Use this U-Boot patch, or wait until it's merged upstream.
• Explicitly skip cleanup tests with pytest args -k not (test_clean or test_mrproper).

certain tests [...] are hanging and producing an error that says "channel is currently borrowed by another machine,"

I've not looked into it, but my guess would be it's unrelated to the build directory stuff we just discussed here in #107.

[Rahix]

I've not looked into it, but my guess would be it's unrelated to the build directory stuff we just discussed here in #107.

This is true.