macOS namespace/setuid container issues

[LawrenceHunter]

Builds do not work on macOS inside containers, specifically when going cross-arch through the Podman VM.

Rootful VM

# Install brew, podman, & podman-compose
podman machine init --rootful
podman machine start
podman run --privileged --rm tonistiigi/binfmt --install all

podman run -it \
    --privileged \
    --platform=linux/amd64 \
    --volume ~/.cache/buildstream:/root/.cache/buildstream \
    --volume $(pwd):/src \
    --security-opt apparmor=unconfined \
    --security-opt seccomp=unconfined \
    --workdir /src \
    bst:latest \
    /bin/bash -c "\
        source /myenv/bin/activate && \
        bst build gnomeos/live-image.bst && \
        bst artifact checkout gnomeos/live-image.bst --directory ./iso \
    "

>  bwrap: Creating new namespace failed, likely because the kernel does not support user namespaces.  bwrap must be installed setuid on such systems.

Rootless VM

# Install brew, podman, & podman-compose
podman machine init 
podman machine init
podman machine start
podman machine ssh
$ sudo -i
$ rpm-ostree install qemu-user-static
$ systemctl reboot

podman run -it \
    --privileged \
    --platform=linux/amd64 \
    --volume ~/.cache/buildstream:/root/.cache/buildstream \
    --volume $(pwd):/src \
    --security-opt apparmor=unconfined \
    --security-opt seccomp=unconfined \
    --userns=keep-id \
    --workdir /src \
    bst:latest \
    /bin/bash -c "\
        source /myenv/bin/activate && \
        bst build gnomeos/live-image.bst && \
        bst artifact checkout gnomeos/live-image.bst --directory ./iso \
    "

> bwrap: Unexpected capabilities but not setuid, old file caps config?

[abderrahim]

Just a passing comment: When using qemu-user, I would recommend installing native buildstream, and instructing it to build for the requested architecture. This way you only incur the translation overhead in the builds, not when running buildstream itself.

(this might actually fix your issue if it is related to the fact that bubblewrap doesn't work correctly under qemu-user)

[LawrenceHunter]

Just a passing comment: When using qemu-user, I would recommend installing native buildstream, and instructing it to build for the requested architecture. This way you only incur the translation overhead in the builds, not when running buildstream itself.

(this might actually fix your issue if it is related to the fact that bubblewrap doesn't work correctly under qemu-user)

Unfortunately it is not our decision and this is a request from a client

[LawrenceHunter]

Seems to be caused by containers/bubblewrap#380

[LawrenceHunter]

I continued investigating this and want to summarise my opinion:

The primary issue is a result of the use of bubblewrap (bwrap) within Buildstream. bwrap is used on build steps to sandbox the build environment and to achieve the sandboxing it uses kernel namespaces. Using namespaces is pretty much a non-issue when working on a traditional Linux environment where a privileged container can be run.

When running Buildstream on macOS it is not possible to do so natively due to buildbox-casd which cannot be built for macOS so some degree of virtualisation is required. A full Linux VM like QEMU or parallels essentially reverts back to a traditional Linux environment so is not an issue; however, the situation is more complex when trying to use containerisation tools such as Docker or Podman. Both of these containerisation tools use a virtual machine in the background to behave as the kernel on non-Linux platforms (e.g. Windows and Mac). This virtual machine is accessible and for Podman is a fedora VM with namespacing. However, when I tried to run Buildstream inside a Podman container no combination of --userns, --user, --cap-add, --privileged, and rootful or rootless machines would allow bwrap to work. The combinations produced two results 1) bwrap exits with an error that the kernel does not support namespaces, 2) bwrap exits that it's capabilities are incorrect and that setuid is not correct. The latter is linked to containers/bubblewrap#380 but the suggestion to check CapAmb did not make a difference for me. I am unsure exactly why despite the Podman VM appearing to support namespacing and capabilities as if it was a normal Linux kernel it was not possible to correctly configure for bwrap to run. I suspect that it is linked to Apple's container framework which is a clear difference between Linux and macOS Podman/Docker; therefore, these is an issue here whether that is incorrect configuration, a bug, or a hard limit.

I do not have any more time currently to investigate this problem but my next steps would be to investigate the fedora container used by Podman machine for it's configuration and if swapping to a similarly modified image from another distro or modification to the fedora image yielded success. Otherwise, I would begin to dive Podman code to understand what happens with namespacing on macOS as unfortunately the internet and docs are fairly sparse on the specifics of the Podman VM interface.

[abderrahim]

When running Buildstream on macOS it is not possible to do so natively due to buildbox-casd which cannot be built for macOS so some degree of virtualisation is required.

I'm pretty sure buildbox-casd works fine on macOS. You probably wanted to refer to bubblewrap here (or buildbox-run-bubblewrap).

Also I suppose the desire to build for Linux is the main reason for using a Linux VM / container in the first place, right?

[LawrenceHunter]

When running Buildstream on macOS it is not possible to do so natively due to buildbox-casd which cannot be built for macOS so some degree of virtualisation is required.

I'm pretty sure buildbox-casd works fine on macOS. You probably wanted to refer to bubblewrap here (or buildbox-run-bubblewrap).

Also I suppose the desire to build for Linux is the main reason for using a Linux VM / container in the first place, right?

Do you have a resource on buildbox-casd for macOS, or running buildstream natively in general? It was not present in homebrew and their repository did not provide instructions for building it.

The desire is to just allow buildstream generally on macOS without a full VM, the limitations of only Linux is a hindrance to adoption as it doesn't map well to organisations.

[abderrahim]

Nothing special. I just built buildbox-casd from sources the same way as on Linux (after installing the dependencies from homebrew). I could also install BuildStream using pip3. With a small change (setting build-os in the sandbox configuration) I built a few elements from freedesktop-sdk using remote execution.