`enroot start` fails with `mount: drop permissions failed.`

[krono]

---

version: 4.1.0
hardened: yes
os: Ubuntu 24.04.3
kernel: 6.8.0-100

Using the overlayfs capabilties fails with mount: drop permissions failed. when enroot tries mounting the overlayfs
in https://github.com/NVIDIA/enroot/blob/main/src/runtime.sh#L158 during enroot start <path to squashfs>.

enroot ist started as a user.

Relevant kernel tunables:

kernel.apparmor_restrict_unprivileged_userns = 0
kernel.unprivileged_userns_apparmor_policy = 1
kernel.unprivileged_userns_clone = 1

How can I debug this?

[krono]

Resorting to fuse-overlayfs works.

[flx42]

Could you share the full error log? And does it happen for all images?

[krono]

That's the only error, actually.

I have only tried with one squashfs tho.

(I can check tomorrow CET time)

[flx42]

Ah, never mind, I can reproduce the issue too.

[flx42]

Yeah so it's a check in the mount utility that escaped my testing unfortunately, sorry about that.

This change should work:

diff --git a/src/runtime.sh b/src/runtime.sh
index ac9afa7..11c6a07 100644
--- a/src/runtime.sh
+++ b/src/runtime.sh
@@ -155,7 +155,7 @@ runtime::_mount_rootfs_shim() {

     # Mount the rootfs by overlaying the image and a tmpfs directory.
     if [ -n "${ENROOT_NATIVE_OVERLAYFS-}" ]; then
-        mount -t overlay overlay -o "lowerdir=${rootfs}/lower,upperdir=${rootfs}/upper,workdir=${rootfs}/work" "${rootfs}" || exit 1
+        enroot-mount - <<< "overlay ${rootfs} overlay lowerdir=${rootfs}/lower,upperdir=${rootfs}/upper,workdir=${rootfs}/work" || exit 1
     else
         FUSE_OVERLAYFS_DISABLE_OVL_WHITEOUT=y \
         fuse-overlayfs -f -o "lowerdir=${rootfs}/lower,upperdir=${rootfs}/upper,workdir=${rootfs}/work" "${rootfs}" &

I will try to release 4.1.1 tomorrow to fix htis.

[krono]

Ah great!

Thanks for the quick response!

[krono]

The patch works for me.

Note: I now get two kernel infos:

kernel: overlayfs: fs on '/run/container/$UID/overlay/lower' does not support file handles, falling back to xino=off.
kernel: evm: overlay not supported

/run is on a tmpfs, which will be mostly the case for ENROOT_RUNTIME_DIR even if hadn't redirected it there.
Maybe xino=off is a good thing to add to the options anyways? Adding this to the list of fs-opts gets rid of both the kernel messages.

[krono]

One more thing I do not understand yet:

If I enroot start the same image two times, is the overlay mount "shared" between the containers, does the latter override the former, or are they independent?

[flx42]

Good point about xino=off, I also see the first error message you reported. I don't see overlay not supported though, even on tmpfs.

The overlay mount is only to make the filesystem writable, so two enroot start will be independent (with pyxis it would be different though).

[krono]

Thanks for the explanation!