Vulkan Not Detecting GPU Inside Container

[clee-ai]

Hi, I'm facing a similar issue as #1021 on v1.18.0. When running vulkaninfo inside the host, I can see all GPUs properly, but running vulkaninfo inside a docker container only shows the CPU, and no GPUs are passed through.

Host:

$ dpkg -l nvidia-container-toolkit
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                     Version      Architecture Description
+++-========================-============-============-=================================
ii  nvidia-container-toolkit 1.18.0-1     amd64        NVIDIA Container toolkit

$ vulkaninfo --summary
'DISPLAY' environment variable not set... skipping surface info
==========
VULKANINFO
==========

Vulkan Instance Version: 1.3.275


...

Devices:
========
GPU0:
        apiVersion         = 1.4.303
        driverVersion      = 575.57.8.0
        vendorID           = 0x10de
        deviceID           = 0x2204
        deviceType         = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
        deviceName         = NVIDIA GeForce RTX 3090
        driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
        driverName         = NVIDIA
        driverInfo         = 575.57.08
        conformanceVersion = 1.4.1.0
        deviceUUID         = b9dda6fa-5d65-79c7-bd79-07df661e567b
        driverUUID         = 5f139ea4-52d6-527b-8f33-b64f559afd6f
GPU1:
        apiVersion         = 1.4.303
        driverVersion      = 575.57.8.0
        vendorID           = 0x10de
        deviceID           = 0x2204
        deviceType         = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
        deviceName         = NVIDIA GeForce RTX 3090
        driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
        driverName         = NVIDIA
        driverInfo         = 575.57.08
        conformanceVersion = 1.4.1.0
        deviceUUID         = 69e1da8b-7a02-547d-be07-57c9c9af7b27
        driverUUID         = 5f139ea4-52d6-527b-8f33-b64f559afd6f
GPU2:
        apiVersion         = 1.4.305
        driverVersion      = 0.0.1
        vendorID           = 0x10005
        deviceID           = 0x0000
        deviceType         = PHYSICAL_DEVICE_TYPE_CPU
        deviceName         = llvmpipe (LLVM 20.1.2, 256 bits)
        driverID           = DRIVER_ID_MESA_LLVMPIPE
        driverName         = llvmpipe
        driverInfo         = Mesa 25.0.7-0ubuntu0.24.04.2 (LLVM 20.1.2)
        conformanceVersion = 1.3.1.1
        deviceUUID         = 6d657361-3235-2e30-2e37-2d3075627500
        driverUUID         = 6c6c766d-7069-7065-5555-494400000000

Running docker:

$ docker run -it --privileged --net=host --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --gpus all --runtime=nvidia nvidia/cuda:12.9.0-cudnn-devel-ubuntu24.04 bash -c "apt update && apt install vulkan-tools -y && vulkaninfo --summary"

...

ERROR: [Loader Message] Code 0 : libXext.so.6: cannot open shared object file: No such file or directory
ERROR: [Loader Message] Code 0 : loader_icd_scan: Failed loading library associated with ICD JSON libGLX_nvidia.so.0. Ignoring this JSON
'DISPLAY' environment variable not set... skipping surface info
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
==========
VULKANINFO
==========

Vulkan Instance Version: 1.3.275

...

Devices:
========
GPU0:
        apiVersion         = 1.4.305
        driverVersion      = 0.0.1
        vendorID           = 0x10005
        deviceID           = 0x0000
        deviceType         = PHYSICAL_DEVICE_TYPE_CPU
        deviceName         = llvmpipe (LLVM 20.1.2, 256 bits)
        driverID           = DRIVER_ID_MESA_LLVMPIPE
        driverName         = llvmpipe
        driverInfo         = Mesa 25.0.7-0ubuntu0.24.04.2 (LLVM 20.1.2)
        conformanceVersion = 1.3.1.1
        deviceUUID         = 6d657361-3235-2e30-2e37-2d3075627500
        driverUUID         = 6c6c766d-7069-7065-5555-494400000000

Is there something special I have to do to get vulkan to work?

[elezar]

@clee-ai could you run:

nvidia-ctk cdi generate 2>nvidia-ctk_cdi_generate.log > nvidia.yaml

and include the two generated files (nvidia-ctk_cdi_generate.log and nvidia.yaml).

It would also be useful to include more information on your system including the content and location of of nvidia_icd.json on this host.

[elezar]

As a sanity check I have run our vulkan sample container:

docker run -it --privileged --net=host --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --gpus all --runtime=nvidia ghcr.io/nvidia/k8s-samples:vulkan-86d426f7 vulkaninfo --summary

which shows the expected output here.
When running

docker run -it --privileged --net=host --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --gpus all --runtime=nvidia nvidia/cuda:12.9.0-cudnn-devel-ubuntu24.04 bash -c "apt update && apt install vulkan-tools -y && vulkaninfo --summary"

I also only see the CPU device, meaning that there is some issue with the cudnn-devel image in this case.

[elezar]

Note that you also have:

ERROR: [Loader Message] Code 0 : libXext.so.6: cannot open shared object file: No such file or directory

which seems to indicate that libext6 is not installed in the container.

With this istalled, the next error was shown:

ERROR:             loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0

which was reported as resolved by installing libegl1 here.

After also installing libegl1 in the container, I was able to run vulkaninfo successfully:

$ docker run -it --privileged --net=host \
    --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
    --gpus all --runtime=nvidia \
    nvidia/cuda:12.9.0-cudnn-devel-ubuntu24.04 \
    bash -c \
        "apt update && \
        apt install -y \
        vulkan-tools \
        libxext6 \
        libegl1 \
        && vulkaninfo --summary"

[clee-ai]

@elezar Thanks for the help!

I was able to get it working with a minimum command of:

docker run -it \
    --gpus all --runtime=nvidia \
    nvidia/cuda:12.9.0-cudnn-devel-ubuntu24.04 \
    bash -c \
        "apt update && \
        apt install -y \
        vulkan-tools \
        libxext6 \
        libegl1 \
        && vulkaninfo --summary"

It would be great to include the packages in the cuda images and/or add some documentation on vulkan requirements but it looks like it's all working well now!