Configurable behavior when resources.requests."nvidia.com/gpu" = 0 (or not set)

[tomzx]

In the documentation/README one can read:

If you do not request GPUs when you use the device plugin, the plugin exposes all the GPUs on the machine inside your container.

Would it be possible to make this behavior configurable?
As in, either all GPUs are visible by default, or none are.

Many on my team have as expectation that submit a kubernetes workload with no GPU requests should result in no GPU being visible by default.
It seems that the current implementation has decided the opposite made more sense but it's not clear to us why.

Furthermore, it appears that the DGCM exporter doesn't work properly if a container does not request GPUs and no metrics are exported.

References

• Existing similar request: Requesting zero GPUs allocates all GPUs #61

[elezar]

The issue is that if one does not request a GPU, the Device Plugin is not involved in the allocation of the container. This means that IF the nvidia container runtime is invoked (e.g. because it is the default runtime), the NVIDIA_VISIBLE_DEVICES=all envvar that is set in the container image causes all devices to be injected.

There are already a number of mechanisms to not have this be the case.

One example that has been recommended for some time is https://docs.google.com/document/d/1zy0key-EL6JH50MZgwg96RPYxxXXnVUdxLZwGiyqLd8/edit?tab=t.0. Here dummy volume mounts are used to indicate requested devices meaning that the NVIDIA_VISIBLE_DEVICES envvar is not respected.

Removing the envvar from any IMAGES (or setting NVIDIA_VISIBLE_DEVICES=void) will also prevent injection.

In the longer term, switching to CDI for explicit revice requests and using native CDI support in runtimes such as Containerd or Crio will also mean that an envvar set in an image does not have the sideeffect of injecting all devices.

[tomzx]

In our case we use nvidia container runtime as our default runtime.
My understanding is that the nvidia container runtime is injecting environment variables and that those would be overriding whatever is defined in an image, are you saying that isn't the case?

In any case, our users do not set this variable in their image so that isn't what we're concerned with.

What we'd like to have is nvidia as default container runtime with the behavior that NVIDIA_VISIBLE_DEVICES=none is set if no GPU were requested from the pod manifest standpoint.

Is this something that could be done, or Nvidia wants users to handle this another way (i.e., use volume mounts instead of environment variables)?

[cdesiniotis]

My understanding is that the nvidia container runtime is injecting environment variables and that those would be overriding whatever is defined in an image, are you saying that isn't the case?

This should not be the case. Note, the CUDA images, e.g. nvcr.io/nvidia/cuda, which are often used as a base image for applications, have NVIDIA_VISIBLE_DEVICES set to all in them. So while your users may not be explicitly setting this variable in their images, it may already be set.