[Question]: Avoid retrying `nvidia-ctk cdi generate` on platforms without Nvidia GPUs

[fbq]

I have a scenario where we build the same Linux image for both platforms with and without Nvidia GPUs, because we want to avoid duplication. But since 4335df0, IIUC, the nvidia-cdi-refresh.service will keep loading Nvidia drivers if there is no Nvidia GPU and generate a lot of error logs that noise out meaningful information in the logs. Would there be a way for this "Restart" behavior to be configurable? Via a config or ENV? Thanks!

[elezar]

@fbq does the image also have the NVIDIA driver modules available and nvidia-smi installed on non-NIVDIA platforms?

With that said, given #1611 using retries may have been the incorrect mechanism to ensure that we're able to generate CDI specs.

Does adding After=multi-user.target to the nvidia-cdi-refresh.service (and removing the restart configuration):

diff --git a/deployments/systemd/nvidia-cdi-refresh.service b/deployments/systemd/nvidia-cdi-refresh.service
index ece65e5f..eff939a6 100644
--- a/deployments/systemd/nvidia-cdi-refresh.service
+++ b/deployments/systemd/nvidia-cdi-refresh.service
@@ -17,9 +17,7 @@ Description=Refresh NVIDIA CDI specification file
 ConditionPathExists=|/usr/bin/nvidia-smi
 ConditionPathExists=|/usr/sbin/nvidia-smi
 ConditionPathExists=/usr/bin/nvidia-ctk
-# Limit the number of successive restarts to 5 in 10 seconds.
-StartLimitBurst=5
-StartLimitIntervalSec=10s
+After=multi-user.target
 
 [Service]
 Type=oneshot
@@ -29,10 +27,6 @@ EnvironmentFile=-/etc/nvidia-container-toolkit/nvidia-cdi-refresh.env
 ExecCondition=/usr/bin/grep -qE '/(nvidia|nvidia-current)\\.ko' /lib/modules/%v/modules.dep
 ExecStart=/usr/bin/nvidia-ctk cdi generate
 CapabilityBoundingSet=CAP_SYS_MODULE CAP_SYS_ADMIN CAP_MKNOD
-# We set the service to restart on failure to ensure that a CDI spec is
-# eventually generated.
-Restart=on-failure
-RestartSec=1s
 
 [Install]
 WantedBy=multi-user.target

fix the thrashing on the non-NVIDIA systems while ensuring that the CDI specs are successfully configured on the systems with the NVIDIA driver installed?

[fbq]

Thanks for the reply!

@fbq does the image also have the NVIDIA driver modules available and nvidia-smi installed on non-NIVDIA platforms?

Yes, we have nvidia-smi installed for non-NVIDIA platforms as well.

With that said, given #1611 using retries may have been the incorrect mechanism to ensure that we're able to generate CDI specs.

Does adding After=multi-user.target to the nvidia-cdi-refresh.service (and removing the restart configuration):

diff --git a/deployments/systemd/nvidia-cdi-refresh.service b/deployments/systemd/nvidia-cdi-refresh.service
index ece65e5f..eff939a6 100644
--- a/deployments/systemd/nvidia-cdi-refresh.service
+++ b/deployments/systemd/nvidia-cdi-refresh.service
@@ -17,9 +17,7 @@ Description=Refresh NVIDIA CDI specification file
 ConditionPathExists=|/usr/bin/nvidia-smi
 ConditionPathExists=|/usr/sbin/nvidia-smi
 ConditionPathExists=/usr/bin/nvidia-ctk
-# Limit the number of successive restarts to 5 in 10 seconds.
-StartLimitBurst=5
-StartLimitIntervalSec=10s
+After=multi-user.target
 
 [Service]
 Type=oneshot
@@ -29,10 +27,6 @@ EnvironmentFile=-/etc/nvidia-container-toolkit/nvidia-cdi-refresh.env
 ExecCondition=/usr/bin/grep -qE '/(nvidia|nvidia-current)\\.ko' /lib/modules/%v/modules.dep
 ExecStart=/usr/bin/nvidia-ctk cdi generate
 CapabilityBoundingSet=CAP_SYS_MODULE CAP_SYS_ADMIN CAP_MKNOD
-# We set the service to restart on failure to ensure that a CDI spec is
-# eventually generated.
-Restart=on-failure
-RestartSec=1s
 
 [Install]
 WantedBy=multi-user.target

fix the thrashing on the non-NVIDIA systems while ensuring that the CDI specs are successfully configured on the systems with the NVIDIA driver installed?

I think it'll work, but we have a lot of instances and we need to update their image frequently. While I can always have a script to apply this modification, it's still fragile to maintain, hence the question here.

[elezar]

I think it'll work, but we have a lot of instances and we need to update their image frequently. While I can always have a script to apply this modification, it's still fragile to maintain, hence the question here.

I understand that. If I could get a signal from you that the changes allow CDI specs to be generated on a system with NVIDIA GPUs as well as fix the restart issue on systems without NVIDIA GPUs then we can consider changing the default implementation to that (pending further testing on some of our own internal use cases).

[elezar]

@fbq I have removed the restarts from the nvidia-cdi-refresh.service in the v1.19.0-rc.3 release. It's available from our experiemental repos:

sudo apt-get update && sudo apt-get install -y --no-install-recommends \
   curl \
   gnupg2

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/experimental/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.19.0~rc.3-1
    sudo apt-get install -y \
        nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
        nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
        libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
        libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

If you're able to confirm that this solves at least some of your problems, that would be great. (We could use this as a signal to backport the changes to an upcoming v1.18.3 release).

[fbq]

@elezar Thank you! I can confirm this fixed our issue (and also for our NV machines, nvidia-cdi-refresh continues to work).