test: add e2e test for NVIDIA device plugin as DaemonSet

[ganeshkumarashok]

Summary

• Add a new e2e test that validates GPU nodes work correctly when the NVIDIA device plugin is deployed as a Kubernetes DaemonSet instead of a systemd service
• This tests the upstream deployment model commonly used by customers who manage their own device plugin deployment

Test Details

The test Test_Ubuntu2204_NvidiaDevicePlugin_Daemonset:

• Provisions a GPU node (Standard_NV6ads_A10_v5) with GPU drivers enabled but systemd device plugin disabled
• Deploys nvidia-device-plugin:v0.18.2 from MCR (mcr.microsoft.com/oss/v2/nvidia/k8s-device-plugin) as a DaemonSet
• Validates:
  • GPU resources are advertised by the device plugin
  • GPU workloads can be scheduled on the node

Test plan

• CI pipeline runs the new GPU e2e test
• Verify the DaemonSet-based device plugin properly registers GPU resources
• Verify GPU workloads can be scheduled

[Copilot]

Pull request overview

This pull request adds a new e2e test (Test_Ubuntu2204_NvidiaDevicePlugin_Daemonset) that validates GPU nodes work correctly when the NVIDIA device plugin is deployed as a Kubernetes DaemonSet instead of a systemd service. This tests the upstream deployment model commonly used by customers who manage their own device plugin deployment.

Changes:

• Adds a new GPU e2e test that provisions a Standard_NV6ads_A10_v5 GPU node with drivers enabled but systemd device plugin disabled
• Deploys nvidia-device-plugin:v0.18.2 from MCR as a DaemonSet
• Validates GPU resource advertisement and workload scheduling with DaemonSet-based device plugin

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 6 comments.

[Copilot]

The wait loop checks if any containers in the pod are not ready, but it only logs which container is not ready without failing immediately on errors like CrashLoopBackOff or ImagePullBackOff. If the pod enters a failed state, the test will wait the full 3 minutes before timing out.

Consider checking pod.Status.ContainerStatuses[].State.Waiting.Reason for error states (ImagePullBackOff, CrashLoopBackOff, etc.) and returning an error immediately rather than continuing to poll. This would make test failures faster and provide clearer error messages.

[Copilot]

The SecurityContext configuration may prevent the device plugin from functioning correctly. The NVIDIA device plugin needs to interact with GPU devices and the kubelet device plugin socket, which typically requires either:

1. Privileged mode (Privileged: true), OR
2. Specific capabilities like SYS_ADMIN

The current configuration only drops all capabilities without granting the necessary ones. The upstream NVIDIA device plugin DaemonSet spec typically does NOT use AllowPrivilegeEscalation: false with dropped capabilities - it usually runs in privileged mode or with specific capabilities.

Without proper privileges, the device plugin won't be able to enumerate GPUs or create device plugin sockets. This could cause the test to fail when it tries to validate GPU resources are advertised. Verify this configuration works, or update it to match the upstream privileged configuration.

Suggested change

	AllowPrivilegeEscalation: to.Ptr(false),
	Capabilities: &corev1.Capabilities{
	Drop: []corev1.Capability{"ALL"},
	},
	Privileged: to.Ptr(true),

[Copilot]

The image tag v0.18.2 does not match the version format used in parts/common/components.json (0.18.2-ubuntu22.04u1). While MCR may support the simplified v0.18.2 tag, it's better to use the exact version that aligns with what's cached on VHDs and referenced in components.json for consistency. This ensures the test validates the same version that production nodes would use.

The components.json shows nvidia-device-plugin versions like "0.18.2-ubuntu22.04u1" for Ubuntu 22.04. Consider using the full version tag or documenting why the simplified tag is intentional.

[Copilot]

The DaemonSet is created but never explicitly cleaned up. While Kubernetes may garbage collect when the test cluster is torn down, it's better practice to add explicit cleanup to prevent resource leaks if the test fails before cluster cleanup. E2E tests should be self-contained and clean up resources they create.

Consider using s.T.Cleanup() or defer to delete the DaemonSet at the end of the validator function, similar to how other e2e tests handle resource cleanup.

Suggested change

	// Ensure the DaemonSet is cleaned up after the test finishes
	s.T.Cleanup(func() {
	s.T.Logf("Cleaning up NVIDIA device plugin DaemonSet %s/%s...", ds.Namespace, ds.Name)
	if err := s.Runtime.Cluster.Kube.Typed.AppsV1().DaemonSets(ds.Namespace).Delete(
	context.Background(),
	ds.Name,
	metav1.DeleteOptions{},
	); err != nil {
	s.T.Logf("Failed to delete NVIDIA device plugin DaemonSet %s/%s: %v", ds.Namespace, ds.Name, err)
	}
	})

[Copilot]

The DaemonSet container spec is missing several configuration options present in the upstream NVIDIA device plugin deployment (https://github.com/NVIDIA/k8s-device-plugin/blob/main/deployments/static/nvidia-device-plugin.yml). Specifically:

1. Missing the /dev/null device volume mount which the device plugin may need
2. The upstream spec typically includes additional environment variables like NVIDIA_MIG_MONITOR_DEVICES, NVIDIA_DRIVER_CAPABILITIES, etc.
3. The upstream spec sets Privileged: true in SecurityContext rather than just dropping capabilities

While the minimal spec may work for basic testing, using a configuration closer to the upstream deployment would make this test more valuable as it validates the real-world customer scenario. Consider either aligning more closely with the upstream spec or adding a comment explaining why this simplified configuration is sufficient for the test's purpose.

[Copilot]

After waiting for the device plugin pod to be ready, consider adding validation to check the pod logs for successful GPU discovery and device plugin registration. This would catch issues where the pod starts but the device plugin fails to function correctly (e.g., driver compatibility issues, incorrect configuration).

The logs should contain messages about discovering GPUs and registering with kubelet's device plugin framework. This would make the test more comprehensive and catch edge cases where the pod runs but doesn't actually register GPU resources properly.

[surajssd]

nvm