[Bug]: NRI doesn't work trying to check nvidia-smi from the validation pod instead of host

Describe the bug
When enabling NRI via --set cdi.nriPluginEnabled=true the toolkit validation pod fails with nvidia-smi not found in path.

I've looked at the code and it seems it's trying to execute nvidia-smi from inside the validation container as opposed to how driver validation validates from the host system.

See:

gpu-operator/cmd/nvidia-validator/main.go

Line 1132 in e6cd031

func (t *Toolkit) validate() error {

This executes nvidia-smi from inside the validation container and it would fail, but for driver validation

see:

gpu-operator/cmd/nvidia-validator/main.go

Line 745 in e6cd031

func validateHostDriver(silent bool) error {

Fix would be to check nvidia-smi similarly to how driver validation works

nvidia-smi is executed from the host

To Reproduce
Install helm chart with --set cdi.nriPluginEnabled=true

Expected behavior
NRI plugin works

Environment (please provide the following information):

GPU Operator Version: v26.3.1
OS: Talos 1.13.0-rc.0
Kernel Version: 6.18.22-talos
Container Runtime Version: containerd 2.2.3
Kubernetes Distro and Version: Talos v1.35.2

Information to attach (optional if deemed irrelevant)

kubernetes pods status: kubectl get pods -n OPERATOR_NAMESPACE
kubernetes daemonset status: kubectl get ds -n OPERATOR_NAMESPACE
If a pod/ds is in an error state or pending state kubectl describe pod -n OPERATOR_NAMESPACE POD_NAME
If a pod/ds is in an error state or pending state kubectl logs -n OPERATOR_NAMESPACE POD_NAME --all-containers
Output from running nvidia-smi from the driver container: kubectl exec DRIVER_POD_NAME -n OPERATOR_NAMESPACE -c nvidia-driver-ctr -- nvidia-smi
containerd logs journalctl -u containerd > containerd.log

Collecting full debug bundle (optional):

curl -o must-gather.sh -L https://raw.githubusercontent.com/NVIDIA/gpu-operator/main/hack/must-gather.sh
chmod +x must-gather.sh
./must-gather.sh

NOTE: please refer to the must-gather script for debug data collected.

This bundle can be submitted to us via email: operator_feedback@nvidia.com

Can you share more details? Which daemonset is failing here?

Please share the yaml manifest of the failing daemonset if you can.

Yes,

❯ kubectl -n gpu-operator get pods
NAME                                                          READY   STATUS       RESTARTS      AGE
gpu-feature-discovery-h9vfg                                   0/1     Init:0/1     0             46s
gpu-operator-7588c78c66-mldjn                                 1/1     Running      0             152m
gpu-operator-node-feature-discovery-gc-585b876f9c-4zrwn       1/1     Running      0             2m51s
gpu-operator-node-feature-discovery-gc-585b876f9c-fpjv7       0/1     Error        0             152m
gpu-operator-node-feature-discovery-master-7f6684fb45-bj5vk   1/1     Running      0             152m
gpu-operator-node-feature-discovery-worker-m8jdr              1/1     Running      0             152m
gpu-operator-node-feature-discovery-worker-zln7l              1/1     Running      0             115s
nvidia-cuda-validator-pzdrp                                   0/1     Completed    0             96s
nvidia-dcgm-exporter-c4856                                    0/1     Init:0/1     0             45s
nvidia-device-plugin-daemonset-cb5mw                          0/1     Init:0/1     0             47s
nvidia-operator-validator-bfmvt                               0/1     Init:Error   2 (14s ago)   17s

logs:

❯ kubectl -n gpu-operator logs nvidia-operator-validator-bfmvt --all-containers
time="2026-04-20T18:52:21Z" level=info msg="version: 5a25fef4-amd64, commit: 5a25fef"
time="2026-04-20T18:52:21Z" level=info msg="Attempting to validate a pre-installed driver on the host"
Mon Apr 20 18:52:21 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.20             Driver Version: 580.126.20     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       On  |   00000000:00:1E.0 Off |                    0 |
| N/A   25C    P8              8W /   70W |       0MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
time="2026-04-20T18:52:22Z" level=info msg="Detected a pre-installed driver on the host"
time="2026-04-20T18:52:22Z" level=info msg="creating symlinks under /dev/char that correspond to NVIDIA character devices"
time="2026-04-20T18:52:22Z" level=info msg="NVIDIA kernel module already loaded in kernel memory (refcnt=15)"
time="2026-04-20T18:52:22Z" level=info msg="Skipping: /dev/nvidiactl already exists"
time="2026-04-20T18:52:22Z" level=info msg="Skipping: /dev/nvidia-modeset already exists"
time="2026-04-20T18:52:22Z" level=info msg="Skipping: /dev/nvidia-uvm already exists"
time="2026-04-20T18:52:22Z" level=info msg="Skipping: /dev/nvidia-uvm-tools already exists"
time="2026-04-20T18:52:22Z" level=warning msg="unable to detect IOMMU FD for 0000:00:1e.0: open /sys/bus/pci/devices/0000:00:1e.0/vfio-dev: no such file or directory"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/195:254 => /dev/nvidia-modeset"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-modeset /host-dev-char/195:254: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/195:255 => /dev/nvidiactl"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidiactl /host-dev-char/195:255: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/236:0 => /dev/nvidia-uvm"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-uvm /host-dev-char/236:0: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/236:1 => /dev/nvidia-uvm-tools"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-uvm-tools /host-dev-char/236:1: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:1 => /dev/nvidia-caps/nvidia-cap1"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap1 /host-dev-char/239:1: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:2 => /dev/nvidia-caps/nvidia-cap2"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap2 /host-dev-char/239:2: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/195:0 => /dev/nvidia0"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia0 /host-dev-char/195:0: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:3 => /dev/nvidia-caps/nvidia-cap3"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap3 /host-dev-char/239:3: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:4 => /dev/nvidia-caps/nvidia-cap4"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap4 /host-dev-char/239:4: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:5 => /dev/nvidia-caps/nvidia-cap5"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap5 /host-dev-char/239:5: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:6 => /dev/nvidia-caps/nvidia-cap6"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap6 /host-dev-char/239:6: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:7 => /dev/nvidia-caps/nvidia-cap7"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap7 /host-dev-char/239:7: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:8 => /dev/nvidia-caps/nvidia-cap8"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap8 /host-dev-char/239:8: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:9 => /dev/nvidia-caps/nvidia-cap9"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap9 /host-dev-char/239:9: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:10 => /dev/nvidia-caps/nvidia-cap10"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap10 /host-dev-char/239:10: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:11 => /dev/nvidia-caps/nvidia-cap11"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap11 /host-dev-char/239:11: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:12 => /dev/nvidia-caps/nvidia-cap12"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap12 /host-dev-char/239:12: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:13 => /dev/nvidia-caps/nvidia-cap13"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap13 /host-dev-char/239:13: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:14 => /dev/nvidia-caps/nvidia-cap14"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap14 /host-dev-char/239:14: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:15 => /dev/nvidia-caps/nvidia-cap15"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap15 /host-dev-char/239:15: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:16 => /dev/nvidia-caps/nvidia-cap16"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap16 /host-dev-char/239:16: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:17 => /dev/nvidia-caps/nvidia-cap17"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap17 /host-dev-char/239:17: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:18 => /dev/nvidia-caps/nvidia-cap18"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap18 /host-dev-char/239:18: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:19 => /dev/nvidia-caps/nvidia-cap19"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap19 /host-dev-char/239:19: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:20 => /dev/nvidia-caps/nvidia-cap20"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap20 /host-dev-char/239:20: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:21 => /dev/nvidia-caps/nvidia-cap21"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap21 /host-dev-char/239:21: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:22 => /dev/nvidia-caps/nvidia-cap22"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap22 /host-dev-char/239:22: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:23 => /dev/nvidia-caps/nvidia-cap23"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap23 /host-dev-char/239:23: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:24 => /dev/nvidia-caps/nvidia-cap24"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap24 /host-dev-char/239:24: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:25 => /dev/nvidia-caps/nvidia-cap25"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap25 /host-dev-char/239:25: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:26 => /dev/nvidia-caps/nvidia-cap26"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap26 /host-dev-char/239:26: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:27 => /dev/nvidia-caps/nvidia-cap27"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap27 /host-dev-char/239:27: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:28 => /dev/nvidia-caps/nvidia-cap28"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap28 /host-dev-char/239:28: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:29 => /dev/nvidia-caps/nvidia-cap29"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap29 /host-dev-char/239:29: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:30 => /dev/nvidia-caps/nvidia-cap30"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap30 /host-dev-char/239:30: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:31 => /dev/nvidia-caps/nvidia-cap31"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap31 /host-dev-char/239:31: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:32 => /dev/nvidia-caps/nvidia-cap32"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap32 /host-dev-char/239:32: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:33 => /dev/nvidia-caps/nvidia-cap33"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap33 /host-dev-char/239:33: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:34 => /dev/nvidia-caps/nvidia-cap34"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap34 /host-dev-char/239:34: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:35 => /dev/nvidia-caps/nvidia-cap35"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap35 /host-dev-char/239:35: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:36 => /dev/nvidia-caps/nvidia-cap36"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap36 /host-dev-char/239:36: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:37 => /dev/nvidia-caps/nvidia-cap37"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap37 /host-dev-char/239:37: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:38 => /dev/nvidia-caps/nvidia-cap38"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap38 /host-dev-char/239:38: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:39 => /dev/nvidia-caps/nvidia-cap39"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap39 /host-dev-char/239:39: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:40 => /dev/nvidia-caps/nvidia-cap40"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap40 /host-dev-char/239:40: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:41 => /dev/nvidia-caps/nvidia-cap41"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap41 /host-dev-char/239:41: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:42 => /dev/nvidia-caps/nvidia-cap42"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap42 /host-dev-char/239:42: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:43 => /dev/nvidia-caps/nvidia-cap43"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap43 /host-dev-char/239:43: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:44 => /dev/nvidia-caps/nvidia-cap44"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap44 /host-dev-char/239:44: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:45 => /dev/nvidia-caps/nvidia-cap45"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap45 /host-dev-char/239:45: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:46 => /dev/nvidia-caps/nvidia-cap46"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap46 /host-dev-char/239:46: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:47 => /dev/nvidia-caps/nvidia-cap47"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap47 /host-dev-char/239:47: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:48 => /dev/nvidia-caps/nvidia-cap48"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap48 /host-dev-char/239:48: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:49 => /dev/nvidia-caps/nvidia-cap49"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap49 /host-dev-char/239:49: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:50 => /dev/nvidia-caps/nvidia-cap50"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap50 /host-dev-char/239:50: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:51 => /dev/nvidia-caps/nvidia-cap51"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap51 /host-dev-char/239:51: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:52 => /dev/nvidia-caps/nvidia-cap52"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap52 /host-dev-char/239:52: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:53 => /dev/nvidia-caps/nvidia-cap53"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap53 /host-dev-char/239:53: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:54 => /dev/nvidia-caps/nvidia-cap54"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap54 /host-dev-char/239:54: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:55 => /dev/nvidia-caps/nvidia-cap55"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap55 /host-dev-char/239:55: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:56 => /dev/nvidia-caps/nvidia-cap56"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap56 /host-dev-char/239:56: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:57 => /dev/nvidia-caps/nvidia-cap57"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap57 /host-dev-char/239:57: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:58 => /dev/nvidia-caps/nvidia-cap58"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap58 /host-dev-char/239:58: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:59 => /dev/nvidia-caps/nvidia-cap59"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap59 /host-dev-char/239:59: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:60 => /dev/nvidia-caps/nvidia-cap60"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap60 /host-dev-char/239:60: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:61 => /dev/nvidia-caps/nvidia-cap61"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap61 /host-dev-char/239:61: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:62 => /dev/nvidia-caps/nvidia-cap62"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap62 /host-dev-char/239:62: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:63 => /dev/nvidia-caps/nvidia-cap63"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap63 /host-dev-char/239:63: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:64 => /dev/nvidia-caps/nvidia-cap64"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap64 /host-dev-char/239:64: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:65 => /dev/nvidia-caps/nvidia-cap65"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap65 /host-dev-char/239:65: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:66 => /dev/nvidia-caps/nvidia-cap66"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap66 /host-dev-char/239:66: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:67 => /dev/nvidia-caps/nvidia-cap67"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap67 /host-dev-char/239:67: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:68 => /dev/nvidia-caps/nvidia-cap68"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap68 /host-dev-char/239:68: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:69 => /dev/nvidia-caps/nvidia-cap69"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap69 /host-dev-char/239:69: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:70 => /dev/nvidia-caps/nvidia-cap70"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap70 /host-dev-char/239:70: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:71 => /dev/nvidia-caps/nvidia-cap71"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap71 /host-dev-char/239:71: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:72 => /dev/nvidia-caps/nvidia-cap72"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap72 /host-dev-char/239:72: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:73 => /dev/nvidia-caps/nvidia-cap73"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap73 /host-dev-char/239:73: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:74 => /dev/nvidia-caps/nvidia-cap74"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap74 /host-dev-char/239:74: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:75 => /dev/nvidia-caps/nvidia-cap75"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap75 /host-dev-char/239:75: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:76 => /dev/nvidia-caps/nvidia-cap76"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap76 /host-dev-char/239:76: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:77 => /dev/nvidia-caps/nvidia-cap77"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap77 /host-dev-char/239:77: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:78 => /dev/nvidia-caps/nvidia-cap78"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap78 /host-dev-char/239:78: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:79 => /dev/nvidia-caps/nvidia-cap79"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap79 /host-dev-char/239:79: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:80 => /dev/nvidia-caps/nvidia-cap80"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap80 /host-dev-char/239:80: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:81 => /dev/nvidia-caps/nvidia-cap81"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap81 /host-dev-char/239:81: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:82 => /dev/nvidia-caps/nvidia-cap82"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap82 /host-dev-char/239:82: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:83 => /dev/nvidia-caps/nvidia-cap83"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap83 /host-dev-char/239:83: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:84 => /dev/nvidia-caps/nvidia-cap84"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap84 /host-dev-char/239:84: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:85 => /dev/nvidia-caps/nvidia-cap85"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap85 /host-dev-char/239:85: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:86 => /dev/nvidia-caps/nvidia-cap86"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap86 /host-dev-char/239:86: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:87 => /dev/nvidia-caps/nvidia-cap87"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap87 /host-dev-char/239:87: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:88 => /dev/nvidia-caps/nvidia-cap88"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap88 /host-dev-char/239:88: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:89 => /dev/nvidia-caps/nvidia-cap89"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap89 /host-dev-char/239:89: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:90 => /dev/nvidia-caps/nvidia-cap90"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap90 /host-dev-char/239:90: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:91 => /dev/nvidia-caps/nvidia-cap91"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap91 /host-dev-char/239:91: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:92 => /dev/nvidia-caps/nvidia-cap92"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap92 /host-dev-char/239:92: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:93 => /dev/nvidia-caps/nvidia-cap93"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap93 /host-dev-char/239:93: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:94 => /dev/nvidia-caps/nvidia-cap94"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap94 /host-dev-char/239:94: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:95 => /dev/nvidia-caps/nvidia-cap95"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap95 /host-dev-char/239:95: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:96 => /dev/nvidia-caps/nvidia-cap96"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap96 /host-dev-char/239:96: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:97 => /dev/nvidia-caps/nvidia-cap97"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap97 /host-dev-char/239:97: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:98 => /dev/nvidia-caps/nvidia-cap98"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap98 /host-dev-char/239:98: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:99 => /dev/nvidia-caps/nvidia-cap99"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap99 /host-dev-char/239:99: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:100 => /dev/nvidia-caps/nvidia-cap100"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap100 /host-dev-char/239:100: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:101 => /dev/nvidia-caps/nvidia-cap101"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap101 /host-dev-char/239:101: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:102 => /dev/nvidia-caps/nvidia-cap102"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap102 /host-dev-char/239:102: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:103 => /dev/nvidia-caps/nvidia-cap103"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap103 /host-dev-char/239:103: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:104 => /dev/nvidia-caps/nvidia-cap104"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap104 /host-dev-char/239:104: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:105 => /dev/nvidia-caps/nvidia-cap105"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap105 /host-dev-char/239:105: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:106 => /dev/nvidia-caps/nvidia-cap106"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap106 /host-dev-char/239:106: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:107 => /dev/nvidia-caps/nvidia-cap107"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap107 /host-dev-char/239:107: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:108 => /dev/nvidia-caps/nvidia-cap108"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap108 /host-dev-char/239:108: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:109 => /dev/nvidia-caps/nvidia-cap109"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap109 /host-dev-char/239:109: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:110 => /dev/nvidia-caps/nvidia-cap110"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap110 /host-dev-char/239:110: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:111 => /dev/nvidia-caps/nvidia-cap111"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap111 /host-dev-char/239:111: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:112 => /dev/nvidia-caps/nvidia-cap112"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap112 /host-dev-char/239:112: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:113 => /dev/nvidia-caps/nvidia-cap113"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap113 /host-dev-char/239:113: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:114 => /dev/nvidia-caps/nvidia-cap114"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap114 /host-dev-char/239:114: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:115 => /dev/nvidia-caps/nvidia-cap115"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap115 /host-dev-char/239:115: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:116 => /dev/nvidia-caps/nvidia-cap116"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap116 /host-dev-char/239:116: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:117 => /dev/nvidia-caps/nvidia-cap117"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap117 /host-dev-char/239:117: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:118 => /dev/nvidia-caps/nvidia-cap118"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap118 /host-dev-char/239:118: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:119 => /dev/nvidia-caps/nvidia-cap119"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap119 /host-dev-char/239:119: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:120 => /dev/nvidia-caps/nvidia-cap120"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap120 /host-dev-char/239:120: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:121 => /dev/nvidia-caps/nvidia-cap121"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap121 /host-dev-char/239:121: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:122 => /dev/nvidia-caps/nvidia-cap122"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap122 /host-dev-char/239:122: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:123 => /dev/nvidia-caps/nvidia-cap123"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap123 /host-dev-char/239:123: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:124 => /dev/nvidia-caps/nvidia-cap124"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap124 /host-dev-char/239:124: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:125 => /dev/nvidia-caps/nvidia-cap125"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap125 /host-dev-char/239:125: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:126 => /dev/nvidia-caps/nvidia-cap126"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap126 /host-dev-char/239:126: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:127 => /dev/nvidia-caps/nvidia-cap127"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap127 /host-dev-char/239:127: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:128 => /dev/nvidia-caps/nvidia-cap128"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap128 /host-dev-char/239:128: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:129 => /dev/nvidia-caps/nvidia-cap129"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap129 /host-dev-char/239:129: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:130 => /dev/nvidia-caps/nvidia-cap130"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap130 /host-dev-char/239:130: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:131 => /dev/nvidia-caps/nvidia-cap131"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap131 /host-dev-char/239:131: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:132 => /dev/nvidia-caps/nvidia-cap132"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap132 /host-dev-char/239:132: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:133 => /dev/nvidia-caps/nvidia-cap133"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap133 /host-dev-char/239:133: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:134 => /dev/nvidia-caps/nvidia-cap134"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap134 /host-dev-char/239:134: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:135 => /dev/nvidia-caps/nvidia-cap135"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap135 /host-dev-char/239:135: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:136 => /dev/nvidia-caps/nvidia-cap136"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap136 /host-dev-char/239:136: file exists"
time="2026-04-20T18:52:22Z" level=info msg="Creating link /host-dev-char/239:137 => /dev/nvidia-caps/nvidia-cap137"
time="2026-04-20T18:52:22Z" level=warning msg="Could not create symlink: symlink /dev/nvidia-caps/nvidia-cap137 /host-dev-char/239:137: file exists"
time="2026-04-20T18:53:05Z" level=info msg="version: 5a25fef4-amd64, commit: 5a25fef"
time="2026-04-20T18:53:05Z" level=info msg="Error: error validating toolkit installation: exec: \"nvidia-smi\": executable file not found in $PATH"
toolkit is not ready
Error from server (BadRequest): container "cuda-validation" in pod "nvidia-operator-validator-bfmvt" is waiting to start: PodInitializing

Others pods in init status are basically waiting for setup

waiting for nvidia container stack to be setup

Yaml dump of validator:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    nvidia.cdi.k8s.io/container.toolkit-validation: management.nvidia.com/gpu=all
  creationTimestamp: "2026-04-20T18:52:21Z"
  generateName: nvidia-operator-validator-
  generation: 1
  labels:
    app: nvidia-operator-validator
    app.kubernetes.io/managed-by: gpu-operator
    app.kubernetes.io/part-of: gpu-operator
    controller-revision-hash: 78b765688f
    helm.sh/chart: gpu-operator-v26.3.1
    pod-template-generation: "3"
  name: nvidia-operator-validator-bfmvt
  namespace: gpu-operator
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: nvidia-operator-validator
    uid: 54075051-4949-436d-901c-5eda9408d40e
  resourceVersion: "24170"
  uid: 699136b5-8fb6-4dff-a748-8dc78dbd931d
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - ip-172-16-33-136.ec2.internal
  containers:
  - args:
    - echo all validations are successful; while true; do sleep 86400; done
    command:
    - sh
    - -c
    image: nvcr.io/nvidia/gpu-operator:v26.3.1
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - sh
          - -c
          - rm -f /run/nvidia/validations/*-ready
    name: nvidia-operator-validator
    resources: {}
    securityContext:
      privileged: true
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /run/nvidia/validations
      mountPropagation: Bidirectional
      name: run-nvidia-validations
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm5p2
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - args:
    - nvidia-validator
    command:
    - sh
    - -c
    env:
    - name: WITH_WAIT
      value: "true"
    - name: COMPONENT
      value: driver
    - name: OPERATOR_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: DRIVER_INSTALL_DIR
      value: /usr/local
    - name: DRIVER_INSTALL_DIR_CTR_PATH
      value: /usr/local
    image: nvcr.io/nvidia/gpu-operator:v26.3.1
    imagePullPolicy: IfNotPresent
    name: driver-validation
    resources: {}
    securityContext:
      privileged: true
      runAsUser: 0
      seLinuxOptions:
        level: s0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /host
      mountPropagation: HostToContainer
      name: host-root
      readOnly: true
    - mountPath: /usr/local
      mountPropagation: HostToContainer
      name: driver-install-dir
    - mountPath: /run/nvidia/validations
      mountPropagation: Bidirectional
      name: run-nvidia-validations
    - mountPath: /host-dev-char
      name: host-dev-char
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm5p2
      readOnly: true
  - args:
    - nvidia-validator
    command:
    - sh
    - -c
    env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: all
    - name: WITH_WAIT
      value: "false"
    - name: COMPONENT
      value: toolkit
    image: nvcr.io/nvidia/gpu-operator:v26.3.1
    imagePullPolicy: IfNotPresent
    name: toolkit-validation
    resources: {}
    securityContext:
      privileged: true
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /run/nvidia/validations
      mountPropagation: Bidirectional
      name: run-nvidia-validations
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm5p2
      readOnly: true
  - args:
    - nvidia-validator
    command:
    - sh
    - -c
    env:
    - name: WITH_WAIT
      value: "false"
    - name: COMPONENT
      value: cuda
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: OPERATOR_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: VALIDATOR_IMAGE
      value: nvcr.io/nvidia/gpu-operator:v26.3.1
    - name: VALIDATOR_IMAGE_PULL_POLICY
      value: IfNotPresent
    image: nvcr.io/nvidia/gpu-operator:v26.3.1
    imagePullPolicy: IfNotPresent
    name: cuda-validation
    resources: {}
    securityContext:
      privileged: true
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /run/nvidia/validations
      mountPropagation: Bidirectional
      name: run-nvidia-validations
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm5p2
      readOnly: true
  - args:
    - nvidia-validator
    command:
    - sh
    - -c
    env:
    - name: COMPONENT
      value: plugin
    - name: WITH_WAIT
      value: "false"
    - name: WITH_WORKLOAD
      value: "false"
    - name: MIG_STRATEGY
      value: single
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: OPERATOR_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: VALIDATOR_IMAGE
      value: nvcr.io/nvidia/gpu-operator:v26.3.1
    - name: VALIDATOR_IMAGE_PULL_POLICY
      value: IfNotPresent
    image: nvcr.io/nvidia/gpu-operator:v26.3.1
    imagePullPolicy: IfNotPresent
    name: plugin-validation
    resources: {}
    securityContext:
      privileged: true
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /run/nvidia/validations
      mountPropagation: Bidirectional
      name: run-nvidia-validations
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm5p2
      readOnly: true
  nodeName: ip-172-16-33-136.ec2.internal
  nodeSelector:
    nvidia.com/gpu.deploy.operator-validator: "true"
  preemptionPolicy: PreemptLowerPriority
  priority: 2000001000
  priorityClassName: system-node-critical
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: nvidia-operator-validator
  serviceAccountName: nvidia-operator-validator
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: nvidia.com/gpu
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/pid-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists
  volumes:
  - hostPath:
      path: /run/nvidia/validations
      type: DirectoryOrCreate
    name: run-nvidia-validations
  - hostPath:
      path: /usr/local
      type: ""
    name: driver-install-dir
  - hostPath:
      path: /
      type: ""
    name: host-root
  - hostPath:
      path: /dev/char
      type: ""
    name: host-dev-char
  - name: kube-api-access-gm5p2
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2026-04-20T18:52:21Z"
    observedGeneration: 1
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2026-04-20T18:52:21Z"
    message: 'containers with incomplete status: [toolkit-validation cuda-validation
      plugin-validation]'
    observedGeneration: 1
    reason: ContainersNotInitialized
    status: "False"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2026-04-20T18:52:21Z"
    message: 'containers with unready status: [nvidia-operator-validator]'
    observedGeneration: 1
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2026-04-20T18:52:21Z"
    message: 'containers with unready status: [nvidia-operator-validator]'
    observedGeneration: 1
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2026-04-20T18:52:21Z"
    observedGeneration: 1
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: nvcr.io/nvidia/gpu-operator:v26.3.1
    imageID: ""
    lastState: {}
    name: nvidia-operator-validator
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
    volumeMounts:
    - mountPath: /run/nvidia/validations
      name: run-nvidia-validations
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm5p2
      readOnly: true
      recursiveReadOnly: Disabled
  hostIP: 172.16.33.136
  hostIPs:
  - ip: 172.16.33.136
  initContainerStatuses:
  - containerID: containerd://12e621dc517c2d7bad494720b6586ecc46725e551389f489e0eedfa2ace8928d
    image: nvcr.io/nvidia/gpu-operator:v26.3.1
    imageID: nvcr.io/nvidia/gpu-operator@sha256:327518e7157e9634eab9aeb51abfba816845fad47177ca985e1b34d202a1f51e
    lastState: {}
    name: driver-validation
    ready: true
    resources: {}
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://12e621dc517c2d7bad494720b6586ecc46725e551389f489e0eedfa2ace8928d
        exitCode: 0
        finishedAt: "2026-04-20T18:52:22Z"
        reason: Completed
        startedAt: "2026-04-20T18:52:21Z"
    user:
      linux:
        gid: 0
        supplementalGroups:
        - 0
        uid: 0
    volumeMounts:
    - mountPath: /host
      name: host-root
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /usr/local
      name: driver-install-dir
    - mountPath: /run/nvidia/validations
      name: run-nvidia-validations
    - mountPath: /host-dev-char
      name: host-dev-char
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm5p2
      readOnly: true
      recursiveReadOnly: Disabled
  - containerID: containerd://b2894c109a5bd3d5608e249d0db5025610215c2b5351d4da2b20f32dce5110bf
    image: nvcr.io/nvidia/gpu-operator:v26.3.1
    imageID: nvcr.io/nvidia/gpu-operator@sha256:327518e7157e9634eab9aeb51abfba816845fad47177ca985e1b34d202a1f51e
    lastState:
      terminated:
        containerID: containerd://b2894c109a5bd3d5608e249d0db5025610215c2b5351d4da2b20f32dce5110bf
        exitCode: 1
        finishedAt: "2026-04-20T18:53:59Z"
        reason: Error
        startedAt: "2026-04-20T18:53:59Z"
    name: toolkit-validation
    ready: false
    resources: {}
    restartCount: 4
    started: false
    state:
      waiting:
        message: back-off 1m20s restarting failed container=toolkit-validation pod=nvidia-operator-validator-bfmvt_gpu-operator(699136b5-8fb6-4dff-a748-8dc78dbd931d)
        reason: CrashLoopBackOff
    user:
      linux:
        gid: 0
        supplementalGroups:
        - 0
        uid: 0
    volumeMounts:
    - mountPath: /run/nvidia/validations
      name: run-nvidia-validations
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm5p2
      readOnly: true
      recursiveReadOnly: Disabled
  - image: nvcr.io/nvidia/gpu-operator:v26.3.1
    imageID: ""
    lastState: {}
    name: cuda-validation
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
    volumeMounts:
    - mountPath: /run/nvidia/validations
      name: run-nvidia-validations
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm5p2
      readOnly: true
      recursiveReadOnly: Disabled
  - image: nvcr.io/nvidia/gpu-operator:v26.3.1
    imageID: ""
    lastState: {}
    name: plugin-validation
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
    volumeMounts:
    - mountPath: /run/nvidia/validations
      name: run-nvidia-validations
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm5p2
      readOnly: true
      recursiveReadOnly: Disabled
  observedGeneration: 1
  phase: Pending
  podIP: 10.244.1.27
  podIPs:
  - ip: 10.244.1.27
  qosClass: BestEffort
  startTime: "2026-04-20T18:52:21Z"

happy to provide further information is needed

It looks like you don't deploy the nvidia-container-toolkit. Is that expected?

Please note that the toolkit daemonset pod is what runs the NRI Plugin server and performs the CDI device injection. Without the toolkit pod, this will not work

It looks like you don't deploy the nvidia-container-toolkit. Is that expected?

Please note that the toolkit daemonset pod is what runs the NRI Plugin server and performs the CDI device injection. Without the toolkit pod, this will not work

yes, that's correct, the host os already ships the toolkit components

helm command:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
  helm repo update
  helm upgrade --wait  --install -n gpu-operator gpu-operator nvidia/gpu-operator \
        --set driver.enabled=false \
        --set toolkit.enabled=false \
        --set hostPaths.driverInstallDir=/usr/local \
        --set cdi.nriPluginEnabled=true

❯ talosctl -n 172.16.33.136 ls /usr/local/bin | grep nvidia
172.16.33.136   nvidia-bug-report.sh
172.16.33.136   nvidia-cdi-hook
172.16.33.136   nvidia-container-runtime
172.16.33.136   nvidia-container-runtime-hook
172.16.33.136   nvidia-container-runtime.cdi
172.16.33.136   nvidia-container-runtime.legacy
172.16.33.136   nvidia-ctk
172.16.33.136   nvidia-ctk-installer
172.16.33.136   nvidia-cuda-mps-control
172.16.33.136   nvidia-cuda-mps-server
172.16.33.136   nvidia-debugdump
172.16.33.136   nvidia-installer
172.16.33.136   nvidia-modprobe
172.16.33.136   nvidia-ngx-updater
172.16.33.136   nvidia-pcc
172.16.33.136   nvidia-persistenced
172.16.33.136   nvidia-persistenced-wrapper
172.16.33.136   nvidia-powerd
172.16.33.136   nvidia-settings
172.16.33.136   nvidia-sleep.sh
172.16.33.136   nvidia-smi
172.16.33.136   nvidia-uninstall
172.16.33.136   nvidia-xconfig

Please note that the toolkit daemonset pod is what runs the NRI Plugin server and performs the CDI device injection. Without the toolkit pod, this will not work

Does this mean the host doesn't need to ship the toolkit ? Just trying to wrap my head around here

is my understanding correct that the toolkit validation with error:

time="2026-04-20T18:53:05Z" level=info msg="version: 5a25fef4-amd64, commit: 5a25fef"
time="2026-04-20T18:53:05Z" level=info msg="Error: error validating toolkit installation: exec: \"nvidia-smi\": executable file not found in $PATH"
toolkit is not ready
Error from server (BadRequest): container "cuda-validation" in pod "nvidia-operator-validator-bfmvt" is waiting to start: PodInitializing

failed to find nvidia-smi since cdi injection did not happen?

Does this mean the host doesn't need to ship the toolkit ? Just trying to wrap my head around here

Well, the host's pre-installed toolkit will not help as it doesn't bring up an NRI Plugin server. To leverage the NRI Plugin feature here, you will need to enable nvidia-container-toolkit in the ClusterPolicy CR. So, the NRI Plugin is only useable if the toolkit is deployed a daemonset.

failed to find nvidia-smi since cdi injection did not happen?

Yes, If the toolkit-validation fails with this error, It most likely points to the toolkit not being able to inject the gpu driver binaries, shared libraries and gpu device nodes into the container successfully.

Does this mean the host doesn't need to ship the toolkit ? Just trying to wrap my head around here

Well, the host's pre-installed toolkit will not help as it doesn't bring up an NRI Plugin server. To leverage the NRI Plugin feature here, you will need to enable nvidia-container-toolkit in the ClusterPolicy CR. So, the NRI Plugin is only useable if the toolkit is deployed a daemonset.

failed to find nvidia-smi since cdi injection did not happen?

Yes, If the toolkit-validation fails with this error, It most likely points to the toolkit not being able to inject the gpu driver binaries, shared libraries and gpu device nodes into the container successfully.

Thanks this clears up stuff, I would like to move this issue to a github discussion if possible

Since Talos is an immtuable host OS, the toolkit binaries are brought in as part of OS, so I would like to have a general discussion on how to then enable NRI if that comes as part of toolkit

Since when i enable toolkit, it fails on Talos due to host OS being immutable

  Warning  Failed     20s               kubelet            Error: failed to generate container "cdc186ceff6e307e2631bfa8f0597c857d055f3c3e68bc7b51a275de5d644d77" spec: failed to apply OCI options: failed to mkdir "/etc/containerd/conf.d": mkdir /etc/containerd/conf.d: read-only file system
  Normal   Pulled     5s (x2 over 20s)  kubelet            Container image "nvcr.io/nvidia/k8s/container-toolkit:v1.19.0" already present on machine and can be accessed by the pod
  Warning  Failed     5s                kubelet            Error: failed to generate container "5786e13b709ecbedb2aad72a60578f8357bbd5d7a0c8c1ef3158c25ed5695d91" spec: failed to apply OCI options: failed to mkdir "/etc/containerd/conf.d": mkdir /etc/containerd/conf.d: read-only file system

I'm not sure why it tries to create containerd config when using NRI since there's no need to inject runtimeclass anymore

I'm not sure why it tries to create containerd config when using NRI since there's no need to inject runtimeclass anymore

From docs https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/cdi.html#about-the-node-resource-interface-nri-plugin

Additionally, with the NRI Plugin enabled, modifications to the container runtime configuration are no longer needed. For example, no modifications are made to containerd’s config.toml file. This means that on platforms that configure containerd in a non-standard way, like k3s, k0s, and Rancher Kubernetes Engine 2, users no longer need to configure environment variables like CONTAINERD_CONFIG, CONTAINERD_SOCKET, or RUNTIME_CONFIG_SOURCE.

Navigation Menu

Repository navigation

[Bug]: NRI doesn't work trying to check nvidia-smi from the validation pod instead of host #2382

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: NRI doesn't work trying to check nvidia-smi from the validation pod instead of host #2382

Description

Activity

tariq1890 commented on Apr 20, 2026

frezbo commented on Apr 20, 2026

frezbo commented on Apr 20, 2026

tariq1890 commented on Apr 20, 2026

frezbo commented on Apr 20, 2026

frezbo commented on Apr 20, 2026

tariq1890 commented on Apr 20, 2026

frezbo commented on Apr 21, 2026

frezbo commented on Apr 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions