You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: Driver upgrade does not evict pods that use nvidia.com/gpu only in a native sidecar #3026
Describe the bug
During an automatic driver upgrade, the upgrade controller does not delete workload pods whose nvidia.com/gpu resource is declared only in a native sidecar container (an entry in spec.initContainers with restartPolicy: Always).
The GPU pod filter, gpuPodSpecFilter in cmd/gpu-operator/main.go, only iterates pod.Spec.Containers, so such pods are treated as non-GPU pods ("No pods require deletion" logged). They keep the GPU open, k8s-driver-manager cannot unload the driver, the driver pod goes into Init:CrashLoopBackOff and the node ends up in upgrade-failed.
To Reproduce
Run a Deployment whose main container has no GPU request, and a native sidecar with one:
Expected behavior
Pods that declare nvidia.com/gpu* in resources of any container, including native sidecars, are selected by gpuPodSpecFilter and deleted in pod-deletion-required.
Environment (please provide the following information):
Activity
JunAr7112 commentedon Oct 15, 2026
Merged #3028 to address this issue