Skip to content

[Bug]: Driver upgrade does not evict pods that use nvidia.com/gpu only in a native sidecar #3026

Closed

Description

@sznpjaronek

Describe the bug
During an automatic driver upgrade, the upgrade controller does not delete workload pods whose nvidia.com/gpu resource is declared only in a native sidecar container (an entry in spec.initContainers with restartPolicy: Always).
The GPU pod filter, gpuPodSpecFilter in cmd/gpu-operator/main.go, only iterates pod.Spec.Containers, so such pods are treated as non-GPU pods ("No pods require deletion" logged). They keep the GPU open, k8s-driver-manager cannot unload the driver, the driver pod goes into Init:CrashLoopBackOff and the node ends up in upgrade-failed.

To Reproduce

  1. Run a Deployment whose main container has no GPU request, and a native sidecar with one:
  initContainers:
  - name: gpu-sidecar
    restartPolicy: Always
    resources:
      limits: 
        nvidia.com/gpu: 1
  1. Change driver.version in the ClusterPolicy.
  2. Observe the operator log and the driver pod.

Expected behavior
Pods that declare nvidia.com/gpu* in resources of any container, including native sidecars, are selected by gpuPodSpecFilter and deleted in pod-deletion-required.

Environment (please provide the following information):

  • GPU Operator Version: v26.7.1
  • OS: Ubuntu24.04
  • Kernel Version: 6.17.0-19-generic
  • Container Runtime Version: containerd 2.2.6
  • Kubernetes Distro and Version: K8s v1.36.1

Activity

  1. added
    bugIssue/PR to expose/discuss/fix a bug
    needs-triageissue or PR has not been assigned a priority-px label
    on Oct 8, 2026
  2. JunAr7112 commented on Oct 15, 2026

    @JunAr7112
    Contributor

    Merged #3028 to address this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugIssue/PR to expose/discuss/fix a bugneeds-triageissue or PR has not been assigned a priority-px label

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions