Skip to content

Fix GPU pod detection for native sidecars during driver upgrades - #3028

Merged
JunAr7112 merged 1 commit into
NVIDIA:mainfrom
shin4141:codex/3026-native-sidecar-gpu-filter
Oct 15, 2026
Merged

JunAr7112 merged 1 commit into
NVIDIA:mainfrom
shin4141:codex/3026-native-sidecar-gpu-filter

Conversation

@shin4141

@shin4141 shin4141 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Description

Addresses #3026. During a driver upgrade, gpuPodSpecFilter checks GPU resources in pod.Spec.Containers but misses native sidecars in pod.Spec.InitContainers (restartPolicy: Always). A pod whose main container has no GPU request and whose native sidecar requests a GPU can therefore be skipped by pod deletion.

Check GPU and MIG limits and requests for native sidecars using the existing resource predicate. The ordinary container check, phase gate, DRA fallback, and handling of regular init containers remain unchanged.

Known limitation: A Pending pod with a currently running regular (non-sidecar) GPU init container is still not selected. This is an existing behavior outside this change. Scanning all init containers unconditionally would also select Running pods whose GPU init has already completed; a status-aware fix needs separate decisions for missing or stale status and retries. Please advise whether that case should be tracked separately.

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint) — not run locally; golangci-lint is unavailable in this environment
  • Generated assets in-sync (make validate-generated-assets) — not run; no generated assets changed
  • Go mod artifacts in-sync (make validate-modules) — not run; no module or vendor files changed
  • Test cases are added for new code paths

Testing

  • Before the fix, three native sidecar cases failed: GPU limit, GPU request, and MIG request. Their non-GPU and regular-init controls passed.
  • After the fix, GOTOOLCHAIN=local GOFLAGS=-mod=vendor go test -count=1 ./cmd/gpu-operator ./controllers passed: 20 operator subtests and 387 controller subtests (407 total). Run with Go 1.26.3 on Darwin arm64 using vendored dependencies.
  • Focused -race filter tests passed (20 subtests) on the same production logic before the final comment-only edit. The final two-file diff passes gofmt and git diff --check.
  • No GPU hardware or Kubernetes cluster upgrade test was run. Driver unload and end-to-end upgrade recovery remain unverified.

@shin4141
shin4141 requested a review from a team as a code owner October 8, 2026 12:29
@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

📝 Walkthrough

Walkthrough

gpuPodSpecFilter now checks GPU and MIG resource requests and limits in init containers with restart policy Always. Tests cover GPU and MIG resources in native sidecars, regular init containers, CPU-only native sidecars, and succeeded pods.

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to f4d11

A native-sidecar MIG-limit test would protect this behavior from regression. The supplied evidence shows no current production failure, so this test improvement does not block merging.

  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
cmd/gpu-operator/main_test.go (1)

211-240: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add a native-sidecar MIG-limit case.

The changed tests cover a native sidecar with a MIG request, but not one with a MIG limit. A regression that removes the native-sidecar Limits check would pass the new test. The ordinary-container MIG-limit test does not cover that branch.

Suggested fix
 		{
 			name:       "running pod with MIG request only in native sidecar -> GPU pod",
 			pod:        podWithInitContainer(corev1.PodRunning, nativeSidecar(nil, corev1.ResourceList{"nvidia.com/mig-1g.5gb": {}})),
 			wantGPUPod: true,
 		},
+		{
+			name:       "running pod with MIG limit only in native sidecar -> GPU pod",
+			pod:        podWithInitContainer(corev1.PodRunning, nativeSidecar(corev1.ResourceList{"nvidia.com/mig-1g.5gb": {}}, nil)),
+			wantGPUPod: true,
+		},

ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/gpu-operator/.coderabbit.yaml
  • Review profile: QUIET
  • Plan: Enterprise
  • Run ID: 36b8feb3-700b-4a14-8bc8-0a12613c0a7b
📥 Commits

Reviewing files that changed from the base of the PR and between 57378e9 and f4d11f6.

📒 Files selected for processing (2)
  • cmd/gpu-operator/main.go
  • cmd/gpu-operator/main_test.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@JunAr7112
JunAr7112 self-requested a review October 12, 2026 22:52
@kvalliyurnatt

Copy link
Copy Markdown
Contributor

@shin4141 thanks for your contribution! can you rebase and make sure commit is signed using a GPG key.

@shin4141
shin4141 force-pushed the codex/3026-native-sidecar-gpu-filter branch from f4d11f6 to 6eded4d Compare October 13, 2026 16:46
@kvalliyurnatt

Copy link
Copy Markdown
Contributor

/ok to test 6eded4d

@JunAr7112 JunAr7112 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR LGTM

@JunAr7112

Copy link
Copy Markdown
Contributor

@shin4141 could you rebase this PR?

Signed-off-by: Nagata Shinichi <128954611+shin4141@users.noreply.github.com>
@shin4141
shin4141 force-pushed the codex/3026-native-sidecar-gpu-filter branch from 6eded4d to 0cd4c46 Compare October 14, 2026 16:00
@JunAr7112

Copy link
Copy Markdown
Contributor

/ok-to-test 0cd4c46

@JunAr7112
JunAr7112 merged commit dbb11e8 into NVIDIA:main Oct 15, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants