Skip to content

docs: document GPU NUMA locality with DRA - #850

Open
yankay wants to merge 1 commit into
ai-dynamo:mainfrom
yankay:codex/docs-644-numa-validation
Open

yankay wants to merge 1 commit into
ai-dynamo:mainfrom
yankay:codex/docs-644-numa-validation

Conversation

@yankay

@yankay yankay commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind documentation

What this PR does / why we need it:

Documents GPU NUMA locality with DRA: a minimal PodCliqueSet example, a short user-guide section, and a link from GREP-244. No API or runtime changes.

Which issue(s) this PR fixes:

Related to #644 (Case 1 only).

Special notes for your reviewer:

Checks passed: server-side dry-run, make verify-toc, and git diff --check. Earlier mock DRA runtime tests passed 7/7 scenarios; physical NVIDIA multi-GPU validation remains outstanding.

Does this PR introduce a API change?

NONE

Additional documentation e.g., enhancement proposals, usage docs, etc.:

GREP-244: Topology-Aware Scheduling

docs/user-guide/topology-aware-scheduling.md#gpu-numa-locality
operator/samples/user-guide/04_topology-aware-scheduling/single-pod.yaml

Signed-off-by: Kay Yan <kay.yan@daocloud.io>
Copilot AI lite review requested due to automatic review settings September 24, 2026 05:48
@copy-pr-bot

copy-pr-bot Bot commented Sep 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Address the four unresolved moderate review findings before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 Medium severity

Open (2)
What changed in this PR

Documents GPU NUMA locality using Kubernetes DRA without runtime changes.

Changes:

  • Adds a DRA-based GPU locality sample.
  • Adds user-guide guidance and prerequisites.
  • Links GREP-244 to the new approach.

Unresolved moderate issues include the missing namespace, an image without nvidia-smi, and incomplete NVIDIA DRA NUMA attribute prerequisites.

File Description
operator/​samples/​user-guide/​04_topology-aware-scheduling/​single-pod.yaml Adds the DRA GPU locality example.
docs/​user-guide/​topology-aware-scheduling.md Documents GPU NUMA locality usage.
docs/​proposals/​244-topology-aware-scheduling/​README.md Links the proposal story to the DRA pattern.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

kind: ResourceClaimTemplate
metadata:
name: decode-four-gpus
namespace: numa-locality
resourceClaimTemplateName: decode-four-gpus
containers:
- name: decode
image: ubuntu:24.04
@aylonsk

aylonsk commented Sep 24, 2026

Copy link
Copy Markdown

Looks mostly good besides copilot comments above. Two things that might be worth mentioning in the user guide section:

  • If no NUMA node has enough free GPUs, the pod stays Pending. DRA decides this at scheduling time, so there's no kubelet TopologyAffinityError the way there is with Topology Manager.
  • Devices whose NUMA node isn't known (numa_node = -1, e.g. on some VMs) don't publish resource.kubernetes.io/numaNode. matchAttribute never selects a device without the attribute, so on those nodes the pod stays Pending even with free GPUs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants