feat(nrl-k8s): Kubernetes infrastructure and CLI tooling - #2330
Merged
Conversation
Local K8s GPU dev environment using nvkind (NVIDIA's kind wrapper): - nvkind cluster setup scripts (install-nvkind.sh, create-cluster.sh) - Custom config template with extraMounts for dev code mounting - Helmfile with kind/prod environments (device plugin vs GPU operator) - KAI scheduler for gang scheduling, KubeRay for RayCluster management - Example manifests: gang-scheduled pods, RayClusters, SFT RayJobs - SETUP.md with prerequisites, quick start, and architecture docs Tested: SFT RayJob (train/loss 4.06 < 5.9), KAI all-or-nothing gang scheduling, two simultaneous 1-GPU SFT jobs. Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
…overy (infra part) Signed-off-by: Terry Kong <terryk@nvidia.com>
When RL and Gym run on separate RayClusters, either cluster failing or being deleted triggers teardown of both clusters to release resources. - peer-watcher.py: pure Python sidecar (no deps beyond stdlib), deployed as a ConfigMap volume mount on each head pod - Monitors peer RayCluster status via K8s API (polls every 10s) - Tears down after MAX_PEER_FAILURES (default 3) consecutive failures - Also monitors ConfigMap "error" key for application-level error signaling - Handles transient K8s API errors as failures (not false-healthy) - Added signal_error() to K8sEndpointRegistry - Updated disagg manifests with peer-watcher sidecar containers - Updated RBAC with "delete" verb for rayclusters Tested: deleting either cluster triggers teardown of both within ~10s. Signed-off-by: Terry Kong <terryk@nvidia.com>
…share configs - Kyverno policy: RayCluster/RayJob must have kai.scheduler/queue label. Validates at CRD level (not pod) since KubeRay operator creates pods. Optional Policy 2 for user→queue access control via ConfigMap. - kube-prometheus-stack: Prometheus + Grafana for fairshare monitoring. Pre-built Grafana dashboard showing GPU allocation vs fair share, preemption events, and scheduling latency per queue. - ServiceMonitors for KAI scheduler, binder, and queue-controller. - Example queue configs: - kai-queue.yaml: 2-GPU kind cluster (2 teams, equal quotas) - kai-queue-prod.yaml: 256-GPU prod (3 departments, 6 teams) - preemptMinRuntime: 4h (protect long training runs from priority preemption) - reclaimMinRuntime: 15m (fast fairness reclaim of over-quota resources) - SETUP.md: fairshare docs, preempt vs reclaim explanation, Grafana access. Tested: Kyverno rejects RayCluster without queue label, accepts with. Team A 2-GPU job reclaimed when Team B submitted to its guaranteed quota. Signed-off-by: Terry Kong <terryk@nvidia.com>
- Upgrade KAI scheduler v0.13.4 → v0.14.0 (adds Ray topology-aware
scheduling, segment-size annotation support for PyTorchJob)
- Update chart URL from NVIDIA/KAI-Scheduler to kai-scheduler/KAI-Scheduler
- Fix Grafana dashboard metric names (add kai_ prefix to match actual
Prometheus metric names). Verified: Grafana queries return live data.
- New: extensions/k8s_cli/ — standalone Python CLI (pip installable):
- nrl-k8s fairshare — show queue config (quota, limit, weight, priority)
- nrl-k8s occupancy — show GPU allocation per node and per queue
- nrl-k8s submit — submit gang-scheduled RayJob with optional
--segment-size for topology-aware scheduling
- 6 unit tests (mocked K8s API), all passing
- Add TODO for NVL72 topology testing with links to relevant PRs/issues
Tested: KAI v0.14.0 gang scheduling works, CLI commands verified against
live cluster, Grafana dashboard loads and queries return data.
Signed-off-by: Terry Kong <terryk@nvidia.com>
- Merge disagg_rl_raycluster.yaml + disagg_gym_raycluster.yaml into single disagg-rayclusters.yaml (always deployed together) - Inline peer-watcher Python script directly in sidecar container args (eliminates ConfigMap setup step, each deployment is self-contained) - Remove 7 redundant workload YAMLs (sft_rayjob, kai_scheduled_*, raycluster-blocker, standalone peer-watcher.py) - Update SETUP.md: simplified quick start, updated architecture tree, removed ConfigMap peer-watcher setup step 15 files → 8 files in examples/. Infrastructure configs unchanged. Tested: inlined peer-watcher works — deleting either cluster triggers teardown of both within 10s. Signed-off-by: Terry Kong <terryk@nvidia.com>
- Remove extensions/k8s_cli/ (not needed for now) - Rename queues: org → root-org, priority-team → high-prio, community → low-prio - Simplify Kyverno policy comments - Update SETUP.md to remove CLI section Signed-off-by: Terry Kong <terryk@nvidia.com>
Not needed right now — queue enforcement can be added back later if required. Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
…a part) Signed-off-by: Terry Kong <terryk@nvidia.com>
Three workload deployment patterns: 1. rayjob-monolithic.yaml — single-cluster RayJob (1 GPU, KubeRay) 2. disagg-rayclusters.yaml — two KubeRay RayClusters + peer-watcher 3. disagg-jobset.yaml — single JobSet with native failure/startup policies Also adds JobSet controller (v0.11.1) to the helmfile. Signed-off-by: Terry Kong <terryk@nvidia.com>
Shows the full Ray cluster pattern for Gym: separate head and worker pods within the JobSet, with dependsOn ordering and DNS discovery. Signed-off-by: Terry Kong <terryk@nvidia.com>
KAI gang-schedules all pods in a JobSet together (one PodGroup with minMember=total pods). This deadlocks with dependsOn: KAI waits for all pods to exist, but JobSet won't create dependent pods until the head is Ready. Fix: drop dependsOn, use init containers that poll ray health-check (same pattern KubeRay uses). Tested: all 6 pods schedule, init containers wait for heads, driver submits a Ray job successfully, successPolicy triggers on driver exit 0, failurePolicy tears down everything on gym-head crash. Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
- Add git safe.directory for Gym submodule (uv build fails otherwise) - Add uv pip install kubernetes (needed by endpoint registry) - Increase readiness probe failureThreshold (uv install takes time) Signed-off-by: Terry Kong <terryk@nvidia.com>
Replace --working-dir and --runtime-env-json with a simple cd wrapper. Both --working-dir and runtime_env.working_dir cause Ray to zip and upload the entire directory to GCS, which is extremely slow for large repos (1GB+). Since the code is already on all nodes via hostPath, wrapping the entrypoint with cd avoids the upload entirely. Before: ray job submit --working-dir /workspace/nemo-rl -- python ... → scans entire tree, uploads 1GB+ to GCS, takes minutes After: ray job submit -- bash -c "cd /workspace/nemo-rl && python ..." → instant submission, no upload Signed-off-by: Terry Kong <terryk@nvidia.com>
…pefail - Use timestamp-based submission_id to avoid GCS collision across redeploys - Disable wandb (not configured in kind dev cluster) - Add set -eo pipefail for proper exit code propagation through tee - Persist driver logs to hostPath for post-mortem debugging Tested: GRPO training loads config, connects to Ray cluster, loads datasets, initializes compute cluster. Fails with "Not enough GPUs" (expected — kind cluster has 2 GPUs, config expects 8). Signed-off-by: Terry Kong <terryk@nvidia.com>
- Add active development disclaimer (GitHub admonition) - Add production guidance (adapt manifests, use Terraform, not helmfile) - Document colocated vs disaggregated architecture with diagrams - Compare KubeRay RayClusters vs JobSet for disagg deployment - Explain why ConfigMap is still needed for JobSet (vLLM URL exchange) - Document dependsOn + KAI deadlock and init container workaround - Add local kind testing instructions - Add comparison table (failure cascading, gang scheduling, discovery) Signed-off-by: Terry Kong <terryk@nvidia.com>
Add all parallelism overrides needed for Qwen3-0.6B on a 2-GPU cluster: - tensor_model_parallel_size=1, pipeline=1, expert=1, context=1 - sequence_parallel=false (requires TP>1) - colocated.enabled=false, gpus_per_node=1 - max_new_tokens=512, max_total_sequence_length=512 - max_num_steps=2 for quick smoke testing Tested: GRPO initializes vLLM workers, captures CUDA graphs, starts Megatron LM workers. Fails at k8s_endpoint_registry import because the container image predates the tk/infra branch — the hostPath mount has newer code than the baked-in worker venvs. Will work with a matching container build. Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
Mount the local nemo-rl source at both /workspace/nemo-rl and /opt/nemo-rl. The container's editable install points to /opt/nemo-rl, so this ensures all imports (Ray job driver, worker venvs) use the same code from the hostPath mount. Tested: GRPO setup completes (471s), vLLM workers initialized with CUDA graphs, Megatron workers started, endpoint registry created and vLLM URLs published. Full disagg E2E validated up to Gym handshake. Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
…part) Signed-off-by: Terry Kong <terryk@nvidia.com>
…hboard
Adds a production-grade launch path, a dedicated dashboard command, and
an onboarding guide. All opt-in — existing interactive dev flows keep
working unchanged.
`--mode {interactive, batch}` on `launch` / `run`:
interactive (default): port-forward + working_dir upload + tail logs.
batch: kubectl exec + code from image + nohup/disown + no wait.
Each axis is independently overridable via `--submitter`,
`--code-source`, `--code-path`, `--run-id`, `--wait/--no-wait`.
New submitters package (`src/nrl_k8s/submitters/`):
* JobSubmitter Protocol with submit/follow/status/stop.
* PortForwardSubmitter (Ray Job SDK) — inlines env vars as shell
`export`s instead of `runtime_env.env_vars` to sidestep Ray's
"Failed to merge the Job's runtime env" error when ray.init captures
os.environ.
* ExecSubmitter — `kubectl cp` launcher script, `nohup bash ... &
disown` via Popen, bounded-timeout pidfile poll (~17 s wall time
vs 14 min previously for detach-clean return on flaky EKS).
tail -F, kill -0 / exitcode sentinel, SIGTERM/SIGKILL stop.
* SubmissionHandle cached under ~/.cache/nrl-k8s/runs/<id>.json so
job logs / job stop dispatch on transport without re-probing.
Schema additions (back-compat — every default preserved):
* SubmitSpec.submitter: portForward | exec
* SubmitSpec.execTmpDir: /tmp
* LaunchSpec.codeSource: upload | image | lustre
* LaunchSpec.codePath + validator requiring it for image / lustre
* LaunchSpec.runMode: interactive | batch
* orchestrate.submit_training always injects NRL_K8S_RUN_ID into the
entrypoint env so recipes can reference it for wandb name etc.
`nrl-k8s cluster dashboard <name>` command:
Port-forwards svc/<name>-head-svc:8265 to localhost:8265 and opens a
browser. `--fix` (default) runs a one-time reinstall of ray[default]
with `--link-mode=copy` on the head pod when dashboard static assets
are symlinks into the uv cache (aiohttp's follow_symlinks=False else
404s every JS/CSS file → blank page). The permanent fix is in the
image build (`ENV UV_LINK_MODE=copy` before the ray install).
Docs + examples:
* docs/onboarding.md — step-by-step for standing up nrl-k8s on a
fresh cluster (RBAC SA, pull secrets, wandb secret, node pool
mapping, first run).
* examples/qwen3_4b_if_gym_disagg.prod.infra.yaml — production
variant of the canonical gym-disagg example:
submitter=exec, codeSource=image, codePath=/opt/nemo-rl.
* README: "Modes: interactive vs batch" section with the full batch
walkthrough, env-var conventions, and blank-dashboard remediation.
Testing:
* 137 unit tests pass (was 134). New coverage: submitter protocol,
exec command shape + env-escape + pidfile plumbing, port-forward
back-compat, mode resolution, cluster dashboard invocation.
Verified end-to-end against the live raycluster-single-qwen3-4b:
* exec-mode run → SUCCEEDED, wandb: nemorl-single-k8s/runs/f9oisnfd
* portForward+image run → SUCCEEDED, wandb: runs/wgm3txe7
Both submit wall-time 9–17 s, laptop detachable as soon as the driver
is running.
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
The GPU Operator is pre-installed by NKX on production clusters, so we should not deploy our own. Rename prod gpu_backend to existing-nkx-gpu-operator, remove the gpu-operator helm release and values file, and update SETUP.md accordingly. Signed-off-by: Terry Kong <terryk@nvidia.com>
Increase binder memory from 200Mi to 1Gi — the default OOMs on large clusters (78+ nodes), which breaks DRA resource claim allocation. Set admission.gpuPodRuntimeClassName to "" to prevent the KAI admission webhook from injecting runtimeClassName: nvidia onto GPU pods. GPU Operator v25.10.0+ with cdi.enabled=true installs a nvidia RuntimeClass that triggers the management.nvidia.com CDI path, which fails with "unresolvable CDI devices" on clusters where CDI spec files are not fully configured. Requires KAI >= v0.13.0 (PR #1035). Signed-off-by: Terry Kong <terryk@nvidia.com>
Add two JobSet variants for running monolithic (colocated) training on GB300 nodes with DRA topology-aware scheduling: - monolithic-jobset.yaml: uses KAI scheduler with queue support - monolithic-jobset-no-kai.yaml: uses default scheduler (for comparing DRA behavior between schedulers) Both use ComputeDomain + RoCE ResourceClaimTemplates for topology-aware placement and run SFT with Qwen3-0.6B as a smoke test. Signed-off-by: Terry Kong <terryk@nvidia.com>
Add step-by-step onboarding for the aws-cmh (GB300) and nemo-ci-h100 EKS clusters, covering access requests, nvsec/AWS SSO auth, kubeconfig setup, shared PVC workspace, and running jobs with nrl-k8s. Signed-off-by: Terry Kong <terryk@nvidia.com>
terrykong
added
CI:Lfast
Collaborator
Author
|
/ok to test 3d846a6 |
Collaborator
Author
|
/ok to test debb067 |
The v4.34.0 versioned URL returns 500; the unversioned URL works. Signed-off-by: Terry Kong <terryk@nvidia.com>
github-actions
Bot
added
the
Documentation
Collaborator
Author
|
/ok to test 5c29d48 |
The chat_templating page was split; "how to create" content moved to chat_templating_writing. Also add the nsight-systems InstallationGuide URL to false positives (transient connection reset). Signed-off-by: Terry Kong <terryk@nvidia.com>
Collaborator
Author
|
/ok to test 0b05895 |
hemildesai
previously approved these changes
Apr 28, 2026
HuggingFace docs intermittently return 500. Drop the `main/` version pin from the chat_templating link and add all HF doc URLs to the linkcheck false-positives list to stop transient failures. Signed-off-by: Terry Kong <terryk@nvidia.com>
Collaborator
Author
|
/ok to test feabeea |
hemildesai
approved these changes
Apr 28, 2026
snivertynv
pushed a commit
to snivertynv/RL
that referenced
this pull request
May 5, 2026
…#2330) Signed-off-by: Terry Kong <terryk@nvidia.com> Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes RL-417
Test plan
cd tools/nrl_k8s && uv run --extra test pytest -x -qnrl-k8s run <recipe> --infra <infra> --dry-runproduces valid manifestnrl-k8s dev connect/nrl-k8s dev stoplifecycle works