Repository navigation
fix(orchestration): forward explicit runtime variables to remote Slurm - #4760
Merged
Merged
Conversation
Signed-off-by: Igor Gitman <igitman@nvidia.com>
yaoyu-33
added
area:orchestration
prokotg
approved these changes
Oct 20, 2026
Contributor
|
/ok to test 84fa363 |
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-4760-84fa36386a9b.docs.buildwithfern.com/nemo/gym |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39382615061 |
qiaochuz-nv
added a commit
that referenced
this pull request
Oct 22, 2026
The slug-label guard added in this PR flags `[how-to](...)` in orchestration/configuration.mdx, a page that landed after this branch's base (#4760). Use the target page's title as the label. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
khazic
pushed a commit
to khazic/Gym-lao
that referenced
this pull request
Oct 23, 2026
…xamples to v0.7.0 (NVIDIA-NeMo#4579) * docs: make r0.7.0 training commands copy-pasteable and bump version examples to v0.7.0 Training blocks export credentials first and reference $WANDB_API_KEY, $HF_TOKEN, $SLURM_* and $USER instead of space-containing placeholders that Bash splits into commands. Install, container, version-output and telemetry examples move from v0.5.1 to v0.7.0 (published on PyPI and NGC; v0.6.0 has no NGC image). Also fixes the FAQ dataset-name en dash and revision wording, slug link labels, grammar, and British spellings. Adds regression tests. Fixes NVBug 6875737 / NVIDIA-NeMo#4499 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: qiaochuz <qiaochuz@nvidia.com> * docs: address review on r0.7.0 docs fix - Pin the version test to v0.7.0 (snapshots land before PyPI/NGC) and check backticked `vX.Y.Z` examples too. - setup.mdx step 2: `ACCOUNT=<ACCOUNT_NAME>` is a Bash syntax error; use sample values like the multi-node page. The placeholder test now flags `VAR=<...>` assignments. - FAQ dataset commands: multineedle was renamed to example_multi_step (NVIDIA-NeMo#291); point at the existing config and example data, and define resource_config_path in the create-pr block. New test: dataset commands use a defined, existing resource config. - vllm.mdx: the command sets --model-url, not policy_base_url=. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: qiaochuz <qiaochuz@nvidia.com> * docs(tests): name the two placeholder shapes the bash-block test covers Review note on NVIDIA-NeMo#4579: the test catches `{your ...}` placeholders and `VAR=<...>` assignments, not every angle-bracket argument. Say so in the docstring instead of claiming all placeholders bash splits. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: qiaochuz <qiaochuz@nvidia.com> * docs(orchestration): label the prefill/decode link with its page title The slug-label guard added in this PR flags `[how-to](...)` in orchestration/configuration.mdx, a page that landed after this branch's base (NVIDIA-NeMo#4760). Use the target page's title as the label. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: qiaochuz <qiaochuz@nvidia.com> --------- Signed-off-by: qiaochuz <qiaochuz@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Remote Slurm submissions do not inherit the submitting machine's environment. A driver configured with
runtime:INFERENCE_API_KEYcan therefore start successfully but fail judge calls because its credential is missing.Add opt-in
compute.<cluster>.forward_envnames. The executor validates names, fails before staging if a listed variable is unset, and exports shell-quoted values before remotesbatchthrough SSH stdin. Values are not written into resolved configuration or job artifacts. No separate issue is needed for this focused bug fix.Validation
pre-commit run --all-files: passed.Rollouts
A real remote Slurm HLE judge canary completed with 1/1 rollout scored, no failure rows, and a parsed judge response. This validates credential delivery through SSH, submission, and the driver to the judge endpoint; it is not a benchmark score measurement.
Compatibility and benchmark impact
The default empty list preserves existing behavior. Remote jobs needing local credentials opt in by name and continue to use
runtime:references in driver environment configuration. Scheduler-provided variables such asCUDA_VISIBLE_DEVICESare not implicitly forwarded. Slurm must export the submission environment;--export=NONEprevents inheritance. Dry runs and resolution do not require these credentials.No benchmark definitions or grading rules change.
Checklist