test: [3/3] add L3 Nano 4B Gym training E2E by yaoyu-33 · Pull Request #4015 · NVIDIA-NeMo/RL

yaoyu-33 · 2026-09-05T06:14:36Z

Purpose

Add the final L3 layer of the Gym/RL acceptance-test series: a real one-step NeMo RL training job driven by a NeMo Gym environment and verifier.

This covers the gap between L2 rollout compatibility and successful optimizer/refit execution with a bounded deterministic workload.

Coverage

loads nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 at exact revision d552708a9d575fa8d4a690b988fd870d65279f98 for FSDP2 policy, tokenizer, and vLLM
runs on a two-H100 functional runner with colocated TP1 vLLM and two-rank FSDP2
starts Gym math_with_judge in deterministic no-judge mode
loads a checked-in two-row NemoGymDataset fixture with one accepted and one rejected rollout
requires exact train reward min/max/mean of 0/1/0.5 and signed Reinforce++ advantages
completes exactly one optimizer step with finite loss, finite nonzero gradient norm, and finite positive learning rate
requires nonempty explicit generation-worker refit acknowledgements through refit/generation_workers_updated
runs post-refit validation and requires exact accuracy 0.5
checks finite bounded token-probability parity and positive generation/validation timing

The test exposed generation/refit correctness gaps. This PR normalizes temperature=0 to vLLM's unscaled/unfiltered logprob semantics across AutoModel, Megatron, and DTensor paths; scopes tiny-positive clamping to vLLM/Dynamo; rejects unsupported non-unit temperature on fused Megatron logprobs; allows generation configs that omit temperature; and makes direct vLLM/TRT refit success require nonempty, explicit worker acknowledgements.

Validation

Current head: 4e6e44ecde057de137ae9cafc9e24d0550a5dce7

Exact-head x2-H100 Gym training E2E passed: https://github.com/NVIDIA-NeMo/RL/actions/runs/34014189087
- train/grad_norm: finite and positive (110.023094...)
- train/lr: finite and positive (4.999999987e-07)
- train reward min/max/mean: 0 / 1 / 0.5
- refit/generation_workers_updated: 2.0
- post-refit validation/accuracy: 0.5
- validation time: 3.264s
All three current-head Vllm unit-test shards passed: https://github.com/NVIDIA-NeMo/RL/actions/runs/34016095608
Complete exact-head L0/H100 run: 22 of 23 unit-test shards passed and all coverage uploads succeeded: https://github.com/NVIDIA-NeMo/RL/actions/runs/34049400947
Codecov patch coverage passed at 94.12% (32 hit / 2 miss; target 80%)
Ruff/lint, script coverage, YAML/JSONL parsing, shell syntax, Python compile, git diff --check, and DCO sign-off: passed
Multiple independent review/fix rounds; final evidence review found no P0-P2 issues

The functional job took about 12 minutes; the training process itself took about 4.5 minutes. The workflow's fresh exact-head image build took additional time.

Explicit scope boundaries

This proves successful rollout, verifier, optimizer, refit-acknowledgement, and post-refit validation execution; it does not checksum model parameters before/after optimizer or refit.
This configuration exercises colocated legacy IPC refit, not the non-colocated reload_weights transport.
It covers one environment/verifier and one deterministic two-row workload; broad environment coverage belongs to L1/L2, and this is not a convergence or performance-regression test.
GB200 has not been validated by this PR.
The only failed shard in the exact-head L0 run is the unrelated main-branch Megatron model-owned-cp-slicing failure tracked by ci: Bump Megatron-Bridge to 5ed9799 #4022; that fix is intentionally not mixed into this PR.

Three-PR series

test: [1/3] add L1 environment verifier acceptance coverage Gym#3127 - L1 environment/verifier acceptance coverage
test: [2/3] add L2 Gym-to-RL rollout acceptance #4014 - L2 Gym-to-RL rollout acceptance
this PR - L3 full RL training E2E

Draft: the author will review before this is ready for merge.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

copy-pr-bot · 2026-09-05T06:14:41Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-05T06:46:48Z

/ok to test 585004a

yaoyu-33 · 2026-09-05T06:47:22Z

Targeted L3 CI (L1 / Gym_Training / H100 / main image): https://github.com/NVIDIA-NeMo/RL/actions/runs/33950761022

yaoyu-33 · 2026-09-05T06:54:55Z

The first targeted dispatch failed before tests because reusing image_tag=main also sets FAST=1; the runner then looked for the run-id image and never started training. No code failure occurred. Corrected run (branch image build, FAST unset): https://github.com/NVIDIA-NeMo/RL/actions/runs/33951100603

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-05T08:34:05Z

/ok to test d961928

yaoyu-33 · 2026-09-05T08:34:35Z

Corrected L3 x2-H100 run for : https://github.com/NVIDIA-NeMo/RL/actions/runs/33955622296\n\nThe previous real run exposed two test-contract bugs before training: it requested four Ray GPUs on the two-GPU functional runner, and the inherited train dataset remained OpenMathInstruct-2. The current head uses a 1-train + 1-non-colocated-vLLM split and explicitly binds the two-row NemoGymDataset fixture. Independent final reviews found no remaining P0/P1/P2.

yaoyu-33 · 2026-09-05T08:34:46Z

Correction: the new reviewed head is d961928. Corrected L3 x2-H100 run: https://github.com/NVIDIA-NeMo/RL/actions/runs/33955622296. The current head uses 1 training GPU plus 1 non-colocated vLLM GPU and explicitly binds the two-row NemoGymDataset fixture.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-05T18:21:11Z

/ok to test 669c8c7

yaoyu-33 · 2026-09-05T18:31:17Z

Targeted L3 x2-H100 CI for the reviewed two-GPU colocated FSDP2 head: https://github.com/NVIDIA-NeMo/RL/actions/runs/33983791731

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-05T20:11:22Z

/ok to test d759166

yaoyu-33 · 2026-09-05T20:11:24Z

Targeted L3 x2-H100 CI for the reviewed finite, reproducible training-sampling fix: https://github.com/NVIDIA-NeMo/RL/actions/runs/33989221264

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-05T22:15:14Z

/ok to test 174f3d3

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-05T22:31:27Z

/ok to test 9b35225

yaoyu-33 · 2026-09-06T02:12:37Z

Latest-head x2-H100 Gym training E2E passed: https://github.com/NVIDIA-NeMo/RL/actions/runs/33997578151 (target job L1_Functional_Tests_Gym_Training, ~18 minutes). A complete L0 unit matrix is now running at https://github.com/NVIDIA-NeMo/RL/actions/runs/34005722087.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-08T17:50:03Z

/ok to test 4e6e44e

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-08T22:43:31Z

/ok to test e655949

yaoyu-33 · 2026-09-08T23:58:50Z

/ok to test 41dd194

…ining-e2e Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-09T17:58:38Z

/ok to test 1a6d7ad

…ining-e2e Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> # Conflicts: # nemo_rl/models/generation/__init__.py # tests/unit/models/generation/test_vllm_generation.py

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-11T05:21:32Z

/ok to test 6ea3c06

## Purpose Add focused L1 standalone environment/verifier acceptance coverage for three environments in the active RL training mix whose prior tests were empty or helper-only. ## Coverage - `jailbreak_detection`: validates all five checked-in rows, standard and Nemotron verdicts, combined reward, all five production policy mappings, both reward polarities for all six policy verifiers, final-verdict precedence, empty responses, and judge failures - `over_refusal_detection`: validates all five checked-in rows and all represented categories, complied/refused/unclear goldens, standard and Nemotron verdicts, final-verdict precedence, empty responses, and judge failures - `multichallenge`: validates all five checked-in rows and represented challenge shapes, production-config judge behavior, full verification, context construction, rubric aggregation, negative pass criteria, empty rubrics, and judge failures All external judge calls are mocked at the service boundary. The tests exercise real checked-in data schemas and verifier reward logic on CPU without adding test-specific APIs to production code. ## Validation - local focused tests: 33 jailbreak + 11 over-refusal + 12 MultiChallenge = 56 passed - scoped pre-commit: passed - Ruff and `git diff --check`: passed - multiple independent review/fix rounds; closure review found no P0-P2 issues - latest full Gym CI matrix passed: core unit tests and all 8/8 environment server shards ## Three-PR test stack 1. this PR - L1 standalone Gym environment and verifier behavior 2. NVIDIA-NeMo/RL#4014 - L2 Gym-to-RL rollout acceptance 3. NVIDIA-NeMo/RL#4015 - L3 Nano 4B Gym-backed training E2E Draft: the author will review before this is ready for merge. --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

test: add Nano 4B Gym training E2E

e263f48

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

This was referenced Sep 5, 2026

test: [1/3] add L1 environment verifier acceptance coverage NVIDIA-NeMo/Gym#3127

Merged

test: [2/3] add L2 Gym-to-RL rollout acceptance #4014

Merged

yaoyu-33 added 3 commits September 4, 2026 23:26

test: exercise post-update Gym rollout

933b72b

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

test: allowlist pinned model revision

2a3edcb

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

test: prove post-update generation refit

585004a

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

test: fit Nano 4B Gym E2E to H100 runner

d961928

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

test: shard Nano 4B E2E training across two GPUs

669c8c7

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

test: make Nano 4B training sampling finite and reproducible

d759166

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

fix: align greedy rollout and training logprobs

174f3d3

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

fix: validate fused sampling and refit acknowledgements

9b35225

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 added 3 commits September 5, 2026 21:53

fix: allow generation configs without temperature

ee56dd6

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Merge origin/main into L3 Gym training E2E

e1cb603

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

test: require a positive training learning rate

4e6e44e

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

refactor: inline GRPO sampling validation setup

579e298

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 added 2 commits September 8, 2026 13:36

refactor: inline fused sampling validation

b22c4e8

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

refactor: keep fused validation in loss setup

ac7f9ea

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 marked this pull request as ready for review September 8, 2026 22:43

yaoyu-33 requested review from a team as code owners September 8, 2026 22:43

Merge branch 'main' into codex/nano-4b-gym-training-e2e

e655949

yaoyu-33 marked this pull request as draft September 8, 2026 22:53

yaoyu-33 added the CI:L1 Run doctests, unit tests, and functional tests label Sep 8, 2026

Merge branch 'main' into codex/nano-4b-gym-training-e2e

41dd194

yaoyu-33 marked this pull request as ready for review September 9, 2026 00:16

yaoyu-33 added 2 commits September 9, 2026 10:38

Merge remote-tracking branch 'origin/main' into codex/nano-4b-gym-tra…

292e4c6

…ining-e2e Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

test: align fused sampling validation params

1a6d7ad

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 added 2 commits September 9, 2026 15:19

Merge remote-tracking branch 'origin/main' into codex/nano-4b-gym-tra…

9aa1dce

…ining-e2e Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> # Conflicts: # nemo_rl/models/generation/__init__.py # tests/unit/models/generation/test_vllm_generation.py

test: run Gym training E2E on GB200

4358882

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yfw reviewed Sep 10, 2026

View reviewed changes

Comment thread nemo_rl/models/generation/__init__.py Outdated

Comment thread tests/unit/models/generation/test_vllm_generation.py Outdated

Comment thread tests/functional/L1_Functional_Tests_GB200_Gym_Training.sh

Comment thread nemo_rl/algorithms/grpo.py Outdated

fix: address Gym training review feedback

6ea3c06

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Conversation

yaoyu-33 commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Coverage

Validation

Explicit scope boundaries

Three-PR series

Uh oh!

copy-pr-bot Bot commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 5, 2026

Uh oh!

yaoyu-33 commented Sep 6, 2026

Uh oh!

yaoyu-33 commented Sep 8, 2026

Uh oh!

yaoyu-33 commented Sep 8, 2026

Uh oh!

yaoyu-33 commented Sep 8, 2026

Uh oh!

yaoyu-33 commented Sep 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yaoyu-33 commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yaoyu-33 commented Sep 5, 2026 •

edited

Loading