[svcnemo-autobot]

Background

Nightly sync of main into dev for 2026-10-30. The slice has N=46 commits, which were all of dev..main at selection time, oldest first. Dev base: c4220af3. Slice tip (main): 58124853. Merge base: 364eef70.

What changed

• These 46 original main commits are included unchanged and in this order: 53015a0 57018e4 8e13411 31c7230 47715bb 4a55e6f d1eec51 a296523 73f17bc dc36b1a 2723d2f 2cfbc2c 6463d38 3e47f31 a153d5d f77f360 e4a643d 51039e5 a714709 79f5594 26020f4 d5cc5e6 57103cf babb33e b9d4e2b 01ac4e8 928fe9d ccea662 5de526f 0a51326 92e1d6a 53944a3 128d097 7d6895a 6dca79c 7196f83 bc9a10e 1ac85e9 579f710 f677840 0f3e582 a77288b 952fdb7 90e4049 902feb8 5812485
• One merge commit, 44a87715. Its parents are c4220af3 (dev) and 58124853 (main).
• One signed repair commit, 2ca8104c. It fixes the main/dev API mismatches that validation found. It keeps dev's kda_safe_gate ([Dev] Add Ling-V3 Tiny hybrid model training support #6556), restores dev's distribute_saved_activations=False, and refreshes 4 mHC golden-value files that Add mHC precision controls, mean output contract, and full activation recompute with mHC #7521 changed. It also fixes two CI failures from the previous head, 08b2dcef:
  • MoK: the mok-wheel stage in docker/Dockerfile.ci.dev now pins MoK 7a8cfb7f, as main does. Dev's pin, a2ad2d0c, does not accept the shared_output_gate_weight argument that runtime.py:128 passes. That is why GB200 test_real_mok_ddp_numerics failed. 7a8cfb7f already contains docker/patches/mok.patch, so the patch file and its install step are removed. Only the ARM64 image builds MoK.
  • NeMo RL: NVIDIA-NeMo/RL Allow fine-grained offloading with MC impl of full-CG. #4253 renamed L1_Functional_Tests_Megatron_4.sh to L1_Functional_Tests_Megatron_Generation.sh. cicd-main.yml now selects the new name. Dev and main still use the old pattern.

graph LR
  D["dev c4220af3"] --> M["merge 44a87715"]
  S["main: 46 commits to 58124853"] --> M
  M --> R["repair 2ca8104c"]

Loading

Details

• Relative to dev, the graph has 48 commits: 46 imports, 1 merge and 1 repair.
• 64 files had conflicts. Each was resolved individually, and no dev files needed to be restored. This PR accepts main's deletion of nightly-sync-main-to-dev.yml (ci(sync): remove scheduled main-to-dev workflow #8603). CODEOWNERS is identical to dev.
• Remerge-diff: 491,938 bytes across 71 files.
• Dependencies: pyproject.toml and uv.lock are identical to dev. Dockerfile.ci.dev differs from dev only in the MoK pin. The Fused tanh-clamped sqrelu activation for inference #8013 change (FlashInfer 0.7.2, cuDNN frontend 1.32) is deferred. On dev, FlashInfer with a clamped squared_relu raises a RuntimeError. The torch and vllm backends still work.
• Open advisories for reviewers: the graphed split of mHC packed_seq_params; report_theoretical_memory (a bug already in main); KDA with CP>1; deepseek_proxy golden values kept from dev. determinism-exempt covers 6 kernel-coverage findings from imported commits.

Tested

Local H100 runs used image 766267172432.dkr.ecr.us-east-1.amazonaws.com/nemo-autobot/agent-sandbox@sha256:6a14e041d26a8ebc12e1add41d331ab7e90f01be06262b0245b543f0c6cdc28a. No CI results were reused for local rows.

• Unit: 71 rows. 69 passed. 2 were skipped upstream (test_batch_invariant_handoff.py and test_disagg_coordinator_pairwise.py), because NIXL is not installed. Upstream run 40846739249 skips them the same way.
• Functional: 84 of 84 passed.
• These results still apply to 2ca8104c. Since 08b2dcef, the repair changes no Python, test or lock file, and the AMD64 H100 image does not build MoK. The lint mirror passed on 08b2dcef.
• GitHub CICD run 41299281492 on 08b2dcef had 4 failures and 1 cancelled job. This head fixes the MoK and NeMo RL failures. test_virtual_expert_planner_graph_replay[dtype1-8] looks flaky: its test and source are unchanged from dev, and it passes on dev, on main and locally. a2a_overlap/test_schedule_chunk_1f1b.py passed 189 tests and was then cancelled at the job time limit. CI on 2ca8104c is pending.
• Still needed: the Run MBridge tests label and a DCO resolution for 18 imported main commits.

[svcnvidia-nemo-ci]

/ok to test 08b2dce

[svcnemo-autobot]

CI status for head 08b2dcef. This head needs no code change. All three CI failures need a maintainer action.

• linting: The only failure is check_kernel_determinism_coverage. Its 6 findings come from kernel changes imported from main (Integrate five inference pairwise campaigns and prefix-cache prompt scores #8426, Add KDA (Kimi Delta Attention) hybrid layers with GLM-style two-stage gates #7054, Reduce DSv4 output RoPE activation memory #8399, Add first-class HybridModel config lists in Core #7555, Add mHC precision controls, mean output contract, and full activation recompute with mHC #7521). The earlier sync PRs chore: AUT-2655 nightly sync main into dev (2026-10-27) #8643, chore: AUT-2616 nightly sync main into dev (2026-10-23) #8461 and chore: AUT-2598 nightly sync main into dev (2026-10-20) #8402 used the determinism-exempt label for this check.
• Nemo_CICD_Test: This failed only because linting failed. In CICD run 41299281492 the unit, integration, MBridge and NeMo-RL jobs were skipped.
• DCO / DCO gate: These fail on 18 commits imported from main, for example 6463d38, 128d097 and 0f3e582. In each one, the sign-off name or email does not match the commit author. The sync must keep the original main commit SHAs, so these commits cannot be rewritten. chore: AUT-2616 nightly sync main into dev (2026-10-23) #8461 passed DCO after an override. chore: AUT-2655 nightly sync main into dev (2026-10-27) #8643 was merged with the same DCO result.

Requested maintainer actions:

1. Review the 6 determinism findings. If you accept them, add determinism-exempt.
2. Add Run MBridge tests.
3. Resolve DCO for the imported commits.
4. Rerun CICD on this head.

Local H100 validation of this head: 84 of 84 functional rows passed. Of the 71 unit rows, 69 passed and 2 were skipped upstream because NIXL is not installed.

[FDecaYed]

Please reconcile the dependency requirements of #8013 before merging this sync.

The imported clamped squared-ReLU FlashInfer path needs ActivationType.ClampedRelu2 and clamped_relu2_limit (resolver). Main brought that feature in with FlashInfer >=0.7.2, but this PR still requires <0.7.0 and locks 0.6.18.post1. That version lacks both APIs, so selecting FlashInfer + squared_relu + a positive tanh clamp fails during model construction. The explicit RuntimeError prevents silent corruption, but does not preserve the incoming main capability.

Please reconcile the compatible dependency pins, lockfile and image inputs, checking the accompanying cuDNN frontend requirement too; or explicitly defer this feature with its supported configuration boundary. A note that the dependency triple matches dev does not resolve this mismatch.

Separately, this head's CICD stopped at the determinism-coverage gate, so the unit, functional and downstream jobs were skipped. Those are validation gaps, not observed training failures. The local H100 results and refreshed mHC goldens still need accessible source/environment-bound evidence before approval.

[svcnemo-autobot]

@FDecaYed, thank you for the review. This reply is for head 08b2dcef.

1. #8013 dependencies

I confirmed the finding. #8013 (a7147091) changes three dependency files on main:

• pyproject.toml: flashinfer-python>=0.7.2,<0.8.0 and nvidia-cudnn-frontend[cutedsl]==1.32.0.
• uv.lock: FlashInfer 0.7.2.post1 and cuDNN frontend 1.32.0.
• docker/Dockerfile.ci.dev: a new uv cache id, so that TE rebuilds against the cuDNN frontend 1.32 headers.

The sync rule keeps these three files identical to dev. So this head locks FlashInfer 0.6.18.post1 and cuDNN frontend 1.29.0. Dev was already behind main here: dev pins <0.7.0 (#4760), and main pinned <0.8.0 before #8013.

I took the deferral option and did not change the dependency files. I asked for this boundary to be added to the PR description:

• Supported on dev: the tanh clamp with inference_grouped_gemm_backend torch or vllm, and FlashInfer with unclamped squared_relu.
• Not supported on dev: FlashInfer with squared_relu and a tanh clamp. Model construction stops with the RuntimeError in _resolve_flashinfer_activation_type. There is no silent fallback.
• The Fused tanh-clamped sqrelu activation for inference #8013 code is identical to main. A later dev dependency change enables this path without a code change.

The local H100 unit run covered the error path: test_flashinfer_clamped_relu2_backend_capability passed for supported, missing-enum and missing-keyword.

If you want dev to adopt FlashInfer 0.7.2 and cuDNN frontend 1.32 in this PR, reply with that decision. That change goes outside the sync rule for the dependency files. It needs a regenerated uv.lock, a new CI image with a TE rebuild, and a new full validation run.

2. CI evidence on this head

I agree. The local H100 results do not replace CICD on this head. CICD run 41299281492 stopped at linting, and the only linting failure is the 6 kernel-determinism-coverage findings. So the unit, functional, MBridge and NeMo-RL jobs were skipped.

To get CI evidence on 08b2dcef, a maintainer must:

1. Review the 6 findings and apply determinism-exempt if they are acceptable.
2. Add the Run MBridge tests label.
3. Resolve DCO for the 18 imported main commits.
4. Rerun CICD.

The functional jobs will then check the 4 refreshed mHC golden-value files in 08b2dcef on this exact head. I will diagnose any failure from that run.

[FDecaYed]

Thanks, the explicit deferral addresses my dependency finding. For clamped squared-ReLU, the documented torch/vLLM alternatives retain the clamp, while the locked FlashInfer version rejects that combination at construction rather than silently changing behavior. I’m no longer requesting a dependency upgrade in this sync.

Approval is still pending the validation evidence noted in my earlier review: accessible runtime results tied to this source and environment, and the generation logs/inputs for the four refreshed mHC goldens. The current CICD attempt stopped before unit, functional and downstream tests; skipped jobs are not runtime failures. Existing local artifacts or appropriately scoped CI evidence can address the gap—this clarification alone does not require another full matrix.

[svcnemo-autobot]

@FDecaYed Thank you for the second review. This reply answers the open item: runtime results, and the inputs and logs for the 4 refreshed mHC golden files. It does not change code. The PR head is still 08b2dcef.

Runtime environment (all local runs)

• Hardware: 1 node with 8×H100.
• Image: 766267172432.dkr.ecr.us-east-1.amazonaws.com/nemo-autobot/agent-sandbox@sha256:6a14e041d26a8ebc12e1add41d331ab7e90f01be06262b0245b543f0c6cdc28a.
• Test-launcher profile digest: 1828fd871eee24c95b1f6815c7b8fe364b39560fcfbb97435ecc59a3a4308e9a.
• Each run log prints its source revision as Megatron-Core version 0.21.0+<sha>.

How the 4 golden files were made

1. On merge commit 44a87715, before any repair, each test ran with its own model_config.yaml. The inputs include --deterministic-mode true, --train-iters 50, --enable-hyper-connections true, --mhc-sinkhorn-iterations 20, --recompute-modules [mhc] and --mhc-recompute-layer-num 2. gpt3_mcore_te_tp2_pp2_mhc is a ckpt-resume test. The other 3 are regular tests.
2. Training finished in each run. Then the DETERMINISTIC comparison against the old dev golden values failed.
3. Each run wrote mlm/golden_values_dev_dgx_h100.json from its TensorBoard log. Each committed file is byte-identical to the file from its run.

Test case (gpt/)	Generation run on 44a87715	Verification run on f583febb
gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph	80c51021 / 0441297f	ac9213dd: passed
..._ep4_mhc_overlap_attn_cudagraph_nosplit	c323c461 / fb35ae3f	6e74959e: passed
gpt3_mcore_te_tp2_pp2_mhc	f0e98266 / 3fce929a	d399e739: passed
gpt3_mcore_te_tp2_pp2_mhc_attn_cudagraph	26daa41e / 722e1198	4c18b23c: passed

Verification of the new files
Commit f583febb is the repair commit that carries the refreshed files. Each of the 4 tests ran again on it. In all 4 runs, the DETERMINISTIC and APPROXIMATE checks passed for lm loss and for num-zeros, and pytest reported 1 passed. git diff f583febb..08b2dcef changes only megatron/core/ssm/gated_delta_net/kda.py, megatron/core/ssm/kda_layer_config.py and 3 unit-test files. The golden files and hyper_connection.py did not change.

Why the values changed
#7521 (8e134118) changes the mHC arithmetic in hyper_connection.py:

• The Sinkhorn forward now runs on a detached input.
• h_aggregate now casts back to the input dtype.
• The h_post bmm now runs in FP32 with autocast off.

In #7521, main refreshed its own mHC golden values. These 4 test cases exist only on dev.

Test case	lm loss max abs diff (50 steps)	num-zeros max diff	mem-max-allocated-bytes
ep4_mhc_overlap_attn_cudagraph	5.1e-4	290	309 MB lower
..._nosplit	6.6e-4	332	309 MB lower
tp2_pp2_mhc	5.9e-5 (47 of 50 steps changed)	140	204 MB lower
mhc_attn_cudagraph	5.5e-5	226	204 MB lower

Access
The new values are in this PR diff, and the old values are on dev. The full run logs are stored in this automation's artifact store, not on GitHub. On request, I can post any log section here.

GitHub CI
GitHub CICD on 08b2dcef stopped at the linting job. Only the determinism-coverage check failed. So GitHub has no runtime results for this head yet. To get them, a maintainer must do these steps:

1. Decide about determinism-exempt.
2. Add the Run MBridge tests label.
3. Resolve DCO for the 18 imported main commits.
4. Run CICD again.

[FDecaYed]

Thanks, this clarifies the environment and how the four mHC goldens were generated. Could you post the existing log excerpts (or accessible artifact links) for those runs: source/image and executed recipe, training completion, and the DETERMINISTIC/APPROXIMATE results? Please include the generated-file hashes so we can match them to the committed goldens.

I couldn't resolve f583febb in the fork. Please include its full SHA and the source diff to 08b2dcef, or an accessible source snapshot if that commit is no longer published. The result summaries for the previously reported unit/functional runs would also help close the runtime evidence gap. No new test matrix requested here—just the existing evidence.

[svcnemo-autobot]

@FDecaYed, this comment answers review 5727959735 with evidence that I already have. I did not run new tests.

GitHub cannot read the session storage that holds the run logs. So I copied the relevant log lines into this comment. All runs used one node with 8 H100 GPUs, the same test profile, and this image:
766267172432.dkr.ecr.us-east-1.amazonaws.com/nemo-autobot/agent-sandbox@sha256:6a14e041d26a8ebc12e1add41d331ab7e90f01be06262b0245b543f0c6cdc28a

The run IDs are short workload IDs. The line numbers in the excerpts are from the rank-0 stdout.log of each run.

1. mHC golden-value runs (R3)

I did these steps for each of the 4 test cases:

1. I ran the test case on the merge commit 44a87715 (log: Megatron-Core version .......... 0.21.0+44a877155). Training completed 50 of 50 iterations, with skipped: 0 and nan: 0.
2. The test compared the run with the old dev golden file. The DETERMINISTIC check failed. The APPROXIMATE check passed: 0 of 50 samples were outside tolerance (rtol 0.05 for lm loss, rtol 0.2 for num-zeros). So pytest reported 1 failed.
3. I committed the golden file that the run generated, without changes. Its sha256 is equal to the committed file on f583febb and on 08b2dcef.
4. I ran the test case again on f583febb, which contains the new golden files (log: 0.21.0+f583febbd). pytest passed. The final lm loss and num-zeros values are equal to the values of the generation run.

Test case	Generation run on 44a87715	Comparison with the old dev golden file	Verification run on f583febb
tp2_pp2_ep4_mhc_overlap_attn_cudagraph	80c51021: iteration 50/50, lm loss 1.077237E+01, num zeros 11535. pytest: 1 failed	DETERMINISTIC failed (lm loss 50/50, num-zeros 50/50 samples); APPROXIMATE passed (0/50)	ac9213dd: iteration 50/50, lm loss 1.077237E+01, num zeros 11535. pytest: 1 passed
tp2_pp2_ep4_mhc_overlap_attn_cudagraph_nosplit	c323c461: iteration 50/50, lm loss 1.077225E+01, num zeros 11821. pytest: 1 failed	DETERMINISTIC failed (lm loss 50/50, num-zeros 50/50 samples); APPROXIMATE passed (0/50)	6e74959e: iteration 50/50, lm loss 1.077225E+01, num zeros 11821. pytest: 1 passed
tp2_pp2_mhc	f0e98266: iteration 50/50, lm loss 9.866622E+00, num zeros 2445; lm loss 9.866622E+00, num zeros 2445. pytest: 1 failed	DETERMINISTIC failed (lm loss 47/50, num-zeros 50/50 samples); APPROXIMATE passed (0/50)	d399e739: iteration 50/50, lm loss 9.866622E+00, num zeros 2445; lm loss 9.866622E+00, num zeros 2445. pytest: 1 passed; 1 passed
tp2_pp2_mhc_attn_cudagraph	26daa41e: iteration 50/50, lm loss 1.065367E+01, num zeros 2584. pytest: 1 failed	DETERMINISTIC failed (lm loss 50/50, num-zeros 50/50 samples); APPROXIMATE passed (0/50)	4c18b23c: iteration 50/50, lm loss 1.065367E+01, num zeros 2584. pytest: 1 passed

The tp2_pp2_mhc log has two training passes, so it shows two iteration 50/50 lines. Its verification run ran pytest two times.

Recipe and command. Each run used the model_config.yaml of its test case. This file has the same Git blob on c4220af3 (dev), 44a87715, f583febb and 08b2dcef. For each test case, the generation run and the verification run used the same launch command. I compared the sha256 of the normalized command line.

Test case	model_config.yaml blob	Normalized command sha256 (first 16 hex digits)
tp2_pp2_ep4_mhc_overlap_attn_cudagraph	9a7d0a89b522	81638c2896e04279
tp2_pp2_ep4_mhc_overlap_attn_cudagraph_nosplit	eba1c28e4f7e	f8df4513dd162899
tp2_pp2_mhc	686c8bdbb59d	0255f97c15559748, 6d02609f09e045fa
tp2_pp2_mhc_attn_cudagraph	dfd22d2da653	571c1eb46cc3af3d

Golden-file hashes. These are sha256 values of golden_values_dev_dgx_h100.json (git show <sha>:<path> | sha256sum for committed files):

tp2_pp2_ep4_mhc_overlap_attn_cudagraph
  old dev golden (c4220af3):          04aeea4728dcc2165d899c16b4a2b7b5b7a6d010b0e4fa42046295a54e221b45
  generated by 80c51021 (44a87715):  1860318b24ef2096d8d67a58ba5859631ea26e98c9d57cc771e47b6a25b6d957
  committed on f583febb and 08b2dcef: 1860318b24ef2096d8d67a58ba5859631ea26e98c9d57cc771e47b6a25b6d957
tp2_pp2_ep4_mhc_overlap_attn_cudagraph_nosplit
  old dev golden (c4220af3):          bfeb17d161c53c861a611d663540304f71eacdf36fa87cac3ce69234610f8a27
  generated by c323c461 (44a87715):  4d8ce5b52b95e1a549bfde664552447d8a0357681103160b47233baf35c73dbf
  committed on f583febb and 08b2dcef: 4d8ce5b52b95e1a549bfde664552447d8a0357681103160b47233baf35c73dbf
tp2_pp2_mhc
  old dev golden (c4220af3):          a1646d2b4f2a8319edd3770a57371b1dba508f77dca27aa14aa0f6c3d37c78d0
  generated by f0e98266 (44a87715):  629ece714108f15da534914f01d5686502c231dfd34af998447cdf6b41efa44e
  committed on f583febb and 08b2dcef: 629ece714108f15da534914f01d5686502c231dfd34af998447cdf6b41efa44e
tp2_pp2_mhc_attn_cudagraph
  old dev golden (c4220af3):          42bb1fde087adf90c71f29acb267233c125ddeb7ee39a2be38b2c769373d5c6d
  generated by 26daa41e (44a87715):  4de5088aefd5e577c08fc248395251ce89f7274bfb5c43433d007cdf374f0292
  committed on f583febb and 08b2dcef: 4de5088aefd5e577c08fc248395251ce89f7274bfb5c43433d007cdf374f0292

The verification runs also wrote a golden file. Compared with the committed file, it is different only in iteration-time, and for 3 test cases also in mem-allocated-bytes and mem-max-allocated-bytes. All other metrics, including lm loss and num-zeros, are equal.

Log excerpts (8 runs)

tp2_pp2_ep4_mhc_overlap_attn_cudagraph

Generation run 80c51021 on 44a87715:

 1522: [default0]:> Megatron-Core version .......... 0.21.0+44a877155
 2704: [default7]: [2026-10-31 11:18:36.625988] iteration 50/50 | lm loss: 1.077237E+01 | num zeros: 11535.0 | skipped: 0 | nan: 0
 2769: + pytest test_pretraining_regular_pipeline.py --golden-values-path .../gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph/golden_values_dev_dgx_h100.json --actual-values-path /mnt/artifacts/mlm/golden_values_dev_dgx_h100.json
 2781: Golden comparison for lm loss [DETERMINISTIC]: FAILED; 50/50 samples outside tolerance; atol=0, rtol=0, precision=full
 2787: Golden comparison for lm loss [APPROXIMATE]: PASSED; 0/50 samples outside tolerance; atol=0, rtol=0.05, precision=rounded_5_decimal_places
 2790: Golden comparison for num-zeros [DETERMINISTIC]: FAILED; 50/50 samples outside tolerance; atol=0, rtol=0, precision=full
 2796: Golden comparison for num-zeros [APPROXIMATE]: PASSED; 0/50 samples outside tolerance; atol=0, rtol=0.2, precision=rounded_5_decimal_places
 3035: ============================== 1 failed in 0.10s ===============================

Verification run ac9213dd on f583febb:

 1522: [default0]:> Megatron-Core version .......... 0.21.0+f583febbd
 2706: [default7]: [2026-10-31 18:39:39.692561] iteration 50/50 | lm loss: 1.077237E+01 | num zeros: 11535.0 | skipped: 0 | nan: 0
 2799: ============================== 1 passed in 0.03s ===============================

tp2_pp2_ep4_mhc_overlap_attn_cudagraph_nosplit

Generation run c323c461 on 44a87715:

 1511: [default0]:> Megatron-Core version .......... 0.21.0+44a877155
 2691: [default7]: [2026-10-31 11:29:09.648897] iteration 50/50 | lm loss: 1.077225E+01 | num zeros: 11821.0 | skipped: 0 | nan: 0
 2756: + pytest test_pretraining_regular_pipeline.py --golden-values-path .../gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph_nosplit/golden_values_dev_dgx_h100.json --actual-values-path /mnt/artifacts/mlm/golden_values_dev_dgx_h100.json
 2768: Golden comparison for lm loss [DETERMINISTIC]: FAILED; 50/50 samples outside tolerance; atol=0, rtol=0, precision=full
 2774: Golden comparison for lm loss [APPROXIMATE]: PASSED; 0/50 samples outside tolerance; atol=0, rtol=0.05, precision=rounded_5_decimal_places
 2777: Golden comparison for num-zeros [DETERMINISTIC]: FAILED; 50/50 samples outside tolerance; atol=0, rtol=0, precision=full
 2784: Golden comparison for num-zeros [APPROXIMATE]: PASSED; 0/50 samples outside tolerance; atol=0, rtol=0.2, precision=rounded_5_decimal_places
 3026: ============================== 1 failed in 0.10s ===============================

Verification run 6e74959e on f583febb:

 1509: [default0]:> Megatron-Core version .......... 0.21.0+f583febbd
 2689: [default7]: [2026-10-31 18:55:29.435303] iteration 50/50 | lm loss: 1.077225E+01 | num zeros: 11821.0 | skipped: 0 | nan: 0
 2784: ============================== 1 passed in 0.03s ===============================

tp2_pp2_mhc

Generation run f0e98266 on 44a87715:

 1238: [default0]:> Megatron-Core version .......... 0.21.0+44a877155
 2404: [default7]: [2026-10-31 11:29:18.959988] iteration 50/50 | lm loss: 9.866622E+00 | num zeros: 2445.0 | skipped: 0 | nan: 0
 4668: [default7]: [2026-10-31 11:30:44.445005] iteration 50/50 | lm loss: 9.866622E+00 | num zeros: 2445.0 | skipped: 0 | nan: 0
 4751: + pytest test_pretraining_regular_pipeline.py --golden-values-path .../gpt3_mcore_te_tp2_pp2_mhc/golden_values_dev_dgx_h100.json --actual-values-path /mnt/artifacts/mlm/golden_values_dev_dgx_h100.json
 4763: Golden comparison for lm loss [DETERMINISTIC]: FAILED; 47/50 samples outside tolerance; atol=0, rtol=0, precision=full
 4769: Golden comparison for lm loss [APPROXIMATE]: PASSED; 0/50 samples outside tolerance; atol=0, rtol=0.05, precision=rounded_5_decimal_places
 4772: Golden comparison for num-zeros [DETERMINISTIC]: FAILED; 50/50 samples outside tolerance; atol=0, rtol=0, precision=full
 4778: Golden comparison for num-zeros [APPROXIMATE]: PASSED; 0/50 samples outside tolerance; atol=0, rtol=0.2, precision=rounded_5_decimal_places
 5016: ============================== 1 failed in 0.10s ===============================

Verification run d399e739 on f583febb:

 1242: [default0]:> Megatron-Core version .......... 0.21.0+f583febbd
 2408: [default7]: [2026-10-31 19:03:01.814609] iteration 50/50 | lm loss: 9.866622E+00 | num zeros: 2445.0 | skipped: 0 | nan: 0
 4673: [default7]: [2026-10-31 19:04:25.201785] iteration 50/50 | lm loss: 9.866622E+00 | num zeros: 2445.0 | skipped: 0 | nan: 0
 4785: ============================== 1 passed in 0.03s ===============================

tp2_pp2_mhc_attn_cudagraph

Generation run 26daa41e on 44a87715:

 1297: [default0]:> Megatron-Core version .......... 0.21.0+44a877155
 2468: [default7]: [2026-10-31 11:28:51.109026] iteration 50/50 | lm loss: 1.065367E+01 | num zeros: 2584.0 | skipped: 0 | nan: 0
 2533: + pytest test_pretraining_regular_pipeline.py --golden-values-path .../gpt3_mcore_te_tp2_pp2_mhc_attn_cudagraph/golden_values_dev_dgx_h100.json --actual-values-path /mnt/artifacts/mlm/golden_values_dev_dgx_h100.json
 2545: Golden comparison for lm loss [DETERMINISTIC]: FAILED; 50/50 samples outside tolerance; atol=0, rtol=0, precision=full
 2551: Golden comparison for lm loss [APPROXIMATE]: PASSED; 0/50 samples outside tolerance; atol=0, rtol=0.05, precision=rounded_5_decimal_places
 2555: Golden comparison for num-zeros [DETERMINISTIC]: FAILED; 50/50 samples outside tolerance; atol=0, rtol=0, precision=full
 2561: Golden comparison for num-zeros [APPROXIMATE]: PASSED; 0/50 samples outside tolerance; atol=0, rtol=0.2, precision=rounded_5_decimal_places
 2801: ============================== 1 failed in 0.10s ===============================

Verification run 4c18b23c on f583febb:

 1299: [default0]:> Megatron-Core version .......... 0.21.0+f583febbd
 2469: [default7]: [2026-10-31 19:02:32.313080] iteration 50/50 | lm loss: 1.065367E+01 | num zeros: 2584.0 | skipped: 0 | nan: 0
 2565: ============================== 1 passed in 0.03s ===============================

2. Commit f583febb (R4)

• Full SHA: f583febbdd9fd2b88997434072a75dcca9a12dfb.
• Parent: 44a877155dfe3372d6b367f420842bb36d2bed15 (the merge commit). The signature is good (%G? = G).
• It was a local candidate for the repair commit. I did not push it, so GitHub does not have it. I amended later fixes into the same repair commit, and the result is 08b2dcef.
• The 4 golden files are the same on f583febb and 08b2dcef.

The diff from f583febb to 08b2dcef changes 5 files:

• kda.py: on the direct-gate path (two_stage_gates is false), _apply_gated_norm reshapes the gate after out_norm again. This restores the dev op order. It fixes test_kda_gated_norm_preserves_upstream_rounding[norm-bf16-fp32] in transformer/**, which failed on f583febb. Main Add KDA (Kimi Delta Attention) hybrid layers with GLM-style two-stage gates #7054 had moved the reshape before out_norm.
• kda_layer_config.py: KDALayerConfig.__post_init__ calls validate_kda() one more time, before super().__post_init__(). Thus an invalid KDA setting raises the KDA-specific ValueError.
• 3 unit-test files get dev compatibility fixes. Dev keeps kda_safe_gate and kda_lower_bound on TransformerConfig ([Dev] Add Ling-V3 Tiny hybrid model training support #6556). The data test patches get_tokenizer (main refactor(training): Refactor Tokenizer Initialization #8391). A test double gets hybrid_layer_config_list=None.

Diff f583febb..08b2dce

 megatron/core/ssm/gated_delta_net/kda.py                  |  5 ++++-
 megatron/core/ssm/kda_layer_config.py                     |  5 +++++
 tests/unit_tests/data/test_pretrain_gpt_dataset_config.py |  2 +-
 tests/unit_tests/ssm/test_kda.py                          | 12 +++++++++++-
 tests/unit_tests/transformer/test_thd_cuda_graph.py       |  1 +
 5 files changed, 22 insertions(+), 3 deletions(-)

diff --git a/megatron/core/ssm/gated_delta_net/kda.py b/megatron/core/ssm/gated_delta_net/kda.py
index 72d8e3204..38531a03e 100644
--- a/megatron/core/ssm/gated_delta_net/kda.py
+++ b/megatron/core/ssm/gated_delta_net/kda.py
@@ -254,16 +254,19 @@ class KimiDeltaAttention(_GDNBase):
 
         x_dtype = x.dtype
         x = x.reshape(-1, self.value_head_dim)
-        gate = gate.reshape(-1, self.value_head_dim)
         if self.two_stage_gates:
             # Round only after both normalization and sigmoid gating.
+            gate = gate.reshape(-1, self.value_head_dim)
             weight = self.out_norm.weight
             if self.config.layernorm_zero_centered_gamma:
                 weight = weight + 1
             return rms_norm_gated(
                 x, gate, weight, None, activation="sigmoid", eps=self.config.layernorm_epsilon
             )
+        # Keep the direct-gate graph order (norm, then gate reshape) so the compiled
+        # rounding matches the pinned upstream primitive.
         x = self.out_norm(x)
+        gate = gate.reshape(-1, self.value_head_dim)
         return apply_attention_output_gate(x, gate, 'elementwise', cast_mode='none').to(x_dtype)
 
     def _proj(self, proj, x):
diff --git a/megatron/core/ssm/kda_layer_config.py b/megatron/core/ssm/kda_layer_config.py
index b84617669..20d742fb1 100644
--- a/megatron/core/ssm/kda_layer_config.py
+++ b/megatron/core/ssm/kda_layer_config.py
@@ -31,6 +31,11 @@ class KDALayerConfig(GDNLayerConfig):
     """Use low-rank f_b(f_a(x)) and g_b(g_a(x)) gates with a QKV-only input projection."""
 
     def __post_init__(self) -> None:
+        # Validate the KDA contract before the shared TransformerConfig checks, which also
+        # cover GDN-family fields with generic assertions, so invalid KDA settings raise the
+        # KDA-specific ValueError. An unset linear_cp_mode skips the CP-mode checks here; it is
+        # normalized by super().__post_init__() and validated again afterwards.
+        self.validate_kda()
         super().__post_init__()
         self.validate_kda()
 
diff --git a/tests/unit_tests/data/test_pretrain_gpt_dataset_config.py b/tests/unit_tests/data/test_pretrain_gpt_dataset_config.py
index 770f2cd6c..88d2cca43 100644
--- a/tests/unit_tests/data/test_pretrain_gpt_dataset_config.py
+++ b/tests/unit_tests/data/test_pretrain_gpt_dataset_config.py
@@ -25,7 +25,7 @@ def test_core_gpt_dataset_config_forwards_sft_mock_dataset_config(monkeypatch, r
         * args.pipeline_model_parallel_size
         * args.context_parallel_size
     )
-    monkeypatch.setattr(pretrain_gpt, "build_tokenizer", lambda _: SimpleNamespace(vocab_size=2048))
+    monkeypatch.setattr(pretrain_gpt, "get_tokenizer", lambda: SimpleNamespace(vocab_size=2048))
     monkeypatch.setattr(pretrain_gpt, "get_blend_and_blend_per_split", lambda _: (None, None))
 
     config = pretrain_gpt.core_gpt_dataset_config_from_args(args)
diff --git a/tests/unit_tests/ssm/test_kda.py b/tests/unit_tests/ssm/test_kda.py
index 2462156c1..21ce90b7d 100644
--- a/tests/unit_tests/ssm/test_kda.py
+++ b/tests/unit_tests/ssm/test_kda.py
@@ -471,7 +471,13 @@ def test_kda_layer_config_preserves_settings(explicit_settings):
 
     assert type(layer_config) is KDALayerConfig
     assert layer_config is not config
-    assert set(expected).isdisjoint(field.name for field in fields(TransformerConfig))
+    # dev keeps kda_safe_gate/kda_lower_bound on TransformerConfig for the
+    # --kda-safe-gate/--kda-lower-bound CLI flags (#6556); the other KDA-only
+    # settings must stay off the stack config.
+    dev_stack_fields = {"kda_safe_gate", "kda_lower_bound"}
+    transformer_fields = {field.name for field in fields(TransformerConfig)}
+    assert dev_stack_fields <= transformer_fields
+    assert (set(expected) - dev_stack_fields).isdisjoint(transformer_fields)
     assert set(expected) <= {field.name for field in fields(KDALayerConfig)}
     for name, value in expected.items():
         assert getattr(layer_config, name) == value
@@ -634,11 +640,15 @@ def test_parallel_kda_correctness(
 @pytest.mark.parametrize("recompute_module", ["gdn_norm_out", "gdn"])
 @pytest.mark.parametrize("variant", [None, "kda"])
 def test_kda_layer_config_accepts_recompute_modules(recompute_module, variant):
+    # dev (#6556) registers "kda" as a GDN-family variant, so a non-hybrid KDA config is a
+    # standalone linear-attention model and must declare its LA/SDPA layer pattern.
+    linear_attention_freq = [1] if variant == "kda" else None
     config = _make_config(
         recompute_granularity="selective",
         recompute_modules=[recompute_module],
         experimental_attention_variant=variant,
         is_hybrid_model=False,
+        linear_attention_freq=linear_attention_freq,
     )
     assert config.recompute_modules == [recompute_module]
 
diff --git a/tests/unit_tests/transformer/test_thd_cuda_graph.py b/tests/unit_tests/transformer/test_thd_cuda_graph.py
index 203f75e27..68d31338b 100644
--- a/tests/unit_tests/transformer/test_thd_cuda_graph.py
+++ b/tests/unit_tests/transformer/test_thd_cuda_graph.py
@@ -201,6 +201,7 @@ def test_hybrid_padding_mask_sp_scatter_uses_model_tp_group(monkeypatch, input_p
             mtp_num_layers=1 if mtp_enabled else None,
             freeze_base_model_for_mtp=False,
         ),
+        hybrid_layer_config_list=None,
         pre_process=input_path != "pipeline",
         post_process=False,
         embedding=FakeEmbedding(),

3. Unit and functional results (R5)

The matrix has 71 unit rows and 84 functional rows. Each row has one result on one tested SHA:

Tested SHA	Unit rows	Functional rows	Files that change from this SHA to 08b2dcef
08b2dcef	2 passed	–	none
04709f9f	1 passed	–	tests/unit_tests/ssm/test_kda.py
dcd760ab	1 passed	–	ssm/test_kda.py, transformer/test_thd_cuda_graph.py
f583febb	2 passed	4 passed	the 5 files in section 2
44a87715	63 passed, 2 skipped upstream	80 passed	13 files: kda.py, kda_layer_config.py, the 4 mHC golden files and 7 unit-test files
Total	71: 69 passed, 2 skipped upstream	84: 84 passed

Full SHAs: 04709f9fa044172b9e0b98dc8d1a814f16f9fb14, dcd760aba32de57a468565d809418760fa6afb28. No retained result is from the candidates 3ce3f8d2 (3ce3f8d21203af614174481952121771e120d138) and 258bb660 (258bb660eda749cca46b50316d65bec2cd80a935).

Why a pass on an older SHA applies to 08b2dcef. I kept a pass only if the diff from its tested SHA to 08b2dcef does not change an input of that row:

• After 44a87715, no dependency, lock, Docker, CI or test-utility file changed (pyproject.toml, uv.lock, docker/, .github/, tests/test_utils/). All runs used the same image and test profile.
• Only the 4 mHC rows read the 4 golden files. These rows passed on f583febb, which has the same golden files as 08b2dcef.
• Each of the 7 changed unit-test files belongs to one row. That row ran on a SHA that has the final version of the file: data/** on dcd760ab, models/** and transformer/moe/** on f583febb, transformer/** on 04709f9f, and ssm/** on 08b2dcef.
• kda.py, recompute change: the two checkpoint calls pass False again instead of distribute_saved_activations. This has an effect only during training with recompute_gdn and distribute_saved_activations both set. Outside the rows that ran again, no KDA test sets these flags.
• kda.py, gate-reshape change: this has an effect only when a test runs KimiDeltaAttention._apply_gated_norm. Only the ssm/**, ssm/gated_delta_net/test_kda.py and transformer/** rows do that, and they ran on SHAs with the final kda.py. The other KDA consumers do not run it. The determinism kernel tests call only _compute_gates and chunk_kda, and test_gated_norm.py uses the unchanged GatedDeltaNet._apply_gated_norm. inference/contexts/test_dynamic_context.py checks that KDA inference fails early. The models/** files build only configs and module specs.
• No functional row in this matrix uses KDA. The only KDA functional case, hybrid_mr_mcore_te_tp1_pp1_ep8_cp2_kda_gated_mla_1N8G, is not in the matrix.
• kda_layer_config.py: the extra validate_kda() call cannot reject a config that the later call accepts. Of the fields that validate_kda() reads, TransformerConfig.__post_init__ sets only linear_cp_mode, and the CP checks skip while that field is unset. test_ssm_kernels.py uses KDALayerConfig.from_config, which does not run __post_init__.
• Skipped upstream: inference/engines/test_batch_invariant_handoff.py and inference/engines/test_disagg_coordinator_pairwise.py. The log shows NIXL is not installed; skipping .... Upstream main run 40846739249 skips them the same way (jobs 123357253143 and 123357253359, at 0a513268). Their inputs did not change from 44a87715 to 08b2dcef. These 2 rows are skipped, not passed.

For rows that have no pytest summary line, the lists show the line that the test script prints instead. Most unit rows print two pytest summary lines, one for each pytest call in the row script. The lists show both, in order. ×N means that the row printed the same summary N times.

Unit rows (71)

Rows without a SHA mark ran on 44a87715.

• examples/multimodal_dev/tests/**/*.py: 83 passed / 83 deselected
• **/*.py: 4391 passed, 162 skipped, 94 deselected, 19 subtests passed / 4647 deselected
• a2a_overlap/**/*.py: 128 passed, 1 skipped, 4 deselected / 133 deselected
• a2a_overlap/test_fsdp_1f1b_overlap.py: 69 passed / 69 deselected
• a2a_overlap/test_schedule_chunk_1f1b.py: 189 passed / 189 deselected
• a2a_overlap/test_schedule_chunk_1f1b_layers_1_1.py: 60 passed / 60 deselected
• a2a_overlap/test_schedule_chunk_1f1b_layers_1_2.py: 60 passed / 60 deselected
• a2a_overlap/test_schedule_chunk_1f1b_padding.py: 24 passed / 24 deselected
• a2a_overlap/test_schedule_chunk_1f1b_recompute_block.py: 60 passed / 60 deselected
• a2a_overlap/test_schedule_chunk_1f1b_recompute_uniform.py: 45 passed, 15 skipped / 60 deselected
• data/**/*.py @ dcd760ab: 813 passed, 161 skipped, 2 deselected / 976 deselected
• determinism/correctness/**/*.py: 55 passed, 6 skipped / 61 deselected
• determinism/kernels/**/*.py: 781 passed, 45 skipped, 2 xfailed, 7 xpassed / 835 deselected
• dist_checkpointing/**/*.py: 428 passed, 49 skipped, 21 deselected / 498 deselected
• dist_checkpointing/models/**/*.py: 142 passed / 142 deselected
• dist_checkpointing/models/test_moe_experts.py: 170 passed / 170 deselected
• dist_checkpointing/test_layer_wise_optimizer_common_state_dict.py: 164 passed, 128 skipped / 292 deselected
• dist_checkpointing/test_optimizer.py: 152 passed, 28 skipped / 180 deselected
• distributed/mfsdp_v1/**/*.py: 109 passed, 12 skipped, 21 deselected / 142 deselected
• distributed/mfsdp_v1/test_mfsdp_fully_shard.py: 249 passed, 95 skipped, 6 xfailed / 350 deselected
• distributed/mfsdp_v2/**/*.py: 339 passed, 47 skipped, 7 deselected / 393 deselected
• fusions/**/*.py: 391 passed, 183 skipped, 91 deselected / 83 passed, 582 deselected
• fusions/test_fused_mhc_kernels.py: 74 passed, 8 deselected / 82 deselected
• generalized_tensor_parallel/**/*.py: 250 passed, 59 skipped, 6 deselected / 315 deselected
• inference/**/*.py: 3830 passed, 84 skipped / 3 skipped, 3911 deselected
• inference/engines/**/*.py: 410 passed, 21 skipped, 9 deselected / 440 deselected
• inference/engines/test_batch_invariant_coordinator.py: 9 passed / 9 deselected
• inference/engines/test_batch_invariant_handoff.py: skipped upstream: 3 skipped / 3 deselected
• inference/engines/test_disagg_coordinator_pairwise.py: skipped upstream: NIXL is not installed; skipping tests/unit_tests/inference/engines/test_disagg_coordinator_pairwise.py
• inference/engines/test_dynamic_engine.py: 287 passed, 2 skipped / 289 deselected
• inference/engines/test_dynamic_engine_parallel_and_chunked_prefill.py: 107 passed, 80 skipped / 187 deselected
• inference/test_batch_invariant_fp8.py: 5 passed, 2 deselected / 1 passed, 6 deselected
• inference/test_coordinator_execution_pairwise.py: 10 passed / 10 deselected
• inference/test_coordinator_media_pairwise.py: 12 passed / 12 deselected
• inference/test_coordinator_sampling_pairwise.py: 30 passed / 30 deselected
• inference/test_coordinator_schedule_pairwise.py: 4 passed / 4 deselected
• inference/test_disagg_transfer_pairwise.py: 4 passed ×6
• inference/test_hybrid_moe.py: 130 passed, 1 deselected / 131 deselected
• models/**/*.py @ f583febb: 1243 passed, 49 skipped, 29 deselected / 24 passed, 1297 deselected
• ops/**/*.py: 54 passed, 8 skipped / 62 deselected
• pipeline_parallel/**/*.py: 385 passed, 3 skipped, 22 deselected / 410 deselected
• post_training/**/*.py: 67 passed / 67 deselected
• resharding/**/*.py: 362 passed, 105 skipped / 467 deselected
• rl/**/*.py: 326 passed / 326 deselected
• ssm/**/*.py @ 08b2dcef: 670 passed, 17 skipped, 131 subtests passed / 687 deselected
• ssm/gated_delta_net/test_gdn.py: 87 passed, 28 skipped, 18 deselected / 133 deselected
• ssm/gated_delta_net/test_gdn_fusion.py: 19 passed, 2 deselected / 21 deselected
• ssm/gated_delta_net/test_gdn_parallel.py: 36 passed / 36 deselected
• ssm/gated_delta_net/test_kda.py @ 08b2dcef: 25 passed / 25 deselected
• ssm/test_gated_delta_net2.py: 44 passed, 16 skipped / 60 deselected
• ssm/test_gated_delta_net_padding.py: 8 passed, 4 skipped / 12 deselected
• ssm/test_gated_delta_net_parallel.py: 10 passed, 14 skipped / 24 deselected
• ssm/test_gated_delta_net_recompute.py: 12 passed / 12 deselected
• ssm/test_gdn_gated_output_norm_distributed.py: 16 passed / 16 deselected
• ssm/test_gdn_gated_output_norm_fusion.py: 39 passed, 4 skipped / 43 deselected
• ssm/test_gdp_dynamic_inference.py: 63 passed / 63 deselected
• test_fp8_param.py: 14 passed, 15 skipped / 29 deselected
• test_model_configs.py: 242 passed, 67 skipped / 309 deselected
• transformer/**/*.py @ 04709f9f: 1843 passed, 63 skipped, 15 deselected / 1921 deselected
• transformer/experimental_attention_variant/**/*.py: 1410 passed, 160 skipped, 56 deselected / 1626 deselected
• transformer/experimental_attention_variant/test_csa_fused_sparse_attention_dev.py: 120 passed, 18 skipped / 138 deselected
• transformer/experimental_attention_variant/test_dsa_tilelang_kernels.py: 38 passed / 38 deselected
• transformer/experimental_attention_variant/test_dsv4_hybrid_attention_cp_cuda_graph.py: 3 passed, 3 skipped / 6 deselected
• transformer/moe/**/*.py @ f583febb: 823 passed, 80 skipped, 236 deselected / 204 passed, 935 deselected
• transformer/moe/test_early_wgrad.py: 75 passed, 25 skipped / 100 deselected
• transformer/moe/test_token_dispatcher.py: 82 passed, 29 skipped / 111 deselected
• transformer/moe/test_token_dispatcher_capacity.py: 21 passed, 33 skipped / 54 deselected
• transformer/test_cuda_graphs.py: 157 passed, 3 skipped, 27 deselected / 187 deselected
• transformer/test_multi_latent_attention.py: 80 passed, 8 skipped, 51 deselected / 38 passed, 13 skipped, 88 deselected
• transformer/test_multi_token_prediction.py: 282 passed, 2 skipped, 8 deselected / 292 deselected
• transformer/test_thd_correctness.py: 19 passed / 19 deselected

Functional rows (84)

• @ 44a87715, no pytest summary in log; step_time iter=5: det=54.80ms nondet=52.70ms ratio=1.04x (threshold 1.35x) (1): gpt/determinism_perf
• @ 44a87715, 1 passed (43): gpt/gpt3_mcore_te_mfsdp_v2_cp2, gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset, gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files, gpt/gpt3_mcore_te_tp1_pp4_vp1, gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer, gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline, gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split, gpt/gpt3_mcore_te_tp2_pp2_cp2, gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss, gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic, gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last, gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last, gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last, gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last, gpt/gpt3_mcore_te_tp2_pp2_cp2_nondeterministic, gpt/gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion, gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather, gpt/gpt3_mr_mcore_te_tp1_pp2_core_attn_offload_attn_cudagraph, gpt/gpt_dynamic_inference_tp1_pp1_dp8_583m_long_prompt_logitsmatch_zmq, gpt/gpt_dynamic_inference_tp2_pp2_dp2_583m_logitsmatch_zmq, gpt/gpt_static_inference_tp1_pp1_16b_multiprompt_tokensmatch, gpt/gpt_static_inference_tp1_pp1_583m_cudagraphs, gpt/gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs, gpt/gpt_static_inference_tp1_pp1_583m_logitsmatch, hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m, hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m_chunked_prefill, hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m_flashinfer, hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m_mamba_bf16_states, hybrid/hybrid_gdp_dynamic_inference_tp1_pp1_dp8_800m, hybrid/hybrid_mr_mcore_te_tp1_pp1_cp1_dgx_a100_1N8G, hybrid/hybrid_mr_mcore_te_tp1_pp2_mhc_core_attn_offload_attn_cudagraph, hybrid/hybrid_mr_mcore_te_tp1_pp2_vpp2_cp1_dgx_a100_1N8G, hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G, hybrid/hybrid_mr_mcore_te_tp2_pp1_cp1_dgx_a100_1N8G, hybrid/hybrid_mr_mcore_te_tp2_pp1_cp4_dgx_a100_1N8G, mimo/mimo_hetero_gtp_checkpoint_round_trip, moe/deepseek_proxy_mfsdp_v2_ep2, moe/deepseek_proxy_mfsdp_v2_ep2_ep_overlap, moe/deepseek_proxy_mfsdp_v2_ep2_hfsdp_mb2, moe/deepseek_proxy_mfsdp_v2_ep2_hsdp_mb2, moe/deepseek_proxy_mfsdp_v2_ep2_mb2, moe/gpt3_mcore_te_tp1_pp1_te_4experts_groupedGEMM_op_fuser, moe/gpt_dynamic_inference_tp4_pp1_ep4_16B_mtp_sp_block_cudagraphs
• @ 44a87715, 1 passed ×2 (26): gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute, gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings, gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion, gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear, gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear, gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs, gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss, gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce, gpt/gpt3_mcore_te_tp2_pp1_modelopt_distill_resume, gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic, gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances, gpt/gpt3_mcore_te_tp2_pp2_mla, gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist, gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic, gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion, gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective, gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute, gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone, gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce, gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather, gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode, gpt/gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone, moe/gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer, moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer, moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances, moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph
• @ 44a87715, no pytest summary in log; test script printed EXIT_CODE=0 (7): gpt/gpt3_mcore_te_tp1_pp2_dsv4_hybrid_mhc_mtp, gpt/gpt3_mcore_te_tp2_pp1_cp4_dcp, gpt/gpt3_mcore_te_tp2_pp1_gdn, gpt/gpt3_mcore_te_tp2_pp1_gdn_no_nvrx_sync, gpt/gpt3_mcore_te_tp2_pp2_dsa, gpt/gpt3_mcore_tp2_pp2_uninstall_te, gpt/gpt_grpo_basic_function
• @ 44a87715, expected-failure case: Training failed as expected. Marking test as success. (1): gpt/gpt3_mcore_te_tp2_pp1_gdn_no_nvrx_async
• @ f583febb, 1 passed (3): gpt/gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph, gpt/gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph_nosplit, gpt/gpt3_mcore_te_tp2_pp2_mhc_attn_cudagraph
• @ f583febb, 1 passed ×2 (1): gpt/gpt3_mcore_te_tp2_pp2_mhc
• @ 44a87715, [sse] legacy mode passed, [sse] async mode passed (1): gpt/gpt_inference_server_async_sched_tp2_pp2_dp2_qwen3_8b_sse
• @ 44a87715, Packed static GPT train/eval passed, Packed dynamic GPT train/eval passed (1): gpt/gpt_packed_train_eval

[FDecaYed]

Thanks for the logs and source mapping. The mHC golden hashes, recipes and reported values match, and the affected test rows support carrying the reviewed local results to 08b2dcef. This addresses the remaining evidence request; the explicit FlashInfer deferral remains accepted.

Approving based on the author-provided H100 results and our source checks. GitHub runtime CI has not passed: the determinism-coverage gate still needs to be addressed before merge. The two NIXL rows were skipped; Blackwell MoK and external MBridge coverage are not implied.

[svcnvidia-nemo-ci]

/ok to test 2ca8104

[FDecaYed]

Approved for 2ca8104c. The rebuilt GB200 image uses MoK 7a8cfb7f, and all four gate/main-gradient numerical cases passed on all four ranks. The current run also passes the previously failing VE case on all eight ranks, completes the A2A job, and validates the four mHC golden recipes. The earlier VE failure remains unexplained; this pass does not establish a root-cause fix.

The remaining NeMo-RL failure is in its Git checkout step (shallow file has changed since we read it), before Generation tests. NeMo-RL Generation, MBridge and the skipped NIXL paths remain unvalidated; this approval does not imply a fully green CI run. The explicit FlashInfer deferral remains accepted.