[balasaajay]

• I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Split the combined GDN/GDN2 unit-test job into two independently scheduled buckets, with GDN running exclusively on one GB200 x4 runner and GDN2 on H100.

Bucket under tests/unit_tests/ssm/	Cases	Platform	GPUs per runner	Environments
test_gated_delta_net.py — all GDN module, FLA/TE parallel correctness, and remaining checks	139	GB200 only	4	dev
test_gated_delta_net2.py — GDN2 module and parallel correctness	60 (54 selected in dev)	H100	8	lts, dev

The shared module-test bodies move into a non-collected helper. All 199 cases retain their assertions, shapes, seeds, TP/CP layouts, fixtures, parameter IDs, and existing skip/internal markers, except for the dev-only GDN2 parallel quarantine below. The GDN padding comparison adopts the established tolerance described below; other numerical bounds are unchanged. GDN2 cases change their file path; GDN retains its original node IDs and gains launch_on_gb200. Its TP/CP layouts fit four workers. The original H100 run used all eight ranks, with additional ranks forming data-parallel replicas; moving GDN to four workers changes that replication. Production code is unchanged.

The H100 recipe schedules GDN2 and the GB200 recipe schedules one dedicated GDN bucket. A scoped exclusion in find_test_cases.py keeps GDN out of H100 catch-all buckets; existing child-bucket exclusions prevent duplicate GB200 execution. Other tests marked launch_on_gb200 retain their existing cross-platform behavior. Both buckets use the latest tag and existing per-platform settings. The pinned legacy suite has no GDN tests.

Measured behavior

The original combined H100 job took 55m38s overall, including 52m17s in pytest. The GDN-only portion is approximately 32–33 minutes when reconstructed from its logs.

In run 38284519331, GDN on four GB200 GPUs took 37m48s overall, including 32m41s in pytest, with 118 passed and 21 skipped. GDN2 on H100 passed 44 cases and skipped 16 after an NCCL-timeout retry; its job took 45m30s excluding queue time. These observations do not establish a hardware speedup or a reduction in total workflow duration. Separate buckets allow scheduling independently; runner availability, compilation, and retries affect elapsed time.

GDN padding tolerance

Run 38305226282 exposed a stale tolerance in the shared test_gpu_forward_thd_padding_correctness[1-False-1-gdn] case: rank 0 had 19/245,760 output elements outside atol=rtol=3e-4, with maximum absolute difference 0.001953125. The other three ranks passed. Relevant source and the container image were unchanged from the prior passing GB200 execution; this was a numerical assertion failure, not a timeout.

Set only this GDN padded SBHD-versus-THD comparison to atol=rtol=5e-3, matching the existing standalone padding test. PR #6851's validation note documents intermittent BF16 differences of 0.001953125–0.00390625 and establishes this tolerance for the distinct FLA layouts. The code comment links that precedent. GDN2 keeps its 1e-2 tolerance, and ordinary GDN THD correctness keeps 3e-4. This aligns duplicate test policy; it does not claim a kernel arithmetic fix.

Temporary GDN2 parallel quarantine

Issue #7742 tracks intermittent H100 GDN2 hangs across TP4 and CP2 layouts. The latest occurrences exhausted three launcher attempts or GitHub's one-hour job limit. One captured rank stack waits for an Inductor asynchronous backward compilation; the underlying cause remains unresolved.

Mark test_parallel_gated_delta_net2_correctness as flaky_in_dev in both test_gated_delta_net2.py and test_gated_delta_net2_parallel.py. This temporarily excludes all six parameterizations in each copy from dev CI; the standalone file previously excluded only its two TP4 cases. All six remain eligible in LTS. The other 54 GDN2 module cases and all 139 GDN cases retain their selection behavior. The existing empty-selection handling lets the standalone dev bucket complete successfully.

This contains the known parallel-test stalls; it does not fix the compiler hang. The issue separately records a GDN2 module THD stall, which remains enabled, and the unrelated BF16 GDN padding mismatch addressed above. Re-enable the parallel test after an identified fix passes all six layouts on eight H100 GPUs across repeated cold-cache runs without launcher retries. No compiler setting or timeout is changed here.

Validation

• The two-bucket implementation previously passed 108 existing CPU tests covering Testmon mandatory selection and workflow/matrix behavior. Those routing paths are unchanged by the tolerance fix and quarantine.
• Actual before/after pytest collection previously verified 199 → 139 + 60, with exact mapped node IDs and unchanged effective parameters, fixtures, and existing skip/internal markers, except for the dev-only GDN2 parallel quarantine below. Collection used a temporary architecture-query shim with CUDA unavailable; no GPU fixtures or test bodies executed locally. The tolerance change does not alter collection.
• The padding AST audit verifies the corrected padding method exactly matches the standalone implementation. The shared file's only semantic changes for that fix are the two intended GDN padding tolerance constants. GDN2 and non-padding tolerances remain unchanged.
• Fresh before/after pytest collection for the quarantine preserves all 205 node IDs and their order across GDN (139), GDN2 (60), and the existing standalone GDN2 parallel file (6). Dev selection excludes exactly the six parallel cases in each GDN2 file; the GDN2 module bucket selects 54 cases, and the standalone bucket selects zero with exit code 0. The LTS marker expression still selects all 205 cases; this checks selection only, not execution in an LTS container. AST checks confirm unchanged test bodies, signatures, and parameter values.
• Repository formatting/lint checks (Black, isort, pylint, Ruff) and git diff --check passed for the tolerance fix and quarantine. The autoformat script's optional mypy step was unavailable in the cached CI image.
• GPU validation of this tolerance fix is pending. Local GPU access is blocked by an NVIDIA driver/NVML mismatch. Validate the affected case and full GDN bucket on four GB200 GPUs; the earlier passing run is not validation of this new commit.

Issue tracking

GDN2 parallel quarantine and follow-up diagnostics: #7742 (kept open until the underlying hang is fixed).

Related: #7987 adds compiler-cache reuse for the separate test_gdn_gated_output_norm_fusion.py bucket. It does not currently cache this GDN bucket.

Contribution process

Pre-checks

• I have added relevant unit tests
• I have added relevant functional tests
• I have added proper typing to my code
• I have added relevant documentation
• I have run the autoformatter on my PR

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[balasaajay]

/ok to test 5fc80b5

[nemo-automation-bot]

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/38478361365