[Connor-XY]

Kernel changes need tests that expose scheduling-dependent numerical differences and a PR check that keeps those tests maintained. Add a kernel replay suite, an explicit coverage registry, and a CI gate requiring changed kernel code to carry an updated determinism test.

Full-precision functional enforcement, its regression tests, and all 60 golden updates are isolated in #7217. The two PRs target main independently and can merge in either order. This PR contains 23 files and no golden-data or functional-comparison changes.

Changes

• Add tests/unit_tests/determinism/kernels/: replay helpers compare tensor outputs and gradients byte-for-byte, including signed zeros and NaN payloads. Tests can restore RNG state and add side-stream contention; module replay and negative controls exercise the same contract.
• Cover fused activations, Triton fusions, tensor-parallel/apex operations, MoE, SSM, Transformer Engine wrappers, optimizers, and inference kernels. Register kernel-bearing source files in manifest.py with tests or an explicit exemption explaining a coverage gap.
• Add manifest tests that check coverage, registered paths, and representative kernel-detection patterns. External-library patterns match call sites rather than bare imports; dispatch modules identify the kernels their tests exercise.
• Add tools/check_kernel_determinism_coverage.py to the linting job on PR pushes. Changed kernel-bearing files must be registered. Changed registered kernel sources must also carry a determinism-test update; the determinism-exempt label overrides the test-update requirement for changes that do not affect numerics, but cannot bypass registration.
• Add a dedicated H100 unit-test bucket and document the workflow in the PR template, review prompt, determinism docs, and testing skill. Preserve main's refactored review workflow while retaining the kernel-test review requirement.

Validation

For the split head:

• 215 manifest/gate tests passed after integrating current main, using the unchanged test/source modules in an isolated CPU harness. Only GPU package initialization and repository conftest were bypassed.
• The actual kernel gate passed against main; YAML parsing and assertions verified the gate, dedicated recipe bucket, and separation from full-precision enforcement.
• isort, Black, Ruff, Python AST checks, and git diff --check passed.
• Verified both split patches compose to the same tree in either merge order. Kernel Python files retain their original contents. The split also corrects two documentation statements: the gate runs on PR pushes, and legacy functional goldens still compare at five decimals.

GPU replay tests have not been rerun on this split head.

Historical results reported on the combined PR: the H100 × 8 kernel bucket passed 380 tests with no failures; a four-GPU GB300 run also reported 380 passed. These results predate this split and are not a fresh validation of its head.

Known coverage gaps and follow-ups

• GDP varlen scan replay differences were reported on GB300 with Triton 3.7, while H100 CI reported XPASS. TE fused MoE auxiliary loss and intentionally racy negative controls also remain documented xfail/XPASS cases; these are not asserted fixes to those kernels.
• Create the determinism-exempt label and extend the bucket to GB200/GB300 runners.
• Add coverage for exempt FlashInfer sampling/MXFP8 GEMM paths and the CuTeDSL GDP kernel when the test container supports them.
• SSM autotune policy is selected at import; the suite sets determinism environment variables early. Launch-time policy selection remains separate work.

Related to #5785.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Connor-XY]

/ok to test 74d6667

[Connor-XY]

/ok to test cdfe783

[Connor-XY]

/ok to test 6e91ecc

[Connor-XY]

/ok to test c4f488c

[guihong-nv]

@Connor-XY My concern is that these golden values may be too strict for new kernels. A new kernel can be fully deterministic and still produce slightly different numbers because it uses a different, but valid, computation order or implementation.

With bit-exact golden checks, that kernel would fail even if it is correct. The author would then have to update the golden values, which means the test is mostly checking that the output has not changed from the previous implementation—not necessarily that the new implementation is correct.

I think we should clarify whether the goal is to lock down one exact implementation, or to allow different correct deterministic kernels. If we want to support the latter, we may need a tolerance or a separate correctness/reference test instead of requiring an exact match to the existing golden values.

[Connor-XY]

Hi @guihong-nv, thanks for raising this issue up! The kernel tests do not use golden values. They replay the same kernel on the same inputs and require the outputs to match themselves, so this should work for all kernels. For functional tests, we have the full precision enabled to check bitwise determinism, and if not set to full precision, a determinism break may not be caught due to tolerance. If some functional test uses new kernel and the result changes, then its golden value should be updated. This also allows new kernels to be added. Please let me know your thoughts!

[guihong-nv]

The kernel tests do not use golden values. They replay the same kernel on the same inputs and require the outputs to match themselves, so this should work for all kernels. For functional tests, we have the full precision enabled to check bitwise determinism, and if not set to full precision, a determinism break may not be caught due to tolerance. If some functional test uses new kernel and the result changes, then its golden value should be updated. This also allows new kernels to be added. Please let me know your thoughts!

Thanks, this makes sense.

[Connor-XY]

/ok to test d5e5950

[nemo-automation-bot]

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/34662389346