Repository navigation
Conversation
Compute the per-token decay only for the first Householder copy. Preserve zero decay on later copies and the bitwise eager-reference output. On GB200 at the 800M shape, CUDA-graph decode-preparation latency drops from 3.07 to 2.86 us for one token and from 7.96 to 7.73 us for a four-token speculative step. Add a reproducible graph benchmark and keep the original determinism stress shape alongside the 800M shape. Signed-off-by: Waterpine <biansonghz@gmail.com>
Write zero output for padding slots inside the Triton recurrence and remove the separate masked_fill launch. Preserve state and snapshot masking, and test non-finite padding buffers. Restore token-only padding in dynamic SSM inference for the FA4 CUDA-graph path. Record exact source hashes in the graph benchmark. On GB200, paired graph samples improve B=32 recurrent decode by 1.221x at S=1 and 1.065x at S=4. Kernel, dynamic, FA4 end-to-end, and determinism suites pass. Signed-off-by: Waterpine <biansonghz@gmail.com>
Keep the retained loop for single-token decode and prefetch the next folded step when a decode call includes speculative tokens. Preserve the arithmetic order and bitwise outputs across padding, snapshots, static and dynamic slots, and CUDA graphs. On GB200, paired graph samples improve B=32 S=4 recurrent latency by 1.134x versus the retained kernel, with B=32 S=1 unchanged. Kernel, dynamic inference, FA4 graph end-to-end, and determinism suites pass. Signed-off-by: Waterpine <biansonghz@gmail.com>
Signed-off-by: Waterpine <songb@nvidia.com>
Signed-off-by: Waterpine <songb@nvidia.com>
Signed-off-by: Song B <songb@nvidia.com>
Signed-off-by: Waterpine <biansonghz@gmail.com>
svcnvidia-nemo-ci
marked this pull request as draft
September 28, 2026 04:17
Contributor
|
This PR has been automatically converted to draft because all PRs must start as drafts. When you are ready for review, click Ready for Review to begin the review process. This will:
See the contribution guide for more details. |
Waterpine
changed the title
Waterpine
changed the title
santhnm2
self-requested a review
September 28, 2026 04:43
Contributor
Author
|
This PR is now covered by #7754, so I am closing it |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Improve GDP decode inference on GB200 by reducing redundant recurrent-path work and avoiding repeated transposed-weight copies in the batch-invariant Triton TE GEMM.
Changes
Performance
Measurements use one GB200, batch size 32, BF16 inputs, and FP32 recurrent state.
GB200 GDP decode performance
The isolated recurrent-kernel benchmark compares the original main (80e5f8a) with this PR (bb8ee60). It uses FP32 recurrent state and 12 paired CUDA-graph runs per case.
For the complete GDP mixer and decode step, unmodified main fails before timing in the global batch-invariant configuration. Their baseline is original main with only the runtime fixes needed to handle padded projection rows and non-contiguous speculative-decode output. The candidate is the full PR. Measurements use a fixed three-layer model, one GB200 per paired experiment, CUDA graphs, and FP32 recurrent state.
The mixer results use four paired runs per case. The complete-step results use 24 paired runs for S=1 and eight for S=4; generated-token hashes matched across the paired complete-step runs.
The mixer and complete-step results measure the combined effect of the PR changes beyond the two runtime fixes. They should not be attributed solely to removal of the Triton TE weight copy. These measurements exercise the global batch-invariant Triton path; they do not measure the default te_native path.
Validation
Issue tracking
Linked issue: N/A
Contribution process
Pre-checks
jit_fuser/torch.compile, CUDA extension, TE or external-library dispatch, or a scatter/index accumulation), I have added or updated its bit-exact determinism test and registered it intests/unit_tests/determinism/kernels/manifest.py(guide)Code review
Feel free to message or comment @NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!
All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.
Step 1: Mark PR as "Ready for Review"
.github/CODEOWNERS.Final Review might get declined if these requirements are not fulfilled.
Step 2: Final Review
For PRs that change
megatron/core, once all expert reviewers have approved, theFinal Reviewlabel is applied automatically and final reviewers are assigned.For PRs outside
megatron/core, this step is skipped.Step 3: Approved
Once all required reviewers have approved, the
Approvedlabel is applied automatically.Merge
Any member of mcore-engineers will be able to merge your PR.