Repository navigation
test: Quarantine GDN2 THD correctness in dev CI - #8310
Merged
Merged
Conversation
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
balasaajay
approved these changes
Oct 18, 2026
1 task done
wujingyue
marked this pull request as ready for review
October 18, 2026 03:09
wujingyue
enabled auto-merge
October 18, 2026 03:09
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/39096185836 |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Temporarily quarantine
TestGatedDeltaNet.test_gpu_forward_thd_correctnessin development CI withflaky_in_dev. GDN2 THD stalls have repeatedly exhausted merge-queue jobs, including #8230 and #7727 on October 17.Both jobs stalled in
[2-False-2-gdn2](TP2, CP2, SP disabled). Per-rank artifacts capture outstanding Inductor compilation waits, including backward compilation, while peers time out in NCCL all-to-all/all-reduce. Automatic retries then reach the one-hour job limit. The TP1/CP1 variant has also stalled repeatedly, as recorded in #7742 and #8222's queue job.A GDN2-specific override delegates to the shared test body and adds the marker without marking the shared GDN method. This quarantines the method's six parameterizations as a family; it does not assert that every parameterization independently failed. Two SP parameterizations already skip inside the test body. The other 48 GDN2 module cases remain selected in dev, and the THD cases retain LTS eligibility. Assertions, tolerances, fixtures, and production code are unchanged.
The existing quarantine from #8148 applies to
test_parallel_gated_delta_net2_correctness; it does not cover this separate THD module method.Validation
torch.distributed.runbefore and after the change across GDN, GDN2, and standalone GDN2 parallel buckets. All 205 node IDs remain present; exactly the six GDN2 THD cases gainflaky_in_dev, with every other effective marker unchanged.git diff --checkpassed.tools/autoformat.shencountered the existing unsupported Pylintbad-builtinsoption; the remaining checks were run separately, and Pylint exited successfully with only that obsolete option removed from a temporary config.Issue tracking
Related to #7742; keep the issue open. This is temporary CI containment. Remove the marker after the compiler stall is fixed and the affected THD layouts pass repeated cold-cache and warm-cache runs on the development image without retries or watchdog timeouts.
Contribution process