Skip to content

test: Quarantine GDN2 THD correctness in dev CI - #8310

Merged
wujingyue merged 1 commit into
NVIDIA:mainfrom
wujingyue:test/quarantine-gdn2-thd-in-dev
Oct 18, 2026
Merged

wujingyue merged 1 commit into
NVIDIA:mainfrom
wujingyue:test/quarantine-gdn2-thd-in-dev

Conversation

@wujingyue

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Temporarily quarantine TestGatedDeltaNet.test_gpu_forward_thd_correctness in development CI with flaky_in_dev. GDN2 THD stalls have repeatedly exhausted merge-queue jobs, including #8230 and #7727 on October 17.

Both jobs stalled in [2-False-2-gdn2] (TP2, CP2, SP disabled). Per-rank artifacts capture outstanding Inductor compilation waits, including backward compilation, while peers time out in NCCL all-to-all/all-reduce. Automatic retries then reach the one-hour job limit. The TP1/CP1 variant has also stalled repeatedly, as recorded in #7742 and #8222's queue job.

A GDN2-specific override delegates to the shared test body and adds the marker without marking the shared GDN method. This quarantines the method's six parameterizations as a family; it does not assert that every parameterization independently failed. Two SP parameterizations already skip inside the test body. The other 48 GDN2 module cases remain selected in dev, and the THD cases retain LTS eligibility. Assertions, tolerances, fixtures, and production code are unchanged.

The existing quarantine from #8148 applies to test_parallel_gated_delta_net2_correctness; it does not cover this separate THD module method.

Validation

  • Ran actual pytest collection through torch.distributed.run before and after the change across GDN, GDN2, and standalone GDN2 parallel buckets. All 205 node IDs remain present; exactly the six GDN2 THD cases gain flaky_in_dev, with every other effective marker unchanged.
  • With the dev CI marker expression, GDN remains 139 selected, GDN2 changes 54 → 48 selected, and standalone GDN2 parallel remains deselected. The LTS marker expression still selects all 205 cases, including all 60 in the GDN2 module bucket.
  • Collection validation only: no GPU test bodies or LTS-container execution were run. The override relocates the six THD cases within unfiltered/LTS collection order; other cases retain their relative order.
  • Black, isort, Ruff, mypy, and git diff --check passed. tools/autoformat.sh encountered the existing unsupported Pylint bad-builtins option; the remaining checks were run separately, and Pylint exited successfully with only that obsolete option removed from a temporary config.

Issue tracking

Related to #7742; keep the issue open. This is temporary CI containment. Remove the marker after the compiler stall is fixed and the affected THD layouts pass repeated cold-cache and warm-cache runs on the development image without retries or watchdog timeouts.

Contribution process

  • Updated test selection using the existing development quarantine marker.
  • Verified development and LTS marker selection.
  • Ran formatting and lint checks, with the configuration limitation noted above.

Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 17, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@tdene tdene mentioned this pull request Oct 18, 2026
1 task done
@wujingyue
wujingyue marked this pull request as ready for review October 18, 2026 03:09
@wujingyue
wujingyue enabled auto-merge October 18, 2026 03:09
@svcnvidia-nemo-ci
svcnvidia-nemo-ci requested a review from a team October 18, 2026 03:09
@wujingyue
wujingyue added this pull request to the merge queue Oct 18, 2026
@nemo-automation-bot

Copy link
Copy Markdown

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/39096185836

Merged via the queue into NVIDIA:main with commit 4dc7e41 Oct 18, 2026
117 checks passed
@wujingyue
wujingyue deleted the test/quarantine-gdn2-thd-in-dev branch October 18, 2026 05:21

This branch was previously deployed

2 inactive deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants