Repository navigation
fix(tests): synchronize GTP process-group teardown - #7656
Merged
balasaajay merged 2 commits intoSep 26, 2026
Merged
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
Author
|
/ok to test 0e91132 |
ko3n1g
approved these changes
Sep 25, 2026
ko3n1g
left a comment
ko3n1g
left a comment
Contributor
There was a problem hiding this comment.
pls double check with @NVIDIA/mcore-oncall
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
balasaajay
force-pushed
the
abalasa/fix-gtp-test-group-teardown
branch
from
September 26, 2026 04:06
0e91132 to
c6a12ba
Compare
Contributor
Author
|
/ok to test c6a12ba |
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/36221363541 |
fanshiqing
approved these changes
Sep 27, 2026
xuwchen
pushed a commit
to xuwchen/Megatron-LM
that referenced
this pull request
Oct 14, 2026
Signed-off-by: Ajay Balasa <abalasa@nvidia.com> (cherry picked from commit d113016) Signed-off-by: xuwenc <xuwenc@nvidia.com>
xuwchen
pushed a commit
to xuwchen/Megatron-LM
that referenced
this pull request
Oct 14, 2026
Signed-off-by: Ajay Balasa <abalasa@nvidia.com> (cherry picked from commit d113016) Signed-off-by: xuwenc <xuwenc@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The four-rank GTP/DDP tests intermittently fail while restoring process groups between cases: a faster rank can start the next case and destroy Gloo groups before peers finish connecting. The resulting rank divergence causes long communication waits and one-hour CI timeouts, as in this job.
Synchronize ranks before both teardown points in
_gtp_env(). Before cleanup, also synchronize CUDA work: the grouped-expert test can leave a prefetch outstanding on a separate all-gather stream. This drains that work before destroying groups and clearing the cache. The entry barrier needs no additional CUDA synchronization because previous cleanup already drained GPU work. The change is confined to six added lines in the existing test helper.Validation
CHECK_ONLY=true bash tools/autoformat.sh): Black, isort, pylint, and Ruff.git diff --check.torchstubs; this machine has no CUDA GPUs, so distributed execution has not been validated locally.