Repository navigation
Conversation
FullyShardedDataParallelV1._init_dist_index filled a missing tp or expt_tp field of pg_collection with dist.new_group(ranks=[rank]). Each rank passed a different rank list, which breaks the rule that every rank creates the same groups in the same order. This works only while each group has one member and every rank takes the same branch. If ranks ever disagree, their group counters drift apart and later group creation mismatches or hangs. Dense models also created an expert TP group they never use. - Create the single-rank groups once, with new_subgroups_by_enumeration over every rank, and only for the axes the model uses. - Check the fields the configuration needs (dp_cp; intra_dp_cp and inter_dist_opt with several optimizer instances; ep, expt_dp and intra_expt_dp for MoE) and raise ValueError naming the missing ones, instead of an AttributeError on None deep in mesh construction. - Read collection fields directly instead of getattr with a default. - Comment the global-grid branch as the migration fallback for callers without a collection. Its accessors and groups are unchanged. The collection path keeps the replicate data-parallel fields (dp_cp, expt_dp, intra_dp_cp, intra_expt_dp). With GTP weight remat off they hold the same ranks as the GTP-inclusive groups that the fallback's accessors return. A new parity test checks this on eight standard-grid layouts, and a forbid-globals test runs the adapter on a HyperCommGrid collection laid out unlike the global grid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
Contributor
Author
|
/ok to test 528b9f6 |
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
This was referenced Oct 30, 2026
Connor-XY
added a commit
to Connor-XY/Megatron-LM
that referenced
this pull request
Nov 1, 2026
NVIDIA#8051 creates the Megatron-FSDP v1 TP groups collectively and validates the pg_collection it is given. The Megatron-FSDP v1 fallback warning changes the same adapter code and the dist-index tests that NVIDIA#8051 creates, so merge NVIDIA#8051's head. The merge is clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
Connor-XY
added a commit
to Connor-XY/Megatron-LM
that referenced
this pull request
Nov 1, 2026
The checker classification that follows requires every allowlisted read in megatron/core to be warned or guarded. NVIDIA#8749 warns the global fallbacks of Megatron-FSDP v1, Torch FSDP and the FSDP-DTensor expert offsets, and brings NVIDIA#8017, NVIDIA#8045 and NVIDIA#8051. The guard tests that follow edit the mfsdp_v1 test_mcore_fsdp_dist_index.py that NVIDIA#8749 extends. The merge conflicts only in the process-group usage allowlist, which is regenerated on the merged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
Main's NVIDIA#8471 moves FullyShardedDataParallelV1 and FullyShardedDataParallelV2 out of mcore_fsdp_adapter.py into mcore_adapter_v1.py and mcore_adapter_v2.py, and keeps mcore_fsdp_adapter.py as the factory that re-exports them. This branch's edits to the two classes and their helpers are applied unchanged where the code moved, and the process-group allowlist follows the move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
This was referenced Nov 3, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why.
FullyShardedDataParallelV1fills an unsettporexpt_tpinpg_collectionwithdist.new_group(ranks=[rank]), a different rank list on every rank. Group creation must be identical onall ranks: this works only while every rank takes the same branch, and otherwise later groups mismatch or
hang. Dense models also create an expert TP group they never use, and a collection that lacks a needed
group fails with an
AttributeErroronNonedeep in mesh construction.What.
new_subgroups_by_enumerationover every rank, and only forthe axes the model uses.
dp_cp;intra_dp_cp,inter_dist_optfor HSDP;ep,expt_dp,intra_expt_dpfor MoE) and raiseValueErrornaming the missing ones.unchanged.
The collection path keeps the replicate data-parallel fields. With GTP weight remat off they hold the
same ranks as the fallback's GTP-inclusive groups.
Tests. New
test_mcore_fsdp_dist_index.py(4 GPUs): a HyperCommGrid collection laid out unlike the global grid,with every
parallel_stateaccessor patched to raise; identical group creation on every rank;ValueErrorfor missing groups; explicit-vs-fallback rank parity on eight standard-grid layouts. Before the2026-11-01 sync, 10 of the 20 cases failed on main, and the existing MFSDP v1 tests matched main on 4 and 2 GPUs.
In the integration run, this PR's test file shows no regressions against main: all 20 cases of
test_mcore_fsdp_dist_index.pypass on 4 GPUs, and main does not have the file.Compatibility. No change for the standard grid or for collections that set the needed groups. A collection missing one
now raises
ValueErrorinstead ofAttributeError.The branch was synced with main 77391e7 on 2026-11-01 by a merge commit (no conflicts), so existing commits
Later on 2026-11-01 the branch also merged main bfab98f. Its #8471 moves
FullyShardedDataParallelV1andFullyShardedDataParallelV2frommcore_fsdp_adapter.pyintomcore_adapter_v1.pyandmcore_adapter_v2.py, and keepsmcore_fsdp_adapter.pyas the factory that re-exports them. This PR's edits to those classes are applied unchanged where the code moved, and the process-group allowlist follows the move. On 4 GPUs against its new base, the Megatron-FSDP adapter tests and this PR's own tests (7 files) show no regressions: 78 passed here, 58 on the base. The two CUDA-graph end-to-end cases oftest_mcore_fully_sharded_data_parallel.pyfail on both sides, as they do on main.and review links stay valid.
Part of #6307.
🤖 Generated with Claude Code