Rename FSDP Flat placement to RowAtomic - #7539
Merged
Merged
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
wujingyue
marked this pull request as ready for review
September 21, 2026 06:19
wujingyue
changed the title
shjwudp
approved these changes
Sep 23, 2026
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
wujingyue
enabled auto-merge
September 23, 2026 07:12
wujingyue
force-pushed
the
rename-fsdp-row-atomic
branch
from
September 23, 2026 07:13
9b7ffc1 to
bd4053a
Compare
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/35861658427 |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 23, 2026
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/35875329759 |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Rename the experimental FSDP
Flatplacement toRowAtomicto describe its row-preserving sharding semantics.By default, mFSDP cuts weights at row boundaries, keeping each row intact within a shard. The name
Flatis confusing because it suggests treating a weight as a 1D flat tensor and potentially cutting in the middle of a row.RowAtomicmakes the intended semantics explicit: each row is an indivisible unit of sharding.Update imports, exports, tests, error messages, and design documentation. Sharding behavior is unchanged.
Validation
test_dbuffer.py,test_memory.py,test_owner_planning.py, andtest_quantized_dbuffer.pyfiles with two GPU ranks in the development container: 53 passed, 7 skipped per rank.git diff --check.