Repository navigation
Conversation
check_param_hashes_across_dp_replicas all-gathered parameter hashes over the data-parallel groups of the global parallel state. A model whose process-group collection differs from the global grid then compares its hashes with ranks that hold other shards or another model: the check reports a mismatch for replicas that agree, or hangs when the ranks hold different numbers of parameters. Add keyword-only dp_group and expt_dp_group arguments for the model's replicate groups, pg_collection.dp and pg_collection.expt_dp. These are the fields that match the global accessors called with with_gtp_remat=False. If neither is passed, the function keeps using the global groups as a commented migration fallback. If either is passed, no global group is read. A rank that holds non-expert parameters without dp_group, or expert parameters without expt_dp_group, raises ValueError before the first collective, so a dense model may pass dp_group alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
The training loop checked parameter hashes at startup and every --check-weight-hash-across-dp-replicas-interval iterations across the global data-parallel groups. Pass the data-parallel groups of the model's process-group collection instead, resolved as train_step does: the collection of the model chunks, or the global one for a model that carries none. On the standard grid the model's collection holds the global groups, so the check compares across the same ranks as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
Contributor
Author
|
/ok to test 0ab8e49 |
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
The TrainingConfig migration (NVIDIA#8149) moved train_sync_interval, the weight hash check interval, the GPU sniff test interval and the manual GC settings from args to the run config's train section. The test only set them in args, so after a rebase past NVIDIA#8149, post_training_step_callbacks failed on cfg.train. Set the same values in both, which passes before and after the migration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
This was referenced Oct 30, 2026
Connor-XY
added a commit
to Connor-XY/Megatron-LM
that referenced
this pull request
Nov 1, 2026
NVIDIA#8047 lets check_param_hashes_across_dp_replicas take the model's data-parallel groups and passes them from the training loop. The param-hash helper is to warn when it falls back to the global groups, in the code NVIDIA#8047 changes, so merge NVIDIA#8047 here. The merge is clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
Connor-XY
added a commit
to Connor-XY/Megatron-LM
that referenced
this pull request
Nov 1, 2026
The training loop's fallback warnings that follow cover the weight-hash check that NVIDIA#8047 runs on the model's data-parallel groups, and their tests extend NVIDIA#8047's hash-check test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
Connor-XY
added a commit
to Connor-XY/Megatron-LM
that referenced
this pull request
Nov 1, 2026
The checker classification that follows requires every allowlisted read in megatron/core to be warned or guarded. NVIDIA#8755 warns the global fallbacks of the log and param-hash helpers, RoPE, MIMO partitioning, batch-invariant unpermute, the FP8/FP4 contexts and the CUDA-graph helpers, and brings NVIDIA#8017, NVIDIA#8032, NVIDIA#8035, NVIDIA#8047, NVIDIA#8049 and NVIDIA#8053 (NVIDIA#6303 already came in with NVIDIA#8752). megatron/core/extensions/transformer_engine.py conflicts on its parallel_state import: NVIDIA#8038 drops get_tensor_model_parallel_world_size and keeps model_parallel_is_initialized, and NVIDIA#8755's eac0b1c drops get_amax_reduction_group and model_parallel_is_initialized. The merged file calls none of the three, so the import keeps only get_context_parallel_group, get_hierarchical_context_parallel_groups and get_tensor_model_parallel_group. The process-group usage allowlist is regenerated on the merged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yan Xu <yxu1@nvidia.com>
This was referenced Nov 3, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why.
check_param_hashes_across_dp_replicasalways all-gathers over the global data-parallel groups. A model on its own grid then compares its hashes with ranks that hold other shards or another model:--check-weight-hash-across-dp-replicas-intervalreports a mismatch for replicas that agree, or hangs when those ranks hold different numbers of parameters.What.
check_param_hashes_across_dp_replicas(..., *, dp_group=None, expt_dp_group=None)takes the model's replicate groups,pg_collection.dpandpg_collection.expt_dp. If either is passed, no global group is read. A rank with non-expert parameters and nodp_group, or with expert parameters and noexpt_dp_group, raisesValueErrorbefore any collective, so a dense model may passdp_groupalone. If neither is passed, the global groups are used as before.train_step, at startup and at each check interval.Tests. New tests on 4 GPUs use groups laid out unlike the global grid, with every
parallel_stategroup and rank accessor patched to raise: agreeing replicas pass, a mismatch is detected with and withoutcross_check, and missing groups raise. They fail on main. Unit tests cover the training-side group resolution. In an integration run of main 77391e7 with all #6307 PRs merged, on 4 GPUs,test_utils.pyhas 36 passed vs 31 on main: the existing results match main and the 5 new tests pass. The newtraining/test_param_hash_check_groups.py, absent on main, passes all 5 of its tests there: 3 as this PR wrote them, and 2 from a later #6307 PR, which rewrote this PR's third test and added one. In the integration run, this PR's test files show no regressions against main. On the pre-sync branch,pretrain_gpt.pywith the check every 2 iterations (DP=4 with the distributed optimizer, PP=2 x VPP=2, MoE with EP=2) gives losses bit-identical to main.Compatibility. Callers that pass no groups keep the global groups. No change to numerics, checkpoints or the process-group allowlist.
Synced with main 77391e7 on 2026-11-01 by a merge commit, which had no conflicts. Existing commits and review links stay valid.
Part of #6307.
🤖 Generated with Claude Code