Repository navigation
Fix integrity settings during checkpoint conversion - #6502
Merged
Merged
Conversation
bvolpato
marked this pull request as ready for review
August 14, 2026 04:31
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
bvolpato
force-pushed
the
bvolpato/save-run-config-yaml
branch
from
August 21, 2026 05:43
79ea001 to
930457d
Compare
Contributor
Author
|
Rebased onto current |
svcnvidia-nemo-ci
removed
the
waiting-on-maintainers
bvolpato
force-pushed
the
bvolpato/save-run-config-yaml
branch
from
August 22, 2026 03:37
930457d to
e89e2d7
Compare
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
Contributor
Author
|
@maanug-nv, could you take a look at this PR when you have a chance? It implements #6002 by saving a Bridge-compatible |
svcnvidia-nemo-ci
added
waiting-on-maintainers
bvolpato
force-pushed
the
bvolpato/save-run-config-yaml
branch
from
September 24, 2026 00:44
49c9348 to
8c6bea8
Compare
bvolpato
force-pushed
the
bvolpato/save-run-config-yaml
branch
from
October 3, 2026 06:09
8c6bea8 to
c592cc4
Compare
bvolpato
changed the title
dimapihtar
requested changes
Oct 5, 2026
Contributor
|
Thanks for the PR! |
Contributor
|
/ok to test c592cc4 |
dimapihtar
added
the
Expert Review
Contributor
|
/ok to test 2ea74a3 |
Contributor
|
/ok to test b3a73de |
Contributor
|
/ok to test 1997dd7 |
svcnvidia-nemo-ci
added
the
waiting-on-customer
dimapihtar
approved these changes
Oct 23, 2026
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/40038856702 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/40045640221 |
svcnvidia-nemo-ci
removed
the
waiting-on-customer
asolergi-nv
added a commit
to asolergi-nv/Megatron-LM
that referenced
this pull request
Oct 26, 2026
Five commits landed on main since the last merge (0a2f6ee); one of them conflicts with this branch, NVIDIA#7047 "[DCP V2 3/4] Integrate dynamic CP with standard train and eval schedules" (ea99fdf). The other four (NVIDIA#8381 CI test selection, NVIDIA#8269 PR template, NVIDIA#6502 checkpoint conversion integrity, NVIDIA#7780 gtp scratch lifetime) do not touch this branch's files. Conflicts, all resolved in this branch's structure: - megatron/core/datasets/gpt_dataset.py: imports (warnings for the new deprecation warning; no `field`, token_dtype_code is a property here). NVIDIA#7047's warn-and-rewrite of the deprecated hybrid_context_parallel alias landed in validate_without_tokenizer(), which asserts only: it keeps the warning and the "not both" error and does not rewrite the alias; the SBHD exclusion counts the alias. - pretrain_gpt.py, pretrain_hybrid.py: NVIDIA#7047 edited the legacy factory and rank predicate this branch replaced with argument_utils and megatron.training.datasets.providers; this branch's imports and the removals stay, NVIDIA#7047's get_batch changes (pg_collection, dynamic_cp) merge as they are. - tools/prepare_cache.py: same; the shared factory maps the new field. The renamed field through the data section: dynamic_context_parallel joins CONTAINER_WRITTEN_FIELDS next to the deprecated alias; the container's cross-section check reads either name against model.transformer's dynamic_context_parallel and finalize() writes the canonical field and clears the alias; the standalone factory forwards both from the arguments, which validate_args already normalizes (NVIDIA#7047). The ownership, section, provider and Core config tests follow, with cases for the alias; the frozen legacy factories the provider tests use as their oracle take the same one-line rename NVIDIA#7047 gave the originals. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: asolergi-nv <asolergibert@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Converting a
torch_distcheckpoint totorchwith--verify-integritycarries that flag into the converted checkpoint's saved arguments, even though integrity checks only supporttorch_dist.Clear
args.verify_integrityafter loading the source and before saving an unsupported output format. Keep it enabled when the target istorch_dist.This change does not update the
run_config.yamlsidecar or the separatetools/checkpoint/convert.pyconversion path. The sidecar can still contain the source checkpoint settings.Issue tracking
For PRs from open-source community contributors:
Linked issue: None.
Contribution process
Pre-checks
jit_fuser/torch.compile, CUDA extension, TE or external-library dispatch, or a scatter/index accumulation), I have added or updated its bit-exact determinism test and registered it intests/unit_tests/determinism/kernels/manifest.py(guide)Code review
Feel free to message or comment NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!
All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.
Step 1: Mark PR as "Ready for Review"
.github/CODEOWNERS.Final Review might get declined if these requirements are not fulfilled.
Step 2: Final Review
For PRs that change
megatron/core, once all expert reviewers have approved, theFinal Reviewlabel is applied automatically and final reviewers are assigned.For PRs outside
megatron/core, this step is skipped.Step 3: Approved
Once all required reviewers have approved, the
Approvedlabel is applied automatically.Merge
Any member of mcore-engineers will be able to merge your PR.
Testing
Before this merge, conversion-argument, training, checkpointing, and config checks passed on the pinned CUDA base with one GPU.
After merging
main, focused conversion-argument, checkpoint-selection, and config-mapping tests passed in a CPU-only Python container using one Gloo rank. Model-parallel initialization was stubbed, and the container used PyTorch 2.13.0 instead of the pinned CI image. The conversion regression fails againstmainand passes with this change. It mocks checkpoint loading and saving.Black, isort, pylint, and Ruff checks passed for the changed test. Python compilation and
git diff --checkalso passed. Mypy was not run.GPU/NCCL, end-to-end conversion, and multi-GPU tests were not run after the merge. The standard unit-test command requires the CI container and GPUs:
Upstream build, installation, lint, and documentation checks passed for the current head. GPU tests are waiting for a slot in the automated external-contributor queue.