Skip to content

Initialize multimodal models from language checkpoints - #7316

Merged
tylerpoon merged 17 commits into
NVIDIA:mainfrom
tylerpoon:tpoon/github-vlm-sft-04-llm-only-checkpoint-init
Sep 22, 2026
Merged

tylerpoon merged 17 commits into
NVIDIA:mainfrom
tylerpoon:tpoon/github-vlm-sft-04-llm-only-checkpoint-init

Conversation

@tylerpoon

@tylerpoon tylerpoon commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Support initializing the language component of a multimodal model from a language-only checkpoint while leaving vision parameters locally initialized. Preserve legacy checkpoint enum deserialization compatibility and add regression tests.

Replaces #7251, which was automatically closed when its bot-managed base branch was deleted after the closure of #7250. The original discussion remains in #7251. This replacement reuses the original source branch and commits without code changes.

Production run scripts, data YAMLs, checkpoint conversion tooling, and separate CI compatibility fixes are outside this series.

Dependencies and review scope

Depends on #7315. Merge #7315 first, then this PR. #7249 has already merged.

Both PRs intentionally target main, not bot mirror branches. The author will manage merge order. Until #7315 is integrated and this branch is refreshed, the diff includes its prerequisite vision context-parallelism/metadata changes as well as this PR's checkpoint-initialization and legacy-deserialization changes. The retained history may also show inherited #7249 changes because that parent was squash-merged.

Validation

  • Reuses the existing signed and signed-off head b48769523b47732e89ec0110baa3afb4a685eee3.
  • Prior local Black, isort, pylint, Ruff, and Python syntax checks passed for the original branch; mypy was unavailable locally.
  • Combined-stack Super SFT validation is not an isolated-PR test result. Creating this replacement does not add a new test result or establish that current-main integration is green.
  • Kept as a draft pending dependency integration, review, and CI. No functional-test label or manual full-CI trigger is requested, to limit CI load. Reviewers are selected by the author.

Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 14, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
@tylerpoon
tylerpoon marked this pull request as ready for review September 16, 2026 17:04
@tylerpoon
tylerpoon requested review from a team as code owners September 16, 2026 17:04
Comment thread megatron/core/safe_globals.py Resolved
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
@tylerpoon
tylerpoon enabled auto-merge September 21, 2026 17:02
@tylerpoon

Copy link
Copy Markdown
Contributor Author

/ok to test 7f47fac

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the Final Review PR is in the "final review" stage label Sep 21, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added Approved All necessary approvals have been made and removed Final Review PR is in the "final review" stage labels Sep 21, 2026
@tylerpoon
tylerpoon added this pull request to the merge queue Sep 21, 2026
@nemo-automation-bot

Copy link
Copy Markdown

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/35655152711

Merged via the queue into NVIDIA:main with commit d256386 Sep 22, 2026
106 of 108 checks passed
@tylerpoon
tylerpoon deleted the tpoon/github-vlm-sft-04-llm-only-checkpoint-init branch September 22, 2026 01:11

This branch was previously deployed

2 inactive deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approved All necessary approvals have been made complexity: high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants