Repository navigation
Conversation
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Contributor
Author
|
/ok to test c806aa7 |
This was referenced Sep 11, 2026
tylerpoon
marked this pull request as ready for review
September 11, 2026 19:45
trintamaki
approved these changes
Sep 14, 2026
|
|
||
| def test_get_num_video_embeddings_applies_temporal_and_attention_pooling(): | ||
| assert ( | ||
| get_num_video_embeddings( |
Contributor
There was a problem hiding this comment.
could you add a test for the image case get_num_image_embeddings
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
ericharper
approved these changes
Sep 14, 2026
tylerpoon
enabled auto-merge
September 14, 2026 16:23
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/34874783688 |
tylerpoon
deleted the
tpoon/github-vlm-sft-02-vision-token-utilities
branch
September 14, 2026 17:27
This was referenced Sep 14, 2026
lauradang
added a commit
to tdene/Megatron-LM
that referenced
this pull request
Sep 14, 2026
einops is only declared in the dev extra, so the unconditional top-level import added in NVIDIA#7249 breaks the check-imports CI step for plain pip installs of megatron.core. Wrap the import in a try/except and assert availability inside patchify_image/unpatchify_image, matching the pattern used by the Megatron FSDP adapter. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>
1 of 6 tasks
balasaajay
added a commit
to balasaajay/Megatron-LM
that referenced
this pull request
Sep 15, 2026
This reverts commit c4997e3. Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
nemo-autobot-origin Bot
pushed a commit
to svcnemo-autobot/Megatron-LM
that referenced
this pull request
Sep 15, 2026
The re-applied multimodal vision token utilities (NVIDIA#7249) import einops at module scope. einops is an optional dependency that is absent from the minimal environment used by the 'Pip - NGC PyTorch' import check, so importing megatron.core.models.multimodal.utils raised ModuleNotFoundError. Guard the import the same way the rest of megatron/core does (e.g. transformer/attention.py): fall back to rearrange = None so the module imports cleanly; the symbol is only exercised when the patchify helpers actually run, where einops is installed. Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Add image patchify/unpatchify helpers and consistent vision-token accounting for temporal patches, attention pooling, class tokens, and tile tags. Include round-trip and video accounting tests.
This is part of a review-split multimodal SFT series. Production run scripts, data YAMLs, checkpoint conversion tooling, and separate CI compatibility fixes are outside this series.
Dependencies and review scope
Targets
main.Validation
mainat46fb90ca942a2950c58c5654b3e56877e88a3e35, preserving atomic signed and signed-off commits.megatron/coreandtests). Mypy is unavailable locally. Changed Python files pass syntax parsing; no CI configuration changes are included.Kept as a draft pending CI and dependency integration.