Repository navigation
Add option to run MIMO input projections on LLM ranks - #6843
chtruong814 merged 3 commits into
Conversation
eb7b2ef to
c0f9329
Compare
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
687789c to
4627c10
Compare
4627c10 to
1a387b8
Compare
| module_sd = sharded_state_dict_default( | ||
| inner, child_prefix, sharded_offsets, mod_metadata | ||
| ) | ||
| if ( |
There was a problem hiding this comment.
we need this change so we can still load checkpoints indepedent of if projection layers are used on llm ranks
deepakn94
changed the title
svcnvidia-nemo-ci
added
Approved
0f56e81 to
afc1b15
Compare
|
/ok to test afc1b15 |
afc1b15 to
261ed6f
Compare
|
/ok to test 261ed6f |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/33340891254 |
Signed-off-by: ykarnati <ykarnati@nvidia.com>
|
/ok to test e788d04 |
|
/ok to test 4f02449 |
|
/ok to test 9cd934e |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/35283996265 |
|
Ran CI as part of an aggregated branch that included multiple PRs in the queue. CI passed Will fast merge this |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/35302229442 |
Summary
--mimo-run-input-projections-on-llm-ranksValidation
16634648(8 GPUs): the exact models-bucket optimizer fixture regression, frozen-encoder language-rank BF16 optimizer step, and world-consensus test passed on ranks 0-716634875: Black and isort checks passed on the final files16633675(8 GPUs): frozen-encoder language-rank BF16 optimizer step and world-consensus tests passed16627732(8 GPUs): renamed parser flag, validation, and provider placement tests passed16627995(8 GPUs): exact 20L checkpoint save/load/resave round trip passed with the renamed flag16616723(8 GPUs): empty source payload, frozen-encoder language-rank projection, and real zero-token GTP forward/backward tests passed16606559(8 GPUs): 23 focused parser, construction, freeze/DDP, quantization, empty-payload, and schedule tests passed16606666(8 GPUs, GTP-enabled): zero-token projector forward/backward and cross-placement checkpoint loading in both directions passed16568629/16568652: functional checkpoint round trips passed with language-rank and encoder-rank projection placement16634648isort --check-only,ruff check,compileall, andgit diff --checkpassed