Repository navigation
Add Lion optimizer support - #3813
Merged
Merged
Conversation
Contributor
|
This PR has been automatically converted to draft because all PRs must start as drafts. When you are ready for review, click Ready for Review to begin the review process. This will:
See the contribution guide for more details. |
svcnvidia-nemo-ci
marked this pull request as draft
March 11, 2026 22:49
mchrzanowski
force-pushed
the
add-lion-optimizer-v2
branch
from
March 12, 2026 03:19
457ce16 to
c4a57c3
Compare
mchrzanowski
marked this pull request as ready for review
March 12, 2026 03:20
mchrzanowski
requested review from
mkhona-nvidia
and removed request for
a team
March 12, 2026 03:20
| @@ -2171,6 +1855,16 @@ def _add_regularization_args(parser): | |||
| help='How to perform NS calculation for tensor model parallel weights') | |||
| group.add_argument('--muon-extra-scale-factor', type=float, default=1.0, | |||
| help='Additional scale factor for the muon update') | |||
| group.add_argument('--muon-nonlinear-optimizer', type=str, default='adam', | |||
Contributor
There was a problem hiding this comment.
We need to make sure to ping downstream stuff (Mbridge etc)'s Muon recipe to additionally add --muon-non-linear-optimizer as an extra entry.
Contributor
There was a problem hiding this comment.
@janEbert , maybe you could add this to mbridge?
mkhona-nvidia
requested changes
Mar 12, 2026
mkhona-nvidia
approved these changes
Mar 13, 2026
deepakn94
reviewed
Mar 13, 2026
mchrzanowski
force-pushed
the
add-lion-optimizer-v2
branch
from
March 13, 2026 17:02
6dd614b to
0fb94af
Compare
deepakn94
approved these changes
Mar 14, 2026
ericharper
reviewed
Mar 16, 2026
ericharper
approved these changes
Mar 16, 2026
svcnvidia-nemo-ci
added
Approved
Contributor
Author
|
/ok ebdbe2c |
Contributor
Author
Contributor
Author
|
/ok to test f212469 |
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
/ok to test 207c465 |
This reverts commit f212469.
Contributor
Author
|
/ok to test e911d8a |
asolergi-nv
approved these changes
Mar 17, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/23215212706 |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/23215938659 |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Mar 17, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/23219203590 |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Mar 17, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/23227234166 |
ilml
added a commit
to ilml/Megatron-LM
that referenced
this pull request
Mar 20, 2026
New files: - tests/unit_tests/test_lion_optimizer.py
This was referenced Mar 23, 2026
5 tasks
yangbofun
pushed a commit
to xlm-research/Megatron-LM
that referenced
this pull request
May 22, 2026
Signed-off-by: Mike Chrzanowski <mchrzanowski@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: root <root@pool0-0020.cm.cluster> Co-authored-by: root <root@pool0-0038.cm.cluster> Co-authored-by: root <root@pool0-0022.cm.cluster>
yhgalaxy
pushed a commit
to yhgalaxy/Megatron-LM
that referenced
this pull request
Jun 17, 2026
Signed-off-by: Mike Chrzanowski <mchrzanowski@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: root <root@pool0-0020.cm.cluster> Co-authored-by: root <root@pool0-0038.cm.cluster> Co-authored-by: root <root@pool0-0022.cm.cluster> Signed-off-by: yhgalaxy <yhgalaxy@outlook.com>
jon-barker
pushed a commit
to jon-barker/Megatron-LM
that referenced
this pull request
Jul 10, 2026
Signed-off-by: Mike Chrzanowski <mchrzanowski@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: root <root@pool0-0020.cm.cluster> Co-authored-by: root <root@pool0-0038.cm.cluster> Co-authored-by: root <root@pool0-0022.cm.cluster> Signed-off-by: Jon Barker <jbarker@aws-cmh-slurm-1-vscode-02.cm.cluster>
terminator123
pushed a commit
to 021ai/Megatron-LM
that referenced
this pull request
Aug 3, 2026
Signed-off-by: Mike Chrzanowski <mchrzanowski@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: root <root@pool0-0020.cm.cluster> Co-authored-by: root <root@pool0-0038.cm.cluster> Co-authored-by: root <root@pool0-0022.cm.cluster>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Import Lion optimizer from emerging_optimizers and integrate it as a standalone optimizer (--optimizer lion) and as a Muon nonlinear parameter optimizer (--muon-nonlinear-optimizer lion).