[svcnvidia-nemo-ci]

Summary

Nightly sync of main into dev. Brings in 65 commits from main since the last sync.

Python-only line stats: +14649 / -3066 across 185 files.

Files where main's version was taken (in addition to skill list)

The dev-feature audit's strict line-by-line comparison flagged a number of files where main intentionally rewrote the code that dev's older versions had. Each was verified by tracing back to a specific main-only commit:

Files	Main commit(s)
examples/post_training/modelopt/*.py, *.md, *.sh, megatron/post_training/*.py	14aaa7e0e Modernize post-training modelopt example scripts (#4807)
megatron/core/resharding/**, tests/unit_tests/resharding/test_planner.py	20bf831da refit clean up and refactoring (#4762)
megatron/core/transformer/moe/experts.py, megatron/core/transformer/moe/moe_layer.py, megatron/core/transformer/transformer_layer.py	6815c0fef Combine GEMM + SwiGLU fused MLP PRs (#4636)
megatron/core/transformer/transformer_block.py, megatron/training/arguments.py, plus introduction of megatron/core/recompute.py	e41b37002 Refactor CUDA graph API (#4292)
tests/test_utils/python_scripts/compare_golden_values_kl.py, skills/update-golden-values/SKILL.md	f744215af Add bump-base-image skill and update golden value comparison (#4733)
megatron/core/models/vision/radio.py, megatron/core/models/multimodal/llava_model.py	6853b4793 Add LLaVA audio (sound) model support (#4402)
tests/unit_tests/dist_checkpointing/test_pipeline_parallel_layout.py and MSC default flip in arguments.py	95bba946c Disable MSC by default; opt in via --enable-msc (#4629)
tests/unit_tests/test_layer_wise_optimizer.py, megatron/core/optimizer/__init__.py (model_chunks=... conditional)	c1e938b03 Integrate LayerWiseDistributedOptimizer with DDP buffer infrastructure (#4509)
tests/unit_tests/training/config/test_container_base.py	0dc36dfc6 [training migration] Migrate mamba builder (#4550)
megatron/training/training.py, initialize.py, utils.py, datasets/data_samplers.py, megatron/core/optimizer/layer_wise_optimizer.py	Skill-list overrides (existing convention)

Dev-only additions preserved across the merge

• TEFusedDenseMLP class in megatron/core/extensions/transformer_engine.py (plus the dense_grouped_gemm selection wiring in gpt_layer_specs.py).
• _maybe_record_overload_factor(...) invocation in moe_layer.py.
• Hyper-Connection config block (enable_hyper_connections, num_residual_streams, mhc_*, use_fused_mhc) and HyperConnectionTransformerLayer references in transformer_config.py / transformer_layer.py.
• paged_stash_* imports and stash_context wrapping inside TEGroupedMLP._fused_forward.
• LinearCrossEntropyModule and paged_stash_init_chunk_handler imports in gpt_model.py.
• mhc_multistream, mhc_enabled paths in multi_token_prediction.py (combined with main's InferenceMode.is_active() and padding_mask plumbing).
• cp_group swap in MTPBlock.forward driven by packed_seq_params.cp_group.
• input_ids plumbing through MoE forward stack (router.py, moe_layer.py, transformer_layer.py, and the recompute path via a new input_ids kwarg added to megatron/core/recompute.py:checkpointed_forward).
• --inference-dynamic-batching-cuda-graph-max-tokens and --cuda-graph-impl=full_iteration in gpt_dynamic_inference_..._zmq test config.
• moe_router added to cuda_graph_modules in nemotron3_super_release_gb200.
• New gpt3_mcore_te_tp2_pp2_mhc test case alongside main's new gpt3_mcore_te_tp2_pp1_gdn_no_nvrx_* cases in tests/test_utils/recipes/h100/gpt.yaml.

Files restored / handled specially

• megatron/training/checkpointing.py: dropped dev's except ModuleNotFoundError legacy fallback because megatron/legacy/fp16_deprecated/loss_scaler.py was intentionally deleted on main (10b514bf5 "fully remove legacy code (fully remove legacy code #4759)").
• Kept dev's pyproject.toml, uv.lock, docker/Dockerfile.ci.dev, and .github/CODEOWNERS verbatim per skill convention.

API mismatches resolved

• attention.py / multi_latent_attention.py: kept dev's attn_proj_manager.group_offload(...) API (the merged tree's FineGrainedActivationOffloadingInterface exposes group_offload, not main's group_commit).
• gated_delta_net.py: took main's _resolve_cu_seqlens(..., cp_size=...) signature including the divisibility check, since callers now pass cp_size=self.cp_size.
• transformer_layer.py: switched to main's as_mlp_submodule / submodules.mlp(...) construction pattern; kept dev's **moe_kwargs (including input_ids) plumbing.
• gpt_layer_specs.py get_mlp_module_spec_for_backend: combined dev's dense_grouped_gemm selection with main's as_mlp_submodule classmethod dispatch and MlpBuilder return type.

How this diff was produced

Conflict resolution followed the nightly-sync skill: started from origin/dev, ran git merge origin/main --no-edit, resolved 47 conflicting paths surgically (preserving dev-only features unless I could identify a specific main commit removing them), ran black + isort over the 185 changed Python files, regenerated nothing in uv.lock, and audited with the pre-push hook before pushing.

🤖 Generated with Claude Code

[svcnvidia-nemo-ci]

/ok to test e2bb47c

[Phlip79]

/ok to test 66191ac

[Phlip79]

/ok to test a921dff

[FDecaYed]

/ok to test cc2a48b

[FDecaYed]

2 UT still failing. One seems nccl error that might be machine or CI related, does not seem due to the merge. The other I seems due to code accidentally dropped during merge, which I added back and merge with tot. Launching UT again now

[svcnvidia-nemo-ci]

Superseded by today's nightly sync.