[sudhu2k]

MoE

• Quantile Balancing router: adds auxiliary-loss-free load balancing using per-expert routing biases derived from quantile estimates (Implement Quantile Balancing in MoE NVIDIA/Megatron-LM#5349).
• Fused shared-expert MLP: enables shared experts through grouped GEMM, providing the path for fused GEMM, SwiGLU, and quantized execution (Fuse shared expert MLP with grouped GEMM NVIDIA/Megatron-LM#5604).
• Packed-sequence MoE dispatch: adds THD sequence-packing support to the DeepEP and HybridEP dispatcher backends ([main] add thd sequence packing dispatcher support for main NVIDIA/Megatron-LM#5008).
• MoE routing analysis: adds training/inference routing traces plus expert-concentration and one-layer-ahead predictability analysis tools (MoE routing analysis and metrics capture NVIDIA/Megatron-LM#5220).

Model Architecture and Support

• HybridModel: next-gen model class.
  • GPTModel to HybridModel guide published.
  • Added ability to directly load GPTModel checkpoints (Add load-time GPT-to-Hybrid checkpoint translation (#5675) NVIDIA/Megatron-LM#5792).
  • Part 1 of the design doc is complete; part 2 will be completed for 0.20.0.
  • DeepSeek v4 migrated to HybridModel on the dev branch (Enable Deepseek-v4 hybrid_model in dev branch Part (1/N) NVIDIA/Megatron-LM#5042, Enable Deepseek-v4 hybrid_model in dev branch Part (2/N) NVIDIA/Megatron-LM#5485, Add mHC support for HybridModel on dev NVIDIA/Megatron-LM#4949).
• GPTModel: deprecation announcement.
• DeepSeek Sparse Attention (DSA): adds backend-neutral CP and THD support, pipeline-aware indexer-loss scaling, IndexShare, and fused cuDNN and TileLang implementations ([split 2/4] Scale DSA indexer loss in pipeline schedules NVIDIA/Megatron-LM#5244, [split 4/4] Enable DSA CP and THD hooks NVIDIA/Megatron-LM#5246, [2/2] Wiring cuDNN fused DSA kernels support with THD, CP and IndexShare (GLM5.2) NVIDIA/Megatron-LM#5099, [2/2] Add TileLang fused DSA kernels support with THD and CP & Clean up NVIDIA/Megatron-LM#5049).
• Multi-Latent Attention in HybridModel: adds MLA layer support, Q/K norm spec resolution, and fused MLA projection handling to HybridModel (Port Multi-Latent Attention to HybridModel NVIDIA/Megatron-LM#4452).

Parallelism

• FSDP: MFSDP v2 is under development. See the design doc for links to the project tracker and implementation code.
• NCCL EP support: adds NCCL EP token dispatcher backend (NCCL EP support NVIDIA/Megatron-LM#5129, NCCL EP zero copy NVIDIA/Megatron-LM#5735).
• Fuse per-sequence AlltoAll: creates a unified AlltoAll (Fuse per-sequence AlltoAll into a unified one in GDN forward NVIDIA/Megatron-LM#4913).

Performance and Memory

• CUDA Graph-compatible activation offloading: enables fine-grained activation offloading during capture and replay, including deferred transfers and fused grouped-MLP offloading ([Main][feat] Support CUDA Graph capture offloading modules NVIDIA/Megatron-LM#3697, [feat] Support fine-grained activation offloading in fused group mlp NVIDIA/Megatron-LM#5082).
• Streaming quantized checkpoint loading: dequantizes FP8, MXFP8, blockwise-FP8, and NVFP4 tensors incrementally instead of allocating BF16 scratch space for the entire model (dist_ckpt: add --stream-ckpt-dequant to fix OOM on large FP8/MXFP8 loads with --fp8-param-gather NVIDIA/Megatron-LM#4451).
• Partial CUDA Graph memory optimization: moves graph surfaces into the global CUDA Graph memory pool and removes references and copies that retained excess buffers (Optimize memory usage of partial CUDA graphs NVIDIA/Megatron-LM#5451).

Precision and Optimizers

• MXFP8 training paths: take a look at our Nemotron 3 Super MXFP8 example.
• Lion distributed-optimizer support: routes Lion through DistributedOptimizer and adds single-moment checkpoint support, including Muon configurations that use Lion for scalar parameters (Route Lion through DistributedOptimizer and support single-moment checkpointing NVIDIA/Megatron-LM#5742).

Inference

• Disaggregated-inference foundations: adds heterogeneous inference-shard specifications, multi-destination refit, KV/Mamba-state resharding, local deterministic planning, and a NIXL copy backend (Disag MR1: Add inference shard specs and pg-collection building NVIDIA/Megatron-LM#5186, Disag MR2: Refit into multiple destination pools and tied-embedding + UVM fixes NVIDIA/Megatron-LM#5187, Disag MR3: Add heterogeneous KV/Mamba reshard planners NVIDIA/Megatron-LM#5188, Refit: local plan building, node-add cache key, and NIXL backend NVIDIA/Megatron-LM#5872).
• Full-model CUDA Graphs for MTP inference: enables full-model graph execution for Multi-Token Prediction inference (Add full model cuda graph support for MTP inference NVIDIA/Megatron-LM#4950).
• Load-aware prefix-cache routing: routes requests across inference workers using prefix-cache load information (Inference: Add load aware routing to prefix caching.  NVIDIA/Megatron-LM#5607).

Datasets

• Inter-document attention masking: GPTDataset now emits document-boundary metadata, resets positions per document, and uses THD attention to prevent cross-document attention within packed samples, including TP and CP support (Add inter-document attention masking to GPTDataset NVIDIA/Megatron-LM#5298).

Multimodal

• MIMO heterogeneous training: expands the current MIMO bullet with end-to-end non-colocated execution, stock training-loop integration, encoder prefetch, DDP overlap, and Nemotron MoE VLM/RADIO examples (E2E heterogenous non colocated MiMo training NVIDIA/Megatron-LM#5602, Add encoder prefetch for heterogeneous MIMO training NVIDIA/Megatron-LM#5833, Enable DDP communication overlap for MIMO training NVIDIA/Megatron-LM#5979).

Known Issues

• TE fused cross-entropy may degrade stability: setting cross_entropy_fusion_impl to te causes convergence issues on some models. Use native instead, or upgrade Transformer Engine to 2.18, which carries the fix (TE PR Add absorbed-mla NVIDIA/Megatron-LM#3193).
• SPCX grouped all-to-all payload corruption at high expert parallelism: in the affected NCCL 2.30.5/SPCX stack, stale or uninitialized ncclIbRequest.send.onlyWriteImm state can misclassify sends in grouped variable-split BF16 all-to-all operations, causing payload loss or corruption. MCore's MoE alltoall dispatcher can expose this issue at high EP configurations, but the fault originates in the SPCX transport rather than the model or router. As a workaround, disable the external NCCL network plugin and use NCCL's internal IB transport (NCCL_NET_PLUGIN=none NCCL_NET=IB), or use the HybridEP dispatcher (Megatron-Bridge issue [Megatron-FSDP] MaxPoolAllocator for double-buffering hybrid architectures. NVIDIA/Megatron-LM#5462).