[ROADMAP][2026 Q3] Megatron Core Roadmap #5676

This roadmap tracks the key Megatron Core features, enhancements, and improvements originally planned for Q3 2026, including their carry-over status as of October 15, 2026. The 26.08 release has been delivered; remaining items continue beyond Q3 without a new delivery commitment. This is a tentative roadmap and subject to change.

Last updated: October 15, 2026.

Status convention: “delivered on main” or “delivered on dev” means the linked implementation has merged into that branch; it does not imply inclusion in 26.08 or complete end-to-end qualification. Items marked “in progress” may have merged foundations with integration or validation still outstanding.

For detailed information on past releases, see the Megatron Core release notes. For the previous quarterly roadmap, see Q2 2026 Roadmap #4997. For component-specific plans and status, see the Q3 MoE Roadmap #6757 and Q3 Megatron FSDP v2 Roadmap #6812.

Q3 Roadmap (26.08 shipped)

MoE

• DeepSeek-V4 architecture support — in progress - Core text-model architecture, packed-sequence training, context parallelism, Muon, and primary fusion paths are available on dev. Native/fused hybrid attention, mHC, HybridModel hash routing, and ClampedSwiGLU foundations have landed on main; packed THD/context parallelism, mHC PP/VPP and CUDA Graph coverage, performance-recipe ports, and broader end-to-end qualification remain in progress. See DeepSeek-V4 training support #4468 for branch-specific readiness.
  • DeepSeek-V4 training support #4468
  • Add native DeepSeek-V4 hybrid attention orchestration #6402
  • Add HybridModel hash MoE routing #6403
  • Add fused mHC support for HybridModel #6401
  • [dev] [DeepSeek-v4] Part 1: Hybrid Attention with CSA and HCA #4458
  • [dev] [DeepSeek-v4] Part 2: Hash MoE and SwiGLU clamp #4481
  • [dev] [DeepSeek-v4] Part 3: MTP support with mHC and new mHC contract #4518
  • [dev] Load-balanced context-parallel DSA indexer #6058
• Hash MoE and ClampedSwiGLU — foundations delivered on main and dev - HybridModel hash routing has merged on main (Add HybridModel hash MoE routing #6403), alongside ClampedSwiGLU across expert paths (Complete clamped SwiGLU across expert paths #5940). Broader DeepSeek-V4 training composition remains tracked separately.
  • [dev] [DeepSeek-v4] Part 2: Hash MoE and SwiGLU clamp #4481
  • [dev] [DeepSeek-v4] Add ClampedSwiGLU to MoE mlp_op_fuser and add force balance to hash routing #5130
  • Complete clamped SwiGLU across expert paths #5940
  • Add HybridModel hash MoE routing #6403
• MoE communication overlap - ShortcutMoE and THD dispatcher support are delivered. ShortcutMoE overlaps MoE all-to-all communication with paired attention, Mamba, or GDN computation; its PR reports 4.7–12.1% higher TFLOP/s and 82–95% overlap efficiency across four evaluated configurations. DeepEP v2 flex-dispatcher support is delivered on main (Add DeepEP v2 flex dispatcher backend #5153) and dev ([Dev] Add DeepEP v2 flex dispatcher backend #4793); latent shared-expert overlap remains in progress.
  • Add implementation for ShortcutMoE #6959
  • [main] add thd sequence packing dispatcher support for main #5008
  • Support latent MoE shared expert overlap #5135
  • Add DeepEP v2 flex dispatcher backend #5153
  • [Dev] Add DeepEP v2 flex dispatcher backend #4793
  • moe: add DeepEP V2 ElasticBuffer support to MoE flex dispatcher #4632 (additional ElasticBuffer work remains open)
• Optimized MoE kernels - Optimize Blackwell performance by integrating GEMM + SwiGLU fused MLP, mHC, and batchedGEMM fusion techniques.
• MegaMoE / ultra-sparse MoE - Research expanded expert counts, sparsity-driven activations, predictive routing mechanisms, and adaptable recomputation strategies.
  • MegaMOE Integration #6810
  • [Pytorch] MegaMOE Fused Op Integration TransformerEngine#3350
  • MOK implementation — delivered on dev - Initial Mixture-of-Kittens integration is available as a pluggable MoE megakernel backend. CuTe DSL MegaMoE integration (MegaMOE Integration #6810) and broader workload qualification remain in progress.
    • feat(moe): add a megakernel backend with MOK integration #6572

Model Architecture and Support

• DeepSeek-V4 recipes - Complete recipes with end-to-end convergence validation across Megatron-LM and Megatron Bridge (Megatron Bridge DeepSeek-V4 examples).
• Long-context validation - Verify convergence for 4K/16K dense and 64K/1M sparse training setups.
• HybridModel — recipe DSL in progress - HybridModel is already available for heterogeneous-layer models. The linked recipe layer schema, loading/lowering, model configuration, Nemotron-3 recipe, and CLI integration remain draft work.
  • Hybrid recipe layer config schema #4537
  • Hybrid recipe loading and lowering #5030
  • Support HybridModelConfig recipes in HybridModel #5031
  • Nemotron-3 Nano recipe with HybridModel DSL #4538
  • Add --model-recipe CLI flag for HybridModel #4539
• Gated Delta Product (GDP) — delivered - Add GDP as a first-class HybridModel mixer with training, static and dynamic inference, tensor and context parallelism, packed-sequence support, distributed checkpoint resharding, and optimizer integration.
  • Gated Delta Product (GDP) implementation #6074
• * Ling-V3 Tiny support — delivered on dev - Add training support for the KDA-MLA hybrid sparse-MoE architecture, including packed THD inputs, context parallelism, gated MLA, MTP, and checkpoint-resume validation.
  • [Dev] Add Ling-V3 Tiny hybrid model training support #6556
• Nemotron 3 on HybridModel - Make Nemotron 3 native to HybridModel.
• * DeepSeek-V4 transition to HybridModel — foundation delivered on dev - Initial DeepSeek-V4 HybridModel support has merged (Enable Deepseek-v4 hybrid_model in dev branch Part (1/N) #5042). Mainline architecture foundations are also available; end-to-end parity and qualification remain tracked in DeepSeek-V4 training support #4468.
  • Enable Deepseek-v4 hybrid_model in dev branch Part (1/N) #5042
• Qwen3-Next support — delivered - Support the GDN-based hybrid architecture, attention and shared-expert output gates, QK LayerNorm weight-decay handling, context parallelism, and packed sequences.
  • [main] feat(moe): Support gated delta net for Qwen3-Next (1/4) #1989
  • [main] feat(moe): Support moe shared expert gate for Qwen3-Next (2/4) #2751
  • [main] feat(moe): Support attention output gate for Qwen3-Next (3/4) #2752
  • [main] feat(moe): Support apply wd to qk layernorm for Qwen3-Next (4/4) #2753
  • [Main] Feat(moe): Gated delta net context parallel (CP) #2642
  • [main] feat(moe): Support packed sequence for gated delta net (GDN) #2645
  • GDN architecture and parallelism — delivered - Support Gated Delta Net for Qwen3-Next, including context parallelism, packed sequences, and unified per-sequence AlltoAll.
    • [main] feat(moe): Support gated delta net for Qwen3-Next (1/4) #1989
    • [Main] Feat(moe): Gated delta net context parallel (CP) #2642
    • [main] feat(moe): Support packed sequence for gated delta net (GDN) #2645
    • Fuse per-sequence AlltoAll into a unified one in GDN forward #4913
  • Fused GDN attention — delivered on main - Transformer Engine integration was restored in Re-add fused GDN TransformerEngine integration #7583 on October 10 after the earlier combined revert. The previous reapplication PR feat(ssm): AUT-2316 reapply fused GDN attention support #7327 was closed without merging.
    • [PyTorch] Fused GDN attention TransformerEngine#3351
    • Re-add fused GDN TransformerEngine integration #7583
    • Fused GDN attention support #6645
  • GDN2 — in progress - GDN2 architecture support is available in Megatron Core through Flash Linear Attention. Optimized GDN2 kernels are available in cuDNN Frontend, and Megatron Core integration remains planned; no delivery date is committed here.
    • Bump FLA to 0.5.1 and integrate GDN2 kernels #5765
• Qwen3.5 / Qwen3.5-VL support — in progress - Add model architecture logic, packed sequence handling, Qwen3.5-VL validation, and bidirectional checkpoint conversion between Hugging Face and Megatron Core. Architecture foundations, HybridModel migration ([Dev] feat(multimodal): migrate Qwen3.5-VL to HybridModel #6315), and pipeline-parallel foundations ([Dev] Add pipeline-parallel support to multimodal training in multimodal_dev code path #6081) have landed on dev. Full-model validation, the variable-length multimodal data path, and broader PP/VPP, CP/DCP, CUDA Graph, and FSDP v2 composition remain in progress; see [ROADMAP][2026 Q3] Megatron Core MoE Roadmap #6757 for detailed qualification work.
  • [dev] [2/5] Qwen3.5 support: FSDP DTensor Bridge checkpoint compatibility #4748
  • [dev] [3/5] Qwen3.5 support: SharedExpertMLP meta init #4749
  • [dev] [4/5] Qwen3.5 support: Interleaved MRoPE layout #4750
  • [dev] [5/5] Qwen3.5 support: Qwen3.5-VL training example #4751
  • [Dev] feat(multimodal): migrate Qwen3.5-VL to HybridModel #6315
  • [Dev] Add pipeline-parallel support to multimodal training in multimodal_dev code path #6081
• SFTDataset class — proposed implementation closed, unmerged - The proposed pre-tokenized, packed implementation in Add new SFTDataset to megatron.core.datasets #5017 was closed without merging; it is no longer an active implementation PR.
  • Add new SFTDataset to megatron.core.datasets #5017

Parallelism

• Megatron FSDP — experimental, in progress - Continue maturing the per-module fully_shard() API, communication-compute overlap, activation recompute support, and pooled memory allocation. Merged MFSDP v2 milestones include full-iteration CUDA Graphs, context parallelism, PP/VPP support (Enable Pipeline Parallelism Support for M-FSDP v2 #6486), MXFP8 compute and parameter gathering through the MCore adapter (feat: Enable MXFP8 through the MFSDP v2 MCore adapter #7588), and independent dense/expert sharding policies (Support independent dense and expert MFSDP sharding policies #7728). Broader composition and qualification remain tracked in #6812; see also the MFSDP project tracker.
  • Support full-iteration CUDA graphs with MFSDP v2 #7075
  • Allow context parallelism with MFSDP v2 #7220
  • Support independent HSDP instances for expert parameters #7012
  • Enable Pipeline Parallelism Support for M-FSDP v2 #6486
  • feat: Enable MXFP8 through the MFSDP v2 MCore adapter #7588
  • Support independent dense and expert MFSDP sharding policies #7728
• Generalized Tensor Parallelism (GTP) — delivered - Add fine-grained per-weight sharding and on-demand asynchronous all-gather with communication overlap; support CUDA Graphs, low-precision parameter gathers, composition with TP/SP/EP/DDP, Adam and Muon, and distributed checkpointing.
  • [feat] Generalized Tensor Parallelism (GTP) #4967
• NCCL EP support — delivered - Add an expert-parallel transport path and dispatch-manager support.
  • NCCL EP support #5129
  • [QUESTION] Do we have any plan to introduce NCCL EP? #4644
• Next-generation dynamic DP - Support multimodal data and improve performance.
• Communication enhancements / DeepEPv2 — delivered on main - DeepEP v2 is integrated as a flex-dispatcher backend (Add DeepEP v2 flex dispatcher backend #5153, merged October 9); the base NCCL EP transport path is also available (NCCL EP support #5129).
  • Add DeepEP v2 flex dispatcher backend #5153

Performance and Memory

• Deterministic training - Track fixed-environment repeatability, baseline equivalence, restart replay, scale validation, and performance overhead.
  • Determinism roadmap tracking #5785
• HybridModel 1F1B overlap — in progress - The common schedule-plan foundation is delivered; grouped HybridStack support, EP-overlap scheduling, FSDP, and training integration continue.
  • [refactor] Common combined-1F1B schedule-plan base (1/4 of #4798) #4941
  • [feat] HybridStack grouped syntax and EP-overlap (2/4 of #4798) #4942
  • [feat] FSDP support for HybridStack EP-overlap (3/4 of #4798) #4943
  • [fix] Training wire-up for hybrid EP-overlap (4/4 of #4798) #4944
• Checkpoint load performance - Reduce cross-rank reads for torch_dist load.
  • dist-ckpt: add local-replica mode to eliminate cross-rank reads of replicated params on torch_dist load #4628
• Reduce MoE peak memory - Avoid full intermediate list/cat buffers and chunk the return all-to-all/unpermute path.
• Full-model CUDA Graph with paged stashing — in progress - TE whole-MoE paged-stash foundations are delivered on dev ([Dev] Support paged stash with TE whole-MoE CUDA graphs #6022); the main port (Support paged stash with TE whole-MoE CUDA graphs #6870) remains open. Chunk-wise capture with dynamic gradient accumulation, full-iteration THD capture, and broader training composition remain in progress; see [ROADMAP][2026 Q3] Megatron Core MoE Roadmap #6757.
  • [Dev] Support paged stash with TE whole-MoE CUDA graphs #6022
  • Support paged stash with TE whole-MoE CUDA graphs #6870
• Enhanced attention mechanisms - Advance kernel development for GDN, NSA, and associated Transformer Engine or cuDNN Frontend integrations.
• GDN memory optimization — in progress - Whole-module selective recompute is available on dev (feat(ssm): whole-module 'gdn' selective recompute for GatedDeltaNet #5296). Fine-grained input-projection and conv/QKV recompute/offload (Add GDN input-projection and conv/QKV selective recompute and offload #7852), BF16/FLA recurrence offload (Add opt-in BF16/FLA GDN recurrence activation offloading #7908), and mainline whole-block recompute (GDN, whole-block attention recompute #7994) remain open.
  • feat(ssm): whole-module 'gdn' selective recompute for GatedDeltaNet #5296
  • Add GDN input-projection and conv/QKV selective recompute and offload #7852
  • Add opt-in BF16/FLA GDN recurrence activation offloading #7908
  • GDN, whole-block attention recompute #7994

Precision and Optimizers

• MXFP8 and NVFP4 training paths - DeepSeek-V3 NVFP4 training and FP4 execution in the 1F1B expert-parallel all-to-all overlap schedule are delivered; continue improving MXFP8/NVFP4 parameter gathering and low-precision GroupedGEMM support.
  • feat(optimizer): AUT-2316 reapply DeepSeek-V3 NVFP4 training support #7328 (restores Add DeepSeek-V3 NVFP4 training support #6841 after the earlier combined revert)
  • Support FP4 in 1F1B A2A overlap #6135
• Low-precision guide - Document practical low-precision training recipes and optimizer configuration recipes.
• Advanced Muon optimizer features — in progress - Muon MXFP8 primary-weight gathering and buffer-reuse support is available on main ([Main] Fix Numerical issue for GTP + Muon + MXFP8 param gather + use layerwise optimizer layout  #6754). Continue precision-aware Muon, compact-layout parity, and DeepSeek-V4 checkpoint/evaluation qualification; see DeepSeek-V4 training support #4468 and [ROADMAP][2026 Q3] Megatron Core MoE Roadmap #6757.
  • [Main] Fix Numerical issue for GTP + Muon + MXFP8 param gather + use layerwise optimizer layout  #6754
• MLA support for Muon - Extend the existing metadata-tagging scheme for splitting MLA up-projections.
  • Add MLA support to Muon #5015
• Model-Optimizer PTQ and QAD - Surface Model-Optimizer-based FP8/NVFP4 post-training quantization (PTQ) and quantization-aware distillation (QAD).
• PTQ and QAD documentation — in progress - A user guide using Nemotron 3 examples is under review in Add a user guide for post-training quantization and distillation #7274; it has not merged.
  • Add a user guide for post-training quantization and distillation #7274

Inference

• Sliding-window attention in dynamic batching - Bring dynamic inference to parity with static Transformer Engine attention for SWA models.
  • feat(inference): support sliding-window attention in the dynamic batc… #5138
• ETP - Add support for expert tensor parallelism.
  • Inference: Add support for expert tensor parallelism #4743
• Async scheduling — delivered - Introduce asynchronous dynamic-inference scheduling and overlap, then enable it by default.
  • Implement async scheduling for dynamic inference #5453
  • Overlap async scheduling phases #5549
  • Extend dynamic inference asynchronous scheduling support #5939
  • Enable async scheduling by default for dynamic inference #6851
• Batch-invariant MoE inference — delivered - Add fused vLLM and Transformer Engine backends for gated MoE plus routed block-scale MXFP8 FlashInfer execution with graph-safe bounded decode capacity.
  • Batch-invariance generation with vLLM fused-MoE backend at improved performance #6521
  • Inference: Add FlashInfer routed MXFP8 MoE with bounded decode capacity #6864

Reinforcement Learning

• Durable rollout storage — delivered - Persist completed rollout groups in a checksummed write-through bank with recovery, compaction, and resume integration so completed work survives process interruptions.
  • rl: add durable rollout bank for completed groups #6352
• ReFIT and mutable low-precision weights — delivered - Add NCCL many-to-many model transfer, in-place MXFP8 weight updates that preserve CUDA Graph buffer addresses, and parameter-aligned bounded-memory ReFIT execution.
  • M2n refit backend #6712
  • feat(refit): expose mutable MXFP8 destinations #6826
  • Mxfp8 refit bounded memory #6878

Multimodal

• MIMO (Multimodal In, Multimodal Out) extensions - Expand the TP/DP heterogeneous primitive and enable autonomous nD parallelism for submodules.
• Pipeline parallel support for the language model - Allow colocated layouts where the destination language grid has PP > 1 while the source encoder grid remains PP = 1.
  • Add LLM PP>1 support for colocated MIMO training (NMFW-19) #4784
• VLM CUDA Graph support - Capture the language model submodule.
  • [feat] enable cuda graph for the language_model sub module in VLM models #4519

* : Feature was developed on the dev branch first. Branch availability is stated above; check release notes for inclusion in a packaged release.

How to Provide Feedback

We welcome community input on prioritization. Please:

1. React to items you would like prioritized.
2. Comment on this issue with use cases, constraints, and hardware / model configurations.
3. Open focused feature requests with the enhancement label.
4. Contribute pull requests for roadmap items where possible.

Credits

This roadmap reflects the collective efforts of NVIDIA, external contributors, and the Megatron Core community.