Feature Request: Support per-layer active expert count (top-k) for MoE architectures #28380

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Version disclosure: b10797
Inspiration: nemotron-3-puzzle discussions and implemnetation.
To my knowledge, there is no issue discussing this exact idea. If I am wrong, feel free to close this issue.

• Add support for configuring active expert count (top-k aka expert_used_count) per layer in EXISTING(!!!) MoE architectures, rather than enforcing a global scalar across all blocks.
• Support setting this configuration via CLI param (e.g. --layer-expert-count N-N:N-N:etc or short form -lec N-N:N-N:etc, but requires 2D array entries, which to my knowledge does not exist in llama.cpp) or by parsing a 1D uint32 array for [arch].expert_used_count in gguf metadata for each layer individually (or any other valid method that yields the same results).

Motivation

• MoE layers do not contribute uniformly across model depth. Some layers may exhibit higher routing redundancy and can operate with reduced top-k (e.g. k=4 or even k=2 instead of k=8, or even higher values than trained on if it yields better results. See this ) with possibly a negligible perplexity degradation compared to other layers, as shown by nemotron-3-puzzle.

• Selectively reducing active experts on tolerant layers decreases memory bandwidth pressure and FLOPs without a quality loss that (could) occur when lowering k on ALL layers globally.

• Currently, --override-kv [arch].expert_used_count=int:N exists but only allows for a uniform scalar across the entire model.

• The CLI key-value override system does not parse array types.

• The model loader in llama-model.cpp strictly expects a scalar for the active expert count key.

• Reduces memory bandw. overhead on CPU offloaded layers, where DDR throughput is the primary bottleneck and pipeline synchronization asymmetry is not a constraint. (Might be an issue for GPUs)

Possible Implementation

• llama_hparams: Change n_expert_used into a per-layer array or std::vector<uint32_t> sized to n_layer. If a single scalar is provided, fill all elements with that value to preserve backward compatibility or use a different path.
• llama-model.cpp: Allow [arch].expert_used_count to read either GGUF_TYPE_UINT32 or GGUF_TYPE_ARRAY (type uint32). Verify array length equals block_count or n_layer before filling the per-layer hyperparameter.
• llama-graph.cpp: In build_moe_ffn, pass the layer-indexed value (n_expert_used[il]) to ggml_top_k rather than the global scalar.
• CLI: Add an argument to pass custom layer schedules at runtime without altering GGUF files (e.g. --expert-layers "0-7:8,8-20:4,21-27:8" or a comma-separatde list of integers matching layer count, or whatever any of you genuine geniuses can come up with).
• Router normalization: Ensure compatibility across router types or allow only for specific router types. Architectures with norm_topk_prob renormalize routing weights to 1.0 automatically when k changes, while architectures without top-k normalization may have the activation magnitudes changed when dropping or increasing k. Alternatively, output error for unsupported router types.
  (Some models like deepseek V4 do not support changing active expert count at all and outputs errors currently when changing n_expert_used globally. Qwen models accept different values, as well as bailing3moe etc. with no issues.)

Edge cases:

• k=0 needs explicit bypass in build_moe_ffn to avoid crashing ggml_top_k aka ggml_mul_mat_id
• Models without norm_topk_prob lose activation scale if k drops, but e.g. Qwen renormalizes automatically so its fine for some models.
• Uneven k across layers creates workload asymmetry across pipeline-parallel GPU splits. Might need optimizations.

Note:

These are possible ideas and might be genuinely flawed. Any other ideas that achieve the same outcome or better outcomes are very welcome. I will leave creating a PR to someone else as I personally may not find the time to implement this.