Skip to content

MiMo-V2.6-Flash-RL: sidecar MTP draft (-md mtp-*.gguf --spec-type draft-mtp) fails with "blk.0.attn_q not found" — draft should reuse main model weights #29345

Open
Open
MiMo-V2.6-Flash-RL: sidecar MTP draft (-md mtp-*.gguf --spec-type draft-mtp) fails with "blk.0.attn_q not found" — draft should reuse main model weights#29345
Labels
bug-unconfirmed

Description

@Ryan-W-lab
Ryan-W-lab
opened on Sep 23, 2026

Name and Version

llama.cpp 0.4.1-dev (build 50, commit 9575389); reproduced on latest master (same commit) as well
Ubuntu 22.04.5 LTS (Linux)

Operating systems

Linux

GGML backends

CUDA

Hardware

2× Intel Xeon Platinum 8269 (Cascade Lake, 52C/104T), 768 GB DDR4-2133 ECC,
8× NVIDIA RTX 3080 20GB (sm86), dual NUMA

Models

Custom (official GGUF release): ggml-org/MiMo-V2.6-Flash-RL-GGUF
(main: MiMo-V2.6-Flash-RL-MXFP4)

Problem description & steps to reproduce

Running speculative decoding with the officially released MTP draft for
MiMo-V2.6-Flash-RL, per the model repo README:
main model via -m, mtp sidecar via -md, plus --spec-type draft-mtp.
The main model loads and runs fine without the MTP flags (~18.6 tok/s decode
on this box with 40/48 expert layers offloaded to GPU). Crash occurs as soon
as the MTP draft is enabled. Command:

numactl --interleave=all ./build/bin/llama-server -m /mnt/data4t/models/mimo-v26rl-gguf/MiMo-V2.6-Flash-RL-MXFP4-00001-of-00002.gguf -md /mnt/data4t/models/mimo-v26rl-gguf/mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf --spec-type draft-mtp --spec-draft-n-max 7 --spec-draft-ngl 999 -ngl 999 -t 48 -c 262144 -np 1 -fa on -b 4096 -ub 1024 -ctk q8_0 -ctv q8_0 -ot 'blk\.(0|1|2|3)\.ffn_.*_exps\..*=CUDA0,blk\.(4|5|6|7)\.ffn_.*_exps\..*=CUDA1,blk\.(8|9|10|11)\.ffn_.*_exps\..*=CUDA2,blk\.(12|13|14|15)\.ffn_.*_exps\..*=CUDA3,blk\.(32|33|34|35)\.ffn_.*_exps\..*=CUDA4,blk\.(36|37|38|39)\.ffn_.*_exps\..*=CUDA5,blk\.(40|41|42|43)\.ffn_.*_exps\..*=CUDA6,blk\.(44|45|46|47)\.ffn_.*_exps\..*=CUDA7,blk\..*\.ffn_.*_exps\..*=CPU' -ts 1,1,1,1,1,1,1,1 --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 --chat-template-kwargs '{"enable_thinking":false}' --path /home/cw/llama-ui-dist --port 8899 --host 0.0.0.0 --jinja

### First Bad Commit

## Expected Behavior

The MTP draft loads (reusing the main model's base weights, per the MTP
architecture) and speculative decoding runs.

## Current Behavior

Crash during model load. With MTP enabled, the main model itself fails first:
loader reports `check_tensor_dims: tensor 'blk.0.attn_q.weight' not found`,
then the draft sidecar load also fails with the same error. A retry pass then
attempts a ~50 GB CUDA0 allocation and OOMs. No request is ever served.
(The sidecar alone, loaded as `-m`, also fails with the same
`blk.0.attn_q.weight not found`.)

Analysis: the MTP draft is designed to reuse the main model's weights
(attention Q/K/V from base layers) and only adds its own small MTP layer set
(blk.48-50 + MTP head); the sidecar therefore only contains MTP-specific
tensors. The `-md` sidecar loading path appears to treat the draft as a
complete standalone model, requiring base-layer tensors not present in the
sidecar. The embedded-MTP packaging (MTP tensors inside the main GGUF, as in
tnhnyzc/MiMo-V2.5-MTP-GGUF) reportedly works with `--spec-type draft-mtp` on
the same build, so this is specific to the sidecar (`-md`) path.
Reproduced on latest master (9575389) as well.

### Relevant log output

## Failure Logs

0.00.000.704 I srv llama_server: initializing ...
0.00.295.789 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.295.993 W srv llama_server: security: no API key is set and CORS allows all origins (see #25655)
0.00.313.694 I srv load_model: loading model '/mnt/data4t/models/mimo-v26rl-gguf/MiMo-V2.6-Flash-RL-MXFP4-00001-of-00002.gguf'
0.02.576.976 E llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' not found
0.02.576.995 E llama_model_load_from_file_impl: failed to load model
0.02.577.065 W operator(): failed to measure the memory of the extra model, fitting without it: failed to load model
0.02.577.105 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 999, abort
0.02.839.485 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden
0.09.486.699 E ggml_backend_cuda_buffer_type_alloc_buffer: cudaMalloc failed: out of memory
0.09.486.705 E alloc_tensor_range: failed to allocate CUDA0 buffer of size 53203257600
0.11.920.771 E llama_model_load: error loading model: unable to allocate CUDA0 buffer
0.11.920.806 E llama_model_load_from_file_impl: failed to load model
0.11.920.886 E cmn common_init_: failed to load model '/mnt/data4t/models/mimo-v26rl-gguf/MiMo-V2.6-Flash-RL-MXFP4-00001-of-00002.gguf'
0.11.920.928 E srv load_model: failed to load model, '/mnt/data4t/models/mimo-v26rl-gguf/MiMo-V2.6-Flash-RL-MXFP4-00001-of-00002.gguf'
0.11.920.959 I srv operator(): operator(): cleaning up before exit...
0.11.925.110 E srv llama_server: exiting due to model loading error


## Additional Context

Related: #25924 ("Feature Request: MTP support for MiMo V2.5") and #26228
("mimo2: add MTP draft support") — the same sidecar-vs-embedded MTP packaging
question. Issue #26605 / #25574 touch MTP layer memory allocation, possibly
relevant to the ~50 GB retry OOM above.<details>
<summary>Logs</summary>
<!-- Copy-pasted short logs go into the "console" area here -->

```console

Activity

  1. Ryan-W-lab
    added
    bug-unconfirmed
    on Sep 23, 2026

  2. DanMurdoch2025 commented on Sep 27, 2026

    @DanMurdoch2025
    DanMurdoch2025
    on Sep 27, 2026
    Can confirm the identical issue on an independent hardware configuration (Intel Cascade Lake / Pascal GPUs on Ubuntu 24.04). Passing the official MiMo-V2.6-Flash-RL sidecar via `-md` with `--spec-type draft-mtp` triggers `check_tensor_dims: tensor 'blk.0.attn_q.weight' not found` during the draft loader phase.

### Environment & Build Details
- **OS:** Ubuntu Server 24.04 LTS (Kernel Linux x86_64)
- **CPU:** Intel Xeon W-2235 (Cascade Lake)
- **RAM:** 256 GB ECC DDR4-2933 (8×32GB)
- **GPUs:** 3× NVIDIA Tesla P40 24GB (Driver 580.178.04 / CUDA 12.9)
- **Version / Commit:** `0.5.0-dev (build 11151, commit bd4f514db14d87fded667787a7a963bfbaa98e89)` built with GCC 13.3.0
- **CMake Flags:**
  ```bash
  cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
    -DGGML_CUDA_ARCHITECTURES=61 \
    -DGGML_NATIVE=ON \
    -DBUILD_SHARED_LIBS=OFF \
    -DLLAMA_BUILD_TESTS=OFF \
    -DLLAMA_OPENSSL=ON \
    -DCMAKE_INSTALL_PREFIX=/opt/llama.cpp/cuda

    Models Tested

    • Base: ggml-org/MiMo-V2.6-Flash-RL-GGUF (MiMo-V2.6-Flash-RL-MXFP4-00001-of-00002.gguf)
    • Draft Sidecar: ggml-org/MiMo-V2.6-Flash-RL-GGUF (mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf)

    Test 1: Base Model Only (Succeeds)

    The main model loads and runs without issue using manual expert tensor offloading across the 3× P40s and CPU:

    P40S="GPU-c865d3e1-3e86-3ddd-68e7-1b6ea8176b11,GPU-b90b1823-adcf-f229-69df-35e9d777333e,GPU-49bf775d-d694-5425-81cd-720f3de8f105"

CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=$P40S /opt/llama.cpp/cuda/bin/llama-server \
  -m MiMo-V2.6-Flash-RL-MXFP4-00001-of-00002.gguf \
  --jinja -c 65536 -t 12 -fa on -ngl 99 -ts 1,1,1 \
  -ot "blk\.(36|39|42|45)\.ffn_(gate|up|down)_exps\.weight=CUDA0,blk\.(37|40|43|46)\.ffn_(gate|up|down)_exps\.weight=CUDA1,blk\.(38|41|44|47)\.ffn_(gate|up|down)_exps\.weight=CUDA2,ffn_(gate|up|down)_exps\.weight=CPU" \
  --load-mode none --host 0.0.0.0 --port 8080

    0.00.000.446 I srv  llama_server: initializing ...
0.00.031.900 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.032.280 W srv  llama_server: security: no API key is set and CORS allows all origins (see [https://github.com/ggml-org/llama.cpp/pull/25655](https://github.com/ggml-org/llama.cpp/pull/25655))
0.00.034.342 I srv    load_model: loading model '/mnt/.../MiMo-V2.6-Flash-RL-MXFP4-00001-of-00002.gguf'
0.01.152.013 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
3.08.576.258 I cmn          init: llama threadpool init, n_threads = 12
3.09.037.993 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 65536, kv_unified = 'true'
3.09.050.583 W srv          init: chat template supports preserving reasoning, it is enabled by default (may use more tokens, disable via --no-reasoning-preserve)
3.09.050.592 I srv  llama_server: model loaded
3.09.050.593 I srv  llama_server: listening on [http://0.0.0.0:8080](http://0.0.0.0:8080)

    Test 2: Base Model + MTP Sidecar (Fails)

    Enabling the MTP draft sidecar via -md mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf --spec-type draft-mtp --spec-draft-n-max 3:

    P40S="GPU-c865d3e1-3e86-3ddd-68e7-1b6ea8176b11,GPU-b90b1823-adcf-f229-69df-35e9d777333e,GPU-49bf775d-d694-5425-81cd-720f3de8f105"

CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=$P40S /opt/llama.cpp/cuda/bin/llama-server \
  -m MiMo-V2.6-Flash-RL-MXFP4-00001-of-00002.gguf \
  -md mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --jinja -c 65536 -t 12 -fa on -ngl 99 -ts 1,1,1 \
  -ot "blk\.(36|39|42|45)\.ffn_(gate|up|down)_exps\.weight=CUDA0,blk\.(37|40|43|46)\.ffn_(gate|up|down)_exps\.weight=CUDA1,blk\.(38|41|44|47)\.ffn_(gate|up|down)_exps\.weight=CUDA2,ffn_(gate|up|down)_exps\.weight=CPU" \
  --load-mode none --host 0.0.0.0 --port 8080

    0.00.000.520 I srv  llama_server: initializing ...
0.00.034.601 I srv    load_model: loading model '/mnt/.../MiMo-V2.6-Flash-RL-MXFP4-00001-of-00002.gguf'
0.00.767.328 E llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' not found
0.00.767.345 E llama_model_load_from_file_impl: failed to load model
0.00.768.432 W operator(): failed to measure the memory of the extra model, fitting without it: failed to load model
0.00.913.886 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
3.08.011.223 I cmn          init: llama threadpool init, n_threads = 12
3.08.330.581 I common_speculative_init_result: loading draft model '/mnt/.../mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf'
3.08.481.107 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
3.08.588.207 E llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' not found
3.08.588.215 E llama_model_load_from_file_impl: failed to load model
3.08.588.216 E common_speculative_init_result: failed to load draft model, '/mnt/.../mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf'
3.08.588.218 E srv    load_model: failed to load draft model, '/mnt/.../mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf'
3.08.588.225 I srv    operator(): operator(): cleaning up before exit...
3.08.588.660 E srv  llama_server: exiting due to model loading error

    Observations

    • Confirms the OP's analysis: the -md draft initialization route expects standalone model tensor architecture and looks for blk.0.attn_q.weight, rather than binding to the base model's existing attention layers when using --spec-type draft-mtp.

  3. ggerganov commented on Sep 27, 2026

    @ggerganov
    ggerganov
    on Sep 27, 2026
    Member

    Re-download the MTP sidecar - it should now work.

  4. DanMurdoch2025 commented on Sep 28, 2026

    @DanMurdoch2025
    DanMurdoch2025
    on Sep 28, 2026

    Downloaded and tested. Q4 and Q8 MTP drafts look good. I did run into an issue with DFlash:

    GitHub Follow-up Comment

    Following up after testing on the latest `master`:

### Summary
1. **MTP Drafts (`--spec-type draft-mtp`):** Both `mtp-MiMo-V2.6-Flash-RL-Q4_0.gguf` and `mtp-MiMo-V2.6-Flash-RL-Q8_0.gguf` failed with `blk.0.attn_q.weight not found` on build `11151` (`bd4f514db`), but **work cleanly on build `11222` (`a97cce86a`)**. Loading and inference both succeed.
2. **DFlash Draft (`--spec-type draft-dflash`):** `dflash-MiMo-V2.6-Flash-RL-Q8_0.gguf` loads successfully, but crashes immediately upon prompt evaluation with a failed assertion in `llama-graph.cpp`.

---

### Environment
- **Build / Commit:** `0.5.0-dev (build 11222, commit a97cce86a8addeb9f40cba7a261c94b1f0c576cb)`
- **Platform:** Ubuntu Server 24.04 LTS | 256 GB RAM | 3× NVIDIA Tesla P40 24GB (Driver 580.178.04, CUDA 12.9)
- **Base Model:** `MiMo-V2.6-Flash-RL-MXFP4-00001-of-00002.gguf` (snapshot `9bf2e45b`)
- **Draft Snapshot:** `a5d1269f`

---

### 1. Verified Working: MTP Q4_0 & Q8_0 on commit `a97cce86a`

Both draft files successfully load and process inference tasks:

```bash
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=$P40S /opt/llama.cpp/cuda/bin/llama-server \
  -m "$MIMO26_FLASH" -md "$MIMO26_FLASH_MTP_Q4" \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --jinja -c 65536 -t 12 -fa on -ngl 99 -ts 1,1,1 \
  -ot "blk\.(36|39|42|45)\.ffn_(gate|up|down)_exps\.weight=CUDA0,blk\.(37|40|43|46)\.ffn_(gate|up|down)_exps\.weight=CUDA1,blk\.(38|41|44|47)\.ffn_(gate|up|down)_exps\.weight=CUDA2,ffn_(gate|up|down)_exps\.weight=CPU" \
  --load-mode none --host 0.0.0.0 --port 8080

    0.00.040.109 I srv    load_model: loading model '.../MiMo-V2.6-Flash-RL-MXFP4-00001-of-00002.gguf'
3.10.637.397 I common_speculative_init_result: loading draft model '.../mtp-MiMo-V2.6-Flash-RL-Q4_0.gguf'
3.12.142.221 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 65536, kv_unified = 'true'
3.12.192.345 I srv  llama_server: model loaded
3.12.192.346 I srv  llama_server: listening on [http://0.0.0.0:8080](http://0.0.0.0:8080)
3.28.236.700 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1

    2. DFlash Crash on Prompt Evaluation (draft-dflash)

    Using dflash-MiMo-V2.6-Flash-RL-Q8_0.gguf under the exact same environment and build (a97cce86a):

    CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=$P40S /opt/llama.cpp/cuda/bin/llama-server \
  -m "$MIMO26_FLASH" -md "$MIMO26_DFLASH_Q8" \
  --spec-type draft-dflash --spec-draft-n-max 3 \
  --jinja -c 65536 -t 12 -fa on -ngl 99 -ts 1,1,1 \
  -ot "blk\.(36|39|42|45)\.ffn_(gate|up|down)_exps\.weight=CUDA0,blk\.(37|40|43|46)\.ffn_(gate|up|down)_exps\.weight=CUDA1,blk\.(38|41|44|47)\.ffn_(gate|up|down)_exps\.weight=CUDA2,ffn_(gate|up|down)_exps\.weight=CPU" \
  --load-mode none --host 0.0.0.0 --port 8080

    The server initializes and binds the speculative draft engine without error, but the process aborts as soon as the first prompt is launched:

    3.10.248.264 I common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
3.10.248.283 I common_speculative_impl_draft_dflash: - n_max=3, n_min=0, p_min=0.00
3.10.248.283 I common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=151675, n_extract=5, sample_from_anchor=true
3.10.306.810 I srv  llama_server: model loaded
3.10.306.812 I srv  llama_server: listening on [http://0.0.0.0:8080](http://0.0.0.0:8080)
5.17.832.459 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
5.17.832.798 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
/opt/src/llama.cpp/src/llama-graph.cpp:1379: GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null") failed
/opt/llama.cpp/cuda/bin/llama-server(+0x181326b)[0x61acfa11e26b]
...
/opt/llama.cpp/cuda/bin/llama-server(+0x195035)[0x61acf8aa0035]
Aborted (core dumped)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug-unconfirmed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions