Skip to content

llama : fix tensor split for fused qkv with uneven K/V head sizes - #29294

Merged
am17an merged 5 commits into
ggml-org:masterfrom
am17an:aman/fix-mimo-sm-tensor
Sep 25, 2026
Merged

llama : fix tensor split for fused qkv with uneven K/V head sizes#29294
am17an merged 5 commits into
ggml-org:masterfrom
am17an:aman/fix-mimo-sm-tensor

Conversation

@am17an

@am17an am17an commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Overview

Additional information

Requirements

@ggerganov

ggerganov commented Sep 23, 2026

Copy link
Copy Markdown
Member

The MiMo-V2.6-Flash-RL on 2x DGX Sparks proceeds further with this change but still crashes with another error:

llama.cpp/ggml/src/ggml-backend-meta.cpp:592: GGML_ASSERT(split_states_equal(src_ss[0], src_ss[1])) failed

@am17an
am17an marked this pull request as ready for review September 23, 2026 18:14
@am17an
am17an requested a review from CISC as a code owner September 23, 2026 18:14
@am17an
am17an force-pushed the aman/fix-mimo-sm-tensor branch from 24a8465 to a048481 Compare September 24, 2026 07:30
@ggerganov

ggerganov commented Sep 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

The tensor split over RPC seems to work now. However the MTP conversion is still not correct:

3.49.270.506 I load_tensors: loading model tensors, this can take a while... (load_mode = dio)
3.49.291.028 E llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' not found
3.49.291.039 E llama_model_load_from_file_impl: failed to load model
3.49.291.040 E common_speculative_init_result: failed to load draft model, '/home/ggml/.cache/huggingface/hub/models--ggml-org--MiMo-V2.6-Flash-RL-GGUF/snapshots/cac6e14e8c2063aba38f487c2b041d9a5a22c7b6/mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf'
3.49.291.046 E srv    load_model: failed to load draft model, '/home/ggml/.cache/huggingface/hub/models--ggml-org--MiMo-V2.6-Flash-RL-GGUF/snapshots/cac6e14e8c2063aba38f487c2b041d9a5a22c7b6/mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf'
3.49.291.053 I srv    operator(): operator(): cleaning up before exit...
3.49.292.853 E srv  llama_server: exiting due to model loading error

I run with:

llama-server -hf ggml-org/MiMo-V2.6-Flash-RL-GGUF:MXFP4 \
    --spec-default --spec-type draft-mtp \
    -ub 2048 -c 262144 -lv 4 \
    --top-p 0.95 --top-k 0 --temp 1.0 \
    --rpc 192.168.100.1:50052,192.168.100.2:50052 -dev RPC0,RPC1 \
    --host 0.0.0.0 --port 8013 -lm dio --alias agent-dsv4 -sm tensor 2>&1 | tee ~/llama.log

For some reason the converted MTP sidecar has block indices starting from 48, instead of 0:

3.49.131.132 I common_speculative_init_result: loading draft model '/home/ggml/.cache/huggingface/hub/models--ggml-org--MiMo-V2.6-Flash-RL-GGUF/snapshots/cac6e14e8c2063aba38f487c2b041d9a5a22c7b6/mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf'
3.49.161.080 I llama_model_loader: loaded meta data with 42 key-value pairs and 39 tensors from /home/ggml/.cache/huggingface/hub/models--ggml-org--MiMo-V2.6-Flash-RL-GGUF/snapshots/cac6e14e8c2063aba38f487c2b041d9a5a22c7b6/mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf (version GGUF V3 (latest))
3.49.161.090 I llama_model_loader: - tensor split  0:                    output.weight q8_0     [   4096, 152576,      1,      1 ]   633.25 MiB
3.49.161.091 I llama_model_loader: - tensor split  0:               output_norm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.092 I llama_model_loader: - tensor split  0:                token_embd.weight q8_0     [   4096, 152576,      1,      1 ]   633.25 MiB
3.49.161.093 I llama_model_loader: - tensor split  0:          blk.48.attn_norm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.094 I llama_model_loader: - tensor split  0:        blk.48.attn_output.weight q8_0     [   8192,   4096,      1,      1 ]    34.00 MiB
3.49.161.094 I llama_model_loader: - tensor split  0:           blk.48.attn_qkv.weight q8_0     [   4096,  14848,      1,      1 ]    61.62 MiB
3.49.161.095 I llama_model_loader: - tensor split  0:         blk.48.attn_sinks.weight f32      [     64,      1,      1,      1 ]     0.00 MiB
3.49.161.096 I llama_model_loader: - tensor split  0:           blk.48.ffn_down.weight q8_0     [  16384,   4096,      1,      1 ]    68.00 MiB
3.49.161.097 I llama_model_loader: - tensor split  0:           blk.48.ffn_gate.weight q8_0     [   4096,  16384,      1,      1 ]    68.00 MiB
3.49.161.097 I llama_model_loader: - tensor split  0:           blk.48.ffn_norm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.098 I llama_model_loader: - tensor split  0:             blk.48.ffn_up.weight q8_0     [   4096,  16384,      1,      1 ]    68.00 MiB
3.49.161.099 I llama_model_loader: - tensor split  0:  blk.48.layer_output_norm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.100 I llama_model_loader: - tensor split  0:      blk.48.nextn.eh_proj.weight q8_0     [   8192,   4096,      1,      1 ]    34.00 MiB
3.49.161.100 I llama_model_loader: - tensor split  0:        blk.48.nextn.enorm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.101 I llama_model_loader: - tensor split  0:        blk.48.nextn.hnorm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.102 I llama_model_loader: - tensor split  0:          blk.49.attn_norm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.103 I llama_model_loader: - tensor split  0:        blk.49.attn_output.weight q8_0     [   8192,   4096,      1,      1 ]    34.00 MiB
3.49.161.103 I llama_model_loader: - tensor split  0:           blk.49.attn_qkv.weight q8_0     [   4096,  14848,      1,      1 ]    61.62 MiB
3.49.161.104 I llama_model_loader: - tensor split  0:         blk.49.attn_sinks.weight f32      [     64,      1,      1,      1 ]     0.00 MiB
3.49.161.105 I llama_model_loader: - tensor split  0:           blk.49.ffn_down.weight q8_0     [  16384,   4096,      1,      1 ]    68.00 MiB
3.49.161.106 I llama_model_loader: - tensor split  0:           blk.49.ffn_gate.weight q8_0     [   4096,  16384,      1,      1 ]    68.00 MiB
3.49.161.106 I llama_model_loader: - tensor split  0:           blk.49.ffn_norm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.107 I llama_model_loader: - tensor split  0:             blk.49.ffn_up.weight q8_0     [   4096,  16384,      1,      1 ]    68.00 MiB
3.49.161.108 I llama_model_loader: - tensor split  0:  blk.49.layer_output_norm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.109 I llama_model_loader: - tensor split  0:      blk.49.nextn.eh_proj.weight q8_0     [   8192,   4096,      1,      1 ]    34.00 MiB
3.49.161.109 I llama_model_loader: - tensor split  0:        blk.49.nextn.enorm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.110 I llama_model_loader: - tensor split  0:        blk.49.nextn.hnorm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.111 I llama_model_loader: - tensor split  0:          blk.50.attn_norm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.111 I llama_model_loader: - tensor split  0:        blk.50.attn_output.weight q8_0     [   8192,   4096,      1,      1 ]    34.00 MiB
3.49.161.112 I llama_model_loader: - tensor split  0:           blk.50.attn_qkv.weight q8_0     [   4096,  14848,      1,      1 ]    61.62 MiB
3.49.161.113 I llama_model_loader: - tensor split  0:         blk.50.attn_sinks.weight f32      [     64,      1,      1,      1 ]     0.00 MiB
3.49.161.114 I llama_model_loader: - tensor split  0:           blk.50.ffn_down.weight q8_0     [  16384,   4096,      1,      1 ]    68.00 MiB
3.49.161.114 I llama_model_loader: - tensor split  0:           blk.50.ffn_gate.weight q8_0     [   4096,  16384,      1,      1 ]    68.00 MiB
3.49.161.115 I llama_model_loader: - tensor split  0:           blk.50.ffn_norm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.116 I llama_model_loader: - tensor split  0:             blk.50.ffn_up.weight q8_0     [   4096,  16384,      1,      1 ]    68.00 MiB
3.49.161.116 I llama_model_loader: - tensor split  0:  blk.50.layer_output_norm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.117 I llama_model_loader: - tensor split  0:      blk.50.nextn.eh_proj.weight q8_0     [   8192,   4096,      1,      1 ]    34.00 MiB
3.49.161.118 I llama_model_loader: - tensor split  0:        blk.50.nextn.enorm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB
3.49.161.118 I llama_model_loader: - tensor split  0:        blk.50.nextn.hnorm.weight f32      [   4096,      1,      1,      1 ]     0.02 MiB

Used models: https://huggingface.co/ggml-org/MiMo-V2.6-Flash-RL-GGUF

@CISC

CISC commented Sep 24, 2026

Copy link
Copy Markdown
Member

The tensor split over RPC seems to work now. However the MTP conversion is still not correct:

3.49.270.506 I load_tensors: loading model tensors, this can take a while... (load_mode = dio)
3.49.291.028 E llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' not found
3.49.291.039 E llama_model_load_from_file_impl: failed to load model
3.49.291.040 E common_speculative_init_result: failed to load draft model, '/home/ggml/.cache/huggingface/hub/models--ggml-org--MiMo-V2.6-Flash-RL-GGUF/snapshots/cac6e14e8c2063aba38f487c2b041d9a5a22c7b6/mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf'
3.49.291.046 E srv    load_model: failed to load draft model, '/home/ggml/.cache/huggingface/hub/models--ggml-org--MiMo-V2.6-Flash-RL-GGUF/snapshots/cac6e14e8c2063aba38f487c2b041d9a5a22c7b6/mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf'
3.49.291.053 I srv    operator(): operator(): cleaning up before exit...
3.49.292.853 E srv  llama_server: exiting due to model loading error

It looks like mtp-only loading is missing/incorrect?

For some reason the converted MTP sidecar has block indices starting from 48, instead of 0

This is normal, the sidecar's blocks should complement main model blocks.

@am17an

am17an commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

Yeah, just needed 6e0962b

ggerganov
ggerganov approved these changes Sep 24, 2026
View reviewed changes

@ggerganov ggerganov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirm it is working. Thanks

@github-actions github-actions Bot added the model Model specific label Sep 24, 2026
@am17an
am17an merged commit f805c57 into ggml-org:master Sep 25, 2026
15 checks passed
@am17an
am17an deleted the aman/fix-mimo-sm-tensor branch September 25, 2026 09:52
adromir pushed a commit to adromir/llama-cpp-turboquant that referenced this pull request Sep 25, 2026
@am17an Adromir
llama : fix tensor split for fused qkv with uneven K/V head sizes (gg…
a357099 
…ml-org#29294)

* llama : fix tensor split for fused qkv with uneven K/V head sizes

Assisted-by: Qwen3.8-27B

* fix v granularity

* convert: fix mtp conversion

* convert: add support for mtp flags

* fix loader
feal87 added a commit to feal87/myllama.cpp that referenced this pull request Sep 25, 2026
@feal87
Merge upstream/master into master
a6d9d7a 
Merge upstream commits:
  - llama: add llama_prec_policy + model-driven W4A4 path (ggml-org#24364)
  - llama: fix tensor split for fused qkv with uneven K/V head sizes (ggml-org#29294)
  - metal: split fa kernels into per-dtype libraries (ggml-org#29329)
  - metal: FWHT kernels for block widths above 512 (ggml-org#29095)
  - CUDA: fuse RMS_NORM + SCALE into one kernel (ggml-org#29393)
  - common: extract shared unicode path/string helpers (ggml-org#29415)
  - common,rpc: simplify fs_create_directory_with_parents() (ggml-org#29432)
  - rpc: include nb in the get_alloc_size cache key (ggml-org#29283)
  - [SYCL] support sparse FA (ggml-org#28796)
  - musa: fix PH1 operator failures and build issues (ggml-org#29193)
  - HIP: bump HIP_VERSION required for fp8 (ggml-org#29231)
  - opencl: add q5_k bin kernel (ggml-org#29401)
  - hexagon: add q5_k quant type support (ggml-org#29123)
  - hexagon: use DMA for contiguous dim1 CONCAT (ggml-org#29404)
  - mtmd: fix mel preprocessor in LFM2 audio (ggml-org#29403)
  - vulkan: fix legacy GLSLC without cooperativeMatrix (ggml-org#29409)
  - gguf-py: ByteLevel processing defaults bos/eos to False (ggml-org#29422)
  - gguf-py: TemplateProcessing has final word on add_special_token (ggml-org#29417)

Assisted-by: Pi
sky-mighty pushed a commit to sky-mighty/llama.cpp that referenced this pull request Sep 26, 2026
@am17an @sky-mighty
llama : fix tensor split for fused qkv with uneven K/V head sizes (gg…
709f597 
…ml-org#29294)

* llama : fix tensor split for fused qkv with uneven K/V head sizes

Assisted-by: Qwen3.8-27B

* fix v granularity

* convert: fix mtp conversion

* convert: add support for mtp flags

* fix loader
@ggerganov ggerganov mentioned this pull request Oct 5, 2026
Merged
frostyautumnleaf pushed a commit to frostyautumnleaf/llama.cpp that referenced this pull request Oct 5, 2026
@am17an
llama : fix tensor split for fused qkv with uneven K/V head sizes (gg…
d2397fe 
…ml-org#29294)

* llama : fix tensor split for fused qkv with uneven K/V head sizes

Assisted-by: Qwen3.8-27B

* fix v granularity

* convert: fix mtp conversion

* convert: add support for mtp flags

* fix loader
edwardyoon pushed a commit to edwardyoon/focus-llama that referenced this pull request Oct 7, 2026
@am17an edwardyoon
llama : fix tensor split for fused qkv with uneven K/V head sizes (gg…
b98dbfc 
…ml-org#29294)

* llama : fix tensor split for fused qkv with uneven K/V head sizes

Assisted-by: Qwen3.8-27B

* fix v granularity

* convert: fix mtp conversion

* convert: add support for mtp flags

* fix loader

(cherry picked from commit f805c57)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CISC CISC CISC approved these changes
@ggerganov ggerganov ggerganov approved these changes

Assignees

No one assigned

Labels

conversion model Model specific

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@am17an @ggerganov @CISC