Eval bug: MTP breaks multimodality in StepFun Step-3.7-Flash "inconsistent sequence positions", "failed to process speculative batch"

[vbooka1]

Name and Version

build 9820 https://github.com/ggml-org/llama.cpp/releases/tag/b9820

Also tried older releases b9736 and b9747 - shortly before and after release number b9745 ( https://github.com/ggml-org/llama.cpp/releases/tag/b9745 #24340 ) where Step3.7 MTP support was added.

Operating systems

Linux

GGML backends

CUDA

Hardware

AMD EPYC 4th gen, Nvidia RTX PRO 6000

Models

Step 3.7 Flash

Problem description & steps to reproduce

Enabling MTP breaks multimodality, specifically image recognition, "the tokens of sequence 0 in the input batch have inconsistent sequence positions", "error: decode() failed: failed to process speculative batch".
Image recognition works well if MTP is not enabled.

./llama-server -m Step-3.7-flash-Q8_0-00001-of-00005.gguf --flash-attn on --mmproj Step-3.7-flash--mmproj-f16.gguf --no-mmproj-offload --model-draft Step-3.7-Flash-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 2  --jinja --reasoning-budget 1 --reasoning-budget-message "[reasoning disabled, will answer now]" --ctx-size 32768 ... # other parameters likely are not relevant

Model: https://huggingface.co/stepfun-ai/Step-3.7-Flash-GGUF/tree/main/Q8_0
MMPROJ: https://huggingface.co/stepfun-ai/Step-3.7-Flash-GGUF/blob/main/mmproj-step3.7-flash-f16.gguf

MTP model is from a 3rd party: https://huggingface.co/notSnix/Step-3.7-Flash-MTP-Draft-GGUF/blob/main/Step-3.7-Flash-MTP-Q8_0.gguf
I have just noticed that StepFun have released their own draft model but I did not try to use it yet: https://huggingface.co/stepfun-ai/Step-3.7-Flash-GGUF/blob/main/Step3.7-flash-mtp-Q8_0.gguf
Will try and update this issue report later.

First Bad Commit

No response

Relevant log output

Logs

1.08.390.801 I srv  llama_server: server is listening on http://.....
1.08.390.803 I srv  update_slots: all slots are idle
1.27.475.149 I srv    operator(): Chat format: peg-native
1.27.476.797 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
1.27.476.814 I srv  get_availabl: updating prompt cache
1.27.476.830 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
1.27.476.843 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 32768 tokens, 8589934592 est)
1.27.476.853 I srv  get_availabl: prompt cache update took 0.03 ms
1.27.477.499 I reasoning-budget: activated, budget=1 tokens
1.27.477.502 I reasoning-budget: budget exhausted, forcing end sequence
1.27.477.515 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
1.27.477.906 I srv  stream_sessi: stream_session_attach_pipe: conv_id=0wrsw3luisxh (empty=0)
1.27.611.829 I slot process_mtmd: id  0 | task 0 | encoding mtmd batch from idx = 5, n_chunks = 1
1.34.109.887 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 4
 - the tokens for sequence 0 in the input batch have a starting position of Y = 86
 it is required that the sequence positions remain consecutive: Y = X + 1
1.34.109.894 E decode: failed to initialize batch
1.34.109.894 E llama_decode: failed to decode, ret = -1
1.34.109.895 E process: llama_decode(ctx_dft) head=0 failed rc=-1 (pos=86)
1.34.109.897 E srv        decode: failed to process speculative batch
1.34.112.550 E srv  update_slots: decode() failed: failed to process speculative batch
1.34.112.559 E srv    send_error: task id = 0, error: decode() failed: failed to process speculative batch
1.34.112.565 I slot      release: id  0 | task 0 | stop processing: n_tokens = 88, truncated = 0
1.34.112.584 I srv  update_slots: all slots are idle
1.34.112.596 W srv          stop: cancel task, id_task = 0
1.34.112.639 I srv  update_slots: all slots are idle

[vbooka1]

I have just noticed that StepFun have released their own draft model but I did not try to use it yet: https://huggingface.co/stepfun-ai/Step-3.7-Flash-GGUF/blob/main/Step3.7-flash-mtp-Q8_0.gguf
Will try and update this issue report later.

Same behavior with original GGUF, original MMPROJ and original MTP draft model.

1.56.502.776 E init: the tokens of sequence 0 in the input batch have inconsistent sequence positions:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 4
 - the tokens for sequence 0 in the input batch have a starting position of Y = 86
 it is required that the sequence positions remain consecutive: Y = X + 1
1.56.502.781 E decode: failed to initialize batch
1.56.502.781 E llama_decode: failed to decode, ret = -1
1.56.502.783 E spec      process: llama_decode(ctx_dft) head=0 failed rc=-1 (pos=86)
1.56.502.784 E srv        decode: failed to process speculative batch
1.56.505.075 E srv  update_slots: decode() failed: failed to process speculative batch
1.56.505.087 E srv    send_error: task id = 0, error: decode() failed: failed to process speculative batch

Tested with build version b9840

[ServeurpersoCom]

Reproduced on my PRO 6000 also

[vbooka1]

Just a follow-up: pull request #28715 (build https://github.com/ggml-org/llama.cpp/releases/tag/b10906 ) did not fix this issue.