Eval bug: forcing full prompt re-processing due to lack of cache data unsloth/Qwen3.5-2B-MTP-GGUF:Q4_K_XL

Name and Version

./llama-server --version
version: 9642 (3686e9d)
built with GNU 11.4.0 for Linux x86_64

./llama-server -hf unsloth/Qwen3.5-2B-MTP-GGUF:Q4_K_XL --no-mmproj --reasoning off

full prompt re-processing with every request, slow for bigger prompts

2.00.280.993 I srv  params_from_: Chat format: peg-native
2.00.281.274 I slot get_availabl: id  3 | task -1 | selected slot by LCP similarity, sim_best = 0.500 (> 0.100 thold), f_keep = 0.056
2.00.281.276 I srv  get_availabl: updating prompt cache
2.00.281.784 W srv   prompt_save:  - saving prompt with length 178, total state size = 21.356 MiB (draft: 0.000 MiB)
2.00.310.595 I srv          load:  - looking for better prompt, base f_keep = 0.056, sim = 0.500
2.00.310.604 I srv        update:  - cache state: 7 prompts, 285.314 MiB (limits: 8192.000 MiB, 262144 tokens, 262144 est)
2.00.310.605 I srv        update:    - prompt 0x619d01a2e150:     478 tokens, checkpoints:  1,    44.143 MiB
2.00.310.606 I srv        update:    - prompt 0x619d090dfe60:      51 tokens, checkpoints:  1,    39.131 MiB
2.00.310.606 I srv        update:    - prompt 0x619d01a2a110:     119 tokens, checkpoints:  1,    39.929 MiB
2.00.310.606 I srv        update:    - prompt 0x619d01a29900:     192 tokens, checkpoints:  1,    40.786 MiB
2.00.310.607 I srv        update:    - prompt 0x619d01a28e30:     104 tokens, checkpoints:  1,    39.753 MiB
2.00.310.607 I srv        update:    - prompt 0x619d01a27380:     206 tokens, checkpoints:  1,    40.950 MiB
2.00.310.607 I srv        update:    - prompt 0x619d0708d030:     178 tokens, checkpoints:  1,    40.622 MiB
2.00.310.609 I srv  get_availabl: prompt cache update took 29.33 ms
2.00.310.681 I reasoning-budget: activated, budget=2147483647 tokens
2.00.310.682 I reasoning-budget: deactivated (natural end)
2.00.310.715 I slot launch_slot_: id  3 | task 1207 | processing task, is_child = 0
2.00.310.717 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache
2.00.310.718 I slot prompt_clear: id  0 | task -1 | clearing prompt with 0 tokens
2.00.310.842 I slot process_sing: id  1 | task -1 | saving idle slot to prompt cache
2.00.310.844 I slot prompt_clear: id  1 | task -1 | clearing prompt with 0 tokens
2.00.310.959 I slot process_sing: id  2 | task -1 | saving idle slot to prompt cache
2.00.310.960 I slot prompt_clear: id  2 | task -1 | clearing prompt with 0 tokens
2.00.311.085 I slot update_slots: id  3 | task 1207 | Checking checkpoint with [16, 16] against 10...
2.00.311.087 W slot update_slots: id  3 | task 1207 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
2.00.311.088 W slot update_slots: id  3 | task 1207 | erased invalidated context checkpoint (pos_min = 16, pos_max = 16, n_tokens = 17, n_swa = 0, pos_next = 0, size = 19.266 MiB)
2.00.403.309 I slot create_check: id  3 | task 1207 | created context checkpoint 1 of 32 (pos_min = 15, pos_max = 15, n_tokens = 16, size = 19.266 MiB)
2.02.534.659 I slot print_timing: id  3 | task 1207 | n_decoded =    100, tg =  47.51 t/s
2.04.199.532 I slot print_timing: id  3 | task 1207 | prompt eval time =     118.88 ms /    20 tokens (    5.94 ms per token,   168.24 tokens per second)
2.04.199.536 I slot print_timing: id  3 | task 1207 |        eval time =    3769.54 ms /   177 tokens (   21.30 ms per token,    46.96 tokens per second)
2.04.199.536 I slot print_timing: id  3 | task 1207 |       total time =    3888.42 ms /   197 tokens
2.04.199.537 I slot print_timing: id  3 | task 1207 |    graphs reused =       1357
2.04.199.559 I slot      release: id  3 | task 1207 | stop processing: n_tokens = 196, truncated = 0
2.04.199.577 I srv  update_slots: all slots are idle


2.10.519.351 I srv  params_from_: Chat format: peg-native
2.10.519.512 I slot get_availabl: id  3 | task -1 | selected slot by LCP similarity, sim_best = 0.524 (> 0.100 thold), f_keep = 0.056
2.10.519.514 I srv  get_availabl: updating prompt cache
2.10.519.705 W srv   prompt_save:  - saving prompt with length 196, total state size = 21.567 MiB (draft: 0.000 MiB)
2.10.547.211 I srv          load:  - looking for better prompt, base f_keep = 0.056, sim = 0.524
2.10.547.222 I srv        update:  - cache state: 8 prompts, 326.147 MiB (limits: 8192.000 MiB, 262144 tokens, 262144 est)
2.10.547.222 I srv        update:    - prompt 0x619d01a2e150:     478 tokens, checkpoints:  1,    44.143 MiB
2.10.547.223 I srv        update:    - prompt 0x619d090dfe60:      51 tokens, checkpoints:  1,    39.131 MiB
2.10.547.223 I srv        update:    - prompt 0x619d01a2a110:     119 tokens, checkpoints:  1,    39.929 MiB
2.10.547.224 I srv        update:    - prompt 0x619d01a29900:     192 tokens, checkpoints:  1,    40.786 MiB
2.10.547.224 I srv        update:    - prompt 0x619d01a28e30:     104 tokens, checkpoints:  1,    39.753 MiB
2.10.547.224 I srv        update:    - prompt 0x619d01a27380:     206 tokens, checkpoints:  1,    40.950 MiB
2.10.547.225 I srv        update:    - prompt 0x619d0708d030:     178 tokens, checkpoints:  1,    40.622 MiB
2.10.547.225 I srv        update:    - prompt 0x619d0916baf0:     196 tokens, checkpoints:  1,    40.833 MiB
2.10.547.226 I srv  get_availabl: prompt cache update took 27.71 ms
2.10.547.302 I reasoning-budget: activated, budget=2147483647 tokens
2.10.547.302 I reasoning-budget: deactivated (natural end)
2.10.547.338 I slot launch_slot_: id  3 | task 1386 | processing task, is_child = 0
2.10.547.338 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache
2.10.547.340 I slot prompt_clear: id  0 | task -1 | clearing prompt with 0 tokens
2.10.547.499 I slot process_sing: id  1 | task -1 | saving idle slot to prompt cache
2.10.547.500 I slot prompt_clear: id  1 | task -1 | clearing prompt with 0 tokens
2.10.547.611 I slot process_sing: id  2 | task -1 | saving idle slot to prompt cache
2.10.547.613 I slot prompt_clear: id  2 | task -1 | clearing prompt with 0 tokens
2.10.547.736 I slot update_slots: id  3 | task 1386 | Checking checkpoint with [15, 15] against 11...
2.10.547.737 W slot update_slots: id  3 | task 1386 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
2.10.547.738 W slot update_slots: id  3 | task 1386 | erased invalidated context checkpoint (pos_min = 15, pos_max = 15, n_tokens = 16, n_swa = 0, pos_next = 0, size = 19.266 MiB)
2.10.640.952 I slot create_check: id  3 | task 1386 | created context checkpoint 1 of 32 (pos_min = 16, pos_max = 16, n_tokens = 17, size = 19.266 MiB)
2.11.371.039 I slot print_timing: id  3 | task 1386 | prompt eval time =     119.08 ms /    21 tokens (    5.67 ms per token,   176.35 tokens per second)
2.11.371.043 I slot print_timing: id  3 | task 1386 |        eval time =     704.20 ms /    33 tokens (   21.34 ms per token,    46.86 tokens per second)
2.11.371.044 I slot print_timing: id  3 | task 1386 |       total time =     823.28 ms /    54 tokens
2.11.371.045 I slot print_timing: id  3 | task 1386 |    graphs reused =       1388
2.11.371.066 I slot      release: id  3 | task 1386 | stop processing: n_tokens = 53, truncated = 0
2.11.371.083 I srv  update_slots: all slots are idle

ref #13194

Operating systems

Linux

GGML backends

Vulkan

Hardware

Ryzen 7840U

Models

unsloth/Qwen3.5-2B-MTP-GGUF:Q4_K_XL

Problem description & steps to reproduce

send a prompt

send the same prompt with 1 extra letter

First Bad Commit

No response

Relevant log output

Logs

2.10.547.737 W slot update_slots: id  3 | task 1386 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
2.10.547.738 W slot update_slots: id  3 | task 1386 | erased invalidated context checkpoint (pos_min = 15, pos_max = 15, n_tokens = 16, n_swa = 0, pos_next = 0, size = 19.266 MiB)

same issue with unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL

no issue with unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL, unsloth/gemma-4-E4B-it-qat-GGUF:UD-Q4_K_XL

You are sending 20 tokens if I am correct. Could you clarify the usecase? Is this the first prompt and the second prompt in the chat?

here is the full case:

./llama-server -hf unsloth/Qwen3.5-2B-GGUF:Q4_K_XL --no-mmproj --reasoning off

curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? A"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? AB"}]}' | jq .

Logs containing "forcing full prompt re-processing due to lack of cache data":

logs.txt

But this is the first prompt and you are changing it, there is nothing to cache.
What is the expected behaviour (what happens when you observe "no issue")?

The "forcing full prompt re-processing" error comes on the second and third prompt, all 3 have a common prefix.

When you run the same test with 3 prompts having a common prefix with Gemma4 E4B, you see in the logs, that there is no "full prompt re-processing" and prompt_save / restored context checkpoint work correctly.

./llama-server -hf unsloth/gemma-4-E4B-it-qat-GGUF:UD-Q4_K_XL --no-mmproj --reasoning off

curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? A"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? AB"}]}' | jq .

Logs NOT containing "forcing full prompt re-processing due to lack of cache data" with Gemma4:

logs2.txt

For bigger prompts, the "full prompt re-processing" error hurts the performance badly.

@pwilkin

cc @pwilkin

here is a screenshot of the logs side by side using qwen3.5 on the left and gemma4 on the right running the same 3 prompts:

The difference is “against 9” vs “against 0”. “9” leads to invalidation, while “0” is empty.

Actually, the right side looks strange, because a checkpoint should not be created at 0.

see #22746

Same issue with Qwen3.6-35b-a3b-mtp and llama.cpp b9694

More testing: issue happens with Vulkan and Cpu-only

When sending messages with same prefix, "forcing full prompt re-processing" happens:

["When was Beethoven born?"]
["When was Beethoven born? A"]
["When was Beethoven born? AB"]

When sending multiple messages, "forcing full prompt re-processing" does not happen

["When was Beethoven born?"]
["When was Beethoven born?", "A"]
["When was Beethoven born?", "A", "B"]

To reproduce:
/llama-server -hf unsloth/Qwen3.5-2B-GGUF:Q4_K_XL --no-mmproj --reasoning off

Problem happens:
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? A"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born? AB"}]}' | jq .

Problem does not happen:
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}, {"role": "user", "content": "ABC"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}, {"role": "user", "content": "ABC"}, {"role": "user", "content": "DEF"}]}' | jq .
curl -s http://127.0.0.1:8080/v1/chat/completions -d '{"stream": false, "messages": [{"role": "user", "content": "When was Beethoven born?"}, {"role": "user", "content": "ABC"}, {"role": "user", "content": "DEF"}, {"role": "user", "content": "GHI"}]}' | jq .

As I wrote this is expected behavior, there is no bug.

And why does it happen with Qwen3.5/3.6 and not with Gemma4?

Something is wrong with the Gemma log, because checkpoint shouldn't be created at zero. But I would need to reproduce/debug it to tell you more

In my tests with bigger prompts, "forcing full prompt re-processing" always causes 2-3s extra response time per request with Qwen3.5.
Gemma4 E4B is slower than Qwen3.5 for single prompt processing, but in total much faster for multiple prompts because it's not doing "forcing full prompt re-processing".

Basically having a common prefix like

prompt 1: ["When was Beethoven born? Datetime1"]
prompt 2: ["When was Beethoven born? Datetime2"]
prompt 3: ["When was Beethoven born? Datetime3"]

is also what ngxson mentioned: #23808 (comment)

I have been experimenting with writing stories with AI and using AI to write a .cpp program that directly submits my stories so I can generate very long stories without being at the computer. Anyway tonight I had a regression where I started having that issue. It was because it started submitting the new parts of the story as Assistant, where previously it was all submitted User. Now, while the order of the data stayed the same, evidently it is more picky about Assistant data than User data in some way, because when I went back to it all being one big User submission the kv cache was mostly valid and everything was fine.

I am guessing that the json keys are getting out of order, either on my end or on llama.cpp's end. orrr also on clients end like sillytavern. But in the interests of sharing information, there ya go.

99% valid
step_payload["messages"].push_back({{"role", "user"}, {"content", total_ai_context + "\n\n" + directive}});

0% valid
step_payload["messages"].push_back({{"role", "assistant"}, {"content", total_ai_context}});
step_payload["messages"].push_back({{"role", "user"}, {"content", directive });

@jacekpoplawski

Dug into this a bit more since the "expected behavior" vs "bug" question never really got resolved above, ran a few controlled tests on unsloth/Qwen3.5-2B-MTP-GGUF:Q4_K_M (build e3546c7, Apple M1/Metal, llama-server --port 8097 --parallel 1 -lv 4 --reasoning off) to check each explanation raised in this thread. None of them held up.

First tried --checkpoint-min-step 0 (the fix suggested for a similar complaint in #23808), set to its most permissive value. It still breaks:

selected slot by LCP similarity, sim_best = 0.545
checking checkpoint with [17, 17] against 12...
forcing full prompt re-processing due to lack of cache data

Ran the identical harness against a normal, non-hybrid model
(Qwen3-4B-Instruct-2507) as a control:

request 1: 14 tokens (cold)
request 2: 6 tokens <- correctly reused
request 3: 6 tokens <- correctly reused

No "forcing full re-processing" anywhere. So the caching/LCP logic itself is fine ---> this is isolated to the hybrid-memory checkpoint path specifically.

A real multi-turn conversation (actual assistant reply in history, not just edited text) hits it too — turn 2 of a genuine 2-turn exchange processed 47 tokens fresh, basically the whole thing again. And a bigger prompt with a genuinely strong match doesn't help either:

sim_best = 0.831, f_keep = 0.721
checking checkpoint with [54, 54] against 49...
forcing full prompt re-processing due to lack of cache data

83% match, thrown away entirely.

@jacekpoplawski — I also replicated your "problem does not happen" example (stacked user messages, no assistant turn) and got inconsistent results: 2
stacked messages triggered the full reprocess, 3 stacked messages didn't. Same pattern, one more message, different outcome — depending on what checkpoint happened to still be around at that moment.

That inconsistency is what convinced me this isn't intended behavior. A real architectural limit (recurrent state can't rewind) should fail the same way every time for the same input shape; getting lucky sometimes points at a gap in the checkpoint-selection logic itself (find_if over slot.prompt.checkpoints, around the "checking checkpoint with [...] against ..." trace line), not a fundamental constraint of the memory type.

Same log signature as #22746 and probably #25259 — all hybrid/recurrent
models.

@ggerganov — since this backs right up to TAG_CHECKPOINTS_FIX_POS_MIN and your regression note on #23981/#24411: is there a preferred direction for the recurrent-memory case, or should I just start poking at checkpoint placement density for these models?

issue-24714-test-logs.md

I have attached the tests that I run with the logs and my findings in the code can you verify those also

This issue was closed because it has been inactive for 14 days since being marked as stale.

Navigation Menu

Repository navigation

Eval bug: forcing full prompt re-processing due to lack of cache data unsloth/Qwen3.5-2B-MTP-GGUF:Q4_K_XL #24714

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Eval bug: forcing full prompt re-processing due to lack of cache data unsloth/Qwen3.5-2B-MTP-GGUF:Q4_K_XL #24714

Description

Name and Version

Operating systems

GGML backends

Hardware

Models

Problem description & steps to reproduce

First Bad Commit

Relevant log output

Activity

thomas-0816 commented on Jun 17, 2026

jacekpoplawski commented on Jun 17, 2026

thomas-0816 commented on Jun 17, 2026

jacekpoplawski commented on Jun 17, 2026

thomas-0816 commented on Jun 17, 2026

thomas-0816 commented on Jun 17, 2026

thomas-0816 commented on Jun 17, 2026

jacekpoplawski commented on Jun 17, 2026

thomas-0816 commented on Jun 17, 2026

AbelVM commented on Jun 18, 2026

thomas-0816 commented on Jun 18, 2026

jacekpoplawski commented on Jun 18, 2026

thomas-0816 commented on Jun 18, 2026

jacekpoplawski commented on Jun 18, 2026

thomas-0816 commented on Jun 18, 2026

thomas-0816 commented on Jun 18, 2026

GoldenWH commented on Jun 29, 2026

anant-lad commented on Jul 12, 2026

github-actions commented on Aug 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions