Repository navigation
Eval bug: "Chunk not found" crash with hybrid attention models (Qwen3.5) under parallel load #20222
Description
Name and Version
Native build from commit 2850bc6 (build 8212) and commit c5a7788 (latest master as of 2026-03-07). Also reproduced with Docker image ghcr.io/ggml-org/llama.cpp:server-cuda (build 8212).
Operating systems
Linux (Debian 12, LXC container)
Which llama.cpp modules do you know to be affected?
llama-server
Command line
llama-server \
--model /models/Qwen3.5-35B-A3B-UD-Q6_K_XL.gguf \
--mmproj /models/mmproj-BF16.gguf \
--host 0.0.0.0 --port 8000 \
--ctx-size 196608 --parallel 6 \
--tensor-split 24,12 \
--cache-type-k q4_0 --cache-type-v q4_0 \
--batch-size 2048 --ubatch-size 512 \
-fa 1
Problem description & steps to reproduce
Qwen3.5-35B-A3B crashes with std::runtime_error: Chunk not found when running with multiple parallel slots (tested with 3 and 6). The crash occurs during context checkpoint restoration under parallel load.
Qwen3.5 uses hybrid attention (sliding window + DeltaNet/recurrent layers), which appears to interact badly with the checkpoint restore path when multiple slots are active simultaneously.
Steps to reproduce
- Start llama-server with Qwen3.5-35B-A3B and
--parallel 3or higher - Send 3+ concurrent
/v1/chat/completionsrequests - Wait for one request to complete and a new request to arrive on the same slot
- Server crashes when it tries to restore the context checkpoint for the reused slot
Crash log
slot update_slots: id 2 | task 47 | restored context checkpoint (pos_min = 1460, pos_max = 1460, size = 1.250 MiB)
terminate called after throwing an instance of 'std::runtime_error'
what(): Chunk not found
The crash always follows a "restored context checkpoint" log entry.
What I've tested
| Config | Result |
|---|---|
--parallel 6 (latest master c5a7788) |
Crashes within minutes |
--parallel 3 (latest master c5a7788) |
Crashes within minutes |
--parallel 6 (rollback to 2850bc6) |
Crashes within ~20 min |
--parallel 6 + --ctx-checkpoints 0 (2850bc6) |
Reduced but still crashes |
--parallel 6 + --ctx-checkpoints 0 --no-cache-prompt |
Stable (45+ min stress test, 0 crashes) |
--parallel 1 (any commit) |
Stable (no parallel slot reuse) |
Workaround
--ctx-checkpoints 0 --no-cache-prompt prevents the crash entirely but sacrifices prompt caching performance (~1s extra per request).
Related issues/PRs
- Misc. bug: Server: LFM2-VL crashes on checkpoint restoring with image input #16590 -- Same "Chunk not found" crash reported for LFM2-VL (another hybrid attention model), fixed by server : fix mtmd checkpoints #16591
- server : fix ctx checkpoint restore logic #19924 -- "fix ctx checkpoint restore logic" (merged Feb 26)
- Hybrid model cache: add
--checkpoint-every-nb#20087 -- "Checkpoint every n tokens" (merged Mar 6)
None of these fully resolve the crash for Qwen3.5 under parallel load.
Hardware
- 2x NVIDIA GPUs: RTX 3090 (24GB) + RTX 3060 (12GB), tensor-split 24,12
- CUDA 13.0, compute capability 8.6 (Ampere)
- 80GB system RAM
Note: This issue was filed with assistance from an AI agent based on hands-on testing and debugging.
Activity
ggerganov commentedon Mar 8, 2026
Can you confirm that #20232 fixes the problem?
SubAllStar commentedon Mar 8, 2026
Confirmed — PR #20232 fixes the crash.
Test setup:
--parallel 6 --ctx-size 196608 --cache-type-k q4_0 --cache-type-v q4_0 -fa 1Results:
--no-cache-promptand--ctx-checkpoints 0workarounds)Previously crashed within minutes under the same load. The fix resolves it completely.
jcartu commentedon Mar 10, 2026
The hybrid attention (sliding window + DeltaNet) in Qwen3.5 doesn't play well with context checkpointing when slots get reused. The DeltaNet recurrent state isn't properly saved/restored in the checkpoint path.
Your workaround (
--ctx-checkpoints 0 --no-cache-prompt) is the right call for now. In my setup I run Qwen3.5 Q8_0 with:Also FYI: speculative decoding doesn't work with Qwen3.5 in llama.cpp either — same DeltaNet recurrent state issue. Save yourself the debugging time.