Eval bug: Gemma 4 (26B-A4B MoE) crashes the server on the 2nd request with -np 4 on Vulkan (GGML_ASSERT "tensor not allocated"); -np 3 is fine, CPU backend is fine #29221

Hello,
Thank you for your amazing job.

Official prebuilt llama-b11065-bin-win-vulkan-x64 binaries.

Operating systems
Windows 11 (10.0.26200), x86_64

GGML backends
Vulkan

Hardware
AMD Ryzen AI 9 HX 370 + Radeon 890M iGPU (Strix Point, GFX115x, unified memory / UMA — 64 GB LPDDR5X-8000 shared CPU/GPU). Single GPU, no split mode.

Models
Main: unsloth/gemma-4-26B-A4B-it-qat-GGUF → gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf (arch gemma4, MoE 25.2B total / 3.8B active, 30 blocks = 25 SWA-1024 + 5 global layers, 2 KV heads × dim 512 on global layers, ctx train 262144)
mmproj-F16.gguf optional (crashes with and without it)
Also reproduced with a second gemma4-26B quant (llmfan46 ...-ultra-uncensored-heretic-Q4_K_M.gguf, same arch).

Problem description & steps to reproduce
With -np 4 (and -np 5), the server loads fine and the first request completes normally, but the second request aborts the whole server:

D:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:359: GGML_ASSERT(tensor->data != NULL && "tensor not allocated") failed
(process exits with status -1073740791 / 0xC0000409)

Minimal repro (no mmproj, no speculative decoding):

llama-server.exe -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf
-c 1048576 -np 4 -ngl 999 -fa on -ctk q8_0 -ctv q8_0 --port 9999
then two sequential POST /v1/chat/completions with short prompts (~25 tokens): request 1 → normal completion; request 2 → assert + server death.

What we ruled out (all still crash at -np 4):

speculative decoding: crashes with and without --spec-type draft-mtp + external mtp-gemma-4-26B-A4B-it.gguf (gemma4-assistant) drafter
prompt cache: crashes with --no-cache-prompt
batch sizes: crashes with -b 2048 -ub 1024
vision: crashes with and without --mmproj
multi-model router mode (llama-server serve --models-preset ...): also reproduces with plain single-instance llama-server (commands above)
What works:

Vulkan -np 1, -np 2, -np 3: fully stable (sequential + 3-concurrent requests, ~20 requests incl. prompt-cache reuse and an external MTP drafter with 0.52–0.66 acceptance). Only from -np 4 upward it crashes.
CPU backend, same version, same machine: -np 4 fully stable (4-concurrent
sequential, zero crash) → looks Vulkan-backend-specific, not a scheduler issue per se.
qwen35moe (Qwen3.6-35B-A3B) at -np 4 works fine on Vulkan on the same machine/build → gemma4-specific, not MoE-generic.
Diagnostic hint: in every crashing run, the crashing request had just been assigned to slot id 3 (LRU or LCP-similarity selection), whether that slot was fresh or had a reusable prefix. With -np 3 (slot ids 0–2) no crash was ever observed. This suggests something about the 4th slot's reserved buffers/graph for gemma4 on Vulkan, but we have no way to pin a request to a slot to confirm. Every crash we captured happened during the 2nd request's prompt processing, right after a slot get_availabl: ... selected slot ... f_keep = ... line.

Not a memory-size problem: at -np 4 the model + KV use ~25 GB of the 48.8 GB Vulkan device heap (device reports 46.3 GB free at init), and the same build at -np 5 also loads and answers before crashing, while the system has 24+ GB of free RAM throughout.

First Bad Commit
No response (official release binaries; not bisected)

Relevant log output
Load is clean, first request OK, second request dies:

console
0.10.824.972 I srv load_model: loaded multimodal model, '...\mmproj-F16.gguf' <- only in the mmproj variant
0.11.221.236 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'false'
0.11.239.674 I srv llama_server: model loaded
0.11.239.688 I srv llama_server: listening on http://127.0.0.1:9999
0.48.520.356 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.48.520.436 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
0.54.217.138 I slot print_timing: id 3 | task 0 | prompt eval time = 5074.18 ms / 30 tokens
0.54.217.149 I slot print_timing: id 3 | task 0 | eval time = 622.17 ms / 20 tokens
0.54.217.153 I slot print_timing: id 3 | task 0 | graphs reused = 19
0.54.217.212 I slot release: id 3 | task 0 | stop processing: n_tokens = 49, truncated = 0
0.54.234.189 I slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, f_sim_best = 0.467 (> 0.100 thold), f_keep = 0.286
D:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:359: GGML_ASSERT(tensor->data != NULL && "tensor not allocated") failed
(The assert line is the last line; no stack trace beyond it on Windows release binaries. Exit code -1073740791 / 0xC0000409.)

Related issues (possibly same root area — gemma4 shared-KV global layers vs backend schedulers)
#25522 — Gemma 4 crashes with MTP on the 2nd prompt (CUDA, multi-GPU tensor-split + MTP; still open, "racy rather than always the 2nd prompt"). Ours: single GPU, no MTP required, Vulkan-only.
#24492 — Gemma 4 31B MTP crashes on Vulkan at init (draft shared-KV cache_k_l58 pre-allocated tensor, NONE op). Ours: crashes at runtime without any MTP drafter.
#28917 — gemma4:26b concurrent decode loses EOS (same model, concurrency related, different symptom).
Workaround for our deployment: keep gemma4 at -np 3 on Vulkan (or use the CPU backend, which is stable at -np 4).