llama-server: first requests of a new conversation re-prefill full prompt instead of reusing cached prefix #29

Problem

When driving llama.cpp from an agent harness (Pithagoras / pi, plain OpenAI-compatible chat completions), the first two requests of every conversation are fully re-prefilled, even though each request is a strict extension of the previous one. From request 3 on, prefix caching works normally — only the start of the conversation is broken.

Observed flow

1. Request 1: [system, user] → full prefill (expected — nothing cached yet).
2. Assistant replies, tool call follows.
3. Request 2: [system, user, assistant, tool_call] → full re-prefill again instead of reusing the shared prefix from request 1.
4. Request 3+ ([system, user, assistant, ..., tool, ...]) → cache is reused correctly (no re-prefill).

So: the initial system prompt should be available as a cached prefix by the time request 2 arrives, but it isn't — after that point everything behaves as expected.

Expected behavior

Since request 2 shares its entire beginning with request 1, the slot/context cache should hold the KV from request 1 and only prefill the delta (n_cached > 0 for the prefix). This works steady-state; only requests 1→2 lose the cached context.

Environment

• Model: Qwen3.8-27b (ctx 262144, large RAM KV cache)
• Served through llama-swap → matrix-router gateway (evict-cost based eviction with a 5-minute evict lock; relevant if slot reuse interacts with routing/eviction)
• llama.cpp: current fork main (upstream base + native kernel upgrades + cache prompt hierarchy)

To fill in

• Exact llama-server launch flags (--ctx-size, --cache-type-k/v, --n-cpu-moe, parallel slot settings, etc.)
• Server log lines / n_cached values for requests 1 and 2 of the conversation
• Whether it reproduces with a raw curl sequence (no gateway in front)

Hypotheses to check

• Slot/context gets evicted or reloaded between request 1 and 2 (e.g. due to the long generation of request 1's response exceeding some retention window, or slot release not keeping the prefix).
• Cache-prompt-hierarchy feature dropping level-1 (system) prefixes on the first reuse.
• Interaction with MoE expert offload state after the first request completes.