feat(server): make the default serve config actually reuse prefix KV and validate footprint end-to-end #228

Measurement context

Found during a cross-version server memory-footprint measurement (v0.1.4 vs main, 2026-06-12) using scripts/bench_memory_footprint.py (commit 18c88c1). Method: one fresh mlxcel serve process per (config, scenario), fixed HTTP workloads at temperature 0, sampling the process phys-footprint via /usr/bin/footprint at 2.5 s (RSS does not see MLX Metal buffers at all). Model: llama-3.2-1b-4bit on M1 Ultra, --max-batch-size 8. Scenarios: idle, shared_prefix_burst (8 concurrent requests sharing a ~3.7k-token system prompt), seqshare (same prefix, 8 sequential requests), multiturn (one conversation, 8 turns), churn (32 distinct prompts). Run-to-run noise: idle/multiturn within 0.5 percent, burst/seqshare/churn peaks within 10 to 25 percent.

Problem

With default flags the prompt-cache lookup requires the stored entry (prompt plus generated tokens) to be fully contained in the new request (the legacy whole-prefix-contained check in src/server/prompt_cache/store.rs around line 556, active whenever APC is off). Entries always carry a generated tail, so divergent siblings structurally miss, and multiturn continuation only hits when the echoed assistant text retokenizes byte-identically (a mid-word max_tokens cut breaks it; observed in measurement).

Net measured effect of the last two weeks on default config: only multiturn improved (peak 2104 to 1485 MiB vs v0.1.4) while shared-prefix scenarios regressed 68 to 70 percent in peak footprint. Every designed win currently requires --apc-enabled --decode-storage-backend dense (seqshare: peak 1506 vs 2774 MiB, settle 988 vs 2255 MiB, and 1.36x faster than the default config).

Fix

After the partial-adoption and non-consuming-adoption issues land: enable APC by default (apc_enabled = true), then re-validate end to end: full matrix of scripts/bench_memory_footprint.py, the byte-parity suites, and a docs update (docs/turbo-kv-cache.md, docs/CONTINUOUS_BATCHING.md) recording the measured numbers and the new default.

Acceptance criteria

• Default mlxcel serve shows cached>0 on burst, seqshare, and multiturn.
• Shared-prefix scenarios' footprint at or below the v0.1.4 baseline.
• No throughput regression (scenario wall times within run noise).
• Docs updated with the measured before/after table.

Depends on the partial-adoption fix and the non-consuming adoption issue.

Related

Filed together from the same measurement: #224, #225, #226, #227.