Repository navigation
perf(server): decode runs ~11% slower after a prompt-cache hit than after a cold prefill #730
Description
Context
While validating the #632 lookahead decode pipeline on GB10 (CUDA 13.0, branch perf/632-scheduler-async-eval-lookahead, 2026-07-10), repeated bench_serving_concurrency.py --concurrency 1 --prompt-tokens 512 --max-tokens 512 runs against one server instance (qwen2.5-0.5b-bf16) are bimodal on decode throughput depending only on whether the synthetic prompt hit the server prompt cache.
Evidence
Cache-miss runs (TTFT ~285-296 ms, full prefill): decode 182.4-185.0 tok/s. Cache-hit runs (TTFT 13-44 ms, prompt-cache: cached=448/461): decode 161.9-164.6 tok/s, i.e. ~11% slower decode for the identical request. The /metrics lookahead counter increments 509 per 512-token run in BOTH modes, so the #632 pipeline engages fully either way; the penalty is not pipeline disengagement. Reproduced 5 times (3+2 samples).
Suspects (unverified)
The restored/donated KV cache's internal layout after a prompt-cache hit (block rounding, fragmentation, or a dense-concat restore that grows differently during decode), per-step cache-append cost differing between a freshly prefilled cache and a restored one, or attention kernel behavior at slightly different internal cache capacities.
Suggested next step
Profile one hit run and one miss run with nsys (--cuda-graph-trace=node) and diff the per-step kernel composition and any reallocation/copy activity in the decode phase; check whether the restored cache path skips a capacity reservation the prefill path performs.
Activity
docs(server): document lookahead sync kill switch and acceptance data
perf(server): pipeline scheduler decode with lookahead async_eval (#729)
inureyes commentedon Jul 12, 2026
Not reproduced on Apple Silicon (metal)
I tried to reproduce the ~11% cache-hit decode penalty on metal before touching any code. It does not reproduce here on either decode backend: cache-hit decode stays within 0.5% of cache-miss decode with fully overlapping sample ranges. Details below so the finding is auditable, plus where the restore and prefill paths genuinely differ in the code.
Environment
metal,acceleraterelease build.mainate63dae0b1. PR perf(server): pipeline scheduler decode with lookahead async_eval #729 (1ad7aeedd, pipeline scheduler decode with lookaheadasync_eval) is an ancestor, so the perf(server): port lookahead async_eval pipelining into BatchScheduler decode #632 pipeline under test is present.qwen2.5-0.5b-bf16(same as the report), 471-token synthetic prompt,max_tokens=512,temperature=0(greedy), streaming/v1/chat/completions.Methodology
To isolate cache provenance from context length, I alternated MISS/HIT samples with identical geometry rather than restarting the server:
cached=0/471).cached=448..464/471, matching the report'scached=448/461).Because the prompt length is identical and greedy decoding makes the continuation identical, both requests decode the same 471 to ~983 token window doing the same work. The only variable is fresh-prefill vs restored cache. Decode throughput is computed client-side exactly as
bench_serving_concurrency.pydoes:(completion_tokens - 1) / (t_last_token - t_first_token). Seven clean sample pairs per run after one discarded warmup pair, 2 s cooldown between requests, machine otherwise idle. The/metricslookahead counter is sampled before and after every request.Results
Default backend (
--parallel 4, paged decode pool,max_batch_size=4, matching the GB10 default). Two independent runs, decode tok/s:HIT vs MISS median delta: +0.2% (run 1) and +0.1% (run 2), i.e. hit is marginally slower than the noise floor, and the sample ranges overlap completely. TTFT tracked hit vs miss as expected (paged run 2: MISS median 93 ms, HIT median 36 ms).
Dense backend (
--parallel 1,max_batch_size=1), to also exercise the "dense-concat restore" suspicion. One run, decode tok/s:HIT vs MISS median delta: +0.4%, ranges overlap. TTFT MISS median 103 ms vs HIT median 43 ms.
Lookahead and cache-hit confirmation
mlxcel_batch_decode_lookahead_steps_totalincremented by exactly 509 per 512-token request in every mode and run, MISS and HIT alike, reproducing the report's observation that the #632 pipeline engages either way. Theprompt-cache: request completedlog confirmedcached=0/471on every miss andcached=448..464/471on every hit.Where restore and prefill actually differ (and why metal shows no delta)
effective_decode_storage_backend(src/server/batch/scheduler.rs:156) setspaged_available = max_batch_size > 1 && supports_batching && supports_paged_decode_backend. With the default--parallel 4, the effective backend for this model is Paged, so the single-concurrency default runs on the paged pool, not the dense cache.adoptable = (matched_len.min(paged.seq_len()) / block_size) * block_size(src/server/batch/scheduler.rs:1624; the take-path repeats it at:1727). A fresh prefill fills the same pool in 32-token blocks. So on the default backend a restored cache and a fresh cache are both block-granular at the same block size, and there is no dense-concat / step-rounding geometry difference at all.matched_len(src/server/batch/scheduler.rs:1700), which is not 256-aligned. But the dense cache grows in a 256-token step (step: 256atsrc/lib/mlxcel-core/src/cache.rs:487,:535; grow atupdate_fp16,n_steps = (step + new_seq_len - 1) / step; buf_size = n_steps * step,cache.rs:866). Whenmatched_len < prompt_len(the partial-hit case here, 448/471), the divergent tail is re-prefilled immediately after adoption, and that re-prefill re-rounds the buffer back up to the sameceil(471/256)*256 = 512capacity a fresh prefill lands on. So the restored dense cache converges to the same geometry as a fresh one before decode begins. Only an exact full-prefix match (matched_len == seq_len, truncate skipped) keeps the donor's padding, and that path is bit-exact by construction. This is consistent with the +0.4% dense measurement.Lookahead priming appends and trims one speculative KV position through the same
update/trimmachinery (record_lookahead_stepatsrc/server/batch/scheduler.rs:5514), orthogonal to the restore path, which is why the counter is identical in both modes.Conclusion and next step
The penalty did not reproduce on metal on either backend, so the nsys profiling step on GB10 remains the right next action. On the paged default path, block-floored adoption keeps the restored cache block-aligned exactly like a fresh prefill, so the geometry-mismatch suspicions do not hold on Apple Silicon's MLX gather + SDPA attention.
One CUDA-specific hypothesis for the nsys diff to check: after a partial paged adopt the sequence's block table is
[block-floored donor blocks shared copy-on-write] + [fresh tail block] + [decode blocks], whose block ids can be non-contiguous, whereas a fresh prefill tends to draw a contiguous run from the pool free list. If the CUDA paged-attention gather kernel is sensitive to block-table contiguity (coalescing / cache locality per gather) while MLX gather + SDPA on metal is not, that would explain an ~11% decode delta visible only on GB10. The suggestednsys --cuda-graph-trace=nodediff of per-step gather/copy activity and block-table layout between one hit run and one miss run should confirm or refute this.No code change is proposed from the metal side; this is a not-reproduced report to keep the investigation honest.
perf(server): pipeline scheduler decode with lookahead async_eval (la…