[inureyes]

Summary

KVCache::trim re-sliced the entire live window to discard a few rejected tail tokens, so every speculative rollback paid an O(context) copy, and then a second one: the re-slice threw away the step-aligned capacity, forcing the next update_* through its regrow path (another full-buffer concatenate). The cost grew with the conversation while the work it undoes stayed constant at a few tokens, which is exactly what #1209 filed.

Fp16 and Int8 trims are now a logical rewind: only offset moves, the buffers (scale sidecars included) keep their allocation, and the next slice_update overwrites the abandoned slots. This is the contract RotatingKVCache::trim has always had ("it moves no data at all, it only rewinds offset and idx and lets the next append overwrite"), applied to the dense cache. The regrow guard in update_fp16/update_int8 changes from prev % step != 0 to prev != buffer length, so a retained capacity is normalized to the live prefix on the rare genuine growth.

Related issues

Closes #1209.

Type of change

• perf

Why every reader stays correct

Capacity larger than the live window is already the normal state of these buffers between growths: update_* writes into step-aligned pre-allocated buffers and every reader derives the window from the offsets, not from the physical length. update_and_fetch and keys_view slice to buffer_idx(); trim_front_keep_sink slices [keep, live_len) computed from offsets; gather_within_tail works in buffer_idx() coordinates; the detached-handle trim_to bounds by offset. What the logical trim adds is stale content (instead of zeros) beyond the live window, and nothing reads past it.

The modes that do not share this property keep the physical trim, with the reason recorded in the code:

• Turbo4Asym / Turbo4 / Turbo3Asym: their packed sidecars feed fused kernels and fold/dequant helpers that have not been audited for a stale tail, and the speculative rollbacks this rewind serves run on Fp16/Int8 caches.
• Turbo4Delegated: unchanged (hot/cold split bookkeeping).
• Pool-backed (paged): unchanged (the block table is authoritative; dense-side trim was already a no-op).

gather_positions

The acceptance criteria ask for the same treatment or a note. Note: a compaction has to write its survivors, so the rewrite cannot be avoided outright; what can be avoided is its scope, and #1214's gather_within_tail already did that for the production tree rollback (it writes O(block) through the same slice_update the append path uses and keeps the capacity). gather_positions has no production callers left, only tests.

Measurements

Apple M5 Max 128 GB, macOS 26.6.1, offline CLI, temperature 0, ABBA per context, warm-up discarded, 6 s cooldowns. Arms: main at 9e2c6675 vs this branch.

Rollback cost against context length (acceptance criterion 1), measured as the A/B it costs end to end. Pairing: qwen2.5-7b-instruct-4bit + qwen2.5-0.5b-4bit classic speculative (--num-draft-tokens 3), a dense 28-layer full-attention target, chosen because every layer's KV pays the rewrite. 300 new tokens on ~1k/8k/24k-token prompts:

context	before tok/s	after tok/s	speedup	saved per decoded token
~1k	55.70, 55.81	56.24, 56.29	+0.9%	0.16 ms
~8k	37.28, 37.30	38.22, 38.21	+2.5%	0.65 ms
~24k	18.65, 18.65	19.63, 19.63	+5.3%	2.68 ms

The saved term scales linearly with context, which is the O(context)-per-round copy this PR removes. The issue's arithmetic estimated sub-millisecond per round from one buffer's bytes; the measured cost is larger because it is paid per layer and twice (trim, then regrow).

Byte identity (acceptance criterion 4): generated text diffed at all three contexts, before vs after: identical. Also identical on both MTP pairings below.

MTP pairings, short context (regression check on the paths #1185/#1204 measure):

pairing	before tok/s	after tok/s
qwen3.8-27b-4bit + mtp-4bit, block 3	53.87, 53.89	54.83, 52.88 (wash)
gemma-4-12b-it-4bit + assistant, block 5	121.60, 121.77	121.94, 121.91 (wash)

Test plan

• cargo fmt --check
• cargo clippy --workspace --all-targets --features metal,accelerate -- -D warnings
• cargo test --release -p mlxcel-core --features metal,accelerate — failure set unchanged from main on this machine (4 pre-existing numeric-tolerance reds, re-verified on unmodified main)
• New contract test kv_cache_trim_then_append_matches_untrimmed_reference: a trimmed-then-overwritten cache fetches a window byte-identical to a reference that never saw the rejected block, in Fp16 and Int8
• test_kv_cache_trim_basic updated to pin the new contract (offset moves, capacity stays, fetched window is offset-sized)
• Real-checkpoint validation as in Measurements