feat(server): route production batched paged decode through the fused v2 kernel

docs/CONTINUOUS_BATCHING.md

@@ -181,6 +181,70 @@ savings, the decode throughput, and `--kv-cache-budget` are documented in
 byte-identical to the dense backend; it is the storage backend the disaggregated
 roles below build on.
 
+#### The fused decode kernel
+
+Since v0.4.4 a batched paged decode step runs as **one fused attention launch
+over the whole batch** (issue #899) rather than as a per-sequence loop that
+copied each sequence's visible KV out of the pool before attending. The kernel
+reads the pool's scattered pages in place through a CSR page table and splits
+the KV range across thread blocks, so its parallelism grows with context
+instead of saturating. Measured against the previous path on an Apple M1 Ultra:
+1.4x to 3.1x at batch 4 across 1K to 32K of context, and 1.3x to 1.5x for a
+single sequence at 16K and 32K
+(`docs/benchmark_results/paged-decode-v2-m1ultra-2026-07-31.md`).
+
+What it does *not* serve falls back to the previous gather-then-SDPA path, per
+launch, with identical output:
+
+| case | why |
+|---|---|
+| a lone request under 4096 visible KV tokens | measured slower there (0.91x at batch 1 with 1K of context) |
+| a multi-request launch under 512 visible tokens per request | below the measured evidence, and gather is trivially cheap at that size |
+| a layer whose pool rows span more than one slab | the kernel reads one contiguous buffer per side |
+| multi-token steps: batched prefill, speculative / MTP verify | the kernel serves a single query token per sequence |
+| logit-softcap families, explicit attention masks | no soft-cap or mask term in the kernel |
+| Int8 / Turbo KV modes | those sequences keep dense caches and are never pool-backed |
+
+The floor is two-regime because the measured loss is a property of batch 1, not
+of total work: at 1024 tokens of context per request the kernel runs 0.91x at
+batch 1 but 1.41x at batch 4 and 1.47x at batch 8, since a batched launch
+spreads the same chunk count over more requests and amortizes the merge pass.
+
+#### Seeing which path ran
+
+The server announces the first occurrence of each distinct dispatch outcome at
+**info**, with no `RUST_LOG` needed:
+
+```text
+INFO ...: paged decode v2: fused v2 launch (batch 4, 3828 visible KV tokens, 16 chunks, merge on)
+INFO ...: paged decode v2: gather: the layer spans 3 K / 3 V pool slabs and the fused kernels read one buffer per side (slab_blocks=32); raise --ctx-size or MLXCEL_PAGED_SLAB_BLOCKS so a layer's rows fit one slab
+INFO ...: paged decode v2: gather: 1024 visible KV tokens across 1 request(s) is below the 4096-token dispatch floor
+INFO ...: paged decode v2: gather: pinned by MLXCEL_PAGED_ATTENTION_NATIVE
+```
+
+A run that never prints `fused v2 launch` never used the fused kernel, whatever
+its throughput looks like. Check for that line before drawing any conclusion
+from a before/after comparison.
+
+Two knobs, neither normally needed:
+
+- `MLXCEL_PAGED_ATTENTION_NATIVE=0` is the kill switch. It restores the
+  pre-#899 gather behaviour end to end, for bisecting a suspected kernel
+  regression. `=1` forces the fused path for every servable shape, bypassing
+  the token floor; that is the supported way to benchmark the declined corner.
+  The floors themselves move with `MLXCEL_PAGED_V2_MIN_KV_TOKENS` (lone
+  request, default 4096) and `MLXCEL_PAGED_V2_MIN_KV_TOKENS_PER_REQUEST`
+  (multi-request, default 512).
+- `MLXCEL_PAGED_SLAB_BLOCKS` overrides the pool's slab size in blocks. The
+  server derives it from `--ctx-size` and `--parallel`, clamped by the KV
+  budget, and logs the resolved value at startup (`Paged KV slab size: N blocks
+  per layer`). A layer's first write allocates its whole slab, so this
+  front-loads exactly the KV bytes the startup memory estimate already reports
+  for that batch and context. **Serving contexts longer than `--ctx-size`
+  implies makes layers outgrow the slab and silently returns them to the gather
+  path**; raise `--ctx-size` or set this variable if the startup line looks too
+  small. `0` pins the historical 32-block default.
+
 Recurrent and hybrid SSM / linear-attention families cannot safely reuse
 arbitrary KV blocks, so they keep the hybrid-SSM/APC exclusion. Families that
 opt into `supports_snapshot_reuse()` instead use a separate exact-prefix

docs/adr/0001-paged-attention-gather-vs-fused-kernel.md

@@ -1,6 +1,8 @@
 # ADR 0001: Paged-attention gather strategy and KV pool tensor layout
 
-**Status:** Accepted (2026-05-31). Part of epic #116 (unified paged KV cache), Phase 0 (#117).
+**Status:** Superseded in part by issue #899 (2026-07-31). Accepted (2026-05-31), part of epic #116 (unified paged KV cache), Phase 0 (#117).
+
+The layout and block-size decisions below still stand. The *attention strategy* decision does not: this ADR chose gather-then-SDPA and named the trigger that would justify building a fused kernel, that trigger has now fired and been measured, and since #899 the fused path is what serves production batched paged decode. See the 2026-07-31 addendum on issue #899 at the end of this document; the historical reasoning is kept intact above it because the measurements are what justified the switch.
 
 ## Context
 
@@ -188,6 +190,20 @@ Scope of #898: library only. `MLXCEL_PAGED_ATTENTION_V2=1` selects it inside `Pa
 
 Correctness on an M1 Ultra: 68 configurations against the gather-then-SDPA reference across head dims 64/128, GQA 1/4/8, page sizes 16/32/64, contexts 512 to 32768, and batches 1/4/8, worst relative deviation 5.2e-4 against a 2e-2 tolerance.
 
+## Addendum (2026-07-31): the trigger fired, the fused path is production (#899)
+
+This ADR set the trigger for building a fused path at "single-sequence context past ~16384 tokens, or any sustained batched decode". Since v0.4 the server defaults to `--parallel 4`, which is that second condition, and #898's v2 kernel was measured against the production gather path on an Apple M1 Ultra (`docs/benchmark_results/paged-decode-v2-m1ultra-2026-07-31.md`). Both trigger points are met: batch 4 runs 1.41x to 3.08x, and single-sequence decode runs 1.29x at 16384 and 1.47x at 32768. The 2x-3x figure this ADR predicted for batch 4 past 4096 tokens is reproduced almost exactly (2.04x, 2.78x, 3.08x).
+
+**The decision is therefore reversed for batched paged decode.** `PagedBlockPool::paged_decode_batched`, reached from `cache::paged_batch_decode::paged_batch_decode_attention`, is the production path for `--decode-storage paged`; `gather_visible` + SDPA remains as the fallback and the parity reference.
+
+Three qualifications, each with its own reason:
+
+**The switch is not unconditional.** Exactly one measured cell loses: batch 1 at 1024 tokens, 0.91x median and below parity in all three repetitions, where the plan degenerates to two pages per chunk and the merge pass plus the workspace round trip costs more than the attention. `paged_v2::dispatch` keeps that regime on gather with a two-regime floor: a lone request needs 4096 visible KV tokens, a multi-request launch needs 512 per request. The first formulation used a single 4096-token total and had to be corrected: the loss is a property of batch 1, not of total work (the same 1024-token context runs 0.91x at batch 1 and 1.41x at batch 4), and a total-token floor separated the two only by accident, declining a real batch-4 launch that delivered 956 tokens per request.
+
+**The single-slab constraint from #235 had to be relaxed, not removed.** The fused kernels read one contiguous pool buffer per side, and 32-row slabs capped them at 1024 tokens across the whole batch, below the dispatch floor. `PagedBlockPool` now carries its own slab size (default unchanged) and the server sizes it from the configuration the KV budget was already computed from. A layer that outgrows its slab still appends without copying and simply stops being eligible, so the degradation is to this ADR's original strategy rather than to a failure.
+
+**The library-only entry point retired in #710 stays retired.** `paged_decode_attention_pooled` and `select_pooled_paged_dispatch` are untouched: they still serve external `mlxcel-core` consumers with the v1 kernel and the #331 selector. #899 adds a separate production entry point rather than reviving that one, because the two have different inputs (a whole batch versus one sequence at a time) and different dispatch evidence.
+
 ## References
 
 - Epic #116, unified KV cache.

docs/benchmark_results/paged-decode-production-m1ultra-2026-08-02.md

@@ -0,0 +1,121 @@
+# Production paged decode through fused v2: Apple M1 Ultra, 2026-08-02
+
+Validation run for issue #899 (epic #909), which routes the server's batched
+paged decode through the fused v2 kernel from #898 and retires gather-then-SDPA
+as the default.
+
+**Result: decode throughput improves in four of five scenarios, 1.15x to 1.57x,
+with parity in the fifth and no regression anywhere.** Aggregate throughput moves
+much less, for a reason explained below that is a property of the metric rather
+than of the kernel.
+
+## Environment
+
+| Field | Value |
+|---|---|
+| Hardware | Apple M1 Ultra, 20 cores (16P + 4E), 128 GB unified memory |
+| OS | macOS 26.5.2 (Darwin 25.5.0) |
+| Backend | Metal |
+| mlxcel | 0.4.3, branch `feature/issue-899-production-paged-v2-dispatch` |
+| Model | `mlx-community/Llama-3.2-1B-Instruct-4bit` (2048 hidden, 32 q heads, 8 kv heads, head_dim 64, 16 layers) |
+| Harness | `scripts/benchmark_paged_decode_production.sh`, `--parallel 4`, 128 decode tokens |
+| Arms | `before` = `MLXCEL_PAGED_ATTENTION_NATIVE=0` (gather), `after` = default (fused v2) |
+| Load average | 5.0 to 5.5 throughout, from unrelated concurrent work |
+| Repetitions | 3 full sweeps, fresh server per arm per sweep |
+
+CUDA was not available. The CUDA JIT bodies inherited from #898 remain
+uncompiled and unrun.
+
+## Dispatch was verified, not assumed
+
+This is recorded first because the first attempt at this benchmark produced a
+completely null result (0.98x to 1.07x everywhere) that looked like a legitimate
+finding and was not. **Both arms had run the gather path**: the fused kernel never
+launched once across the entire sweep, for two independent reasons (a token floor
+whose shape excluded the measured scenarios, and a single-sequence decode path
+that was never wired). A before/after comparison of gather against gather agrees
+to within noise, which is indistinguishable from "the kernel does not help".
+
+Every run below therefore carries proof of which kernel it executed, taken from
+the server logs:
+
+```
+before: paged decode v2: gather: pinned by MLXCEL_PAGED_ATTENTION_NATIVE
+after:  paged decode v2: fused v2 launch (batch 4, 3828 visible KV tokens, 16 chunks, merge on)
+```
+
+The harness now fails the run if the after arm never logs a fused launch, or if
+the before arm does. A null sweep can no longer be produced silently.
+
+## Decode throughput, after / before
+
+The metric the kernel actually affects. Three independent sweeps.
+
+| scenario | rep 1 | rep 2 | rep 3 | median |
+|---|---|---|---|---|
+| 4 clients, ctx 1024 | 1.33x | 1.82x | 1.57x | **1.57x** |
+| 4 clients, ctx 4096 | 0.98x | 1.00x | 1.02x | 1.00x |
+| 4 clients, ctx 16384 | 1.15x | 1.13x | 1.15x | **1.15x** |
+| 1 client, ctx 16384 | 1.24x | 1.42x | 1.22x | **1.24x** |
+| 1 client, ctx 32768 | 1.41x | 1.41x | 1.44x | **1.41x** |
+
+Two cells are tight enough to quote without hedging: 4 clients at 16384 spans
+1.13x to 1.15x, and 1 client at 32768 spans 1.41x to 1.44x. Both are large
+relative to their own spread and both reproduce in every sweep.
+
+The 4-client / 1024 cell is the noisiest (1.33x to 1.82x) but never drops below
+1.33x, so the direction is not in doubt even though the magnitude is.
+
+## Aggregate throughput, after / before
+
+| scenario | rep 1 | rep 2 | rep 3 |
+|---|---|---|---|
+| 4 clients, ctx 1024 | 1.21x | 1.20x | 1.23x |
+| 4 clients, ctx 4096 | 1.00x | 1.00x | 1.01x |
+| 4 clients, ctx 16384 | 1.00x | 1.00x | 1.00x |
+| 1 client, ctx 16384 | 1.09x | 1.26x | 1.18x |
+| 1 client, ctx 32768 | 1.04x | 1.05x | 1.06x |
+
+Aggregate includes time to first token, so at long context it is dominated by
+prefill, which this change does not touch. At 4 clients and 16384 tokens the
+server spends several seconds prefilling four prompts before decoding 128 tokens
+each, so a 1.15x decode improvement is diluted to 1.00x. Read the decode column
+for the kernel's effect and this column for what a user of that particular
+workload shape would feel.
+
+## Against the issue's required outcomes
+
+- **"No scenario regresses beyond noise (3%)."** Met. The weakest cell is 4
+  clients at 4096, whose median is exactly 1.00x (0.98x, 1.00x, 1.02x across
+  sweeps), which is parity rather than regression.
+- **"Batched decode aggregate throughput improves at 4K+ context."** Partially
+  met, and stated precisely rather than rounded up: batched *decode* improves at
+  16384 (1.15x, reproducible) but is at parity at 4096, and batched *aggregate*
+  does not improve at either because prefill dominates it. ADR-0001's prediction
+  that removing gather overhead is worth 2x-3x of SDPA time is an op-level claim,
+  and #898 reproduced it at the op level (2.78x at this shape); it does not follow
+  that end-to-end serving throughput moves by the same factor, because attention
+  is one component of a decode step.
+
+## The 4096 cell
+
+Batched decode at 4096 tokens is the one scenario that gains nothing, sitting
+between a 1.57x gain at 1024 and a 1.15x gain at 16384. That non-monotonicity is
+not explained. It is reproducible (three sweeps within 4%), so it is a real
+property of that shape rather than noise, and it is worth understanding before
+the dispatch floor is tuned further. Recorded as an open question rather than
+smoothed over.
+
+## Open items
+
+- The non-monotonic 4096 cell above.
+- Batch 2 and 3 are interpolated in the dispatch floor: every multi-request cell
+  #898 measured was batch 4 or 8.
+- The 1-client / 4096 crossover cell has the least margin of any win (1.08x at
+  the op level). If single-sequence 4K regresses in a future run,
+  `MIN_SINGLE_REQUEST_KV_TOKENS` can be raised to 8192, which gives up only that
+  cell.
+- CUDA remains entirely unvalidated across #898 and #899.
+- One model family measured end to end. Greedy parity covers three families
+  (Llama3, Qwen2.5, Qwen3), all exact against the gather baseline, but throughput
+  was measured only on Llama-3.2-1B.