perf(cuda/quant): amortize small-M quantized matmul with a multirow qmv path

docs/CONTINUOUS_BATCHING.md

@@ -48,18 +48,19 @@ Escape hatches restore the previous single-client behavior: `--parallel 1`
 (single decode slot), `--no-batch` (legacy sequential worker, no scheduler),
 `--max-batch-prefill 1` (sequential prefill), and `--no-prompt-cache`.
 
-> Backend note (CUDA / Blackwell, e.g. GB10): the "aggregate throughput scales
-> with concurrency" statement above holds on Metal but not on CUDA. On CUDA the
-> quantized batched-decode matmul does not amortize weight bandwidth (a batch of
-> B decodes takes about B times a single decode), so aggregate decode throughput
-> is flat across `--parallel` (measured ~45 tok/s at B=1/2/4 on llama-3.1-8b-4bit
-> on GB10). There, `--parallel` is a TTFT / concurrency knob, not an aggregate
-> decode-throughput lever, and batched vs sequential decode is a throughput wash.
-> The aggregate lever on CUDA is prefill: see the sm_120/121 qmm tile tuning in
-> `docs/benchmark_results/qmm-sm121-tile-tuning-gb10-2026-07-10.md` (+38% prefill).
-> `--max-batch-prefill` (large-M, tile-tuned) and the prompt cache remain real
-> wins on CUDA. The root cause is the absence of an amortizing small-M quantized
-> GEMM (or a weight-reusing batched qmv) on Blackwell, an upstream MLX follow-up.
+> Backend note (CUDA / Blackwell, e.g. GB10): batched decode used to be a
+> throughput wash on CUDA because the `M*B < 8` quantized matmul fell back to
+> per-row qmv, re-reading the weights once per sequence (aggregate flat at
+> ~45-50 tok/s across B=1/2/4 on llama-3.1-8b-4bit). The multirow qmv path
+> (issue #725, `MLXCEL_QMV_MULTIROW`, on by default) amortizes the weight reads
+> across the batch, so aggregate decode now scales with concurrency on CUDA
+> too: 40.4 / 68.5 / 74.2 tok/s at 1/2/4 clients on llama-3.1-8b-4bit on GB10
+> (vs 40.5 / 49.8 / 49.6 with the kill switch off). Scaling stays sublinear
+> versus Metal because attention KV reads still grow with B; the small-M
+> `qmm_sm80` tile shape (`M*B >= 8`) remains upstream MLX territory. See
+> `docs/benchmark_results/qmv-multirow-gb10-2026-07-11.md`. `--max-batch-prefill`
+> (large-M, tile-tuned, `docs/benchmark_results/qmm-sm121-tile-tuning-gb10-2026-07-10.md`)
+> and the prompt cache remain independent wins on CUDA.
 
 Memory sizing: the KV footprint grows with the active batch, so budget for up to
 `--parallel` concurrent sequences' KV. When `--ctx-size` is set it is divided

docs/benchmark_results/model_tests.md

@@ -255,26 +255,33 @@ tuning derivation are in
 | Pairing (GB10 CUDA)                     | Kind | K | tok/s | speedup vs no-drafter | acceptance | mean accepted len | status |
 |-----------------------------------------|------|---|------:|----------------------:|-----------:|------------------:|--------|
 | Gemma 4 Unified 12B (no drafter)        | none | — | 14.5  | 1.00×                 | —          | —                 | ok |
+| Gemma 4 Unified 12B + MTP assistant     | mtp  | 2 | 19.0  | 1.31×                 | 56.6%      | 0.57              | ok (multirow qmv, #725) |
+| Gemma 4 Unified 12B + MTP assistant     | mtp  | 4 | 21.2  | 1.46×                 | 35.8%      | 1.07              | ok (multirow qmv, #725) |
+| Gemma 4 Unified 12B + MTP assistant     | mtp  | 8 | 20.4  | 1.41×                 | 34.1%      | 1.02              | ok (effective K=4) |
+
+Pre-#725 record (per-row qmv verify; reproducible with `MLXCEL_QMV_MULTIROW=0`,
+which measures 7.7 tok/s at K=4 on the same binary):
+
+| Pairing (GB10 CUDA, pre-#725)           | Kind | K | tok/s | speedup vs no-drafter | acceptance | mean accepted len | status |
+|-----------------------------------------|------|---|------:|----------------------:|-----------:|------------------:|--------|
 | Gemma 4 Unified 12B + MTP assistant     | mtp  | 2 | 11.1  | 0.77×                 | 55.6%      | 0.56              | regression |
 | Gemma 4 Unified 12B + MTP assistant     | mtp  | 4 | 7.6   | 0.52×                 | 35.0%      | 1.05              | regression |
 | Gemma 4 Unified 12B + MTP assistant     | mtp  | 8 | 7.5   | 0.52×                 | 35.0%      | 1.05              | regression (effective K=4) |
 
-The 12B Unified MTP pairing is a consistent regression on GB10 across K. The
-verify `[1, K]` forward hits the CUDA quantized dispatch's `M*B < 8` per-row
-qmv fallback (the #725 kernel gap), so its cost is roughly K classic forwards
-instead of amortizing to one the way it does on Apple Silicon (the same
-pairing measures ~1.87× on M5 Max), and the assistant's acceptance (35-56%)
-is too low to pay for that. K=8 collapses onto K=4 because the drafter's
-configured block size is 4 and the acceptance never clears the adaptive
-block-expansion gate. The adaptive policy de-rates its speedup estimate by
-`sqrt(K)` on non-Apple-Silicon hosts (issue #638) so it settles to a decline
-for this pairing after its bounded profiling window, while Apple Silicon
-verdicts stay byte-identical (#736 tracks replacing the heuristic with the
-measured round cost). This is a kernel gap, not a hardware limit: SGLang on
-the same DGX Spark hardware shows near-linear batched decode scaling and up
-to 2× EAGLE-3 speculative speedups. No local Gemma-4 or Qwen-3.5 pairing on
-this host reaches the issue's 1.4× target; the DFlash and 31B rows remain
-deferred (no checkpoint / wrong target family, see the dated note).
+The pre-#725 regression came from the verify `[1, K]` forward hitting the CUDA
+quantized dispatch's `M*B < 8` per-row qmv fallback, which costs roughly K
+classic forwards instead of amortizing to one the way it does on Apple Silicon
+(the same pairing measures ~1.87× on M5 Max). The multirow qmv path (#725,
+`MLXCEL_QMV_MULTIROW`) removes that fallback's weight re-reads, and B=1 MTP on
+GB10 now clears the 1.4× target from issue #638 at K=4/K=8 at unchanged
+acceptance (see `qmv-multirow-gb10-2026-07-11.md`). K=8 collapses onto K=4
+because the drafter's configured block size is 4 and the acceptance never
+clears the adaptive block-expansion gate. Serving caveat: the adaptive policy
+still de-rates non-Apple-Silicon hosts by `sqrt(K)` (issue #638), calibrated
+against the pre-#725 verify; it can now wrongly decline this favourable
+pairing, and #736 tracks the measured-round-cost re-calibration
+(`MLXCEL_ENABLE_MTP_B1=1` pins B=1 MTP on until then). The DFlash and 31B rows
+remain deferred (no checkpoint / wrong target family, see the dated note).
 
 ### Deferred pairings
 

docs/benchmark_results/qmm-sm121-tile-tuning-gb10-2026-07-10.md

@@ -3,6 +3,11 @@
 Date: 2026-07-10. Host: NVIDIA GB10 (Grace-Blackwell, sm_121 / cc 12.1), CUDA 13.0, MLX 0.32.1 (pin `57c66cac`).
 Binary: `make release-cuda`. Bench: `mlxcel-bench-decode` (`--prompt-tokens` for prefill), `mlxcel generate` (greedy parity).
 
+> **2026-07-11 update (#725):** the "NOT addressed" small-M batched-decode gap
+> in the Scope section below is now closed by the multirow qmv path
+> (`MLXCEL_QMV_MULTIROW`); see `qmv-multirow-gb10-2026-07-11.md`. The large-M
+> prefill tile fix on this page is unaffected and stands on its own.
+
 ## TL;DR
 
 The CUDA quantized prefill GEMM (`qmm_sm80`) ran at ~0.55x the bf16 ceiling on GB10 because it used the Ampere CTA tile (M=64), which underutilizes Blackwell's SMs. Raising the CTA tile's M cap to 128 on sm_120/121 (a one-line, arch-gated heuristic change in the overlay `qmm_sm80.cu`) recovers **+31-38% prefill** throughput, greedy-parity identical, no decode regression. Wider-N / deeper-K tiles were also swept but break the fixed-MMA shared-memory layout (JIT failure), so `tile_m` is the one safely-tunable axis in-tree.
@@ -71,9 +76,9 @@ Meets #637's acceptance (>= 25% prefill @8192).
 ## Scope: addressed vs not
 
 - **Addressed**: large-M prefill and batched prefill on sm_120/121 (+32%).
-- **NOT addressed**: small-M batched decode (`M*B` in [2,8)) still uses per-row `qmv` (no weight amortization across the batch), and the `tile_m` cap does not raise small-M tiles. This is the batched-decode non-amortization behind #714's `--parallel 4` CUDA regression, and needs a dedicated small-M kernel (or a weight-reusing batched `qmv`) - upstream MLX territory. Until then, #714's `--parallel` default should be made backend-aware (fall back to 1 on CUDA/Blackwell).
+- **NOT addressed at the time of this report, closed 2026-07-11 (#725)**: small-M batched decode (`M*B` in [2,8)) used per-row `qmv` with no weight amortization across the batch, the batched-decode non-amortization behind #714's `--parallel 4` CUDA regression. The `tile_m` cap here still does not raise small-M tiles, but a weight-reusing multirow `qmv` variant now covers that window instead (`qmv-multirow-gb10-2026-07-11.md`), so the CUDA-backend-aware `--parallel` fallback floated below is no longer needed for this gap.
 - **Further prefill gains** toward the bf16 ceiling (0.72x -> 1.0x) need a Blackwell-tuned kernel (5th-gen tensor-core MMA, wider tiles with a compatible smem layout) - upstream MLX.
 
 ## Upstream
 
-MLX's CUDA quantized GEMM has no sm_120/121 tile specialization: the Ampere tile underutilizes Blackwell (ncu evidence above), and there is no amortizing small-M (batched decode) quantized path on Blackwell. A tracking issue on lablup/mlxcel should carry this for upstream follow-up.
+MLX's CUDA quantized GEMM still has no sm_120/121 tile specialization: the Ampere tile underutilizes Blackwell (ncu evidence above), and stock MLX itself still has no amortizing small-M (batched decode) quantized path on Blackwell; the multirow `qmv` fix (#725) is a mlxcel overlay, not an upstream MLX change. A tracking issue on lablup/mlxcel should carry both gaps for upstream follow-up.

docs/benchmark_results/qmv-multirow-gb10-2026-07-11.md

@@ -0,0 +1,111 @@
+# Multirow qmv: amortizing small-M quantized matmul on GB10 (CUDA), 2026-07-11
+
+Issue #725. On Blackwell sm_120/121 the CUDA quantized-matmul dispatch routes
+`M*B < 8` to per-row `qmv`, which re-reads the full weight matrix once per
+input row (one row per `grid.y`/`grid.z` block). That made batched decode
+aggregate throughput flat in B and made the speculative-verify `[1, K]`
+forward cost ~K classic forwards, the root cause of the GB10 MTP regression
+measured in `speculative-pairing-gb10-2026-07-10.md`.
+
+The fix is a multirow qmv variant in the `qmv.cu` overlay
+(`src/lib/mlx-cpp/patches/mlx/backend/cuda/quantized/qmm/qmv.cu`): when the
+weights are broadcast and `2 <= M*B <= 8`, one warp keeps per-row accumulator
+arrays and applies each dequantized weight tile to every input row, so weight
+traffic is O(1) in the row count instead of O(R). The per-row arithmetic
+(dequant order, fma order, accumulator types, final float reduction) is kept
+exactly equal to the stock kernel, so per-row outputs are bit-identical to the
+per-row launches it replaces. `M*B == 1` (classic single-token decode) keeps
+the stock kernel unchanged. Kill switch: `MLXCEL_QMV_MULTIROW=0`.
+
+## Hardware and build
+
+- Host: NVIDIA GB10 (Grace-Blackwell, sm_121), CUDA backend.
+- Build: `cargo build --release --features cuda`.
+- Greedy (`temperature = 0`), decode-only tok/s.
+
+## Parity
+
+- `qmv_multirow_matches_per_row_qmv_bitwise` (ffi test, `-p mlxcel-core
+  --features cuda`): bf16/f16/f32, 4-bit and 8-bit affine, group sizes 32/64,
+  rows 2/3/4/7, including a residue-k shape; every multirow row is
+  byte-identical to the matching single-row qmv launch.
+- End-to-end: on a quiet GPU the K=4 MTP runs with `MLXCEL_QMV_MULTIROW=0` and
+  `=1` produce the identical token stream (85 generated, 41 rounds, acceptance
+  0.358). Note the `[1, K]` verify path itself is load-sensitive run-to-run on
+  CUDA (near-tie argmax flips under concurrent GPU work, the known FP-reduction
+  jitter also documented for `MLXCEL_FUSED_QK_NORM`); the jitter pre-dates this
+  change and reproduces with the kill switch on.
+
+## Speculative decoding (MTP): the #638 matrix, re-run
+
+Target `models/gemma-4-12b-it-4bit` (`gemma4_unified`), drafter
+`models/gemma-4-12b-it-assistant-4bit`, 14-token `DEFAULT_PROMPT`,
+`--max-tokens 128`, one process per row (same invocations as the 2026-07-10
+matrix).
+
+| Kind | K | tok/s | speedup vs no-drafter | 2026-07-10 (per-row qmv) | acceptance rate | mean accepted len |
+|------|---|------:|----------------------:|-------------------------:|----------------:|------------------:|
+| none (baseline) | — | 14.5 | 1.00x | 14.5 (1.00x) | — | — |
+| mtp | 2 | 19.0 | 1.31x | 11.1 (0.77x) | 56.6% | 0.57 |
+| mtp | 4 | 21.2 | 1.46x | 7.6 (0.52x) | 35.8% | 1.07 |
+| mtp | 8 | 20.4 | 1.41x | 7.5 (0.52x) | 34.1% | 1.02 |
+| mtp | 4 (`MLXCEL_QMV_MULTIROW=0`) | 7.7 | 0.53x | — | 35.8% | 1.07 |
+
+The K=4 round cost drops from ~268 ms (per-row: verify M=4 ran as 4 narrow
+GEMVs, ~3.9 classic forwards) to ~98 ms (verify amortizes toward one classic
+forward plus the drafter and per-round overhead). B=1 MTP on GB10 flips from a
+0.52-0.77x regression to a 1.31-1.46x win at the same acceptance, clearing the
+1.4x target from issue #638 at K=4/K=8. K=8 still runs at an effective K=4
+(the adaptive block controller holds the drafter's configured block size at
+this acceptance).
+
+Policy note (#736): the adaptive MTP policy still de-rates compute-bound hosts
+by `sqrt(K)`, which was calibrated against the pre-#725 per-row verify. With
+the multirow verify the de-rating is now pessimistic in the opposite direction
+(it can decline this now-favourable pairing in serving). #736 (measured round
+cost vs measured classic step) covers the re-calibration; until it lands,
+`MLXCEL_ENABLE_MTP_B1=1` pins B=1 MTP on for serving.
+
+## Batched serving decode: aggregate scaling
+
+`mlxcel serve -m models/llama-3.1-8b-4bit --n-parallel 4` +
+`scripts/bench_serving_concurrency.py --concurrency 1,2,4 --prompt-tokens 512
+--max-tokens 128`:
+
+| clients | aggregate tok/s (`MLXCEL_QMV_MULTIROW=0`) | aggregate tok/s (multirow) | per-request decode tok/s (off -> on) |
+|--------:|------------------------------------------:|---------------------------:|-------------------------------------:|
+| 1 | 40.5 | 40.4 | 46.5 -> 46.5 |
+| 2 | 49.8 | 68.5 | 25.1 -> 34.7 |
+| 4 | 49.6 | 74.2 | 12.5 -> 18.8 |
+
+Aggregate decode now scales with concurrency (1.84x from 1 to 4 clients)
+instead of staying flat (1.22x before, the "throughput wash" documented in
+`docs/CONTINUOUS_BATCHING.md`). Scaling is sublinear because only the
+quantized-matmul weight reads amortize: attention KV reads still grow with B,
+and the 4-client row interleaves prefill with decode. The remaining gap is the
+small-M `qmm_sm80` tile territory tracked upstream (see below).
+
+## Kernel evidence
+
+`nsys profile --cuda-graph-trace=node` over a K=4 MTP run shows the verify-side
+`qmv` work collapsing from per-row launches into single `qmv_multirow_kernel`
+launches (see the PR for the capture summary). `ncu` counter capture on this
+host requires `sudo` (driver `RmProfilingAdminOnly=1`); the nsys kernel-time
+comparison plus the wall-clock A/B above stand in as the amortization
+evidence.
+
+## Scope and follow-ups
+
+- `fp_qmv` (non-affine fp modes, sm120+) still maps one row per block for
+  `vec_batch` up to 8; the same multirow treatment applies but is not needed
+  for the affine checkpoints measured here.
+- `gather_qmv` (MoE decode) keeps per-row launches: rows select different
+  experts, so there is no shared weight tile to amortize in general.
+- `M*B == 8` exactly still dispatches to `qmm_sm80` (the `< 8` gate is
+  unchanged); the multirow kernel accepts up to 8 rows, so widening the gate is
+  a measurement away if an 8-row shape shows up hot.
+- Upstream MLX issue: the gap is upstream territory (`qmv.cu` is stock MLX);
+  an issue draft with this evidence is attached to the PR for filing against
+  ml-explore/mlx.
+- #735 (pad the MTP verify past the qmv threshold) is superseded for K in
+  [2,7]: the verify now amortizes without padding.

docs/benchmark_results/scheduler-decode-lookahead-gb10-2026-07-10.md

@@ -3,6 +3,12 @@
 Date: 2026-07-10. Host: NVIDIA GB10 (Grace-Blackwell), CUDA build.
 Bench: `bench_serving_concurrency.py` (512 prompt tokens, 512 max tokens) against `mlxcel-bench-decode --prompt-tokens 512` as the matched-context CLI reference; sync baseline via `MLXCEL_FORCE_SYNC=1`. Full run and methodology: PR #729.
 
+> **2026-07-11 update (#725):** the "#724 CUDA batched-decode caveat" bounding
+> the B=4 llama row below is closed by the multirow qmv path
+> (`MLXCEL_QMV_MULTIROW`); re-measured aggregate decode now scales with
+> concurrency instead of staying flat (`qmv-multirow-gb10-2026-07-11.md`). The
+> B=4 llama numbers below predate that fix and are kept as the pre-#725 record.
+
 ## TL;DR
 
 Porting the CLI generation loop's lookahead `async_eval` pipelining into the server `BatchScheduler` decode tick closes most of the structural decode-throughput gap between server and CLI on CUDA. The lookahead pipeline engages 509/511 decode steps per 512-token request, and greedy output stays byte-identical to the `MLXCEL_FORCE_SYNC=1` synchronous path.

docs/benchmark_results/speculative-pairing-gb10-2026-07-10.md

@@ -1,5 +1,13 @@
 # Speculative drafter pairing matrix on GB10 (CUDA), 2026-07-10
 
+> **2026-07-11 update (#725):** the multirow qmv path landed and flips this
+> matrix: the same pairing now measures 1.31x (K=2), 1.46x (K=4), 1.41x (K=8)
+> against the same 14.5 tok/s baseline, at unchanged acceptance. The verify
+> forward amortizes as the analysis below predicted once the `M*B < 8` per-row
+> qmv fallback is removed. Current numbers:
+> `qmv-multirow-gb10-2026-07-11.md`. The measurements below are kept as the
+> pre-#725 record.
+
 Measured B=1 MTP speculative-decoding numbers for the Gemma 4 pairing whose
 target and drafter checkpoints are present on the GB10 CUDA host, filling the
 CUDA column of the speculative pairing matrix (issue #638). The headline result: