docs: correct the GB10 MTP regression framing to the small-M qmv kernel gap

docs/benchmark_results/model_tests.md

@@ -259,18 +259,22 @@ tuning derivation are in
 | Gemma 4 Unified 12B + MTP assistant     | mtp  | 4 | 7.6   | 0.52×                 | 35.0%      | 1.05              | regression |
 | Gemma 4 Unified 12B + MTP assistant     | mtp  | 8 | 7.5   | 0.52×                 | 35.0%      | 1.05              | regression (effective K=4) |
 
-The 12B Unified MTP pairing is a consistent regression on GB10 across K: the
-assistant's acceptance (35-56%) is too low for the K-wide verify to pay for
-itself on the compute-bound Blackwell GPU, where verification does not amortize
-to one classic forward the way it does on memory-bandwidth-bound Apple Silicon
-(the same pairing measures ~1.87× on M5 Max). K=8 collapses onto K=4 because the
-drafter's configured block size is 4 and the acceptance never clears the
-adaptive block-expansion gate. The adaptive policy now de-rates its speedup
-estimate by `sqrt(K)` on compute-bound hardware (issue #638) so it settles to a
-decline for this pairing after its bounded profiling window, while Apple Silicon
-verdicts stay byte-identical. No local Gemma-4 or Qwen-3.5 pairing on this host
-reaches the issue's 1.4× target; the DFlash and 31B rows remain deferred (no
-checkpoint / wrong target family, see the dated note).
+The 12B Unified MTP pairing is a consistent regression on GB10 across K. The
+verify `[1, K]` forward hits the CUDA quantized dispatch's `M*B < 8` per-row
+qmv fallback (the #725 kernel gap), so its cost is roughly K classic forwards
+instead of amortizing to one the way it does on Apple Silicon (the same
+pairing measures ~1.87× on M5 Max), and the assistant's acceptance (35-56%)
+is too low to pay for that. K=8 collapses onto K=4 because the drafter's
+configured block size is 4 and the acceptance never clears the adaptive
+block-expansion gate. The adaptive policy de-rates its speedup estimate by
+`sqrt(K)` on non-Apple-Silicon hosts (issue #638) so it settles to a decline
+for this pairing after its bounded profiling window, while Apple Silicon
+verdicts stay byte-identical (#736 tracks replacing the heuristic with the
+measured round cost). This is a kernel gap, not a hardware limit: SGLang on
+the same DGX Spark hardware shows near-linear batched decode scaling and up
+to 2× EAGLE-3 speculative speedups. No local Gemma-4 or Qwen-3.5 pairing on
+this host reaches the issue's 1.4× target; the DFlash and 31B rows remain
+deferred (no checkpoint / wrong target family, see the dated note).
 
 ### Deferred pairings
 

docs/benchmark_results/speculative-pairing-gb10-2026-07-10.md

@@ -4,7 +4,9 @@ Measured B=1 MTP speculative-decoding numbers for the Gemma 4 pairing whose
 target and drafter checkpoints are present on the GB10 CUDA host, filling the
 CUDA column of the speculative pairing matrix (issue #638). The headline result:
 B=1 MTP with the Gemma 4 assistant is a consistent regression on GB10 across
-K, so the adaptive policy is tuned to decline it on compute-bound hardware.
+K, so the adaptive policy is tuned to decline it there. The regression is a
+kernel-dispatch effect (the small-M qmv fallback tracked in #725), not a
+hardware limit; see "Why MTP loses here".
 
 ## Hardware and build
 
@@ -60,34 +62,83 @@ prefix is usually fully accepted. That gate never trips at 35% acceptance, so
 but proposes only one draft per round (mean accepted 0.56), so its best case is
 still 0.77×.
 
-The per-round wall-clock grows sub-linearly with K: ~140 ms/round at K=2 versus
-~271 ms/round at K=8 (a 4x wider verify for ~1.9x the time). The verify forward
-is therefore not free on GB10, but it does not amortize to a single classic
-forward the way the memory-bandwidth-bound Apple path assumes.
+The verify cost is linear in K, and the mechanism is the quantized-matmul
+dispatch, not the GPU's compute budget. The MTP verify is a `[1, K]`
+sequence-axis forward, so every quantized linear in it runs with M = K rows.
+The CUDA dispatch
+(`src/lib/mlx-cpp/patches/mlx/backend/cuda/quantized/quantized.cpp`) routes
+`M*B < 8` to per-row `qmv`, and the sm90 CUTLASS GEMM is Hopper-gated, so on
+sm_121 a K=4 verify executes as four narrow GEMVs that each re-read the full
+weight matrix. This is the same root cause as the flat batched-decode
+throughput tracked in #725.
+
+The measured numbers fit this model exactly. The classic forward is 69 ms
+(14.5 tok/s); the K=2 round is ~140 ms (2.0 forwards) and the K=4 round is
+~271 ms (3.9 forwards), with the 258 MB drafter contributing only a few ms.
+The measured speedups are tokens-per-round divided by round cost:
+1.56 / 2.0 = 0.78x (measured 0.77x) and 2.05 / 3.9 = 0.53x (measured 0.52x).
+An earlier revision of this note read the `--block-size 8` row as evidence of
+sub-linear growth ("a 4x wider verify for ~1.9x the time"); that row runs at
+an effective K=4 (see above), so the datum actually shows a 2x wider verify
+for 1.94x the time, i.e. linear scaling.
 
 ## Contrast with Apple Silicon
 
 The same pairing measures ~1.87x on M5 Max (`gemma4-mtp-speculative-decoding.md`).
-The economics differ because the M5 decode is memory-bandwidth-bound at B=1: the
-K-wide verify reads the target weights once, so verifying K positions is nearly
-as cheap as decoding one token. GB10 at B=1 is comparatively compute-bound, so a
-K-wide verify runs closer to K narrow forwards and the drafter overhead is not
-recovered at this acceptance rate. This is the same compute-vs-bandwidth split
-noted for GB10 batched decode in the epic notes.
+On Metal the K-wide verify reads the target weights once, so verifying K
+positions is nearly as cheap as decoding one token; the speculative path even
+pads the verify up to a 32-token NA tile on M5+ to force the tiled `qmm_nax`
+GEMM instead of GEMV (`speculative/mod.rs`). The CUDA path has no analogous
+padding (#735), and with the `M*B < 8` qmv fallback the verify runs as K
+narrow forwards instead.
+
+This is a kernel gap, not a hardware limit. GB10 has more compute headroom per
+byte than Apple Silicon (~273 GB/s of memory bandwidth alongside ~100 TFLOPS
+dense bf16, versus ~614 GB/s on M5 Max with far less compute), so an
+amortizing kernel would make the K-wide verify nearly free here too. Runtimes
+with mature Blackwell kernels demonstrate it on the same hardware: LMSYS
+measured SGLang batched decode scaling near-linearly on a DGX Spark
+(llama-3.1-8b, 20.5 tok/s at B=1 to 368 tok/s at B=32) and EAGLE-3
+speculative decoding reaching up to 2x end-to-end
+(<https://www.lmsys.org/blog/2025-10-13-nvidia-dgx-spark/>), and NVIDIA ships
+speculative-decoding recipes for the same box
+(<https://build.nvidia.com/spark/speculative-decoding>). Once #725 lands an
+amortizing small-M path, this pairing's arithmetic flips positive even at the
+measured acceptance: ~2.05 tokens per round for ~1.2 forwards of round cost
+is ~1.7x at K=4.
 
 ## Policy tuning applied (issue #638)
 
 The adaptive MTP policy's speedup estimator assumed a bandwidth-bound verify
 (`verify_cost_multiple == 1.0`), which is optimistic on GB10 and would wrongly
 enable this regressing pairing. The estimator is now backend-conditional
-(`mtp_policy::estimate_speedup_scaled`): on compute-bound hardware (detected as
-non-Apple-Silicon, `AppleSiliconGen::Unknown`) the K-wide verify cost is scaled
-by `sqrt(K)`, chosen to match the measured ~1.9x round-cost growth from K=2 to
-K=8. Apple Silicon keeps `verify_cost_multiple == 1.0`, so its estimates and
-verdicts are byte-identical to before. With the de-rating, the profiler settles
-to a decline for this pairing on GB10 after its bounded profiling window, while
-a genuinely favourable compute-bound pairing (high accepted length) still clears
-the enable floor.
+(`mtp_policy::estimate_speedup_scaled`): on non-Apple-Silicon hosts (detected
+as `AppleSiliconGen::Unknown`) the K-wide verify cost is scaled by `sqrt(K)`.
+Apple Silicon keeps `verify_cost_multiple == 1.0`, so its estimates and
+verdicts are byte-identical to before. With the de-rating, the profiler
+settles to a decline for this pairing on GB10 after its bounded profiling
+window, while a pairing with high accepted length still clears the enable
+floor.
+
+Known limitation (#736): the `sqrt(K)` constant was calibrated against the
+"~1.9x growth from K=2 to K=8" datum, which the section above corrects to
+effective K=4; the true verify scaling is linear, so `sqrt(K)` under-costs a
+K=4 round by about 2x (sqrt(4) = 2 versus the measured ~3.9 classic-forward
+equivalents). The decline verdict for this pairing still holds because its
+accepted length is low, but a moderately better pairing could be wrongly
+enabled; #736 tracks replacing the shape heuristic with the measured round
+cost against a measured classic step time.
+
+## Follow-ups
+
+- #725: amortizing small-M quantized GEMM, the root-cause fix. Re-run this
+  matrix once it lands.
+- #735: pad the CUDA MTP verify past the qmv dispatch threshold, the
+  verify-side consumer of #725.
+- #736: measured-round-cost policy estimator replacing `sqrt(K)`.
+- #737: acceptance-rate cross-check on Apple Silicon. GB10's 35-56% is below
+  the 70-87% third parties report for the Gemma 4 assistants, and the M5 Max
+  run predates the bench's acceptance reporting.
 
 ## Deferred rows
 

docs/environment-variables.md

@@ -134,7 +134,7 @@ The OpenAI audio endpoints (`/v1/audio/speech`, `/v1/audio/transcriptions`, `/v1
 |----------|--------|---------|-------|
 | `MLXCEL_DRAFT_KIND` | `dflash`, `mtp` | auto/none | Alias for `--draft-kind` when the CLI flag and `LLAMA_ARG_DRAFT_KIND` are absent. |
 | `MLXCEL_DRAFT_BLOCK_SIZE` | unsigned integer | per drafter (`4` for MTP, `16` for DFlash) | Alias for `--draft-block-size` when the CLI flag and `LLAMA_ARG_DRAFT_BLOCK_SIZE` are absent. |
-| `MLXCEL_MTP_ADAPTIVE` | `0`/`false`/`no`/`off` to disable, any other value (or unset) to enable | on | Adaptive B=1 MTP policy (issue #333). When on, the server profiles the first few B=1 MTP bursts of each (target, drafter, hardware, block_size) pairing (acceptance length, verify latency, drafter latency, batch size, prompt shape) and settles to a data-driven enable/decline verdict that overrides the static per-hardware gate when the measured profile is clearly favorable or unfavorable, falling back to the static default otherwise. The verdict (enable/decline plus the coarse acceptance rate, no prompt data) is persisted at `${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/mtp-policy/<key-hash>.json` (hint format v2), so profiling runs once per pairing and a restart reuses the verdict. Changing `MLXCEL_DRAFT_BLOCK_SIZE` changes the block_size dimension of the key, so the old hint is discarded and profiling restarts for the new K. Set to an off value to disable profiling and use the pre-#333 static per-hardware gates. `MLXCEL_ENABLE_MTP_B1` still pins the decision and, when set, suppresses profiling. The experimental batched (B>1) path is unaffected and stays behind `MLXCEL_ENABLE_MTP_BATCH`. The speedup estimate is backend-conditional (issue #638): on compute-bound hardware (non-Apple-Silicon, e.g. CUDA/GB10) the K-wide verify does not amortize to one classic decode forward, so the estimate is de-rated by `sqrt(K)` (matching the measured GB10 round-cost growth), which makes the profiler decline the Gemma 4 MTP pairings that regress there (~0.52 to 0.77× measured); Apple Silicon keeps the un-scaled estimate and byte-identical verdicts. |
+| `MLXCEL_MTP_ADAPTIVE` | `0`/`false`/`no`/`off` to disable, any other value (or unset) to enable | on | Adaptive B=1 MTP policy (issue #333). When on, the server profiles the first few B=1 MTP bursts of each (target, drafter, hardware, block_size) pairing (acceptance length, verify latency, drafter latency, batch size, prompt shape) and settles to a data-driven enable/decline verdict that overrides the static per-hardware gate when the measured profile is clearly favorable or unfavorable, falling back to the static default otherwise. The verdict (enable/decline plus the coarse acceptance rate, no prompt data) is persisted at `${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/mtp-policy/<key-hash>.json` (hint format v2), so profiling runs once per pairing and a restart reuses the verdict. Changing `MLXCEL_DRAFT_BLOCK_SIZE` changes the block_size dimension of the key, so the old hint is discarded and profiling restarts for the new K. Set to an off value to disable profiling and use the pre-#333 static per-hardware gates. `MLXCEL_ENABLE_MTP_B1` still pins the decision and, when set, suppresses profiling. The experimental batched (B>1) path is unaffected and stays behind `MLXCEL_ENABLE_MTP_BATCH`. The speedup estimate is backend-conditional (issue #638): on non-Apple-Silicon hosts (e.g. CUDA/GB10) the K-wide verify does not amortize to one classic decode forward because it hits the small-M per-row qmv dispatch fallback (the #725 kernel gap; verify cost is roughly linear in K), so the estimate is de-rated by `sqrt(K)`, which makes the profiler decline the Gemma 4 MTP pairings that regress there (~0.52 to 0.77× measured); Apple Silicon keeps the un-scaled estimate and byte-identical verdicts. Note `sqrt(K)` still under-costs the measured linear growth; #736 tracks a measured-round-cost estimator. |
 | `MLXCEL_ENABLE_MTP_B1` | `0`/`false`/`no`/`off` to disable, any other value to force on | adaptive (per hardware) | Manual override for the singleton (B=1) MTP burst, in both directions. When set it pins the decision and disables adaptive profiling (issue #333). When unset, the adaptive policy decides (see `MLXCEL_MTP_ADAPTIVE`); with `MLXCEL_MTP_ADAPTIVE=0` the decision is the static per-hardware default (issue #165): non-batchable targets (`gemma4_unified` 12B pairs, whose only decode path is B=1) default **on** everywhere (measured ~1.87× on M5 Max and ~1.1 to 1.4× on M1 Ultra); batch-capable targets (the 31B + bf16 assistant) default **on only on M5+** (Neural Accelerator generation): M5 Max measured ~1.2 to 1.4×, while M1 Ultra measured a consistent ~0.75 to 0.96× regression, so pre-M5 chips fall back to classic decode. |
 | `MLXCEL_ENABLE_MTP_BATCH` | truthy value | off | **Advanced.** Forces the batched Gemma 4 MTP burst path for parity/debug testing. Not governed by the adaptive policy (issue #333), which scopes to the validated, byte-identical B=1 path. |
 | `MLXCEL_ENABLE_MTP_DEFERRED` | `1` | off | **Advanced.** Enables the deferred greedy verifier path for Gemma 4 MTP when sampling settings allow it. |