bench(speculative): acceptance-rate cross-check for the Gemma 4 MTP pairing on Apple Silicon #737

Context

PR #733 (issue #638) measured B=1 MTP speculative decoding on GB10 (CUDA, sm_121) for the gemma-4-12b-it-4bit + gemma-4-12b-it-assistant-4bit pairing as a consistent regression, with acceptance 35-56% and mean accepted length 0.56-1.05 (docs/benchmark_results/speculative-pairing-gb10-2026-07-10.md). Third-party Gemma 4 MTP measurements report 70-87% acceptance and Google cites ~80% (the assistant is a conditioned drafter that reads the target's final-layer activations and shares its KV cache), so the GB10 acceptance is suspiciously low.

The M5 Max 1.87x measurement (docs/benchmark_results/gemma4-mtp-speculative-decoding.md) predates the acceptance-rate reporting that PR #733 added to speculative_bench, so no same-prompt acceptance comparison exists; back-calculation from the M5 numbers suggests ~47%.

Attribution is unresolved between:

• (a) Host-dependent causes: CUDA numeric differences in the drafter conditioning path (target hidden-state dtype), or [1, K]-verify near-tie argmax flips (the fp jitter documented in perf: validate B=1 MTP default-on on lower-bandwidth Apple Silicon (M1/M2) #165 / fix: per-row position holes break B>1 batched MTP greedy parity after divergent accepts #203). Note the [1, K] verify on GB10 also runs a different kernel path than a classic single-token forward (the per-row qmv fallback, see perf(cuda/quant): amortizing small-M quantized GEMM for batched decode on Blackwell #725 and the 2026-07-11 round-cost investigation), so kernel-level numeric differences on the verify shape are plausible.
• (b) A pairing/prompt property: the bench uses the 14-token DEFAULT_PROMPT, and acceptance is workload-dependent.

Impact

Acceptance rate multiplies directly into any speculative speedup estimate, including the adaptive MTP policy's decision. Until the acceptance gap is attributed, we cannot tell how much of the GB10 regression would remain after the round-cost problem (#725 and its consumers) is fixed.

What's needed

• Run speculative_bench --k-values 2,4,8 with the acceptance / mean-accepted columns on M5 Max, with the identical checkpoints and prompt as the GB10 run.
• Add 2-3 longer, more realistic prompts and measure them on BOTH hosts.
• If acceptance is materially higher on M5 Max for the same prompt, investigate the CUDA drafter conditioning path (hidden-state dtype through the cxx bridge, shared-KV copies) and file a targeted follow-up.
• Document the resulting table in docs/benchmark_results.

Acceptance Criteria

• Same-prompt acceptance table, M5 Max vs GB10, across K in {2, 4, 8} (blocked on hardware: the M5 Max rows need the Apple Silicon host).
• 2-3 longer prompts measured on both hosts (M5 Max rows blocked on hardware).
• Attribution (host vs pairing/prompt) stated with evidence.
• Targeted follow-up issue filed if a CUDA numeric gap is confirmed.
• Table and conclusion documented in docs/benchmark_results.

Technical Considerations

• Refs perf(speculative): CUDA drafter pairing matrix and continuous-batching integration #638, PR perf(speculative): CUDA pairing matrix and burst HOL observability #733, perf: validate B=1 MTP default-on on lower-bandwidth Apple Silicon (M1/M2) #165, fix: per-row position holes break B>1 batched MTP greedy parity after divergent accepts #203.
• The round-cost side of the GB10 regression is tracked separately (perf(cuda/quant): amortizing small-M quantized GEMM for batched decode on Blackwell #725 and its follow-ups); this issue isolates the acceptance-rate axis.