Repository navigation
docs(speculative): cross-check Gemma 4 MTP acceptance on M5 Max (#737) - #744
Merged
Merged
Conversation
Add the Apple Silicon acceptance-rate rows that #737's criteria left blocked on hardware. On the identical checkpoints and the identical 14-token prompt, M5 Max accepts at the same rate as GB10 (35.0% at K=4/8, mean 1.05, to three digits), so the low GB10 acceptance is a prompt property, not a CUDA numeric artifact. Three longer, realistic prompts recover 52-88% acceptance on the same 4-bit pairing, inside the 70-87% band third parties report, with M5 Max speedup up to 2.43x. The 0.52x vs 1.72x host gap on identical 35% acceptance is the qmv verify round cost (#725), not acceptance. No CUDA-numeric follow-up is filed: the same-prompt data refutes a host gap on acceptance rather than confirming one. Cross-link the GB10 matrix and the prior M5 Max doc to the new result.
inureyes
added
type:performance
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #737.
Adds the Apple Silicon acceptance-rate rows that #737's acceptance criteria left blocked on hardware. This host is an M5 Max, so the block is lifted. Measured with the identical on-disk checkpoints (
gemma-4-12b-it-4bit+gemma-4-12b-it-assistant-4bit) and the identical 14-tokenDEFAULT_PROMPTas the GB10 run inspeculative-pairing-gb10-2026-07-10.md.Headline
The GB10 acceptance was not suspiciously low because of the CUDA backend. On the same prompt, M5 Max accepts at the same rate as GB10:
K=4/8 match to three digits; K=2 is within 2.9 points (noise). What differs between hosts is what that acceptance buys: M5 Max turns 35% into a 1.72× speedup, GB10 into a 0.52× regression, because the GB10
[1,K]verify runs as K narrow qmv passes (#725), not because of acceptance.Longer prompts recover the third-party range
Three longer, realistic prompts on M5 Max lift K=4 acceptance to 52.7–74.6% (P3 K=2 hits 88.2%), inside the 70–87% band third parties report, with speedup up to 2.43×. So the 35% was an artifact of an adversarially short 14-token prompt, not the 4-bit drafter or the backend. The adaptive block controller also stops collapsing K=8→K=4 once acceptance is high enough (P1 K=8 mean 2.12 vs K=4 1.91).
Attribution
Follow-up
No targeted CUDA-numeric follow-up issue is filed: #737's condition was to file one only if a CUDA numeric gap is confirmed, and the data refutes a host gap on acceptance. The open CUDA work is the verify round cost, already tracked by #725 and #735.
Changes
docs/benchmark_results/speculative-acceptance-m5max-2026-07-11.md(full tables + methodology + attribution).speculative-pairing-gb10-2026-07-10.md(bench(speculative): acceptance-rate cross-check for the Gemma 4 MTP pairing on Apple Silicon #737 follow-up) andgemma4-mtp-speculative-decoding.md(the prior 1.87× doc that predated acceptance reporting).Validation
Real-checkpoint runs on Apple M5 Max (128 GB),
speculative_benchbuilt--release --features metal,accelerate, greedy,--max-tokens 128. 16 rows total (4 same-prompt + 12 longer-prompt). Docs-only change; no code contract surface touched.