fix(speculative): Gemma 4 MTP breaks byte-identity on generation 15+, and its gate returns unconditionally true

Status: measured and confirmed on 2026-08-18

This issue was filed as "the mechanism should carry over, nobody has run the comparison". The comparison has now been run on an Apple M5 Max and the mechanism does carry over. Gemma 4 MTP is not byte-identical to classic decode on generation 15+ hardware under the default kernel selection, and disabling qmv_wide restores it exactly.

The gate change is no longer speculative work. What is new, and what makes it a real decision rather than a formality, is the price.

The measurement

Host: Apple M5 Max (Mac17,7), macOS 26.6.1, mlxcel at 07ab8612, [profile.test-fast], --features metal,accelerate. Target models/gemma-4-12b-it-4bit, drafter models/gemma-4-12b-it-assistant-4bit, --draft-kind mtp, temperature 0, 250 tokens, two prompts.

comparison	prose prompt	code prompt
MTP vs classic, `qmv_wide` on (default)	differs at char 51	differs at char 55
MTP vs classic, `qmv_wide` off	byte-identical	byte-identical

Block width does not change the answer: block 2 and block 5 both diverge under the default, at the same character. That is consistent with the mechanism, since use_qmv_wide splits at M >= 2 and a block of 2 is already on the wrong side of it.

Classic decode is deterministic run to run here (three runs, identical), so the divergence is systematic rather than the f16 jitter class.

One methodological trap, because it cost us a wrong answer first

Toggling MLXCEL_QMV_WIDE changes classic decode too, so the comparison has to hold the kernel fixed on both sides. Classic's multi-token prefill is itself an M >= 2 forward, so classic(wide on) and classic(wide off) differ from each other at char 51 on the prose prompt. Comparing MTP(wide off) against classic(wide on) conflates the prefill effect with the verify effect and reports a divergence that says nothing about the contract.

Second trap, for whoever reproduces this: the CLI echoes the prompt without a trailing newline and then appends Loading MTP drafter from "..." to that same line, so a naive line filter leaves the banner in the compared text and every MTP run "differs from classic" at the character where the banner starts. Extract the generated text by splitting on the prompt echo and dropping the two drafter banner lines.

What it costs

This is the number the decision turns on. Same host, code prompt, block 5, 300 tokens, three samples per arm alternated:

kernel	decode	vs classic (43.5 tok/s)
`qmv_wide` on (fast, not exact)	121.50 tok/s	2.79x
`qmv_wide` off (exact)	93.19 tok/s	2.14x

Spreads were 0.04 and 0.37 tok/s, so the gap is not measurement noise: exactness costs 23.3% on this pairing. For comparison, the Qwen 3.5 arms pay about 17 to 20% for the same guarantee, and that cost is already being paid in production today because their probe runs.

So the change is not free, and it is not a bug fix that only removes a wrong claim. It converts an advertised 2.80x into an honest 2.14x on this prompt, or it keeps 2.80x and qualifies the claim. Both are defensible; silently advertising byte-identity while not measuring it is not.

What to change

Route the Gemma 4 arms through mtp_exactness_gate instead of returning true. In src/server/batch/speculative_burst.rs:

LoadedModel::Gemma4(_) | LoadedModel::Gemma4VLM(_) | LoadedModel::Gemma4Unified(_) => true,

The probe machinery in src/models/speculative_exactness.rs is already family-agnostic: it compares a T = K verify block against the single-token chain, and on failure retries with qmv_wide disabled before declining. Nothing in it is Qwen-specific.

Decide what a failing probe should do for Gemma. Qwen's answer is "disable the kernel and keep the contract", which is why it pays 17 to 20%. Given the 23.3% measured here, the alternative worth putting on the table is to keep the fast kernel and state plainly that Gemma 4 MTP is not byte-identical on generation 15+, the way MLXCEL_MTP_ALLOW_INEXACT already allows. This issue should not pick that by default; it should make the choice explicit and measured.
Qualify docs/benchmarks.md. The MTP section overview still claims byte-identity with no qualification, and the Gemma 4 rows inherit it. The Qwen rows were qualified by fix(speculative): MTP temperature-0 byte-identity fails on Apple GPU generation 15+ and at block widths above the qmv batch limit #1186; these were not.

Not covered here

The 31B and Unified pairings, and any generation 13 or 14 host. The measurement above is one target, one drafter, one machine. It is enough to settle that the exposure is real; it is not enough to characterise the cost across the family.

Measured: Gemma 4 projection shapes on M5 Max

Filling the "not yet measured" cell, using the cheap first step this issue names. Op-level only; this is not a Gemma 4 model-level chain run.

Apple M5 Max 128 GB, macOS 26.6.1, architecture generation 17, main at e3f322e5. Shapes read from local checkpoints: gemma-4-31b-it-4bit (hidden 5376, intermediate 21504), gemma-4-12b-it-4bit (3840, 15360), both affine 4-bit group 64, vocab 262144.

Affine 4-bit, the shipped configuration

shape	T=2	T=3	T=4	T=8
31B MLP 5376 -> 21504	DIVERGES (2471 / 43008 bytes)	DIVERGES (2471)	DIVERGES (2471)	DIVERGES (2471)
31B attn 5376 -> 5376	DIVERGES (539 / 10752)	DIVERGES (539)	DIVERGES (539)	DIVERGES (539)
12B MLP 3840 -> 15360	DIVERGES (705 / 30720)	DIVERGES (1612)	DIVERGES (1612)	not run

The mechanism carries to Gemma 4. Every Gemma 4 projection shape breaks at T = 2, and the MTP default block width is 4, so the shipped configuration is inside the broken range. The unconditional true arm is wrong on generation 15 and newer.

Both thresholds are visible on one machine

Sweeping the 31B attention shape and reporting the magnitude rather than just equal/diverges shows two steps, not one:

   T=1   equal
   T=2   DIVERGES (worst  539/10752 bytes)
   T=3   DIVERGES (worst  539/10752)
   T=4   DIVERGES (worst  539/10752)
   T=5   DIVERGES (worst  539/10752)
   T=6   DIVERGES (worst  539/10752)
   T=8   DIVERGES (worst  539/10752)
   T=10  DIVERGES (worst 1693/10752)   <- second step
   T=11  DIVERGES (worst 1693/10752)
   T=12  DIVERGES (worst 1693/10752)
   T=16  DIVERGES (worst 1693/10752)

The T = 2 step is use_qmv_wide. The T = 10 step is get_qmv_batch_limit crossing into the matrix-matrix kernel. Note the limit here is 10, not the 12 quoted in #1186: that 12 is the arch_size == 'd' branch, which is the Ultra. A Max lands in the default branch, which returns 10 for both operands above 4096 on a generation that is not 13 or 14.

The mxfp4 claim does not hold on generation 17

#1186 infers from use_qmv_wide(mode, d) = mode != "affine" || arch_gen >= 15 that non-affine modes diverge on every generation, and measured mxfp4 diverging on M1 Ultra (3, 4, 2 of 10240 bytes at T = 3). On M5 Max the opposite is true. Measured at the same 5120 -> 5120 shape #1186 used, so this is directly comparable:

mode	T=1	T=2	T=3	T=4
mxfp4 group 32	equal	equal	equal	equal
affine 4-bit group 64 (control)	equal	DIVERGES (501 / 10240)	DIVERGES (501)	DIVERGES (501)
nvfp4 group 16	equal	equal	equal	DIVERGES (1 / 10752)

The affine control diverging in the same run is what makes this a real result rather than a broken harness.

So the two modes behave oppositely across the two generations: M1 Ultra is affine-clean and mxfp4-broken, M5 Max is affine-broken and mxfp4-clean (nvfp4 marginal, one byte at T = 4). Whatever the full dispatch story is, "mxfp4 diverges everywhere" is not it, and #1186's Impact section should be narrowed accordingly.

This also strengthens the case for #1189's approach over a hardware table. Two generations, two quantization modes, and the answer inverts.

What this does not establish

No Gemma 4 model-level block-vs-chain run. Op-level divergence is necessary but not sufficient for a token flip; the Qwen 3.5 case needed 241 tokens to produce 2 flips. A Gemma 4 chain test is still the thing that proves user-visible impact.
No M1 Ultra control at Gemma 4 shapes. Predicted equal below T = 12 by the same mechanism, unmeasured.
Nothing here touches the structural work this issue identifies as the real cost, namely that Gemma 4's verify path drives model-owned seq_id-keyed cache slots and needs a throwaway slot for the probe.

Reproduction was a scratch harness at these shapes rather than a committed test, since #1189 already carries tests/metal_block_vs_chain_op_parity.rs and adding Gemma 4 shapes to it belongs on that PR or on this issue's own change.

M5 Max confirms the exposure: all four Gemma 4 rows diverge at the production block width

Ran the diagnostic from #1192 on an M5 Max 128 GB (macOS 26.6.1, Xcode 26.5.0), branch test/issue-1188-gemma4-op-parity-shapes at 0e1f9f10. Build 4 minutes, test 0.75 s, no checkpoint.

`T = 3`

gemma4-12b  q_proj    3840 -> 4096  (affine 4-bit g64) : DIVERGES (3147, 3231, 3356 of 8192)
gemma4-12b  kv_proj   3840 -> 2048  (affine 4-bit g64) : DIVERGES (1580, 1617, 1531 of 4096)
gemma4-12b  gate/up   3840 -> 15360 (affine 8-bit g64) : DIVERGES (24, 22, 30 of 30720)
gemma4-12b  down      15360 -> 3840 (affine 8-bit g64) : DIVERGES (8, 12, 9 of 7680)
qwen3.8-27b gate/up   5120 -> 17408 (affine 4-bit g64) : DIVERGES (14002, 14264, 13650 of 34816)
qwen3.8-27b o_proj    5120 -> 5120  (affine 4-bit g64) : DIVERGES (3853, 3951, 4099 of 10240)
fast_rms_norm / fast_rope / SDPA                       : equal

Every Gemma 4 projection diverges, and the Qwen 3.8 affine rows diverge alongside them, which is the expected signature: use_qmv_wide keys on arch_gen >= 15, not on the model family. The Gemma 4 arm of mtp_capable_target is admitting a host where its verify block is not byte-identical to the single-token chain. The model-level probe this issue asks for is justified.

For contrast, the same commit on an M1 Ultra (generation 13) reports all four Gemma 4 rows equal at T = 3.

The 18/12 cliff does not appear here at all

    T   qwen q4 5120->5120   qwen q4 5120->17408   gemma q4 3840->4096   gemma q8 3840->15360   dense f16
    1   equal                equal                 equal                 equal                  equal
    2   DIVERGES             DIVERGES              DIVERGES              DIVERGES               DIVERGES
   ...  every width from 3 to 32: DIVERGES

On an M1 Ultra the same sweep shows Gemma 4's attention shapes holding to 17 and its MLP shapes breaking at 12, the two get_qmv_batch_limit buckets. Here use_qmv_wide catches M >= 2 first, so the batch limit is never reached and never observed. The per-shape cliff is a generation 13/14 phenomenon; from generation 15 there is a single threshold at 2.

Amplitude varies by three orders of magnitude, and the gate must not care

The 8-bit MLP rows move 24 of 30720 bytes and 8 of 7680, while the 4-bit attention rows move about 38% of theirs. Byte-identity is broken in every one of them.

This is the argument for the gate deciding whether the arms differ rather than how much. A tolerance-based gate tuned on the loud rows would pass the quiet ones, and a quiet row still flips a near-tie argmax eventually: it is the same last-ulp mechanism, just rarer per position.

It also has a direct consequence for the probe design in #1186, filed separately below.

Follow-up on this issue

The measurement half of the acceptance criteria is now done for generation 17 and for the M1 Ultra control. What remains is the gate change: giving the Gemma 4 arms the same runtime probe, which needs a throwaway model-owned sequence slot because forward_with_speculative_sinks keys its cache on seq_id rather than taking a cache vector.

test(speculative): extend the block-vs-chain diagnostic to Gemma 4 sh…

Can the exactness cost be made conditional? Measured: no

Before wiring the gate, the obvious question is whether the 23.3% has to be paid unconditionally. The contract only breaks when a kernel difference flips an argmax, and a flip needs the top two logits to sit inside that difference. If near-ties were rare relative to the kernel error, the fast kernel could run every round and only ambiguous positions would be recomputed exactly.

Instrumented and measured. It does not work, and the numbers say why.

The two distributions

MLXCEL_MTP_GAP_LOG=1 (added in e757aac1) prints position, token, top1, top2, gap per verify position. Running a generation under each kernel selection and aligning the logs while the token streams still agree gives both halves. gemma-4-12b-it-4bit with the 4-bit assistant, block 5, 15 prompts, 444 aligned positions:

quantity	p50	p90	p99	max
kernel delta on the winning logit	0.1250	0.3125	0.7500	3.0000

The delta is exactly zero at only 145 of 444 positions. The top-two gap has median 5.875, so typically the gap is about forty times the error, which is why most tokens agree. The problem is the tails: the gap is under 3.0 at 30.6% of positions.

The break-even

A guard recomputes the block exactly whenever any position's gap falls under t. Always-narrow costs 1.180x the fast verify, so a guard has to come in under that.

guard t	covers delta up to	delta coverage	K=5 round ambiguous	expected cost
0.25	0.125	70.9%	12.9%	1.153
0.50	0.250	89.9%	27.2%	1.321
1.00	0.500	98.0%	48.6%	1.573
3.00	1.500	99.8%	83.9%	1.990
6.00	3.000	100%	97.0%	2.145

Sound coverage costs nearly twice what simply running narrow costs. The one row that beats always-narrow, t = 0.25, misses three potential flips in ten.

Break-even needs the per-round ambiguity below 15.3%, which needs the per-position rate below 3.3%. Reaching that means guarding only delta <= 0.125, and 29% of measured deltas exceed it.

Why it fails

The kernel error and the top-two gap are the same order of magnitude. There is no threshold that is both cheap and safe, because the two distributions overlap where it matters rather than being separated by orders of magnitude.

A first pass with a thin sample looked promising, and that is worth flagging for anyone who repeats this: over a 16-position common prefix the max delta read 0.3125, which would have put the guard comfortably in the black. Across 444 positions the max is 3.0. The tail is the whole answer here, so a bound taken from a short run will mislead.

What this means for the decision

The choice in this issue stands as it was, without a third option: either pay the 23.3% and keep the contract, or keep the fast kernel and say plainly that Gemma 4 MTP is not byte-identical on generation 15+. There is no cheap conditional path between them.

Measured on Apple M5 Max (Mac17,7), macOS 26.6.1, mlxcel at e757aac1.

What the 23.3% actually buys, measured at the right shape

The decision here has been stuck between "pay 23.3% and keep byte-identity" and "keep the speed and admit it is not byte-identical", with nothing to say how much the second option costs a user. That is a gap in tooling rather than in judgement, so I built the missing measure: examples/logit_trace plus scripts/compare_logit_traces.py (PR #1215 branch, commit bd97da0c). Teacher-forced, one trace per configuration, and it reports disagreement split by how decided the reference was.

The answer

gemma-4-12b-it-4bit, MLXCEL_QMV_WIDE off as reference against on as candidate, forward width 5 behind 512 tokens of context, 300 positions:


top-1 disagreement, all positions	4.0%
top-1 disagreement, decided positions (reference top-two gap >= 2.0)	0.585%, 1 of 171
where the reference's choice landed when they disagreed	rank 2 in 75%, rank 3 in 17%, beyond top-8 once
logit delta on the reference's own choice	p50 0.125, p90 0.500, max 1.50

Split by decidedness, the disagreement is almost entirely where the model had no preference:

reference top-two gap	positions	disagreed
0 to 0.5	41	21.95%
0.5 to 1	37	5.41%
1 to 2	51	0.00%
2 to 5	71	1.41%
5 to 10	66	0.00%
10 and up	34	0.00%

So the trade is 23.3% of throughput against a 0.585% disagreement rate on positions the model had decided, three quarters of which are it picking its own runner-up. That is a materially different-looking decision from "23.3% for byte-identity", and it is the number this issue was missing.

One methodological point, because it inverts the answer

The forward width selects which kernel is measured, so it selects the answer. The same comparison reads:

forward width	top-1 disagreement
8	20.6%
16	19.5%
32	0.0%
256	0.0%

use_qmv_wide splits at M >= 2 and the batch limit sends larger M to a matrix-matrix kernel both arms share, so anything traced at prefill width reports no difference at all and anything traced at a bare narrow width reports a catastrophe. Neither is the shape a verify runs at, which is a narrow forward behind a long context. The harness takes a prefill argument for exactly that reason.

It also means the exposure is bounded in a useful way: prefill is unaffected, and the break lives only in the small-M window the verify block sits in.

What I am not doing

Picking the default. The measurement narrows the question, it does not settle it: 0.585% on decided positions is small but it is not zero, and one of the twelve disagreements put the reference's choice outside the candidate's top eight. Whether that is worth 23.3% is a product call.

Measured on Apple M5 Max (Mac17,7), macOS 26.6.1.

docs: current speculative-decoding numbers, with the prompt they were…

feat(speculative): report the top-two logit gap at each verify position

docs: current speculative-decoding numbers, with the prompt they were…

docs(bench): record the M1 Ultra speculative decoding numbers

feat(bench): measure speculative decoding properly, and record what i…

fix(speculative): probe Gemma 4 MTP exactness instead of assuming it (#…

docs(make): record the TF32 pin at the verify-test-cuda gate definiti…

fix(tests): seed synthetic model fixtures from a sorted key walk (#1266)

Navigation Menu

Repository navigation

fix(speculative): Gemma 4 MTP breaks byte-identity on generation 15+, and its gate returns unconditionally true #1188

Status: measured and confirmed on 2026-08-18

The measurement

One methodological trap, because it cost us a wrong answer first

What it costs

What to change

Not covered here

Measured: Gemma 4 projection shapes on M5 Max

Affine 4-bit, the shipped configuration

Both thresholds are visible on one machine

The mxfp4 claim does not hold on generation 17

What this does not establish

M5 Max confirms the exposure: all four Gemma 4 rows diverge at the production block width

`T = 3`

The 18/12 cliff does not appear here at all

Amplitude varies by three orders of magnitude, and the gate must not care

Follow-up on this issue

Can the exactness cost be made conditional? Measured: no

The two distributions

The break-even

Why it fails

What this means for the decision

What the 23.3% actually buys, measured at the right shape

The answer

One methodological point, because it inverts the answer

What I am not doing

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fix(speculative): Gemma 4 MTP breaks byte-identity on generation 15+, and its gate returns unconditionally true #1188

Description

Status: measured and confirmed on 2026-08-18

The measurement

One methodological trap, because it cost us a wrong answer first

What it costs

What to change

Not covered here

Activity

inureyes commented on Aug 16, 2026

Measured: Gemma 4 projection shapes on M5 Max

Affine 4-bit, the shipped configuration

Both thresholds are visible on one machine

The mxfp4 claim does not hold on generation 17

What this does not establish

inureyes commented on Aug 16, 2026

M5 Max confirms the exposure: all four Gemma 4 rows diverge at the production block width

T = 3

The 18/12 cliff does not appear here at all

Amplitude varies by three orders of magnitude, and the gate must not care

Follow-up on this issue

inureyes commented on Aug 18, 2026

Can the exactness cost be made conditional? Measured: no

The two distributions

The break-even

Why it fails

What this means for the decision

inureyes commented on Aug 18, 2026

What the 23.3% actually buys, measured at the right shape

The answer

One methodological point, because it inverts the answer

What I am not doing

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`T = 3`