Skip to content

fix(speculative): the MTP round-loop diagnostics report the requested block width, not the one actually used #1206

Closed

Description

@inureyes

Problem

--draft-block-size can be overridden by the adaptive controller, and nothing in the output says so.

Measured on M5 Max, gemma-4-12b-it-4bit with the gemma-4-12b-it-assistant-4bit drafter:

$ mlxcel generate --draft-kind mtp --draft-block-size 6 ... -n 120 --temp 0
MTP round-loop diagnostics block_size=6 ... rounds=44 proposed_tokens=130
                                acceptance_rate=0.5846153846153846
                                emitted_per_verify=2.7045454545454546

The same command with --draft-block-size 4 produces the same run: same round count, same acceptance to four decimals, same emitted-per-verify. proposed_tokens / rounds is 2.95, which is the three proposals a width of 4 makes, not the five a width of 6 would.

So the request was overridden and the diagnostics still report block_size=6.

Why it happens

effective_mtp_block_size pins a drafter at its configured depth until recent accept lengths show that prefix is usually fully accepted. That is deliberate and is tracked separately. What is wrong here is only the reporting: MtpRoundDiagnostics::log emits self.block_size, the requested width, while the width the round loop actually drafted at is computed per round and discarded.

Drafters that set prefer_requested_block_size (Qwen 3.5 MTP) bypass the controller, so their diagnostics happen to be accurate. The Gemma 4 assistant drafter does not, so its diagnostics are not.

Why it matters more than a cosmetic mismatch

It makes the flag untunable and it makes benchmark records wrong.

An operator sweeping --draft-block-size sees distinct configurations reporting distinct widths while several of them are the same run. Two entries in a benchmark table that differ only in a label are worse than one entry, because they read as evidence that the width does not matter.

This cost real time on #1204: block 6 and block 4 producing byte-identical results looked like a bug in the sweep harness until the proposal count gave it away.

Acceptance criteria

  • The round-loop diagnostics carry the effective width, and carry the requested width too when they differ, so the override is visible rather than inferred.
  • A drafter whose configured depth overrides the request logs that once, at the level an operator sees by default, rather than only in a field.
  • Pinned by a test: a request above the configured depth on a non-prefer_requested_block_size drafter must not report the requested width as the width used.

Refs #1204

Activity

  1. added
    type:bugBug fixes, error corrections, or issue resolutions
    area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)
    on Aug 18, 2026
  2. added a commit that references this issue on Aug 18, 2026
    c50ae8d
  3. added a commit that references this issue on Aug 20, 2026
    5dfcb39
  4. self-assigned this
    on Aug 31, 2026
  5. added and removed on Sep 6, 2026
  6. added a commit that references this issue on Oct 27, 2026
    e4c25ee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)priority:mediumMedium prioritystatus:doneCompletedtype:bugBug fixes, error corrections, or issue resolutions

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions