vision-suite: MTP draft-depth as a campaign arm, a low start rung, and two silent-failure fixes

[glennneuber]

Five harness changes, split out of the v0.32.15 sync work (#208) because they are independent of it — disjoint files, and draft_num_predict has been in api/types.go since upstream v0.32.7, so none of this depends on the merge.

Each was found by hitting it during a real campaign, and each failure mode was silent.

ae07fe1d — --prefix for summarize_contract_matrix.py

The only summarizer without it. It called resolve_tag(DIR, model, think) with the prefix argument omitted, so a TAG_PREFIX campaign found no score file and rendered every arm as — — indistinguishable from a model that answered nothing. resolve_tag already took prefix; this threads the flag through.

73d948a4 — CTX_START knob, and escalate on a context-overflow rejection

CTX_START / CTX_START_THINKON set the first rung each think mode runs at; CTX_LADDER still supplies the rungs escalation climbs to. Both default to 16384, so unset behaviour is byte-identical. They are separate because think-on derives num_predict as num_ctx - CTX_PROMPT_RESERVE — an 8192 start would give it num_predict = 0.

A low think-off start is worth real wall clock: a 16384 window allocates KV most think-off cells never reach, and a larger window decodes slower.

What makes it safe is the second half. Escalation previously keyed on eval_count >= num_predict alone. A context-overflow 400 carries no eval_count at all, so the cell broke out of the loop, was recorded as a failure at the low rung, and was never retried — reading as "this model cannot do this test" when the truth was "the harness asked in too small a window". At 8192 with num_predict 2200 that is the difference between gemma4 (3765 + 2200, fits) and nemotron3 (6203 + 2200, rejected).

Unrelated failures deliberately do not escalate — a 500 is not a window problem, and climbing would burn KV to fail again. Unit-tested against capped / both overflow message forms / an unrelated 500 / a non-dict block before use.

7bef3253 + 704db6a8 — DRAFT_NUM_PREDICT passthrough, and actually recording it

The suite honoured twelve request-shaping env vars and draft depth was not one, so MTP could not be a campaign arm at all — the only way to vary it was a bespoke loop outside the harness, which run_engine_compare.sh's own header exists to prevent.

Sent whenever set, including 0: server/routes.go:154 zeroes draft_num_predict unless the model carries a DraftPath or the option was explicitly set, so the api/types.go default of 4 never reaches an embedded-MTP model, and 0 is the only way to state disabled rather than unspecified.

The second commit is a fix to the first, and worth flagging honestly: 7bef3253 stamped _draft_num_predict on the response dict, but vision_suite.py copies only selected fields into the score block, so the stamp landed one layer short of the file. I verified the wrong artefact and called it proven. An MTP on/off A/B then produced four score files that all read draft=[], carrying the arm only in their tag — the exact provenance gap ADR 0012 rule 1 exists to prevent. 704db6a8 records req_draft_num_predict in the block, following the req_image_*_tokens contract immediately above it, and tests not in (None, "") rather than truthiness because the IMAGE_* loop's if os.environ.get(env) idiom would silently drop the 0 off-arm.

3c5a9d28 — docs

All of the above in the README, plus a correction: the context-ladder table still listed five rungs stopping at 65536. 131072 was added 2026-08-19 and a qwen3.6 think-on cell reached it on 2026-08-21.

No ADR change. These apply existing decisions rather than making new ones — ADR 0012 rule 1 already requires provenance to come from score files, and req_draft_num_predict is that rule applied to a new field, not a new rule.

Measured with it

MTP on qwen3.8:27b-q4_K_M (arch qwen35, nextn_predict_layers=1), full vision suite, clean single-build provenance:

arm	gen tok/s	req/h	scene IoU	eval_count
think=false, MTP off	70	390	0.977	544
think=false, MTP on (4)	140	667	0.977	544
think=on, MTP off	70	205	0.975	1117
think=on, MTP on (4)	135	335	1.000	1147

Decode roughly doubles, prefill is untouched, and think=false quality is bit-identical at the same eval_count — same output, half the time. A depth sweep (0/2/4/6/8) put the knee at 4, which is already the built-in default.

[glennneuber]

Reviewing as consolidator. All five are real, every failure mode named is genuinely silent, and the CTX_START half is safe for the reason you give rather than by luck.

CTX_START and the ladder guard

I wrote the exit-2 guard (SPEC H4a), so this is the interaction I checked first. It composes correctly: the guard refuses when CTX_MAX leaves no rung above the think-on start, and CTX_START_THINKON moves that start rather than removing it — so a low start makes the guard more likely to fire usefully, not less. Keeping the two knobs separate because think-on derives num_predict = num_ctx − CTX_PROMPT_RESERVE, and an 8192 start would yield num_predict = 0, is exactly the sort of thing that would otherwise be discovered as a mystifying empty cell.

The second half is what makes it safe, and it closes a hole the guard never covered. A context-overflow 400 carries no eval_count, so escalation keyed on eval_count >= num_predict broke out and recorded a failure at the low rung — "this model cannot do this test" when the truth was "asked in too small a window". That is the same class as #202's capped-cell-rendered-as-a-result: a harness limit published as a model property. My guard stopped a fixed ceiling; it did nothing about a rejection that never produced a count to compare.

Not escalating on unrelated failures is the right restraint — a 500 is not a window problem, and climbing burns KV to fail again.

The self-correction in 704db6a8 is the most useful paragraph here

7bef3253 stamped _draft_num_predict on the response dict, but vision_suite.py copies only selected fields into the score block, so the stamp landed one layer short of the file. I verified the wrong artefact and called it proven.

Recording that is worth more than the fix. It is the same shape as several failures this week — a value that exists somewhere real but not where the reader looks — and "I verified the wrong artefact" is the sentence that stops the next person doing it. The consequence being four score files reading draft=[] with the arm surviving only in the tag makes the ADR 0012 rule 1 point concrete rather than abstract.

Testing not in (None, "") rather than truthiness is the correct fix for exactly the reason given: the neighbouring IMAGE_* loop's if os.environ.get(env) idiom would drop the 0 off-arm, and 0 is the only way to say disabled rather than unspecified given server/routes.go:154. Worth a comment at the IMAGE_* loop too, since it now differs from its neighbour for a non-obvious reason and looks like an inconsistency.

Smaller notes

--prefix on summarize_contract_matrix.py rendering every arm as — is the worst possible failure shape — indistinguishable from a model that answered nothing — and it is the last summarizer without the flag, so this closes the class rather than an instance.

The README rung-table correction (five rungs stopping at 65536, when 131072 was added 2026-08-19 and a qwen3.6 think-on cell reached it on 2026-08-21) is small but load-bearing: a documented ladder that stops short of where cells actually land invites exactly the fixed-ceiling reasoning H4a exists to prevent.