Skip to content

feat(preflight): measure qwen35's think_format, and record why pinned cannot be - #177

Merged
glennneuber merged 1 commit into
mainfrom
feat/qwen35-thinkformat-and-pinned
Aug 17, 2026
Merged

glennneuber merged 1 commit into
mainfrom
feat/qwen35-thinkformat-and-pinned

Conversation

@glennneuber

Copy link
Copy Markdown

Closes the two honest skips on the gfx1151 qwen35 row. One was measurable; the other turned out not to be, and conflating those was the actual problem.

think_format — measured

On 0.32.1-dynres-5d5b7a72: 306 eval tokens, 823 chars of thinking then 517 chars of valid JSON, identical across a 2000 and a 4000 budget — so it terminates well inside the allowance, deterministically. num_predict recorded as 4000 to match this profile's other arches; it is a ceiling, not a measurement, and the headroom exists so a legitimate run never caps.

Worth recording: this arch does not hit the runaway-reasoning trap check_think_format warns about. That gate samples at temperature 0, which is off-policy for thinking and is what stops nemotron-class reasoning terminating. qwen3.8 closed its thinking block at 306 tokens under exactly that sampling.

pinned_budget — not applicable, not unmeasured

visionServerArgs builds gemma4's and nemotron_h_omni's flags from the request options, but the qwen VL family's branch returns a fixed --image-min-tokens and never reads opts. So image_min_tokens/image_max_tokens are inert for this arch.

Verified on the host rather than inferred from the source:

2048x1152 unpinned      -> 2306
2048x1152 pinned @1536  -> 2306     # byte-identical; the pin does nothing

The existing skip read "this arch's probe has not been measured. See README.md to add one" — which would send someone off to measure a value that cannot move. This adds pinned_not_applicable, a per-row reason so the skip states why. Rows without it keep the old message unchanged.

Verification

PASS  think_format [qwen35]: valid JSON after thinking (306 tokens)
skip  pinned_budget [qwen35]: pinned budget does not apply to this arch:
      image_min_tokens/image_max_tokens are arch-gated to gemma4 and
      nemotron_h_omni; qwen35's visionServerArgs branch ignores them.
PASS=18  SKIP=2        VERDICT: PASS

Up from PASS=17 SKIP=3. The one remaining skip is gemma4's pinned probe, which is genuinely unmeasured and untouched. test_verdicts.py 34 green.

🤖 Generated with Claude Code

… cannot be

The two honest skips left on the gfx1151 qwen35 row. One was measurable, one
turned out not to be, and they are different things.

think_format: MEASURED on 0.32.1-dynres-5d5b7a72. 306 eval tokens, 823 chars of
thinking then 517 chars of valid JSON, identical across a 2000 and a 4000 budget,
so it terminates well inside the allowance and deterministically. num_predict is
recorded as 4000 to match this profile's other arches -- it is a ceiling, not a
measurement.

Worth noting this arch does NOT hit the runaway-reasoning trap check_think_format
warns about. That gate samples at temperature 0, which is off-policy for thinking
and is what stops nemotron-class reasoning terminating; qwen3.8 closed its
thinking block at 306 tokens under exactly that sampling.

pinned_budget: NOT APPLICABLE, which the harness could not previously express.
visionServerArgs builds gemma4's and nemotron_h_omni's flags from the request
options, but the qwen VL family's branch returns a fixed --image-min-tokens and
never reads opts -- so image_min_tokens/image_max_tokens are inert. Verified on
the host rather than inferred from the source: 2048x1152 delivers 2306 unpinned
and 2306 pinned at 1536, byte-identical, the pin does nothing.

The existing skip said "this arch's probe has not been measured. See README.md
to add one", which would send someone to measure a value that cannot move. Adds
`pinned_not_applicable`, a per-row reason that makes the skip state why instead.
Rows without it keep the old message unchanged.

Full run against the live host: PASS=18 SKIP=2, up from PASS=17 SKIP=3. The two
remaining skips are gemma4's pinned probe (genuinely unmeasured, unchanged) and
this one, now explained. test_verdicts 34 green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@glennneuber
glennneuber merged commit 8f0f76a into main Aug 17, 2026
1 check passed
@glennneuber
glennneuber deleted the feat/qwen35-thinkformat-and-pinned branch August 17, 2026 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant