Repository navigation
feat(preflight): measure qwen35's think_format, and record why pinned cannot be - #177
Merged
Merged
Conversation
… cannot be The two honest skips left on the gfx1151 qwen35 row. One was measurable, one turned out not to be, and they are different things. think_format: MEASURED on 0.32.1-dynres-5d5b7a72. 306 eval tokens, 823 chars of thinking then 517 chars of valid JSON, identical across a 2000 and a 4000 budget, so it terminates well inside the allowance and deterministically. num_predict is recorded as 4000 to match this profile's other arches -- it is a ceiling, not a measurement. Worth noting this arch does NOT hit the runaway-reasoning trap check_think_format warns about. That gate samples at temperature 0, which is off-policy for thinking and is what stops nemotron-class reasoning terminating; qwen3.8 closed its thinking block at 306 tokens under exactly that sampling. pinned_budget: NOT APPLICABLE, which the harness could not previously express. visionServerArgs builds gemma4's and nemotron_h_omni's flags from the request options, but the qwen VL family's branch returns a fixed --image-min-tokens and never reads opts -- so image_min_tokens/image_max_tokens are inert. Verified on the host rather than inferred from the source: 2048x1152 delivers 2306 unpinned and 2306 pinned at 1536, byte-identical, the pin does nothing. The existing skip said "this arch's probe has not been measured. See README.md to add one", which would send someone to measure a value that cannot move. Adds `pinned_not_applicable`, a per-row reason that makes the skip state why instead. Rows without it keep the old message unchanged. Full run against the live host: PASS=18 SKIP=2, up from PASS=17 SKIP=3. The two remaining skips are gemma4's pinned probe (genuinely unmeasured, unchanged) and this one, now explained. test_verdicts 34 green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the two honest skips on the gfx1151 qwen35 row. One was measurable; the other turned out not to be, and conflating those was the actual problem.
think_format— measuredOn
0.32.1-dynres-5d5b7a72: 306 eval tokens, 823 chars of thinking then 517 chars of valid JSON, identical across a 2000 and a 4000 budget — so it terminates well inside the allowance, deterministically.num_predictrecorded as 4000 to match this profile's other arches; it is a ceiling, not a measurement, and the headroom exists so a legitimate run never caps.Worth recording: this arch does not hit the runaway-reasoning trap
check_think_formatwarns about. That gate samples at temperature 0, which is off-policy for thinking and is what stops nemotron-class reasoning terminating. qwen3.8 closed its thinking block at 306 tokens under exactly that sampling.pinned_budget— not applicable, not unmeasuredvisionServerArgsbuilds gemma4's and nemotron_h_omni's flags from the request options, but the qwen VL family's branch returns a fixed--image-min-tokensand never readsopts. Soimage_min_tokens/image_max_tokensare inert for this arch.Verified on the host rather than inferred from the source:
The existing skip read "this arch's probe has not been measured. See README.md to add one" — which would send someone off to measure a value that cannot move. This adds
pinned_not_applicable, a per-row reason so the skip states why. Rows without it keep the old message unchanged.Verification
Up from PASS=17 SKIP=3. The one remaining skip is gemma4's pinned probe, which is genuinely unmeasured and untouched.
test_verdicts.py34 green.🤖 Generated with Claude Code