Repository navigation
feat(agent): four more models drive every surface, and the engine that serves them joins the record - #522
Merged
Conversation
`nemotron-3.5-lightning:30b` and `muse-glimmer:latest` each clear all six agent
surfaces five consecutive times. Both needed the shipped 90-second turn limit
raised for one cell and neither needed it for the other five, so both carry a
per-model limit rather than moving the default.
nemotron-3.5-lightning:30b — 30/30 with turnTimeoutMs at 150000. Its five agent
surfaces clear 90 seconds untouched (4s, 20s, 57s, 38s, 9s); its plan turn
scored 4/5 there, losing the first run after a cold load to a model-timeout with
zero tools invoked while the four passes landed within three seconds of each
other at 67 to 69.
The entry records what the fix cost as well as what it bought: the plan turn's
median rose from 67 seconds to 144 when the limit rose, both ending
model-stopped. The model is not being cut off at either setting — it spends the
budget it is given, so the cell now costs twice what it did and is won rather
than lost. A reader choosing it for plan work should budget from the 144.
muse-glimmer:latest — 30/30 with three settings, the widest set any model here
carries, because it lost three cells to three different failures:
- plan 0/5, every loss a model-timeout at exactly 90 seconds with an empty
ledger and no tool invoked. Reasoning suppressed on the plan turn, which is
qwen3.5:4b's measured case; the same five runs then finish in 16 to 21s.
- investigate 4/5 and assess 4/5, both single losses model-timeouts at 159s
and 160s against passes at 102-116 and 141-163. A 150-second turn: its
passing assessments call eleven tools and take two and a half minutes.
- assess still lost one run once the clock was not the constraint, to
model-stopped with no-report at 105s having called fourteen tools. A second
report reminder — the distinction between a model that forgot and one that
stops as its habit.
Each setting was found on the cell it fixes, so each was found under a different
configuration from the one that ships. All six surfaces were therefore re-read
in a single pass under the final three, and that pass is what the numbers report.
One run of it ended model-unavailable; a serving outage is not a verdict, so the
cell was re-read clean rather than scored with a gap in it.
Docs and the pinned tables move with the code: a page per model, the README and
setup counts to fourteen, methodology to 420 runs, and both records pinned by
digest in the resolution table so neither can drift unnoticed.
…e it faster
`qwen3.6:27b` runs all six agent surfaces five consecutive times: 30 of 30. It
arrived at 3/6 and 18/30, and every one of its twelve losses across three cells
was a `model-timeout` — plan 0/5, optimize 1/5, assess 2/5, with nothing lost to
a wrong answer. Two settings close all three:
- no reasoning on the plan turn. All five plan losses were the same run: a
timeout at exactly 90 seconds with no tool invoked, the turn spent thinking
rather than answering. That is qwen3.5:4b's measured case and
muse-glimmer:latest's after it; the same runs then finish in 16 to 24s.
- a 150-second turn. Its passing optimizations take four minutes and its
assessments three and a half, so the shipped 90 seconds is the wrong
statement about this model rather than the model being wrong.
The part worth reading twice: the settings also made the three cells that
ALREADY passed faster. Investigate went 51s to 26s, operate 102s to 68s, analyze
110s to 82s, with no change to their verdicts. A model that is not racing a
limit it cannot meet finishes sooner, and nothing in the shipped configuration
made that visible until the limit moved.
Assess was read twice and both readings are in the record. The first pass gave
4/5 with one `model-timeout` at 252s; a second read of the same cell at the same
setting gave 5/5. The cell is reported from the five consecutive passes, and the
outlier is written into the model's page rather than dropped — this cell sits
closest to its limit of any in the model, and a reader on a slower machine
should expect the loss the second pass did not repeat.
Docs and the pinned tables move with it: a page for the model, the counts to
fifteen and 450 runs, the 150-second list from three models to four, and the
record pinned by digest so it cannot drift unnoticed.
…ges under you An Ollama upgrade landed between one day's measurements and the next, and it moved cells in both directions. The method recorded the machine, the database and the code revision, and said nothing about the server that loads the weights. It does now. `gemma4:12b` is the case that found it. Its investigate cell read 5 of 5 at 9 seconds on 0.32.13. On 0.33.0 — same model file, same digest, same objective, same code — it reads 1 of 5, and four of the five losses spend the whole turn without invoking a single tool before the clock ends them. Three configurations were measured before the engine was suspected, because the obvious suspects were mine: with the per-model settings under test, without them, and with the context bound removed. All three read the same. Neither the settings nor the bound was the difference. The upgrade is not simply a regression, which is why this is a note about versions rather than advice to stay on one. The two models measured on 0.32.13 were re-verified on 0.33.0 at their most marginal cell — the cell whose settings exist — and both hold, faster than before: nemotron-3.5-lightning:30b's plan 5/5 at 93s against 144s, muse-glimmer:latest's assess 5/5 at 137s against 167s. The same upgrade that broke one model improved two. So the figures on these pages are claims about a model ON THE ENGINE NAMED BESIDE THEM, and a reader whose numbers disagree should check their own version before concluding anything about the model.
…easured needing it The product could tell a model to spend nothing on reasoning during a PLAN turn and had no way to say it during an AGENT turn. `gemma4:12b` is the model that needed the second sentence, and it goes from 2/6 at 15/30 to 6/6 at 30/30 with it — three times faster than the 26b at half the size. The illness, from its ledger: its investigate cell read 5/5 at 9 seconds on one serving engine and 1/5 on the next, with four losses spending the entire turn without invoking a single tool before the clock ended them, against passing runs that still finished in 9. Bimodal — it answers at once or thinks until the wall. Nothing that already existed reached it, and that was measured rather than assumed. `turnTimeoutMs: 150000` read 1/5 again with the losses simply longer: a turn spent thinking finds the new wall too. Removing the per-model settings entirely read 3/5, and removing the context bound read 1/5, which is how the serving engine was identified as the difference in the first place. `suppressAgentReasoning` is a SECOND switch rather than a widening of the first, and the tests pin why: `qwen3.5:4b` carries the plan one and won its five agent cells while thinking. One field read for both modes would have moved five cells nobody re-measured. What the settings bought this model is not only the four open cells — every cell got faster, several by most of their length: analyze 102s to 11s, assess 118s to 21s, operate 75s to 14s, investigate 9s to 4s. Assess read 4/5 on the first pass and 5/5 on a second read at the same settings. The obvious suspect for the loss was the call ceiling — its losing run made eleven calls, one under the general ceiling of 12, which is `gemma4:26b`'s measured case exactly. The ledger refutes it: two of the passing runs also made eleven. Lowering the ceiling would have cut runs that pass, and the record says so where the next reader will look.
…ve one back to another Ollama moved from 0.32.13 to 0.33.0 under this work, and it did not move every model the same way. Two hundred and fifty runs over the whole supported set say what it did, and this commit is what follows from them. `qwen3.6:27b` gains a third setting. Its optimize cell read 5/5 at 238 seconds on the old engine and 1/5 on the new one, and the losses carry the thinking signature this branch has now seen four times: one tool called, then the entire turn spent before the clock ends it, at 182 to 299 seconds. Reasoning off on its agent turns takes the cell back at 76 seconds — not merely recovered, four times faster than the runs that were passing before. `qwen3.5:4b` comes out. Its plan cell reads 1/5, and the losses stop in two seconds having written nothing at all. Three fixes were measured and none of them reached it: the plan-statement retry read 1/5 unchanged, turning its reasoning-off switch back on read 0/5 with every loss timing out at exactly 90 seconds, and paying for that with a 150-second turn read 0/4 timing out at 150. The model loses at both ends of the one switch that governs the cell — silent in two seconds, or thinking past any limit it is given — so it is withdrawn rather than shipped with a figure the docs cannot stand behind. Neither is this branch's doing, and that was measured rather than argued. Both cells read the same on a clean origin/main worktree carrying none of these commits, and `qwen3.6:27b` is not in that document at all — it ran there on the compiled defaults and lost anyway. Fifteen models, 450 runs. The withdrawn model leaves the profile document, the pinned resolution table, the README, its own page, the e2e sweep list and the prose that cited it as a measured case; `muse-glimmer:latest` and `gemma4:12b` carry that citation now, having been measured on the same shape.
Three claims in these pages were checked against the document they summarise and none of them held. The README said ten of fifteen models cleared their surfaces at the shipped 90-second turn and four carried a 150-second one of their own. That is fourteen. Five carry it - `gemma4:12b` was left out of the sentence while its own entry records the setting and the rationale behind it. `methodology.md` carried the same omission in stronger words, naming `qwen3.5:9b` as the one model that asks for more time by name while four others already did, and `qwen3.5.md` closed its own line with "the limit stays 90 for every other model". The UI sweep is the third and the worst of them. It drove ten models through the rail at one run per surface and read 57 of 60; sixty runs at six surfaces is ten models, and the sentence has said "all twelve" and then "all fifteen" as models were added without the sweep being run again. Nothing in the number beside it let a reader see that. It says ten now, and says why, because the models added since are measured over HTTP only.
|
cevheri
approved these changes
Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four models clear all six agent surfaces five consecutive times each, one that was shipped comes
out, and both halves belong in one PR because of the third thing that happened between them:
Ollama moved from 0.32.13 to 0.33.0 under this work and did not move every model the same way.
Fifteen models, 450 runs, 450 passes.
The four models
nemotron-3.5-lightning:30b— 30/30, one setting. Its five agent surfaces clear the shipped90-second turn untouched (4s, 20s, 57s, 38s, 9s); its plan turn scored 4/5 there, losing the first
run after a cold load to a
model-timeoutwith zero tools invoked while the four passes landedwithin three seconds of each other at 67 to 69. It carries
turnTimeoutMs: 150000, and the entryrecords what that cost as well as what it bought: the plan turn's median rose from 67 seconds to
144, both ending model-stopped. The model spends the budget it is given, so the cell now costs
twice what it did and is won rather than lost. A reader choosing it for plan work should budget
from the 144.
muse-glimmer:latest— 30/30 with three settings, the widest set any model here carries,because it lost three cells to three different failures: plan 0/5, every loss a timeout at exactly
90 seconds with an empty ledger and no tool invoked — reasoning suppressed on the plan turn, and
the same five runs then finish in 16 to 21s; investigate 4/5 and assess 4/5, both single timeouts
at 159s and 160s against passes at 102–116 and 141–163 — a 150-second turn, since its passing
assessments call eleven tools and take two and a half minutes; and assess still losing one run
once the clock was not the constraint, to model-stopped with
no-reportat 105s having calledfourteen tools — a second report reminder, which is the distinction between a model that forgot
and one that stops as its habit. Each setting was found on the cell it fixes and therefore under a
different configuration from the one that ships, so all six surfaces were re-read in a single pass
under the final three, and that pass is what the numbers report.
qwen3.6:27b— 30/30, and its settings made the cells that already passed faster. It arrivedat 3/6 and 18/30 with every one of its twelve losses a
model-timeout— plan 0/5, optimize 1/5,assess 2/5, nothing lost to a wrong answer. No reasoning on the plan turn and a 150-second turn
close all three. The part worth reading twice is what happened to the three cells that were
already passing: investigate 51s to 26s, operate 102s to 68s, analyze 110s to 82s, with no change
to their verdicts. A model that is not racing a limit it cannot meet finishes sooner, and nothing
in the shipped configuration made that visible until the limit moved.
gemma4:12b— 2/6 and 15/30 to 6/6 and 30/30, on a switch the product did not have. Itcarries three settings and the new one is what did it; the other two are the clock (assess and
operate lost to
model-timeoutwith tools called and work done) and its plan turn, which was 0/5with the thinking signature and reads 5/5 at 2 s with reasoning off. It is three times faster than
the 26b at half the size, and the settings did not merely close the four open cells — every cell
got faster, several by most of their length: analyze 102s to 11s, assess 118s to 21s, operate 75s
to 14s, investigate 9s to 4s.
What was wrong
The product could tell a model to spend nothing on reasoning during a PLAN turn and had no way
to say it during an AGENT turn.
gemma4:12bis the model that needed the second sentence: itsinvestigate cell read 5/5 at 9 seconds on one serving engine and 1/5 on the next, with four losses
spending the entire turn without invoking a single tool before the clock ended them, against
passing runs that still finished in 9. Bimodal — it answers at once or thinks until the wall.
Nothing that already existed reached it, and that was measured rather than assumed:
turnTimeoutMs: 150000read 1/5 again with the losses simply longer, because a turn spentthinking finds the new wall too.
suppressAgentReasoningis a SECOND field rather than a wideningof the first, and the tests pin why — a model carrying only the plan switch won its five agent
cells while thinking, so one field read for both modes would have moved five cells nobody
re-measured.
The method recorded the machine, the database and the code revision, and said nothing about the
server that loads the weights. Same model file, same digest, same objective, same code, one
engine version apart: 5 of 5 at 9 seconds, then 1 of 5. Three configurations were measured before
the engine was suspected, because the obvious suspects were ours — with the per-model settings
under test, without them, and with the context bound removed. All three read the same. The upgrade
is not simply a regression, which is why the note is about versions rather than advice to stay on
one: the two models measured on 0.32.13 were re-verified on 0.33.0 at their most marginal cell,
and both hold, faster than before —
nemotron-3.5-lightning:30b's plan 5/5 at 93s against 144s,muse-glimmer:latest's assess 5/5 at 137s against 167s. The same upgrade that broke one modelimproved two. A figure on these pages is a claim about a model on the engine named beside it.
A shipped model reads 1/5 on a cell and no fix reaches it, so it is withdrawn.
qwen3.5:4b's plan losses stop in two seconds having written nothing at all. Three fixes weremeasured and none of them reached it: the plan-statement retry read 1/5 unchanged, turning its
reasoning-off switch back on read 0/5 with every loss timing out at exactly 90 seconds, and paying
for that with a 150-second turn read 0/4 timing out at 150. It loses at both ends of the one
switch that governs the cell — silent in two seconds, or thinking past any limit it is given — so
it is withdrawn rather than shipped with a figure the docs cannot stand behind. It leaves the
profile document, the pinned resolution table, the README, its own page, the e2e sweep list and
the prose that cited it as a measured case;
muse-glimmer:latestandgemma4:12bcarry thatcitation now, having been measured on the same shape.
Neither broken cell is this branch's doing, and that was measured rather than argued. Both
read the same on a clean
origin/mainworktree carrying none of these commits, andqwen3.6:27bis not in that document at all — it ran there on the compiled defaults and lost anyway.
Three claims this repo made about itself, corrected
docs/llms/README.mdsaid every supported model cleared its six surfaces "at the 90-secondper-turn limit the product ships". One of them already did not:
qwen3.5:9bhas carriedturnTimeoutMs: 150000since it was measured. Five models carry it now, and the README andmethodology.mdname them rather than asserting a limit that five of their own rows contradict.methodology.mdsaid the UI sweep drove "all twelve models" through the product's own rail,and reported 57 of 60. Sixty runs at one per surface is ten models, which is what that sweep
covered when it ran in feat(agent): ten models run every agent surface, measured 300 for 300 #465; the models added since have been driven over HTTP only. It says
ten now, and says so explicitly, because "all of them" described a sweep nobody ran.
gemma4:12b's assess loss looked like the call ceiling, and the ledger refutes it. Thelosing run made eleven calls, one under the general ceiling of 12 — which is
gemma4:26b'smeasured case exactly. Two of the passing runs also made eleven. Lowering the ceiling would have
cut runs that pass, and the record now says so where the next reader will look.
New entries, all measured
nemotron-3.5-lightning:30bturnTimeoutMs: 150000muse-glimmer:latestturnTimeoutMs: 150000,suppressPlanReasoning,reportReminderLimit: 2qwen3.6:27bturnTimeoutMs: 150000,suppressPlanReasoning,suppressAgentReasoninggemma4:12bturnTimeoutMs: 150000,suppressPlanReasoning,suppressAgentReasoningEach is pinned by digest in
tests/unit/lib/agent/model-resolution-table.test.ts, so no entry candrift unnoticed, and each has a page under
docs/llms/.What this does not close
nemotron3:33bis untouched here, andB66stays open on the work list. The sweep it asks forhas been run and is recorded on the model's page rather than in the entry: a 150-second turn read
3/5 again, but one of those runs composed its report at 114 s — a run the shipped 90-second limit
would have killed. So the limit is part of the answer and not all of it, and what the cell
actually shows is a report turn whose length varies by a factor of twenty on identical input.
Settling the entry against that record is somebody's next PR, not this one's business.