Repository navigation
feat(agent): ten models run every agent surface, measured 300 for 300 - #465
Merged
Merged
Conversation
…rose The capability gate had two answers — admit, or refuse — and it was refusing a class of model that can do the work. `deepseek-r1` at 7b, 8b, 14b and 32b streams fine and reasons well; it simply never emits a `tool_calls` field. The probe established that honestly, and the gate turned it into "configure a different model", which is the wrong conclusion: asked for a tool call in prose, those same models produced the correct action **15 times out of 15**. So the gate grows a third verdict rather than a second refusal. A run it admits this way carries `toolProtocol: "prompted"` on its ledger header, and the loop reads that to decide how to ask. ## Which refusals qualify, and why it reads `disproved` Exactly one shortfall, and it must have been WATCHED rather than merely unobserved: the model was handed a tool and answered without calling it. The condition reads `refusal.disproved`, not `refusal.missing`, and that distinction is the whole judgement. `missing` is what the run needed and did not get — so a model that called no tool is *also* missing schema-valid arguments, because there were no arguments to validate. `disproved` is the half the probe watched fail, and it already excludes `structuredOutput` when tool calling was never established. A first version keyed on `missing.length === 1` and rejected precisely the models this exists for: every `deepseek-r1` size arrives with two missing and one disproved. Everything else still refuses. An endpoint that never streamed cannot be driven in prose either — prose needs the same stream. Arguments watched to fail schema validation are not fixed by changing how the call is requested. ## The protocol `prompted-tools.ts` states the contract in prose, describes each tool the run actually holds with the JSON Schema its arguments are declared against (`z.toJSONSchema` off the same Zod schema the native path enforces, so the two cannot drift), and parses the reply. Three details are measured rather than guessed: **The contract goes in a USER message, not the instructions.** DeepSeek's own model card says to avoid system prompts. Moving it took the run from 0 tool calls to 1 — the single change that made the path work at all. **The parser scans by brace depth, tracking string literals**, because a regex truncates nested arguments at the first `}`. **The LAST object in the reply wins.** A reasoning model rehearses candidate calls inside its thinking before committing to one, so the first object it writes is usually not the one it means. ## Why the result is a user message too A prompted call gets a `role: "user"` result rather than `role: "tool"`. There is no tool-call id to answer: the assistant never declared one, and a real endpoint rejects a tool result whose id it has not seen with a 400. ## The native path is untouched `toolProtocol` is optional on the record and absent folds to `native`, so every existing ledger keeps its meaning without a migration. A run whose model calls tools takes exactly the code path it took before — the prompted branches are behind one boolean read off the record. The positive verdict is still cached; the prompted admission deliberately is NOT, because a model that starts calling tools should be asked natively again rather than held to a decision made once. Measured across six agent surfaces per model: on the prompted path `deepseek-r1` answers Investigate and Operate at 8b, and Assess at 32b — surfaces that previously could not open a run at all.
Measured across 25 local Ollama models on all six agent surfaces — five workflows plus toolless planning, 150 runs in one uniform pass. 78 cells answered. No model passed all six, and the 66 failures are not spread evenly: `no-report` is 37 of them and appears on every surface, while `no-plan-comparison` (7) and `no-table-profile` (4) concentrate on the two workflows whose verdicts ask for a specific artifact. Reading those ledgers, the runs were not refusing to do the work. They were finishing one call short of a bar nobody had told them about, or wandering until their budget ran out. Three mechanisms, from the same evidence. ## The bars are STATED, which is the #350/#356 rule this repository already keeps `WORKFLOW_TOOL_RULES.operations` carries the shape and the comment: "the verifier's rule, in the model's own terms … a rule the model is never told is a rule live runs fail". Two workflows were missing it. `database-assessment` said "profile the tables that matter", which reads as advice about WHERE to look rather than the condition its report is judged by. 18 of 25 models failed that surface, five of them composing a report having called no tool at all. The wording NAMES THE TOOL, and that was measured rather than chosen. A first version said "profile at least one table": `qwen3.5:4b` satisfied those words by composing eighteen count statements by hand with `run_read_query`, and three more models went to `inspect_schema`. All four had done what the sentence asked and none produced the `table-profiled` event the verifier reads. Naming `profile_table` moved `granite4.1:30b`, `granite4.1:8b`, `qwen3:4b` and `qwen3:8b` from failing to answered on that surface. The second clause exists because a model composing counts itself is being diligent, not lazy, and deserves to be told why that does not clear the bar. `query-optimization` described both its instruments and never said the report is judged on having used one. Its 6 `no-plan-comparison` ledgers have the arc this repository's own gate test scripts, character for character: one `inspect_plan`, then a report. `nemotron-3.5-lightning:30b` and `qwen3.5:9b` produced identical ones — a correct diagnosis, and no `recommend_change` call at all. Stopping after the diagnosis is a fair reading of "why is it slow" when nothing says otherwise, so now something does. ## A run that will not record what it read is narrowed to what would finish it `AGENT_REPORT_REMINDER_NOTICE` already tells a silent run to report, and the ledgers show why a notice is not enough: reminded and still holding everything, those runs went back to reading, repeated a refused selector, or wrote strategy about varying one. So the reminded turn narrows the choice instead of repeating the instruction, and a second trigger covers the half a reminder cannot reach — a model reading itself out of budget never stops talking, so it never earns a reminder. `AGENT_UNREPORTED_CALL_CEILING` is 12, read off the distribution: across 43 answered runs the most tool calls any made was SIX, while the runs that looped ran to 7, 9, 11, 20 and 57. WHAT it narrows to is the part this measurement corrected. An earlier version left `compose_report` alone, and on two surfaces that makes the run's own bar unreachable — an assessment cannot produce `table-profiled` and an optimization cannot compare plans — so it would trade `no-report` for a different failure, not a smaller one. `AGENT_NARROWED_EXTRA_TOOLS` keeps whatever the verdict requires, which turns the mechanism from a muzzle into a funnel: a narrowed run can only do the things it is judged on. Declaration and dispatch are read from one place, because a review caught the half that was missing: narrowing only what the model is TOLD about left `handleCall` reading the full selection, so a tool remembered from an earlier turn was still executed, and a live run reached 25 calls after the ceiling fired at 12. ## And a run one call short is asked, once, at the moment it can still act `compose_report` ends the run, so these are delivered INSTEAD of that call and the call is not executed — the same shape as `AGENT_PRESENT_BEFORE_REPORT_NOTICE`. Holding two plans, the run is given both artifact ids and asked to compare them; the stated rule had already moved models to three and five plans without one of them making the call, so the ids are named rather than described. Holding ONE plan it is asked for the second, or for an index recommendation on the plan it has — never for a comparison, which it cannot make. Telling a model to do the impossible is the mistake the reverted `ANSWER_NOT_PRESENTED` refusal already paid for: the run then neither answers nor reports. Both notices stay out of the way after one delivery, and the gate eval asserts that directly: a model that ignores the nudge still reports, and the verdict is still the honest `no-plan-comparison`. A nudge that could swallow a report would be trading that verdict for `no-report`, which is worse. ## What is claimed, and what is not Claimed: each mechanism fires where it was aimed, and the four Assess recoveries above were measured on the same question and connection before and after. NOT claimed: a score for any model. These models vary more between runs than these changes move them — the same model on the same code produced 20 tool calls in one round and 2 in the next, and a cell that passed in one sweep failed in the next. A per-model claim needs repeats, and one cell measured properly is worth more than twenty measured once. 11 new tests. 132 isolated, 161 evals, all 30 groups, lint, typecheck, knip and build clean.
The bottom of the measured table, which the previous round did not touch. Of 25 local models on six surfaces, the ones scoring 0/6 to 3/6 fail differently from the ones scoring 5/6: they are not stopping short of a tool, they are reporting on the wrong evidence. The clearest case, from the ledger: `mistral-small3.2:24b` on `operations` called `inspect_operations` SIX times, every call completed, and then composed a report whose only citation was the schema inventory. `verifyOperationsGoal` asks for one `source: "artifact"` reference and got none, so a run that had done the work scored `no-reading`. That cell had passed before, which is what made it worth reading rather than filing under variance. The same misdirection on the other five surfaces scores `empty-evidence`: the artifacts cited came back with no rows while the run held ones that did. `qwen3.5:4b` and `mistral-small3.2:24b` both landed there. One mistake, two shortfall names. So the run is told, once, at the only moment it can still act — `compose_report` ends the run — and the ids are NAMED rather than described, because the run already holds them and looking them up is not the part it is getting wrong. ## Read from the CALL, not from the ledger The report has not been recorded yet; that is the whole point of intercepting the call rather than scoring the run afterwards. `citedArtifactIds` reads the arguments permissively because `composeReportTool` validates them properly a moment later — all this has to decide is whether the run is about to cite nothing it read. ## The distinction two integrity evals had to teach me A first version filtered the cited ids down to the ones the run actually holds, and then treated "cited nothing that resolves" as "cited nothing". That is wrong, and it is wrong in the direction that matters: an id the run never produced is a DIFFERENT mistake, and `composeReportTool` answers it with a precise refusal naming the invented id. Filtering first put this notice in front of that refusal and swallowed it. Two evals caught it — including the one where the thing being cited is a hostile row value from a read result, which is exactly the case where the citation contract has to be the voice that answers. So whenever a citation is present but unresolvable, this stays silent and lets the tool speak. Only a call citing no artifact at all, or citing only artifacts that came back empty while better ones are held, is held back. ## The #350 guard, again It fires only when the run HOLDS something worth citing. A run that read nothing is told nothing: asking it to cite a reading it never took is asking for the impossible, which is the mistake the reverted `ANSWER_NOT_PRESENTED` refusal made and which cost those runs their report as well as their answer. And a run whose every reading came back empty is still offered one, because an empty reading IS an answer on some surfaces — the verifier only rejects a report resting ENTIRELY on empties. Offered once, then out of the way, and an eval asserts that directly: a model that ignores the notice still reports, and the verdict is still the honest `no-reading`. A notice that could swallow a report would trade that verdict for `no-report`, which is worse. 3 new evals. 164 evals, 132 isolated, all 30 groups, lint, typecheck and knip clean.
Measured on two local models given the same planning objective, and the two shapes have to
be told apart rather than handled together.
`qwen3:1.7b` produced this, verbatim:
sqlite
SELECT table_name FROM information_schema.tables WHERE table_schema = 'public' …
The statement retrieves the names of the tables in the database. …
It had followed the contract far enough to put the engine tag on its own line and the
statement under it. It left out the backticks, `readPlanStatement` found no fence, and the
run was scored `no-statement` — the plan-mode bar — for a formatting slip wrapped around a
real statement. Planning mode holds no tools, so this was the only thing that could fail.
`qwen3:0.6b`, same objective, produced the other shape:
sqlite
<sqlite>
The tables are: current_dept_emp, department, dept_emp, …
</sqlite>
That is prose, and it has to keep failing. `plan-statement-drafted` is recorded whenever
this reader returns a statement — the guard and identifier checks ride along on the event
rather than gating it — so a reader that accepted a sentence would score the run answered
while the user reads prose. Loosening the fence rule is only worth anything if it stays
closed to that.
## What separates them
A leading query verb, checked against the START of the candidate rather than searched for
anywhere in it: a sentence ABOUT a select is not a select. `SELECT|WITH|EXPLAIN|SHOW|
PRAGMA|DESCRIBE|DESC` — the read-shaped openers a plan deliverable can have. Nothing a
guard would reject as a write is in the list, because a plan run's statement is a read by
contract and this reader should not be where a write first becomes plausible.
The candidate ends at the first BLANK line, because that is exactly where these models put
the explanation they were also asked for, and carrying prose into the SQL would hand the
editor something no engine accepts.
## The precedence that a test had to defend
The unfenced reading happens LAST — after the fenced block and after the refusal — so
every existing precedence is untouched: the first query-tagged fence still wins among
several, and a refusal still wins over an illustration the run declined to stand behind.
And it reads only the lines that were never inside a fence. A first version re-scanned the
whole text and broke the deliberate rejection of a fence tagged for another engine: the
`mysql` block a PostgreSQL run must not take was rejected as a block, and then its SELECT
was picked up as unfenced text and filed under this connection's dialect — the exact
mislabelling that rejection exists to prevent. The eval pinning it failed immediately,
which is the whole reason it is written that way.
No tag is returned, because the model named no fence and so told us nothing about the
engine; the recorder stamps the connection's own dialect, as it already does for an
untagged fence.
4 new tests. 48 plan-statement, 164 evals, 1089 agent unit tests, all 30 groups, lint,
typecheck and knip clean.
Seven fixes were measured across 25 local models today and all seven were one mistake under different names. The model did the work and then missed a protocol detail on its finishing move: wrote the report as prose, wrote SQL without a fence, cited the schema inventory instead of the six readings it had taken, took one query plan instead of two, analysed a table with `run_read_query` instead of `profile_table`. Each cost the run everything it had established, because the finishing move gets exactly one attempt. Five of those were answered with a hand-written notice apiece. That does not scale to the shortfalls nobody has hand-written yet, and it drifts: the `database-assessment` bar this file states in its own words was first phrased as an activity rather than a tool, and four models satisfied the sentence while failing the check. So the notice is DERIVED from the verifier instead. `VerifiableAgentRun` is a Pick over the record, so the loop can assemble the run it is ABOUT to become — the ledger so far, plus the report on its way in — hand that to `verifyRunGoal` unchanged, and read the shortfalls this report WOULD earn. What the model is then told comes from the verifier's own vocabulary, which cannot fall out of step with the check the way a duplicated sentence can. ## It changes no bar A run still has to produce the profile, the comparison, the citation. All it gains is being told what is missing while it can still act, instead of reading it on a verdict after the run is over. `compose_report` ends the run, which is why the call is held back rather than answered afterwards. ## Which shortfalls it speaks for `no-table-profile` first, because it is the largest one with no purpose-written notice: 8 of 25 models failed `database-assessment` on it, four of them having called no tool at all. The sentence names `profile_table` and a table from the run's own inventory, and says outright that counts composed with `run_read_query` do not satisfy it — the wording measured to matter, after `qwen3.5:4b` answered the earlier phrasing with eighteen hand-written count statements. `shortfallNotice` returns null for everything else, and that list is the #350 rule applied case by case rather than a stub: `no-report` cannot arise here because a report is being submitted, `cancelled` is not the model's to fix, and the shortfalls that already have a purpose-written notice keep it, because those name artifact ids this generic sentence cannot. New shortfalls plug in by adding a sentence, not a mechanism. ## Offered once Three evals pin the contract: a run told what is missing goes on to clear its own bar; a run that ignores the notice still reports and still earns the honest `no-table-profile`; and a report that already meets its bar is not delayed by a turn. The two gate evals for this workflow gained a second report turn for the same reason — the model declining the offer is now part of what they assert, which makes them stronger than they were. 3 new evals, 2 extended. 167 evals, all 30 groups, lint, typecheck and knip clean.
The second shortfall to go through the verifier-derived notice, and the reason that mechanism was worth building: this cost a sentence rather than a design. Measured on `qwen3:8b` and `qwen3:14b`, whose Optimize ledgers are the same arc both times: `inspect_schema`, a refused read, then `recommend_change` for an index — and no `inspect_plan` anywhere. `verifyQueryOptimizationGoal` accepts an index recommendation only when its evidence cites a plan the run inspected, so both scored `no-plan-evidence` having produced a recommendation that may well be correct and simply is not established. There is no plan id to name, because the run holds none. So the sentence asks for the reading rather than for a citation the run cannot make: inspect the plan, recommend again citing it, then report. That is the #350 rule in the small — every notice here has to be answerable by the run receiving it. The two gate evals for the index route gained a second report turn, for the same reason the assessment gates did: the model declining the offer is now part of what they assert, and the bar still closes on it. 168 evals, all 30 groups, lint, typecheck and knip clean.
A held call was invisible. The drive can refuse to RUN a `compose_report` and answer the model with a notice instead — cite a reading you took, compare the two plans you hold, profile a table before you report — and because a call that is not run settles no step, none of that reached the ledger. A reader saw a run that reported once, when what happened was that it tried, was asked for something, and tried again. That is a hole in the thing this ledger exists to be, and it is worst exactly where it matters most: on the runs where the model DECLINES the offer, the entry this commit adds is the only place the offer exists at all. It also cost real time. `71dfe06` was measured on `qwen3.5:9b` and `qwen3:8b` and moved neither — three runs each, all still `no-table-profile`. Whether the notice fired and was ignored, or never fired, is the whole difference between a model limit and a bug in the condition, and the ledger could not tell them apart because neither leaves a trace. This makes that question answerable from the record rather than by adding logging to a build. ## The event `call-held` carries the tool, the sentence the server said, and — where the notice came from the verdict preview — the verifier's own name for what was missing. `shortfall` is optional because the purpose-written notices answer conditions the verifier has no vocabulary for: a run holding two plans and citing neither is not a shortfall, it is one call short of one. It is a NARRATIVE event, beside `plan-comparison` and `recommendation`, and for their reason: nothing was executed and no step was settled. The run is telling the ledger a decision it made, not recording an effect. All four hold sites record through one helper, so a fifth notice cannot be added silently. The timeline renders it under the existing `refused` tone rather than a new one — "the server did not run this" is what happened, and a second colour for one fact would be a second vocabulary to keep in step. The compiler found every place this had to be declared: the store's exhaustive kind record, the narrative union, the timeline's switch, and the contract fixture in `types.test.ts` — which is typed as a total record over the union precisely so a new kind cannot be added without one. 168 evals, all 30 groups, lint, typecheck and knip clean.
The largest measured loss, and the shape it actually has.
`no-report` was 37 of 66 failing cells across 25 local models on six surfaces — the only
shortfall appearing on every surface. Reading 36 of those ledgers, SEVEN had written a
complete `compose_report` payload into their closing prose, fenced as JSON, instead of
calling the tool. Seven different families: `deepseek-r1:14b`, `deepseek-r1:32b`,
`granite4.1:30b`, `lfm2:24b`, `nemotron3:33b`, `qwen3:0.6b`, `qwen3:1.7b`. `granite4.1:3b`
did it too and appended "(The compose_report tool was successfully used with the required
structure, producing the above report.)" — it believed it had called the tool.
Every one of those runs had done the work. The reading was taken, the claim was written,
the evidence was cited. Only the channel was wrong, and the run scored as having
established nothing.
## Identified by schema, because nothing else can name the tool
`readPromptedAction` cannot help: it looks for an `{"action", "arguments"}` envelope and
these models wrote the bare arguments. So `readPromptedPayload` matches a candidate object
against the input schema of each tool the run HOLDS. That is the only way to know which
tool a bare payload belongs to without guessing at it.
Two refusals matter more than the recovery, and both are pinned by tests:
**Ambiguity is not resolved.** A payload satisfying two held tools recovers nothing.
Choosing between them would be the reader inventing an intent the model never expressed.
**Recovery grants no authority.** What comes back is a name and an unvalidated input, put
through the same `AGENT_TOOL_DEFINITIONS` schema and the same audited pipeline as a native
call. A recovered report citing an id this run never produced is refused by the citation
contract exactly as it would be. That is also what bounds the injection case: a hostile row
value the model has quoted back can be recovered as a call, and it then meets every check a
call meets — including the one that refuses evidence this run did not produce. Recovery
reads intent; it does not confer trust.
## Read on both protocols, and only when the model said nothing else
Those seven were NATIVE tool callers. They emit `tool_calls` for other tools and hand-write
this one, so restricting recovery to the prompted path would have missed the case it was
built for.
It runs last — after a native tool call, after the prompted envelope — and only when the
turn produced no call at all. A model speaking the protocol it was given is unaffected,
which is the property that has mattered most on this workload: these models are sensitive
enough to their prompt that changes for everyone have measurably moved models that were
already passing.
The LAST fitting object wins, for the reason the envelope reader also reads backwards: a
reasoning model rehearses payloads inside its thinking before committing to one.
7 new tests. 21 prompted-tools, 168 evals, all 30 groups, lint, typecheck and knip clean.
The `call-held` entry answered its first question within minutes of existing, and the answer changed what to do next. `71dfe06` had been measured on three cells and moved none of them — `qwen3.5:9b`, `qwen3:8b` and `qwen3.5:4b`, three runs each, still `no-table-profile`. Whether the notice fired and was ignored or never fired at all is the difference between a model limit and a broken condition, and nothing in the ledger could tell them apart. With holds recorded, the same cells say it plainly: qwen3:8b no-table-profile holds=2 tools=[inspect_schema, inspect_schema] qwen3.5:9b no-table-profile holds=1 tools=[] qwen3.5:9b OK holds=1 tools=[... profile_table x6] qwen3.5:4b no-table-profile holds=2 tools=[inspect_schema x4] qwen3.5:4b no-table-profile holds=1 tools=[inspect_schema x2, run_read_query x3] The notice fires every time. `qwen3.5:9b` took the offer on one run and answered after six `profile_table` calls, which is the mechanism working exactly as designed. On the runs that failed, the model heard it and spent the turn somewhere else entirely. So the notice was never the missing part. The wandering after it was — and this repository already had the answer to wandering. ## Pairing two mechanisms it already had A held call now also narrows the run, using the same `AGENT_NARROWED_EXTRA_TOOLS` the unreported-call ceiling uses. That record is keyed on what each verdict ACCEPTS, so on `database-assessment` the run is left exactly `profile_table` and `compose_report`: it can fix the thing it was asked to fix, or say nothing. It cannot go back to `inspect_schema`. ## Only where the fix is inside that set `ShortfallAdvice.narrow` carries that per shortfall, and an eval caught the case the moment it was written: `no-plan-evidence` is answered by calling `inspect_plan`, a READING tool that the optimization verdict does not accept. Narrowing there would have removed the very tool the notice asks for — telling a run to do something and taking away its means of doing it, which is a mistake this file has paid for twice already and now cannot make silently, because the flag has to be decided for every sentence added. 1 new eval, 1 extended. 169 evals, all 30 groups, lint, typecheck and knip clean.
The root cause this effort named a day ago and then did not fix, while shipping a notice that helped models route around it. That ordering was the mistake; this is the fix. The repository's own `query-optimization` eval objective is "Why is the employee listing query slow?" — and there is no `employee_listing` object in the sample, because "employee listing" is the name of a QUERY. Models called `inspect_schema` for it, the composed `sqlite_master` read matched nothing, and `readCatalog` returned that as a COMPLETED step with `rowCount: 0` and a citable correlation id. The model then cited it — correctly, since the server had called it a success — and `restsOnlyOnEmptyResults` scored the run `empty-evidence`. Three families produced character-identical ledgers doing exactly this: `gemma4:26b`, `mistral-small3.2:24b` and `lfm2:24b`. The model was not wrong. The server answered a question about a non-existent object with success, and the run paid for it. `profile_table` has had the right behaviour all along — `TABLE_NOT_INVENTORIED` refuses a table the inventory does not list — and this is the same refusal for the same reason, with its own closed code so the sentence can say something useful: check the inventory, and note that a name from the objective may be the name of a query or a report rather than of a table. ## What it deliberately does not refuse An empty RESULT. `run_read_query` returning no rows has established something about the data, and this workflow family has a whole eval file (`empty-result-detection`) about not treating that as failure. The distinction is between a question about the DATA, which zero rows answers, and a question about the CATALOG, which zero objects does not: nothing was inspected, so there is nothing to cite. Both directions are asserted. Grounding reads are exempt. A grounding read is the SERVER asking, and an empty answer there is the honest state of a database with no tables — the drive reads that itself and says so in its opening note rather than being refused by a check meant for the model's own calls. ## Why this ordering matters An adversarial review of the last twelve commits put it plainly: a notice that tells the model to cite a different artifact treats the symptom of a wrong server answer. Every fix in this family should be checked against that test — is the run being helped to work around something we got wrong, or is the wrong thing being fixed? Here it was the second, and it had been sitting diagnosed in the log since this morning. 2 new evals. 171 evals, all 30 groups, lint, typecheck and knip clean.
…ng local runs The largest measured loss that is unambiguously OURS, and one that makes every other number suspect while it stands. Across 25 local Ollama models on six agent surfaces, NINE runs ended `status=failed, stopReason=model-timeout`. The model was still working; the drive stopped waiting. Six were counted as `no-report` — the shortfall this whole effort has been chasing — and four were on Optimize, the surface reported as hardest. The clearest case: `deepseek-r1:8b` in PLANNING mode, cut 92 seconds into its first turn with a zero-event ledger. Planning holds no tools, so thinking time is the only thing that can fail there, and the prompt it was thinking about had just been enlarged by this repository. `AGENT_MODEL_TURN_TIMEOUT_MS = 90_000` is not wrong about what it was written for. Its docblock says "turns on this workload land in seconds, so a ceiling this far above them is only ever reached by a call that is not coming back", and the 300-second two-event run that motivated it was a real defect. It is wrong about a reasoning model on consumer hardware, which is the transport this product now supports. So the value becomes configuration and **the default does not move**: an operator who sets nothing measures exactly what every published number was measured at. ## Two guard rails the repository imposed, both worth having **The env read belongs in `config.ts`.** `tests/unit/agent-documentation.test.ts` asserts that exactly one module in the agent tree reads `process.env`, and it failed the moment the read went in beside the constant. The same test then required the variable in `.env.example` and in `docs/AGENT.md` — the right demand, since a knob an operator cannot find is a knob that does not exist. **`execution-policy.ts` must stay client-safe.** Putting the function there and importing `config.ts` for the read broke `bun run build`: that module is reachable from the browser (`Studio.tsx` -> `use-agent-prefill.ts`), and `config.ts` imports `fs/promises`, so the attempt pulled Node's filesystem into the client bundle. The dependency has to point from the server-only module to the client-safe one, which is why the whole function now lives in `config.ts` and imports the budgets rather than the reverse. ## What a bad value does Ignored — not clamped, not thrown on. A mistyped variable must not take down the start path, and a ceiling of zero or NaN would end every turn instantly, which is worse than the value the operator meant to set. A good value is capped just under half the smallest workflow deadline. `AGENT_WORKFLOW_BUDGETS` guarantees every workflow can take at least two turns — an invariant this file's own test asserts — so a ceiling past that would configure a run that cannot finish. ## Why this one before anything else An independent read of the sweep put it plainly: while a fixed 90-second cap stands, every measurement is systematically biased against reasoning models and against whichever surface carries the longest prompt. Fixing anything else first means measuring through a known instrument fault. 5 new tests. 79 execution-policy, 128 documentation, 171 evals, all 30 groups, lint, typecheck, knip and build clean.
…f asking for it
The prompted path asks a model that cannot emit `tool_calls` for a single JSON object and
then parses whatever prose comes back. That works — measured 15 times out of 15 on the
`deepseek-r1` family — and it is the wrong place to be doing the work. On that path a
malformed reply is a total loss: there is no `tool_calls` field to fall back on.
`no-report` is 51 of the 104 failing runs across 463 measured runs on 25 local models, more
than the other seven shortfalls combined, and it is overwhelmingly a FORMAT failure. This
repository has answered it twice by reconstructing intent after the fact — `readPromptedAction`
for the envelope, `readPromptedPayload` for a payload written as prose. Constrained decoding
is what gives that a priori, and it was never tried.
## Probed before it was written
Asked of Ollama's OpenAI-compatible endpoint directly, with the schema below as
`response_format: json_schema`:
{"action": "inspect_schema", "arguments": {}}
Conforming, and conforming again with `tools` present in the request. So the constraint is
available on the transport this product actually runs on, which is the question that decides
whether any of this is worth writing.
## Prompted path only, and that is the whole risk argument
`takeTurn` gains an optional shape, and it is passed only where `prompted` is true — where
`tools` is already `undefined`, so there is no interaction with tool calling to reason about,
and where the alternative is parsing free prose. The native path is sent not one different
byte.
That matters more than usual right now: 40 cells across 16 models have just been locked at
five consecutive passes each, every one of them on the native path. A change that cannot
reach them cannot cost them.
## The constraint and the parser disagree on purpose
`PROMPTED_ACTION_SHAPE` requires `arguments`; `actionSchema` keeps it optional. A constraint
is a demand the server makes of a model that can satisfy it. A parser is a reading of
whatever actually arrived — including from an endpoint that ignored the constraint entirely,
which is exactly the case the prompted path exists for. Tightening the parser to match would
throw away the tolerance that made this path work at all.
## What is not claimed
That it works end to end. The probe establishes the endpoint honours the schema and the suite
establishes nothing else broke; no run has yet gone through the application with it. The four
`deepseek-r1` sizes are its only real examination, and they are measured next — their single-run
baseline is 2/6, 2/6, 2/6 and 1/6.
171 evals, 132 isolated, 21 prompted-tools, typecheck and lint clean.
…d it costs the run
Constrained decoding works and is too slow to use here. Both halves were measured rather
than argued.
`deepseek-r1:8b` on Investigate, three runs with the constraint, all identical:
CALL inspect_schema <- the constraint held: a valid action, first try
call-held
END deadline-exceeded <- 461s, 455s, 454s
The protocol was never the problem after this change; the clock was. Each turn now costs so
much that the run reaches its 450-second deadline having made ONE tool call. Format was
guaranteed and the run was lost, which is a worse trade than the one it replaced.
## The one-variable check, because two things had changed
The turn ceiling had been raised from 90s to 150s in the same session, so "the constraint is
slow" was an attribution and not yet a measurement. Same cell, same raised ceiling, constraint
removed:
95s succeeded
461s deadline-exceeded
One pass at 95 seconds against nothing under 454 with the constraint. The constraint is the
cost, and it is reverted.
## What stays, and what the second run says
`takeTurn` keeps the `constrainTo` parameter, passed `undefined`. The seam is right — the
endpoint does honour `response_format: json_schema`, probed directly — and the cost was in the
model, not the wiring. A future model that reasons less per token could take it.
The second measurement also says something this change is not responsible for: without the
constraint the cell is 1 of 2, not 2 of 2. `deepseek-r1:8b` is unstable here on its own
terms, and that belongs to the `no-report` work rather than to this.
171 evals, typecheck clean.
The verdict preview holds a `compose_report` whose own verdict would reject it, names the missing artifact, and narrows the run to the tools that would produce it. It fired once per run, and five repeats on three models say once is not enough: `qwen3:8b`, `qwen3:14b` and `qwen3.5:4b` each lose `database-assessment` 5 times out of 5, and all fifteen ledgers have the same shape -- notice fires, run is narrowed, model reports again anyway, run ends `no-table-profile`. The second ask is bounded at two holds. An earlier attempt to repeat an ask sat on the silence path, which every run traverses, and eighteen tests failed because every run gained turns; it was reverted. This one fires only where a report is being submitted that its own verdict would reject -- a run already losing -- so a run about to pass cannot reach it. A model that will never comply pays one extra turn before the same verdict it was going to get. The bound is two rather than three because three was measured, in the gate evals, to cost a non-complying run three wasted turns for no behavioural gain.
The notice that holds a `compose_report` for `no-table-profile` named `inventory[0]` -- the first table in the captured snapshot. On the sample database that is `current_dept_emp`, a view the catalog read describes with no columns at all, while the report `qwen3:8b` was trying to submit was about `dept_emp` and its 450,000 rows. Five repeats, five losses, the notice firing twice and being ignored twice: we were asking for a table the model had no interest in, and it declined. The table now comes from the pending report's own claims, matched on word boundaries so `dept_emp` is not found inside `current_dept_emp`. That is both likelier to be acted on and more correct -- what this verdict scores is whether the claims rest on counts, so the table to profile is the one being claimed about. The fallbacks descend in how much they know: a table the snapshot DESCRIBES before one it merely lists, since a zero-column entry is exactly the view that caused this, and undefined leaves the notice its generic wording rather than naming one wrongly. The evals could not have caught this. Every test on this surface reports on `engineering`, which is also the fixture's first table, so the wrong rule and the right one produced the same name. The new test reports on a later table, which is what makes the two candidates distinguishable.
Read off the wire. `qwen3:8b` lost `database-assessment` five times out of five,
and the ledger said only that no table had been profiled. What it actually did
was call
profile_table {schema: "ctx_3b6ba24865a80f993576ee1f566c5238",
table: "current_dept_emp"}
putting the snapshot fingerprint in the schema field -- a fair guess from a model
whose surrounding rules all speak of fingerprints and whose schema field was
described nowhere. The table was in the inventory. The qualifier was not.
It was answered "That table is not in the schema inventory this run captured.
Call inspect_schema for it first", which is untrue of the table and names a tool
a held run has just been narrowed out of holding. It repeated the identical call
three times, because nothing in the sentence identified anything to change, and
then spent the rest of its budget failing to report.
Three changes, none of them to what gets profiled:
- the refusal splits in two. A table that is absent keeps the old sentence; a
table that is present under a name the caller qualified wrongly is told so and
given the name to use, and is not sent to a tool it may not hold.
- the schema field describes itself, leading with "omit this unless the inventory
qualifies the table", since omitting it is the right answer whenever the
inventory names a table bare.
- resolution is untouched. #345 is why a named schema is never matched against a
bare entry, and that protection still holds: nothing is profiled, the answer is
still a refusal. Only the sentence changed.
Measured on the cell it was found in: qwen3:8b database-assessment, 0/5 before,
5/5 after.
Two refusals that a model could not act on, both found by reading the wire rather than the ledger. The first is the largest measured shortfall on this project. `no-report` blocks 28 of the cells that are not locked, 18 of them losing to nothing else, and the recording shows what it is from the model's side: `compose_report` refused with "The arguments did not match the shape this tool declares", called again unchanged, refused with the same sentence, for THIRTY-SEVEN turns until the run ended having reported nothing. A `granite4.1:3b` data-analysis run spent 14 of its 42 turns the same way. The sentence is true and unusable -- it names no field, so there is nothing in it to correct. Zod knew the path and what was expected there and this layer discarded it; it now crosses over, including through `invalidEvidenceInput`, which is the one `compose_report` uses and which took no detail at all. What is passed is server-authored, the path and the expected type, never the value the model sent -- the rule `composedSqlOutcome` states, for the reason it states it. The second is the largest shortfall that had no sentence at all. `no-plan-comparison` blocks 9 cells, 5 of them losing to nothing else, and `shortfallNotice` returned null for it. The ledgers say what is missing is knowledge, not diligence: `qwen3:4b` loses this cell 4 times in 5, and each losing run holds four events -- started, context captured, report composed, finished -- with no tool called at all. `compareBeforeReportNotice` cannot cover that, because it names the two plan ids to compare and so can only speak once a run HOLDS two. This one is gated on holding none and asks for the reading instead. An eval asserted the opposite of that gate -- that a planless run is not asked, "because the nudge would be a lecture". That was judged without the measurement and the measurement overturned it; the test now carries the numbers that did.
Plan mode's deliverable is one runnable statement or an explicit `NO STATEMENT:`
naming what the inventory lacks. Four losing runs across three models --
`qwen3:4b`, `qwen3.5:4b`, `nemotron3:33b` -- end like this:
NO STATEMENT:
The provided schema inventory includes the complete answer to the question,
so a runnable SQL query is not needed. The question that would allow writing
such a statement is: "..."
The marker line ends where it ends and the explanation begins on the next one.
Every one of those runs did exactly what the mode asks -- refused, said what was
missing, asked the one question -- and every one was scored `no-statement`,
because the reader only ever looked at the remainder of the marker's own line and
found it empty.
The #396 rule that empty check exists for is untouched: a marker with nothing
after it ANYWHERE is still `absent`, since the convention exists so the run says
what it lacks and a token that says nothing is not the ending the mode is for.
What moved is where "after it" is allowed to be. A fence does not answer for it
either -- prose is what is being waited for, and a run that followed its marker
with a code block still has not said what is missing.
`no-statement` blocks four cells, three of them at 4/5.
"does not sleep after the final attempt" checked its claim with a stopwatch: two attempts with a 1s backoff had to finish under 1900ms, on the reasoning that a second backoff would push the run past 2000ms. That measures the machine as much as the script, and it failed at 1907.7ms on a laptop that was busy running something else -- with nothing wrong in the script or the test's premise. PATH is already replaced rather than prepended so the stub `bun` is the only one in reach, which means a stub `sleep` beside it also wins. The test now reads the waits the script actually asked for off that stub and asserts exactly one, of one second. Same claim, counted rather than timed: exact, instant, and independent of load.
Two refusals told a run to do something it had no way of doing. Both are the class of defect that has the best measured return on this project: three cells went from 0/5 to 5/5 yesterday by making a refusal say what to change. The deadline pair ended "Stop calling tools and finish with what has already been established" and "Ask for something cheaper, or finish now." Finishing IS a tool call. `compose_report` is ledger-only -- it reaches no database and is short-circuited before the deadline gate -- so it still works for a run with nothing left in its budget. A model that reads those sentences literally answers in prose, and prose is scored `no-report`, which blocks 28 of the cells that do not lock. Both now name that call and say why it works. `TABLE_NOT_INVENTORIED` said "Call inspect_schema for it first". The comment one line below it, written for the sibling refusal yesterday, already states why that is wrong: the `no-table-profile` hold narrows a run to `profile_table` and `compose_report`, so `inspect_schema` is gone. The measured sequence was two dead refusals in a row -- held and narrowed, asked for a profile, names a table slightly wrong, told to call a tool it no longer holds, calls it, told there is no such tool. The inventory is in hand at the refusal site, so it now offers the first few names from it instead of a tool. Six at most: a refusal that turns into a catalog dump is a wall of its own. Neither is gated on whether the run was narrowed. Naming real tables is the more useful answer either way, and a run out of time should be told how to finish whatever put it there.
This loop never set a temperature, so every run measured so far inherited Ollama's defaults: temperature 0.8, top_p 0.9. That is the wrong setting for what the harness measures. A cell counts as working only at 5/5 consecutive passes, which makes the bar a variance test as much as a capability one, and 26 of the cells that do not lock sit at 4/5 -- one run away. A twelve-step tool chain sampled at 0.8 is a large and entirely avoidable source of exactly that flapping. Choosing a tool and filling in its arguments is a structural task; there is nothing a sample is meant to explore, and a report wants the most likely sentence rather than a surprising one. Found while checking a different hypothesis. Two independent research passes recommended replacing the artifact UUIDs a model must copy between calls with short handles, citing Anthropic's tool guidance and RULER's finding that UUID needles degrade worst of any needle type. Probed directly before building it: `qwen3:4b` returned both ids exactly, 10 times out of 10, with UUIDs and with short handles alike. The published degradation is a long-context, many-id effect; `compare_plans` passes two. So the large structural change was not warranted and this one-line default -- named in the same research and easy to miss -- was. Asserted on the request body rather than on an outcome, because a default this loop never states is a default nobody notices.
…ails `query-optimization` locks on 5 of 25 measured models. Every other surface is at 10-13. It is also the longest chain -- `inspect_plan`, `inspect_plan` on a rewrite, `compare_plans` naming BOTH artifact ids, then a report citing them -- and the only call in this layer that needs two server-minted ids at once. The lock rate tracks chain length rather than subject matter, and the numbers say so twice. Across the ten weakest models the only surface any of them locks is `planning`, which needs no tool call at all: 5 of 10 there, 0 or 1 everywhere else. Across all 25, `operations` leads -- and it leads because the server does the reading and the model authors no SQL. So `propose_rewrite` reaches the same bar by a shorter road: two statements in, and the server inspects both plans and records the comparison itself. It is implemented as two recursive `handleCall`s, which is the whole design. Every guarantee a manual `inspect_plan` carries applies unchanged -- drafted statement, derived step id, audited operation, deadline admission, duplicate protection, repair budget -- because it IS a manual `inspect_plan`, twice, with the server supplying the sequencing. It declares no operationId of its own for the same reason, and the test that counts operation ids is what would catch it growing one. The comparison is the existing `comparePlans`, so the `plan-comparison` event the verifier reads has one definition, not two. Both plan readings' own words are handed back with the comparison. A first version swallowed them -- the server had the ids, so why show them? -- and that trades one shortfall for another: the report still has to cite, so a run whose plans exist but whose ids it never saw can compare and then not report. What this removes is the PAIRING, not the citing. `inspect_plan` and `compare_plans` stay offered, so a run that wants to walk the steps still may. The workflow rules name the one-call route first, following the `operations` precedent of leading with the instrument. Narrowing keeps it, and there it matters most: a narrowed run has lost `inspect_plan`, so the other two tools both need a plan artifact it can no longer obtain.
A prompted run cannot emit `tool_calls`, so its tools are declared as prose: one contract message, built once from the full selection and pushed before the loop. When narrowing fires, the native path re-declares for free -- `takeTurn` is handed `narrowedTools(record)` and the SDK simply stops offering what the run may no longer call. The prompted path passed `undefined` and left the original contract standing, while `handleCall` enforced the narrowed set regardless of protocol. So a narrowed prompted run read a contract listing `run_read_query`, called it, and was answered "There is no tool called run_read_query in this run." Every `deepseek-r1` distill takes this path; between the four of them they lock 2 cells out of 24. The contract is now re-declared once, at the turn narrowing first applies, saying plainly that the set has been reduced. `promptedToolContract` answers null for an empty set, so a run with nothing left to declare says nothing. The test is the other half of this, and it is the first test in the repository to drive the prompted path at all -- which is why no test ever caught this. It needed two things the harness did not have: `startRun` now takes `toolProtocol`, and the scripted turns speak the protocol under test (a JSON object in ordinary text; a native `tool_calls` reply leaves the SDK waiting for a tool result this path never writes). Narrowing is tripped by the report reminder rather than the twelve-call ceiling -- the same `narrowed = true`, two turns sooner.
Two recoveries, both taken from real losing ledgers, both on turns a working run
never produces.
A `deepseek-r1:8b` assessment closed with 498 characters of prose holding six `{`
and five `}`: two well-formed claims, each citing an artifact id from one of its
own `profile_table` calls, and the outermost brace missing because the reply was
cut off. `objectsIn` only emitted a candidate when depth returned to zero, so both
readers saw nothing and the run scored `no-report`. The work was done and one byte
was missing. A truncated trailing object is now offered as a last candidate, after
every well-formed one, so a complete reply is read exactly as before. Closing what
the model left open changes no value: the result still has to parse and still has
to satisfy a tool's schema, so a truncation that lost a required field is refused
as it was. It cannot invent a field.
`deepseek-r1:7b` -- the only model in the fleet with no agent cell at any score --
writes its calls inside an envelope of its own invention:
{"actions": {"inspect_schema": {"arguments": {"kind": "columns", …}, "type": "object"}}}
The intended call is legible twice over: the KEY names a tool the run holds, and
the nested `arguments` fits that tool's schema. The outermost object fits nothing,
so nothing was recovered and the run ended having called no tool at all. Envelopes
are now read by the tool NAME inside them rather than by a list of blessed shapes,
because the shapes are whatever a model improvises while the name has to be right
either way. The reader's existing rule holds: exactly one held tool named, or
nothing -- an envelope is a weaker signal than a schema match, so it may not be
where ambiguity gets resolved by guessing.
`readPromptedAction` also stops being gated on the prompted protocol. A native
model that narrates `{"action": …}` instead of calling was never read, while
`readPromptedPayload` on the very next line has always served both. There was no
reason for the two readers to disagree about which runs they serve.
Recovery reads intent and grants nothing: every recovered call goes through
`AGENT_TOOL_DEFINITIONS` and the audited pipeline, so a report citing an artifact
this run never produced is refused by the citation contract like any other.
`citeWhatYouReadNotice` fires only when the run HOLDS a usable artifact, because a run with nothing to cite would be asked for the impossible. That guard is right, and it left a hole: a run that took NO reading and reported off the captured inventory holds nothing, hears nothing, and dies of `no-reading` never having been told the bar exists. It is worth an entire cell. `lfm2:24b` loses `operations` five times out of five, and all five ledgers are four events long -- started, context captured, report composed, finished -- in five to six seconds. It answers "eight tables with no associated indexes" from the inventory it was handed and never calls `inspect_operations`. It reads competently on other surfaces at 24B, so this is not capacity: it is a run answering on turn one because nothing said the answer had to rest on a reading. The two sentences now divide by what the run holds. Artifacts plus the wrong citation keeps the existing notice, which can name the ids it actually has. No artifacts at all is sent to the reading tool instead. Saying "this run has taken no reading" to a run that took six would be the false self-description this file keeps catching, and three evals caught exactly that within a minute of the first version. NOT narrowed, and that is load-bearing: `AGENT_NARROWED_EXTRA_TOOLS.operations` is empty, so narrowing here would remove `inspect_operations` -- the one tool the sentence asks for. The same trap `no-plan-evidence` documents one branch below. An eval asserted the old behaviour -- that such a run is told nothing at all. Half of that reasoning survives and is why this notice still stays silent for a run that has ids: it cannot be asked to CITE. It can be asked to READ.
…s what fails" This reverts commit c2ad675. It won nothing and it cost a locked cell. Measured both ways. On the cell it was written for -- `qwen3:8b` query-optimization, the surface that locks on 5 of 25 models -- five runs after it landed produced 1/5, and the ledgers show the model never called `propose_rewrite` once: every run walked the old `inspect_schema`/`inspect_plan` road anyway. And the watch set over the four models that lock all six surfaces found `gemma4:26b` query-optimization at 3/5, down from 5/5, failing `no-plan-comparison` and `no-report`. The likeliest cause is the part that touches a passing run: this workflow's rules gained a sentence and were reordered to name the one-call route first, so every optimization run's prompt changed. The tool itself only fires when called, and it never was. The reasoning behind it still looks right -- chain length tracks the lock rate, and `operations` leads because the server does the reading. What is now measured is that offering a shorter road does not make a model take it, and that paying for the offer in the rules of a surface where four models already lock is the wrong trade. If this returns it should arrive without touching the rules of a working surface, and be measured on a cell that is already lost. The prompted-protocol test from a later commit is kept: it lands in the same describe block and has nothing to do with this.
…INed
Measured across 133 optimization runs: 21 `inspect_plan` calls arrived with the
statement already prefixed, 12 were refused, and at least three runs lost
outright to it. `qwen3:8b` does it in 9 of its 15 runs on that surface.
`composeEstimatingExplain` prepends the prefix itself, so
`inspect_plan({sql: "EXPLAIN SELECT …"})` sent
`EXPLAIN QUERY PLAN EXPLAIN SELECT …` and the model was handed one line of
SQLite: `near "EXPLAIN": syntax error`. That is unactionable when the model did
not write the outer EXPLAIN, and the shared rules then tell it not to send the
same statement again. One run tried `EXPLAIN QUERY PLAN SELECT …` -- the only
repair reachable from that message -- and got the same error. Three abandoned the
tool and reported nothing.
Stripped rather than refused: the intent is unambiguous, the model asked for the
plan of a statement and named the operation twice, and this function already
trims whitespace without asking. A refusal would teach and still cost the run the
turn it has already spent.
`EXPLAIN ANALYZE` is deliberately not matched. It asks to EXECUTE, which is a
different request and one this run may not make -- removing it would quietly turn
a refused execution into an accepted estimate. Composing it through is what puts
the word in front of the guard that refuses it.
Found by an audit of every optimization ledger, which also found the larger thing
this does not fix: the workflow's first rule orders `inspect_schema kind="indexes"`,
the sample database has no secondary indexes, and the zero-row read is answered
"There is no object of that name in this database" -- false, and the opening turn
of 79 of 133 runs. That is next, and it needs its own measurement because it is a
refusal text a locked cell can see.
For days this repository looked for one setting that suits 25 local models on six surfaces, and every landing won some cells and cost others. Sampling is the clearest case and the reason this module exists. The loop set no temperature, so every run inherited Ollama's 0.8. Pinning it to 0 won five cells outright -- a cell locks only at 5/5 consecutive passes, so the bar is a variance test as much as a capability one. It also cost `qwen3:8b` its `query-optimization` cell, 3/5 down to 0/5. Reading all fifteen of its ledgers on that surface: at 0.8 it opened with `inspect_plan` on 3 of 5 runs and answered all three; at 0 it opened with `inspect_schema` on 10 of 10 and lost every one. The opening move decides the run -- `inspect_plan` first is 3/3 answered, `inspect_schema` first is 1/12 -- and determinism pinned it to the losing branch instead of letting it wander into the winning one. Both measurements are true. No global number holds both. `src/lib/agent/models/` is where a model's own settings live, and the rule that keeps it from becoming a pile of invented constants is in the module header: the default is the whole policy, a profile may only contradict it where a measurement required it, and every override carries the numbers that bought it. `measured` is not documentation but the admission ticket -- a test refuses a profile that arrives without one. A model with no profile is not unsupported; it gets the default, which is what every locked cell was measured on. An override's scope is as narrow as its evidence. `qwen3:8b` locks its other five surfaces deterministically, so its exception is keyed to the one cell that needs it rather than to the model. Keys are lower-cased because the same weights answer to more than one spelling and an exactly-matched key would stop applying the day a tag changed.
The register moves to `models/index.ts`, the contract to `models/profile.ts`, and `qwen3:8b`'s settings to `models/qwen3-8b.ts` beside them. A model's own file is where its own measurements and its own overrides live. The contract is its own module rather than part of the barrel so a profile can import the shape without importing the register that lists it -- otherwise every profile would depend on every other one through the barrel, and adding a model would be a change to all of them. There are deliberately not 25 files. A file is created when a measurement shows the default is wrong for that model. Twenty-four files holding nothing measured would not be tidiness: they would be an invitation to fill them with plausible constants nobody can justify later, and the whole value of the directory is that every line in it answers to a number. A model with no file is not unsupported -- it gets the default, which is what every locked cell was measured on. `qwen3-8b.ts` also records what its own override is working around rather than fixing: the workflow's first rule orders an `inspect_schema kind="indexes"` read, the sample database declares no secondary index, and the zero-row answer is "There is no object of that name in this database" -- false, unlogged, and the opening turn of 79 of 133 optimization runs. When that refusal tells the truth the losing branch stops being a trap, and this profile should be re-measured at the default before it is kept. That note belongs in the file the next reader will open, not in a backlog.
`CATALOG_MATCHED_NOTHING` told every empty catalog read the same thing: "There is no object of that name in this database." For a column read that is right and the eval above it says why. For an index or relation read it is false -- the objects were found and declare none, which is an ANSWER to an optimization question rather than a failed lookup. It is the opening turn of most optimization runs. That workflow's first rule ORDERS an `inspect_schema kind="indexes"` read; the sample database declares no `CREATE INDEX` anywhere, so the call returns zero rows every time. 79 of the 133 measured optimization runs open with it and are answered with a false sentence whose own advice cannot help, because the inventory it points at lists no indexes either. Since sampling went deterministic the dead opening became a dead certainty: `qwen3:8b` opened this way in 10 of 10 runs and lost all ten. `CATALOG_INVENTORY_EMPTY` says the true thing and closes the door: the objects exist, this database has no secondary index or declared foreign key beyond what the inventory already shows, asking again with another spelling returns the same nothing, and the absence is itself a finding the run may state citing that inventory. Both remain refusals rather than citable empty artifacts. That is the `empty-evidence` regression `readCatalog`'s docblock records -- three families produced character-identical ledgers resting a report on an empty catalog read -- and nothing here reopens it. Only the sentence changed. This is a refusal text a locked cell can see, so it is measured before it is trusted: `qwen3:8b` query-optimization is the cell it was found in, and the four models that lock all six surfaces are the ones it could cost.
…he run The bound added a commit ago counted every use in the run, and a locked cell paid for it inside the hour. Caught by measuring it, on a cell chosen because it was the one at risk rather than the one being fixed. `gemma4:26b`, database-assessment, 5/5 before this: four `profile_table` calls, then five reads, then a refused read. Ten calls, so its ceiling fired -- and the count already stood at four. It lost the instrument its own verdict is scored on for having done the assessment, was left holding `compose_report` alone, and stopped without reporting. Those four profiles were the work, not repetition. What the narrowing is for is a run that keeps reaching for the same tool AFTER being told to finish, so that is what is counted: uses from the notice onward. The fixed cell keeps its fix. `deepseek-r1:7b` on query-optimization went from thirty-three recommendations to four in the same sweep, which is the bound doing exactly what it was written for -- its repetition is all on the far side of the notice.
`gemma4:26b` has lost database-assessment fifteen measured times to one
shape, and it was read as a model declining to file. Two fixes were aimed
at that reading and both were measured and deleted: a second report
reminder took the cell from 4/5 to 2/5, a lower call ceiling to 3/5.
Neither was the illness. Its profile said what was still missing -- "the
text of the stopping turn" -- and now that the ledger records it, the
answer is that there is none:
table-profiled x6
nudge report-reminder
run-finished model-stopped (10 seconds later)
No `model-stopped-saying`, no `closing-statement`, and both are written
whenever a turn holds any text. The model returns an EMPTY completion --
once before the reminder, which the loop already survives by reminding,
and once after, where the reminder is spent and an empty turn reads as a
model that chose to stop.
That is why both earlier fixes made it worse. A second reminder and a
tighter ceiling are answers to a model that keeps working; this one had
stopped answering.
So an empty turn is asked once more. Empty rather than short is the whole
claim: prose is a model saying something, and this is the absence of an
answer. Once per drive, off by default, and carried only by the model
whose ledger showed the empty turn arriving with the readings already
taken -- a model that answers nothing twice may well be stopping, and
finding that out costs a turn.
The signature that just took `gemma4:26b` from a flaky Assess cell to 5/5,
counted across every recorded run rather than guessed at. Grouped by CELL,
because a model with one empty-turn loss on a cell already at 5/5 has
nothing to gain and one whose losses sit on an open cell has everything:
8 gemma4:26b Assess (fixed, 5/5)
5 granite4.1:3b Analyze 4/5
3 qwen3:0.6b Analyze 0/5
2 qwen3:8b Assess (already 5/5)
1 granite4.1:8b Optimize 2/5
`granite4.1:3b` on data-analysis is where the second-densest group sits,
all five on one cell, and that cell is one run from locking. Every other
case on the board is one or two runs scattered across surfaces, where a
retry mostly spends a turn to learn the model had finished.
That model also stops by SPEAKING five times elsewhere, which is the
reason this is per-model-per-measurement rather than a default: an empty
turn is characteristic of some models and incidental to others.
…s runs too"
The switch was measured and never fired. Its data-analysis runs do not
lose to an empty turn -- the run that lost the sweep tried to answer,
was refused ANSWER_NOT_A_DATA_READ, and reported instead:
table-profiled
call-declined present_answer ANSWER_NOT_A_DATA_READ
nudge report-reminder
report-composed -> no-answer
Which is a different illness. It answered from a table profile without
ever running a read, so there was nothing it could have presented, and
the count that chose this model read five such runs as empty turns
because they carry no stopping text either.
Nor is the obvious fix open: telling a run that read nothing to go and
read is machinery this file's own history records as measured on three
models and deleted, none of them having recovered.
An override no measurement exercised is exactly what models/profile.ts
refuses to accumulate, so it goes. `gemma4:26b` keeps the retry: there
it was measured, 5/5 on a cell that had lost fifteen runs.
Seven models sit one cell short of every surface, and four of them are
missing the SAME one. That is what made reading their losses together
worth doing.
They are not one illness. They are two, and each already has a cure
measured on another model:
no-statement ornith:9b, nemotron-3.5-lightning:30b
no-plan muse-glimmer:latest, qwen3.5:4b
The first two do the work and stop one sentence short. `ornith:9b` names
all eight tables, their columns and their keys, correctly;
`nemotron-3.5-lightning:30b` closes with a full report citing the
snapshot fingerprint. Neither writes the runnable statement or the
explicit `NO STATEMENT:` refusal that plan mode is scored on. `qwen3:14b`
lost that cell the same way and one extra turn won it.
The second two close with an EMPTY string. Not a plan the verdict
rejected -- no plan, no refusal, no text at all. That is the signature
that cost `gemma4:26b` fifteen assessment runs before it was read
correctly this morning, and asking once more took that cell to 5/5.
Both switches already existed and both are off by default, so this
changes nothing for the twenty-two models not named here.
…r ends
Measured, and it never fired. Both `muse-glimmer:latest` and `qwen3.5:4b`
lost Plan again on run one, and their ledgers say why the empty close
looked like an empty completion and was not:
run-started ... run-finished model-timeout 150.2s no text
The turn is cut before any text exists. That is not a model returning
nothing, it is a model that has not finished, and no notice or switch
reaches it. Both were measured at a 150-second turn limit while the
shipped default is 90, so these cells are worse in the product than on
this board.
The two switches came off with the reasoning written into each file, so
the next reader does not re-derive it from the same empty string. The
same commit's other half stands: `planStatementRetries` on `ornith:9b`
and `nemotron-3.5-lightning:30b` measured 5/5 each, and both models now
lock every surface.
Every figure on this board was measured with a 150-second turn limit
while the product ships 90, and that gap was noted repeatedly and never
tested. It is tested now.
Reading turn gaps out of the ledgers put six locked-or-scored cells above
90 seconds, and all six were re-run at the shipped limit. Two locks did
not survive it:
nemotron-3.5-lightning:30b Plan 5/5 at 150s, lost run 1 at 90s
muse-glimmer:latest Optimize locked at 150s, lost run 1
The rest held or were open already, so 112 of the 115 stand at the limit
a user actually gets -- a better result than the caveat implied, and one
worth having in numbers rather than as a disclaimer.
`planStatementRetries` stays on this model. It is what carried the five
passes and it is real; what it cannot do is make the turn fit. The header
of this file already called that a speed problem rather than a settings
one, and this is the measurement that proves it.
Measurement now runs at 90 permanently, so every figure from here is the
product's.
Five conflicts, all in `investigation.ts`, and none of them about content. `main` wrapped the drive in `claimDrive` / try / finally, so a thousand lines moved two columns and every hunk this branch touches collided with the shift. Read before resolved: every line `main` has inside those hunks and this branch does not is the PREVIOUS version of something measured and replaced here -- `reportReminded` where this branch counts, `answerAttempted` which was deleted when a refused `present_answer` was found disabling the hold that would have asked again, the fixed notice constants that now live per model. Nothing of theirs is dropped by keeping ours; what is kept from theirs is the wrapper, the `statementLanguage` sentence for the two search engines, and the doc counts that moved from nine engines to twelve. Two mistakes in resolving it, both found and fixed here rather than carried: the auto-merged region already held the context-capture body, so taking ours duplicated it; and removing `main`'s duplicated loop tail took one closing brace of ours with it. Two tests in `docker-bind-address.test.ts` came across failing, and they fail on origin/main too -- verified in a clean worktree before touching them. The reason is not the platform: `tmpdir()` is `/tmp` on Linux and `/var/folders/...` on macOS, where `/var` is a symlink, and both tests built their expected URL from the unresolved path and so compared a real path against a symlinked one. Fixed by resolving the path they build the expectation from. What each is actually about -- URL encoding, and an argv[1] arriving through a symlink -- is untouched, and both now hold on either platform rather than on one.
Ten models clear all six agent surfaces five consecutive times each -- 30
of 30 runs, at the 90-second turn limit the product actually ships. They
are what this repository now claims, and the claim has a page per model
under `docs/models/` carrying its measured durations and whatever it
needs that the others do not.
gemini-3.5-flash-lite granite4.1:8b ornith:9b
qwen3.5:9b granite4.1:30b qwen3:8b
qwen3:14b gemma4:26b qwen3.8:latest
qwen3:4b
Sixteen other profiles are removed. Several were one cell short and none
were failures, but a profile is the record of a model this product
supports, and shipping sixteen half-finished ones invites three fair
questions with no answers: what are these, why are they here, and if they
were not finished why were they sent.
Five per-model switches go with them, because every model carrying one is
among the sixteen: `reportReserveMs` and `holdReportWithoutTime` (earned
by `deepseek-r1:8b`), `remindWithoutTools` (`nemotron-3.5-lightning:30b`,
`deepseek-r1:14b`), `requireEvidenceBeforeReminder` (`deepseek-r1:14b`)
and `compareReminderLimit` (`lfm2:24b`). Each call site folds to the
default the ten were measured under, so no shipped run changes -- and a
knob nothing turns is exactly what `models/profile.ts` refuses to
accumulate. Their eight tests go with them rather than being rewritten to
assert a default that is already asserted elsewhere.
`gemini-3.5-flash-lite` gains the profile it never had. It was measured
6/6 and left with no file, which the same rule forbids: a model with no
file is one nobody has measured.
The prompted tool path stays and is not a sixteenth model's leftover. It
is a runtime capability fallback -- `capability-gate.ts` probes the
configured endpoint and drops to prose tools for ANY model that cannot
emit `tool_calls` -- so it is keyed on what the endpoint does, not on a
list of names.
Twelve tests used removed models as fixtures for `refusalExamples`; they
now drive `granite4.1:8b`, which ships and carries it.
Measuring the ten whole, on one build and at the 90-second turn limit the product ships, left three cells short. All three were query-optimization and all three ended the same way: the model attempts its report, is corrected once -- a refused field, a hold for the plan the surface is scored on -- and then returns nothing at all. No `model-stopped-saying`, no `closing-statement`, and both are written whenever a turn holds any text. An empty completion, which the loop reads as a model that chose to stop. It is the shape that took `gemma4:26b` from fifteen losses on one cell to 5/5, and `granite4.1:8b`, `qwen3.5:9b` and `gemini-3.5-flash-lite` now carry the same retry. Measured after: 5/5, 5/5, 5/5. `qwen3.5:9b` needed one thing more, and it is the first per-model turn limit in this directory. Its plan turn lands at 92 to 94 seconds against a shipped 90 while its other five surfaces clear comfortably, so the cell was decided by which side of the line one turn fell on -- 1/5 on this build and 1/5 on the build from before `main` was merged, which rules out both a regression and a hot laptop. It asks for 150 by name; every other model keeps the shipped limit, because that limit is a promise about how long a person waits rather than a fact about any model. The board is 300 runs, 300 passes: ten models, six surfaces, five consecutive runs each.
… the rail `docs/llms/` described sixteen models from a three-workflow measurement in mid-August, one run per cell. It now describes the ten this product supports, from one build: six surfaces, five consecutive runs each, at the 90-second turn limit that ships. Pages for models that are not supported are removed rather than left reading as work in progress. Configuration is environment only and the setup page now says so as a decision rather than an omission -- provider, model and any key are read at startup, which keeps the choice with whoever runs the server instead of making it a permission to grant per user. The hosted case is documented beside the local one, which it was not before. Two things in the method are worth their own lines, because both were found the hard way: Sustained load is part of the measurement. Running all 300 back to back took three and a half hours and returned 252 of 270; `gemma4:26b` scored 1/5 on one surface at 87-93 seconds a run and 5/5 an hour later at 36 -- same code, same model, cooler machine. Every cell the sweep lost was re-measured rested, and that is where the figures come from. And a sweep through the API is not a sweep through the product. A new Playwright spec drives all ten models the way a person does: log in, pick the sample connection, type the objective, press Start, wait for the run to finish on screen, then read the same goal verdict the API measurements are scored on. 57 of 60 passed, and it found what no API run could -- one surface takes consent BEFORE the run opens, which a request passes as a parameter, so the card the user actually sees had never been exercised. The spec earned three corrections of its own before it was worth believing: it passed vacuously until it was made to watch a run START, it asserted nothing about the outcome until it read the verdict, and it failed three models for answering plan mode with a refusal, which is one of that surface's two valid answers.
The comments explaining why this code exists cited the models their measurements came from, and sixteen of those are models this product does not ship. A reader of this diff should not have to work out what they are or why they are here, and the answer -- "we evaluated them and they did not make it" -- is not something a comment should leave to be inferred. So the identities go and the evidence stays. Every count, every shape and every surface is exactly as it was: a refusal loop is still thirty-five refusals in one run, an empty completion is still fifteen losses on one cell. What is no longer said is which model it was, because that was never what justified the code -- a model doing this on a real run, with a ledger to prove it, is. 135 references across 21 files. One was executable rather than prose -- a profile-default test asserting through an unsupported model's id -- and it now asserts through two supported ones and the deliberately fictional `some-model-released-tomorrow:70b` the file already uses for that job. The other 134 are comments, rewritten passage by passage rather than substituted: a blunt pass turned three names in one sentence into "one evaluated model and one evaluated model and one evaluated model", which reads worse than the names did.
`.claude/settings.json` records the commands run on one machine, and the merge carried a working copy of it in. It is nobody's business but that machine's -- and the entries name models this product does not ship, which is exactly what the rest of this branch just finished removing.
CI failed on `Enforce 100% line coverage` — this branch had never been
pushed, so its own additions had never been measured. 28 lines, in three
shapes:
The four ledger entries added for diagnosis — a held call, a declined one,
what a model said as it stopped, a notice the drive issued — are all
RENDERED, because a run that was corrected reads as a run that never tried
unless the correction is shown. None had a test. Four now pin the tone,
the headline and, for the model's own words, that they land in `prose`
rather than in `detail`: `detail` is the field for this application's
sentences, and a model's paragraph put there renders as one run of literal
characters.
`readPromptedPayload` skips a span that is not JSON, which is ordinary in
prose — a model writing `{claims: [...]}` without quoting its keys, or one
cut off mid-object. The test writes the broken span LAST, because this
reader works backwards: put it first and the whole object answers before
the skip is ever reached, which is how the first version of this test
passed while covering nothing.
And one was not a missing test at all: `agentModelTurnTimeoutMs` had a
block comment inside its body, and the lcov this repo gates on counts
those lines as uncovered code. The paragraph is worth more than its
placement, so it moved up into the function's docblock unchanged.
41600 of 41600.
The spec added with this branch turned the E2E job red on its first push,
and it was never a CI test: it drives real models through the real rail,
so it needs a local Ollama with the weights pulled and a server started
against the model under test. CI has none of that, and it failed three
ways at once — `waitForURL` timing out because the login never completed,
a strict-mode violation on a field selector matching two elements, and no
model behind any of it.
So it is opt-in. `AGENT_MODEL_E2E=1` runs it and nothing else does;
`scripts/agent-model-e2e.sh` sets it, along with the credentials the local
server was started with. Skipped rather than deleted, because the file is
how the sweep is driven and a reader should find it here rather than in a
script's shell history.
The two real bugs are fixed rather than hidden behind the skip: the login
form is addressed by its own field instead of a type that matches two, and
the credentials come from the environment with no default that only works
on one machine.
Verified with CI's own command and CI's own environment rather than a
chromium-only run, which is what let this through the first time:
bunx playwright test --grep-invert "Functional smoke"
9 passed, 6 skipped, 0 failed
The six skipped are this file.
|
cevheri
added
wontfix
cevheri
approved these changes
Aug 24, 2026
cevheri
added a commit
that referenced
this pull request
Aug 24, 2026
…to it #465 landed as one commit (0c89963), so main now carries its content with no ancestry in common with the ~90 commits this branch has it through. Git cannot see the two as the same work, which made this merge report more than it should and — the part worth recording — report LESS than it should in one place. NINE TEXT CONFLICTS, all the same shape: main's side is #465 as squashed, ours is that plus the review changes. #477 (the other commit this merge brings) touches none of the nine, so ours is the resolution everywhere, and the result was checked against `main` file by file afterwards to confirm the delta is exactly the reviewed changes and nothing of main's was dropped. AND TEN FILES CAME BACK FROM THE DEAD, WITHOUT CONFLICTING. The per-model modules this PR replaced with a document — `models/qwen3-8b.ts` and its nine siblings — are absent on our side and, against a merge base that predates them, ADDED on main's. Git resolves "added on one side only" by taking the addition, so a merge that reported nine conflicts and no problem had quietly restored the directory the refactor exists to remove. `typecheck` caught it, and only by luck: the restored modules assign `notices`, a field the refactor deleted from `AgentModelProfile`, so they stopped compiling. Had the refactor been purely subtractive they would have compiled, passed every test, and shipped as dead code nobody wrote. The check that finds this class directly is `git diff --name-status HEAD --diff-filter=A` — what the merge ADDS relative to where you were. Here it listed eleven files: ten resurrections and one genuinely new (`PanelUnavailable.tsx`, #477). Anyone merging a squashed base under a stacked branch should run it before trusting a clean merge. Verified on the result: format, lint (0 errors), typecheck, knip, `bun run test` (33/33 groups), build, chart:check (0.1.48 / 0.13.3), and coverage:check at 42537/42537 lines.
This was referenced Aug 24, 2026
cevheri
added a commit
that referenced
this pull request
Aug 24, 2026
Correcting my own claim. #479 removed a duplicate model table from `docs/AGENT.md` and wrote that every one of its ten rows "had drifted from the canonical figures", naming `gemma4:26b` at 46 s/180 s "against a measured 36 s/92 s". Both were measured. The owner's own sweep dashboard matches the removed table almost row for row — `gemma4:26b` 46 s/3:00, `qwen3.8:latest` 1:02/5:47, `qwen3:14b` 42 s/2:31 — so those figures were a real sweep, not a decayed copy of another one. What separates the two sets is the CONDITION, which #465 already recorded and which `docs/llms/methodology.md` states under "Sustained load is part of the measurement": running all 300 back to back takes about three and a half hours, a machine under that load is slower, and one model scored 1/5 on a surface at 87 to 93 seconds a run and 5/5 an hour later at 36. The cells that sweep lost were re-measured rested, and the rested figures are the published ones. So the removal stands and the reason given for it did not. Two tables of one measurement is still the defect — neither said which sweep it was, and a reader had no way to tell — but the fault was an unlabelled condition rather than a wrong number, and calling one set "measured" implied the other was not. The correction matters beyond tidiness: in a repository whose whole claim is that a setting traces to a measurement, "these numbers drifted" and "these numbers are from a loaded machine" send the next person to different places. The first says fix the copy; the second says label the condition, which `methodology.md` had already done and the tables had not. The commit message on #479 cannot be edited; this is the file that can.
cevheri
added a commit
that referenced
this pull request
Aug 24, 2026
…uing against their own settings (#481) B60 asked where the bundled `summary` prose belongs. Moving it turned up something the entry beside it was doing wrong, so both are here. SUMMARY IS GONE, AND NOTHING MOVED WITH IT. Every line of it was already somewhere: the per-surface tallies and the override stories are on the family pages under `docs/llms/`, the argument for each setting is in that entry's own `rationale`, and the paragraph seven entries repeated verbatim — why a model measured at the defaults still gets an entry — is in `models/index.ts` and `docs/llms/model-tuning.md`. Each page was read before its entry's lines were dropped rather than assumed to cover them. Two of the lines should not have moved anywhere. `granite4.1:8b` said "Still open: Investigate, Optimize, Operate" and `ornith:9b` said "Still open: Investigate, Optimize, Assess, Plan", and neither is true: both lock all six. AND THAT IS THE SECOND HALF. `measured` is not decoration — the schema calls it "what was measured to justify this profile", and it is what separates a setting from a guess. Two entries had their PRE-OVERRIDE numbers under the words "at these settings": granite4.1:8b "3/6 modes locked, 24/30 runs passed at these settings" ornith:9b "2/6 modes locked, 25/30 runs passed at these settings" Both carry overrides, and both lock 6/6 at 30/30 with them. So the field whose job is to justify a setting was reporting the numbers that setting was added to fix, as though the setting had produced them — arguing against itself. #465 moved the settings when it took the last cells to 30 of 30; the prose was carried across mechanically and did not move with them. Rewritten to say which numbers came from where. `granite4.1:8b` names the defaults and then the result with its two settings, which the family page supports cell by cell. `ornith:9b` names an earlier round, attributes the plan cell to the extra turn it was added for, and explicitly declines to explain the other four: a plan-mode setting cannot have fixed Investigate, Optimize and Assess, and what did is not recorded anywhere I could read. Saying so is better than a plausible sentence nobody measured. The 6/6 and 30/30 figures come from the owner's sweep dashboard, which also settles that the two "Still open" lines were stale. Both digests in `model-resolution-table.test.ts` were edited BY HAND, beside a note saying why — which is the workflow that file documents for itself, and the reason it refuses to be regenerated. Verified: format, lint (0 errors), typecheck, knip, `bun run test` (33/33 groups), build, and coverage:check at 42544/42544 lines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ten models run every agent surface. This documents them, ships the settings three of
them needed to get there, and removes everything about the models that did not make it.
What was measured
Six surfaces, one question each, five consecutive runs per surface, per model — at the
90-second per-turn limit the product ships. 300 runs, 300 passes.
gemini-3.5-flash-litegranite4.1:8bgranite4.1:30bornith:9bqwen3.5:9bgemma4:26bqwen3:8bqwen3:14bqwen3:4bqwen3.8:latestFive CONSECUTIVE is the bar, and it is the bar because a single run says nothing: cells
that had passed once came back 3 of 5 when asked again.
Three cells needed a fix to get there
All three were query-optimization and all three ended identically — the model attempts its
report, is corrected once, and then returns nothing at all. No stopping prose, no closing
statement, and both are written whenever a turn holds any text. An empty completion, which
the loop reads as a model that chose to stop.
Asking that turn once more took
granite4.1:8bfrom 3/5,qwen3.5:9bfrom 4/5 andgemini-3.5-flash-litefrom 4/5 to 5/5 each.qwen3.5:9bneeded one thing more, and it is the first per-model turn limit here. Its planturn lands at 92 to 94 seconds against a shipped 90 while its other five surfaces clear
comfortably, so the cell was decided by which side of the line one turn fell on — 1/5 on
this build and 1/5 on the build from before this branch merged
main, which rules out botha regression and a hot laptop. It asks for 150 by name. Every other model keeps the shipped
limit, because that limit is a promise about how long a person waits rather than a fact
about any model.
Configuration stays in the environment
There is no settings screen for choosing a model, and this PR adds none. Provider, model
and any key are read at startup, which keeps the choice with whoever operates the server
instead of making it a permission to grant and audit per user.
docs/llms/setup.mdnowsays that as a decision, and documents the hosted case beside the local one, which it did
not before.
Driven through the rail as well
Every figure above was produced over HTTP. A new Playwright spec drives all ten models the
way a person does — log in, pick the sample connection, type the objective, press Start,
wait for the run to finish on screen — then reads the same goal verdict the API runs are
scored on. 57 of 60 passed; the three that did not are the shapes the ledger records
anywhere else, and one run is not five, so the API sweep remains the authority on rates.
It found something no API run could.
data-analysistakes consent BEFORE the run opens —it executes a statement the model wrote and presents the result — and a run opened over
HTTP passes that as a parameter. The card a user actually sees had never been exercised.
The spec earned three corrections before it was worth believing: it passed vacuously until
it was made to watch a run START, asserted nothing about the outcome until it read the
verdict, and failed three models for answering plan mode with a refusal, which is one of
that surface's two valid answers.
What was removed
Sixteen model profiles, and the five per-model switches whose only carriers were among
them. Each call site folds to the default the ten were measured under, so no shipped run
changes — and a knob nothing turns is what
models/profile.tsexists to refuse.gemini-3.5-flash-litegains the profile it never had: it was measured 6/6 and left withno file, which the same rule forbids.
Comments that cited unsupported models by name are rewritten — 135 references across 21
files. The counts, the shapes and the surfaces are unchanged; only the identities are gone,
because what justifies the code is that a model did this on a real run with a ledger to
prove it, not which one.
What this does not claim
captured inventory and is not measured here.
user will ask.
something, presented it and cited it. Whether the SQL was the right SQL is separate.
all 300 back to back took three and a half hours and returned 252 of 270, with one model
scoring 1/5 on a surface at 87-93 seconds a run and 5/5 an hour later at 36. Every cell
that sweep lost was re-measured rested, and that is where the figures come from.