Repository navigation
fix(agent): remind a run that read the database and then narrated - #416
Merged
Merged
Conversation
A turn with no tool calls ends the run. For a model that established nothing that
is the right ending: there is no report to ask for. But a run that HAS read
something and then writes its findings as prose has done the whole job except the
one call that records it, and ending there throws the readings away: the ledger
keeps the artifacts, `compose_report` was never called, and the goal verifier
scores the run `no-report`.
Measured against a local Ollama endpoint, three models did exactly that, each
after reading the database:
mistral-small3.2:24b 4 readings, then prose
lfm2:24b 11 readings on a data-analysis run, then prose
qwen3.5:2b 42 readings, then prose
None was refusing to report. Each had answered in the register a chat model
answers in. Replaying one captured request five times showed the same model
calling tools three times and saying it had none twice, so the ending was not a
property of the model either: it was whichever way that turn happened to land.
The loop now sends one message when a prose turn arrives AFTER at least one tool
call, and takes the turn again. Deliberately narrow:
- never when the run has taken no readings, which is a model that is stopping
- once, because a model that will not call the tool the first time it is asked
is stopping rather than hesitating, and a second telling spends a turn to
learn that
- a message, not a rule change: it spends no statement, consults no deadline
admission and reaches no database
Measured after the change: `mistral-small3.2` operations went from `no-report` to
`answered`, confirmed in `.workflow-data` (four readings, then `report-composed`).
`lfm2:24b` investigation likewise. `lfm2:24b` data-analysis still ends
`no-report`: the reminder was sent after nine readings and the model narrated
again. So this rescues runs that were one call short, not every run that ends
without a report.
Ten eval scripts gain one prose turn. `scriptedModel` treats a script running out
as simulated process death (agent-scripted-model.ts:73), so a script that ended
exactly on its prose turn now needs the turn the reminder asks for. No assertion
changes, including #341 F2's: a competent investigation that never reports is
still `succeeded` and `unanswered`, and now says so about a run that was reminded
and narrated anyway.
…d turns The reminder names `compose_report`, so it may only reach a run that holds that tool, and it costs a turn, so it may only be sent where a turn remains. Neither bound was applied. A PLANNING run holds no tools at all. `anyToolCalled` was set the moment any tool-call part arrived, including one the offered-set check then refused, so a plan run that reached for a tool was told to call `compose_report` — a rule stated to a model whose tool set cannot satisfy it, which is the #350/#356 failure. The same looseness made the sentence untrue in agent mode: a name the model invented reached nothing, and the run was still told it had taken readings and then asked to cite artifacts it has not got. The flag is now set only for a tool this run was actually given, read from the set handed to the SDK. Planning is covered by the same line rather than by a second check: `declaredTools` returns nothing outside agent mode. The turn ceiling was the other gap, and it sits exactly where this fix is aimed. `AGENT_REPORT_RESERVE_NOTICE` already tells a model to wrap up two turns from the end, and a small model wraps up in prose — so the reminder was most likely to be spent on a turn the loop would not grant. Measured with `maxTurns: 2`: a run that read and then narrated ended `succeeded` / `model-stopped` before this change and `failed` / `turn-limit` after it, which is a false account of why the run ended. The reminder now reads the loop's own ceilings before spending one. Five scripted prose turns come back out of the eval scenarios. Every one of them was a run reaching for a tool it was never offered — `read_monitoring`, `recommend_change`, `compare_plans`, a planning `run_read_query` — so they were the defect's own footprint rather than scenarios that changed. Tests: five for the reminder itself, which had none of its own.
cevheri
force-pushed
the
fix/remind-a-run-that-read-and-narrated
branch
from
August 18, 2026 00:51
828c69a to
6d6e594
Compare
Member
|
Rebased onto A planning run was told to call That is the #350/#356 shape, and the same looseness made the sentence untrue in agent mode: a name the model invented reached nothing and the run was still told it had taken readings, then asked to cite artifacts it has not got. The flag now reads the set the SDK was actually handed, which covers planning by construction — The reminder could spend a turn the loop would not grant. Measured with
A model that stopped on its own is not a run that ran out of turns, and the ceiling is exactly where this lands: Five scripted prose turns came back out of the eval scenarios, and every one was a run reaching for a tool it was never offered — Five tests added for the reminder, which had none of its own — the one-shot and the no-call guard were only pinned indirectly, by scripts running out.
|
|
cevheri
added a commit
that referenced
this pull request
Aug 18, 2026
…d what the nudges do not (#421) * docs(agent): say which runtime the measured rows came from, and record what the nudges do not `methodology.md` named one commit, `907e83c`, and the pages under it report figures that commit cannot produce: the rows marked *(fixed)* and every "after" column were taken with the report reminder and the present-before-report notice in place, and those landed as #416 and #417. A reader following the method against `907e83c` alone would fail to reproduce them and have no way to see why. Both commits are named now, along with the review narrowing each notice took before it landed and why the recorded runs sit inside the narrower triggers. B51 records what those two notices do NOT do. `recordEvent` is called for what a run did — the capture, the drafted statement, the closing prose, the answer, the report — and never for what the server said to the model, so a rescued run and a run that needed no rescue leave the same ledger, and a `compose_report` the loop withheld leaves no trace that a call was made. That is a measurement problem before it is a design one, because `docs/llms/` is built by reading those ledgers and its whole claim is that each figure comes from an observed run. The entry carries the second half too: three notices now, each with its own boolean, its own guard set and its own delivery, and the guards are the part that keeps being got wrong — both of the new ones shipped with a condition that named one thing and read another, and both were caught in review rather than by a gate. * docs(agent): name all three measured revisions, and scope "once" to a drive Three findings from review, all three verified against the code and the history. **The revision table was missing one.** #406 (`df15e56`) is an ANCESTOR of the `907e83c` baseline, so the before-figures on `qwen3.8` (`no-answer` in 84.9s) and `muse-glimmer` (`no-answer` after 234.4s) are reproducible from neither commit this page named — they are from `7256f93`, before that fix. "Everything else is the baseline" was therefore wrong for exactly the two rows a reader is most likely to try to reproduce. All three revisions are named now, in a table that says which figures belong to which. **"One-shot" was true of a drive, not of a run.** The three booleans are `let`s inside `runInvestigation`, and that function is also what resumes a run a dead process left running — so a resumed drive starts with all three false and can repeat a notice. Two of the three are partially covered by accident: the present-before-report notice reads `answer-composed` off the ledger, but a REFUSED `present_answer` writes no event and so is not covered; nothing bounds the report reminder or the reserve notice across a resume at all. That is the same defect B51 already describes rather than a second one, so it is folded into that entry: recording delivery is what would make "once" mean once. The declarations in `investigation.ts` say it too — a comment-only change, because a reader reaching for `reserveAnnounced` should not have to derive the scope from the resume path three hundred lines up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A turn with no tool calls ends the run. For a model that established nothing that is the right ending: there is no report to ask for.
But a run that HAS read something and then writes its findings as prose has done the whole job except the one call that records it. Ending there throws the readings away: the ledger keeps the artifacts,
compose_reportwas never called, and the goal verifier scores the runno-report.Measured
Three models did exactly that against a local Ollama endpoint, each after reading the database:
mistral-small3.2:24blfm2:24bqwen3.5:2bNone was refusing to report. Each had answered in the register a chat model answers in.
It is not a fixed property of the model either
mistral-small3.2failed Operate with zero tool calls, which reads as a model that cannot use tools. It can — given the same tools and a simple prompt it calledinspect_operationsimmediately.So the exact request the agent had sent was captured and replayed five times:
Same request, same model, same server: 60%. Because the run ended at the first prose turn, that 40% was being recorded as a model that could not drive an agent run at all.
What this changes
The loop sends one message when a prose turn arrives AFTER at least one tool call, and takes the turn again. Deliberately narrow:
Verification
Measured after the change, confirmed on the ledger in
.workflow-data:mistral-small3.2:24bno-reportanswered— four readings, thenreport-composedlfm2:24bno-reportansweredlfm2:24bno-reportno-reportThe last row is the honest bound: the reminder was delivered after nine readings and the model narrated again. This rescues runs that were one call short, not every run that ends without a report.
Eval scripts
Ten scripts gain one prose turn.
scriptedModeltreats a script running out as simulated process death (agent-scripted-model.ts:73), so a script that ended exactly on its prose turn now needs the turn the reminder asks for.No assertion changes, including #341 F2's: a competent investigation that never reports is still
succeededandunanswered, and now says so about a run that was reminded and narrated anyway.Scope
No provider is touched: every file under
src/lib/llm, plusprovider-registry.tsandmodel-adapter.ts, is unchanged, so Gemini and OpenAI deployments behave exactly as before.format,lint,typecheck,knip,test(all 30 groups) andbuildpass locally.