Skip to content

feat(agent): let a run say what it concluded, and how it stopped - #338

Merged
cevheri merged 3 commits into
mainfrom
feat/agent-run-output
Aug 12, 2026
Merged

cevheri merged 3 commits into
mainfrom
feat/agent-run-output

Conversation

@cevheri

@cevheri cevheri commented Aug 12, 2026

Copy link
Copy Markdown
Member

Phase A1 + the honesty half of A2 from the roadmap. Nine live runs on 2026-08-12 produced zero visible output between them; this is why.

What was wrong

conclude() is every terminal path's single exit, and it held both missing pieces while writing neither:

await service.finish(runId, status);              // stopReason and text dropped here
return { runId, status, stopReason, turns, text };

Planning mode — the default — called the model, received a plan, and discarded it. Its whole ledger was two events. Agent runs read the database competently and ended succeeded with the answer nowhere.

What changed

closing-statement carries the model's prose, under its own event kind rather than a lenient report-composed. It cites nothing and claims nothing, and that is the point: planning mode has no tools, so it can never produce evidence and could never have composed a report — being unable to speak at all was the consequence. Written only when the prose is non-empty.

stopReason on run-finished — report-composed | model-stopped | cancelled | deadline-exceeded | turn-limit. This is what separates succeeded from answered. A run that stops without composing now says so in the rail rather than presenting silence as success. It sits beside reason (why a drive died outside the loop); reason wins when both are present, since a dead drive is the more specific account.

finish takes an ending object rather than two positional optionals of different unions.

What it deliberately does not do

It does not demote an unanswered run to failed. The terminal vocabulary has no partial, and a planning run that produced a good plan did not fail. Making the ending legible is the precondition for that policy call, not the call itself.

Two guards, both earned

  • The store's event-kind whitelist was a hand-written array of the union's own type — which does not catch an omission. Adding closing-statement left it behind, and the first run to write one made its own ledger unreadable: the validator rejected an event the writer had just appended. Derived from a total record now, so the next kind fails typecheck instead.
  • STOP_SENTENCES is total over the stop-reason union, so a reason added later cannot reach a user as an unlabelled ending.

Verification

Live against the dev server with Gemini, planning mode:

Run opened in planning mode
Run started in planning mode
Closing statement — "To determine which departments have the highest average salary…
                    I would follow a systematic investigation plan: 1. Inspect the Schema…"
Run succeeded — "The model stopped without composing a cited report."

Six local gates green, coverage 100% (33711/33711).

Nine live runs on 2026-08-12 produced zero visible output between them. Planning mode
— the default — called the model, received a plan and discarded it; agent runs read
the database competently and ended `succeeded` with the answer nowhere. This is the
first half of finishing M2: the loop already knew both things and threw them away.

`conclude()` is every terminal path's single exit, and it held the model's closing
prose and a five-member stop reason while writing neither:

    await service.finish(runId, status);              // stopReason, text dropped
    return { runId, status, stopReason, turns, text };

Both now reach the ledger, because the ledger is the only thing that outlives a drive.

**`closing-statement`** carries the prose, under its own event kind rather than a
lenient `report-composed`. It cites nothing and claims nothing, and that distinction is
the point: planning mode has no tools, so it can never produce evidence and could never
have composed a report — a mode that cannot speak at all was the consequence. Written
only when the prose is non-empty; an empty entry would record that the model spoke.

**`stopReason` on `run-finished`** says how the loop ended: `report-composed`,
`model-stopped`, `cancelled`, `deadline-exceeded`, `turn-limit`. It is what separates
"succeeded" from "answered". A run that stops without composing now says so in the
rail — "The model stopped without composing a cited report" — instead of presenting
silence as success. It sits beside `reason`, which says why a drive died before or
outside the loop; the two are different questions and `reason` wins when both are
present, since a dead drive is the more specific account.

`finish` takes an ending object rather than two positional optionals of different
unions, which is the call site that eventually passes one in the other's place.

One thing this does NOT do: it does not make an unanswered run `failed`. The terminal
vocabulary has no `partial`, and demoting a planning run that produced a good plan
would be wrong. Making the ending legible is the precondition for that policy decision,
not the decision itself.

Two guards, both earned:

- The store's event-kind whitelist was a hand-written array of the union's own type,
  which does not catch an omission. Adding `closing-statement` left it behind and the
  first run to write one made its own ledger unreadable — the validator rejected an
  event the writer had just appended. It is derived from a total record now, so the
  next kind fails typecheck instead.
- `STOP_SENTENCES` is total over the stop-reason union, so a reason added later cannot
  reach a user as an unlabelled ending.

Verified against the dev server with Gemini: a planning run now renders its plan and
its own admission that it did not compose a report.
Six distinct failures reached the rail as three labels, and the collapse was hiding the
only one a user can act on without an operator.

A live run found it. A Gemini free-tier quota — 15 requests a minute — was exhausted by
a morning of testing, and the rail reported that the model provider "is not configured
or could not be reached". The provider was configured and had answered seconds earlier;
it had asked for less traffic and named the wait: 54.6 seconds. The user was sent to a
settings screen to fix a problem that fixes itself.

`AgentRunFailureReason` gains three members:

- `model-rate-limited` — the provider answered and refused on volume. The only model
  failure that clears on its own, so the sentence says to wait rather than to check
  anything.
- `model-unauthorized` — the credentials were rejected. Retrying will not help; an
  operator is genuinely needed.
- `engine-unsupported` — the connection's engine has no database-native read-only
  execution profile. This is the LibreDB sample, which is the DEFAULT connection of a
  zero-config deployment, so it is the first thing a new user meets. Reported as
  `internal` it read as a server fault and sent them to a log that could only repeat
  what their own connection already said.

The classifier checks the LLM subclasses before `LLMError` itself, which they all
extend, and `ExecutionProfileError` before the generic classes.

This overturns a considered decision rather than filling a gap, so the old reasoning is
recorded next to the new code and in `docs/AGENT.md`: the model failures were one label
because "the user's next move is the same for all of them — look at the model
settings". That held right up until the failure whose next move is to wait a minute.

`FAILURE_SENTENCES` is total over the union, so each new reason had to be given words
before this compiled — which is the property that made the change safe to make at all.

Verified live: the LibreDB sample now fails with "The agent cannot run on this database
engine: it offers no read-only execution profile" instead of the generic internal
sentence.
@cevheri

cevheri commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

A3 pushed to this branch (97c97df)

Six distinct failures were reaching the rail as three labels, and the collapse hid the only one a user can act on without an operator.

A live run found it. A Gemini free-tier quota — 15 requests a minute — was exhausted by a morning of testing, and the rail said the provider "is not configured or could not be reached". It was configured, and had answered seconds earlier; it had asked for less traffic and named the wait (54.6s). The user was sent to a settings screen to fix something that fixes itself.

AgentRunFailureReason gains three members:

Reason Next move
model-rate-limited Wait — clears on its own
model-unauthorized Fix the key; retrying will not help
engine-unsupported Investigate a different connection

engine-unsupported matters more than it looks: it is the LibreDB sample, which is the default connection of a zero-config deployment. Phase 1 targets PostgreSQL and SQLite by design, so this is the first thing a new user meets, and as internal it read as a server fault.

This overturns a considered decision rather than filling a gap, so the old reasoning is recorded beside the new code and in docs/AGENT.md: the model failures were one label because "the user's next move is the same for all of them — look at the model settings". That held right up until the failure whose next move is to wait a minute.

Live verification

Agent mode on the SQLite sample — the first run in this programme to show an answer:

Closing statement — "### Department Average Salaries
                     1. Sales: ~79,643.10   2. Marketing: ~71,251.05
                     3. Finance: ~69,545.40  4. Development: ~61,466.30 …"
Run succeeded — "The model stopped without composing a cited report."

Agent mode on the LibreDB sample:

Run failed — "The agent cannot run on this database engine: it offers no read-only execution profile."

Six gates green, coverage 100% (33718/33718).

A client with no persisted active connection selects the first connection it is
served, so the order of the two built-in samples decides where a brand-new user
starts. It started them on LibreDB.

Phase 1 of the agent targets PostgreSQL and SQLite, and the LibreDB engine has no
database-native read-only execution profile — so a zero-config deployment put every
new user on the one connection an agent run can never execute against. Turning the
runtime on and pressing Start failed immediately, on the default connection, before
the user had chosen anything.

Swapping the two pushes fixes it. An operator's own seed config still leads both:
those are appended before either sample, so this only decides which built-in comes
first.

The test states the reason rather than the order, so it fails for the thing that
matters if the sequence is ever rearranged again.

Verified against a cleared browser: a new session now opens on "Sample (Employees)"
and an agent run on it answers — "There are 1,000 employees in total, and they are
held in the `employee` table."
@sonarqubecloud

Copy link
Copy Markdown

@cevheri
cevheri requested a balanced review from Copilot August 12, 2026 12:24
@cevheri
cevheri merged commit 793cae7 into main Aug 12, 2026
29 of 30 checks passed
@cevheri
cevheri deleted the feat/agent-run-output branch August 12, 2026 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant