Skip to content

M2 completion: make a run answer (record of work already landed) #341

Closed

Description

@cevheri

Part of #325. Opened as a record rather than as a work item: the work is done, and this exists so the epic does not read as though it never needed doing.

Why this issue exists

#329 closed on 2026-08-12 with every task in its scope complete — durable runs, an authenticated HTTP surface, a gated rail, 100% coverage, all six gates green. Its acceptance criteria never said a run answers the question it was asked, so nothing checked whether one did.

Nothing could check, either: until #336 the rail could not start a run on any connection a zero-config deployment ships (backlog B22), so the surface had never been driven end to end against a real model. The gap was invisible while the door was locked.

The three PRs below closed it. They reference no issue, which is what prompted this record.

The evidence

Nine agent runs against Gemini and the bundled SQLite and LibreDB samples, read back from the durable ledger (.workflow-data/streams/chunks/*.bin) rather than from the screen. Two of them were the maintainer's own.

Started Run Mode Status Tools Events Report
13:29:02 arun_23daf33 agent failed 3 7 none
13:29:52 arun_7529b42 agent failed 11 37 none
13:41:11 arun_0cbe5fc planning succeeded 0 3 none
13:42:01 arun_59aa818 agent succeeded 5 17 none
13:48:29 arun_5b75c31 planning succeeded 0 3 none
13:49:16 arun_eceb045 agent unfinished 5 16 none
13:52:05 arun_5b5fb3a planning failed 0 3 none
13:52:28 arun_2868d80 agent failed 0 4 none
13:53:53 arun_bb6cfd4 planning failed 0 3 none

Nine runs, zero reports. A planning run's entire ledger was three events, which is the shape of a run that had nothing to say.

Every layer M2 built did its job — schema capture, drafting, policy-bound read-only execution, EXPLAIN, artifact storage, the ledger, the live timeline. Nothing made a run answer.

Findings

F1 — Planning mode was mute by construction. The default mode called the model, received a plan and discarded it. The only prose-carrying event (report-composed) requires evidence; evidence requires an artifact or a context snapshot; planning has an empty tool set by contract. The plan had nowhere to go. Fixed in #338 by closing-statement, an event whose whole point is that it cites nothing.

F2 — A run could end succeeded without answering. compose_report is a tool the model may simply not call, and nothing checked that it did. One run inspected the schema, ran the average-salary query, ran EXPLAIN on it and read the indexes on salary — a competent investigation of exactly the question asked — then finished succeeded with the answer nowhere. #338 records stopReason, so the rail now states "the model stopped without composing a cited report" and shows what the model did say. The status word itself is deliberately unchanged — see the hand-off below.

F3 — The default connection could not run the agent, and the rail did not say so. Phase 1 targets PostgreSQL and SQLite; the LibreDB sample has no database-native read-only execution profile, and it was the connection a zero-config deployment landed on. The run was accepted, spent a model call and failed as internal. Fixed twice in #338: an engine-unsupported reason with words a user can act on, and the SQLite sample now leads the built-in seeds so a first run starts where the agent works.

F4 — One model call could spend the entire run budget. Originally recorded as a "398-second hang"; measuring from the ledger instead of from polling shows the run ended at 300.0 s — exactly AGENT_RUN_DEADLINE_MS. The deadline worked perfectly, and the 398 was an observation time reported as a duration. What was real: the run deadline was the only bound on a single call. #339 adds AGENT_MODEL_TURN_TIMEOUT_MS (90 s) and reports model-timeout separately from deadline-exceeded. Why that one call never answered is still unknown — it now costs 90 s rather than the run.

F5 — Turn-budget exhaustion lost the work it had done. AGENT_MAX_MODEL_TURNS is 16. One run obtained all eight table counts in a single query, then re-verified each table individually and hit the ceiling, ending failed with no partial statement. #338's stopReason: turn-limit now says so and points at what was gathered. The model's strategy is what wastes turns, and that belongs to #330's goal verifiers.

F6 — A log line that named neither the connection nor the reason. logger.error(message, error?, context?) takes the error second; four call sites passed their context object there, so the formatter rendered Unknown: [object Object] and the context slot stayed empty. The loudest fired on every managed-connections request. Fixed in #339, with a scanning test guarding the class because error?: unknown makes the shape uncatchable by the compiler.

What landed

Handed to #330, not deferred

B24 — a run that answers nothing is still succeeded. AgentRunTerminalStatus has no word for "finished, and did not answer", and #330's acceptance criteria already name needs_input, a status this codebase does not have, plus a policy gate requiring "citation present for every final finding".

failed was deliberately not spent on it: it would be wrong for planning mode, which has no tools and cannot produce evidence, so a planning run that returned a good plan did exactly what the mode is for; changing ledger semantics twice means every past run is re-read under a new meaning twice; and #330 asks for goal verifiers per template, which implies the rule differs by workflow type rather than being one predicate over every run.

Also open, tracked in docs/BACKLOG.md

  • B23 — seed eligibility is decided against the browser's last descriptor fetch, so a seed repointed server-side mid-session is not seen until the next fetch.
  • D3 — POST /api/db/test-connection opens its own provider, so testing a connection to an already-open single-writer file fails on the first handle's exclusive lock. The connection modal tests before it saves, which makes the built-in LibreDB sample uneditable.

The process lesson

M2 passed on unit tests, 100% line coverage and a rail that renders. None of those can observe that a run said nothing, because the code ran correctly — what was missing was a requirement. The existing test asserted the model's prose was returned to the caller, which is a correct test of the wrong thing: a user reads the ledger, not a return value.

That is the argument for #330's eval gates being built before its templates, and it is recorded in that issue.

Activity

  1. cevheri commented on Aug 12, 2026

    @cevheri
    MemberAuthor

    Closed on creation: this is a record, not a work item. All three PRs are merged (#336 25f5b4f, #338 793cae7, #339 f54824c), and what was deliberately not done is tracked as B24 against #330.

    Verified on merged main with a cleared browser: a first run lands on the SQLite sample, and an agent run reports Run failed — "The run reached its step limit before it finished. What it had gathered is above." Before this work the same run showed a bare Run failed with no reason and stored results with no explanation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions