diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a8602a4c..7b61c4f5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -138,6 +138,14 @@ jobs: - name: Lint the tooling scripts run: pnpm lint:tools + # A repo-local SECOND checkout must never be collected by a root run (CR-90). This is the one check + # whose defect CI could not have caught on its own: the leak lives in `.claude/worktrees/`, hidden by a + # LOCAL `.git/info/exclude` entry, so a clean CI checkout never has it. The vitest exclusions defend a + # developer's tree; this step is what stops them being weakened — without it the guard would be the only + # root-`ci` check missing here, which is the #312 divergence the two steps above exist to close. + - name: Test isolation (no repo-local checkout in the run) + run: pnpm lint:test-isolation + # RUN the artifact this job just built, through the SAME `pnpm smoke:cli` script the root `ci` script # calls — a check that exists in only one of the two is exactly the #312 divergence this change closes. # Until now nothing in the required gate executed diff --git a/.gitignore b/.gitignore index 4822ff15..84b48485 100644 --- a/.gitignore +++ b/.gitignore @@ -53,6 +53,19 @@ coverage-tmp/ # Claude Code transient state (skills/agents ARE committed; runtime locks are not) .claude/scheduled_tasks.lock .claude/**/*.lock +# A repo-local SECOND checkout (CR-90). `.claude/worktrees/` was only ever hidden by `.git/info/exclude`, +# which is local and untracked — so a fresh clone did not carry the rule and nobody could see the reason. +# Tracked here so it travels. The vitest side of this is `REPO_LOCAL_CHECKOUTS` in vitest.config.ts; the two +# lists are kept in step by tools/test-isolation, which fails on any collected file that turns out to live in +# a directory carrying its own pnpm-workspace.yaml. +.claude/worktrees/ +.worktrees/ +/worktrees/ +# The guard's own scratch directories. It clears them explicitly on every exit path, including a failure — +# but `.claude/__test_isolation_fixture__/` is a SIBLING of `.claude/worktrees/`, so the entry above would not +# have covered it if a crash ever did leak one, and a leaked fixture is CR-90's defect self-inflicted. +__test_isolation_fixture__/ +__test_isolation_detector_probe__/ # Private analysis — never commit, never reference from tracked files docs/analysis/private/ diff --git a/README.md b/README.md index 13bf182a..9283d547 100644 --- a/README.md +++ b/README.md @@ -1,158 +1,213 @@ # Relavium -> **Start as an agent. Ship the workflow. Own every run.** -> A multi-surface, local-first AI agent workflow platform — a product of [HodeTech](https://github.com/HodeTech). - -Relavium meets you where you already work — in conversation — and gives that -conversation somewhere to go. You **start as an agent**: a multi-turn session in your -terminal, in VS Code, or in a desktop chat panel. When a flow proves itself, you **ship -the workflow**: export the session to a git-committable, multi-agent, multi-model -`.relavium.yaml` pipeline that runs identically in your editor, your terminal, and your -CI. Or author workflows directly. Either way you **own every run** — every step -debuggable, every token and dollar tracked, every artifact yours, nothing leaving your -machine unless you choose it. - -## Why Relavium? - -- **Four surfaces, one engine.** Desktop (Tauri), CLI, VS Code, and (planned) the web - portal run the _identical_ pure-TypeScript engine. No Python sidecar, no single-tool - lock-in — every surface is a first-class execution target. -- **A chat-to-workflow continuum.** Other tools make every session ephemeral. Relavium - sessions are persistent, resumable, and one-click exportable into a reviewed, - committed workflow. -- **You own your LLM seam.** Multi-provider routing with fallback chains - (`[claude → gpt-4o → gemini]`) is first-class through Relavium's own `@relavium/llm` - abstraction over the official provider SDKs — no Vercel AI SDK, no LangChain. -- **Local-first by design.** Zero cloud, no account required. Your API keys live in your - OS keychain — never in plaintext, never in logs. Optional managed inference and cloud - execution are planned extensions on the same engine. -- **Workflows are git objects.** `.relavium.yaml` files are diffable, reviewable, - PR-able, and shareable — team infrastructure, not a proprietary JSON blob or buried - Python. -- **Multimodal, end-to-end.** Image / audio / video as input and output — including - rule-driven media generation — flow through the same seam and engine. - -## Highlights - -- **Chat-to-workflow export** — turn a proven session into a reusable `.relavium.yaml`. -- **Persistent, resumable agent sessions** — no run is ever ephemeral. -- **Live execution** — tokens stream as the run progresses; parallel branches run together. -- **Multi-model fallback chains** — runs survive provider outages and rate limits. -- **Checkpoint & resume** — pause and resume at any node boundary, even across processes. -- **Human gates with timeout policy** — pause for an approve / reject / input decision. -- **Per-node cost waterfall** — token and dollar attribution per node, per model. -- **Interactive Home** — a bare `relavium` invocation opens a management center: start agents, - monitor runs, browse history, manage providers. -- **MCP client** — agents consume tools from external MCP servers over stdio, HTTP, SSE, and - WebSocket, with secrets in the keychain. -- **Live model catalog** — browse and switch models mid-session; per-model cost tracking. -- **Local-first, zero-install posture** — BYOK, OS keychain, no sign-up. - -## Getting started - -The CLI is the first usable surface. It ships as a single npm binary — `npm install -g relavium` -(the public npm publish is the final maintainer step of the **v0.1.1** release; until it lands, build from -source per [local dev setup](docs/runbooks/local-dev-setup.md)). Then **start as an agent → ship the workflow -→ own every run**: +

+ Relavium — start as an agent, ship the workflow, own every run +

+ +

+ Start as an agent. Ship the workflow. Own every run. +

+ +

+ A local-first AI agent platform that turns productive conversations into
+ version-controlled, multi-agent workflows — on one pure-TypeScript engine.
+ A product of HodeTech. +

+ +

+ npm version + CI status + Node.js 22 or newer + Strict TypeScript + Local-first BYOK + Proprietary license +

+ +

+ Get started · + Why Relavium · + Architecture · + Status · + Documentation +

+ +Relavium begins where agent work naturally begins: in conversation. Explore a task in a +persistent session, keep the flow that proves useful, and graduate it into a reviewable +`.relavium.yaml` workflow. The conversation and the workflow are not separate products — +they are two entry points into the same engine, tool registry, model seam, and event stream. + +## From conversation to infrastructure + +

+ A Relavium agent session becoming a committed workflow and observable runs +

+ +Most agent tools make you choose between a flexible chat and an operable workflow. Relavium +treats them as a continuum: + +1. **Explore in an agent session.** Work conversationally with streaming output, tools, + model controls, persistent history, and human approval. +2. **Promote what works.** Export the proven flow into git-native YAML that can be reviewed, + changed in a PR, and shared without embedding provider keys. +3. **Run it deliberately.** Execute locally or in CI with typed events, durable history, + human gates, checkpoints, and cost controls. + +You can also author the workflow directly; conversation is an on-ramp, not a requirement. + +## Get started + +The published CLI is the fastest way to use Relavium. It requires **Node.js 22 or newer**. + +```bash +npm install -g relavium +``` + +1. Connect a provider. The key is read from stdin and stored in the OS keychain — + never passed through argv. ```bash -# 1. Point Relavium at a provider — your key goes to the OS keychain, never a file relavium provider add anthropic -echo "$ANTHROPIC_API_KEY" | relavium provider set-key anthropic # the key is read from stdin, never argv +printf '%s\n' "$ANTHROPIC_API_KEY" | relavium provider set-key anthropic +``` + +2. Start as an agent. -# 2. Start as an agent — a multi-turn session in your terminal +```bash relavium chat -# …converse until a flow proves itself, then run /export inside the REPL -# to ship the session to a git-committable .relavium.yaml +``` + +3. Run `/export` inside the session to create a git-committable workflow. Execute it + interactively, or stream NDJSON for CI with `--json`. -# 3. Own every run — execute the workflow and stream every event (CI-friendly with --json) +```bash relavium run ./my-workflow.relavium.yaml --json ``` -Prefer to author directly? `relavium create` scaffolds an agent or a minimal single-agent workflow, and -`relavium import` / `relavium export` move them between projects. The full surface is the -[CLI command reference](docs/reference/cli/commands.md). +Prefer to author first? `relavium create` scaffolds an agent or a minimal workflow; +`relavium import` and `relavium export` move validated artifacts between projects. See the +[CLI command reference](docs/reference/cli/commands.md) for the complete surface and the +[local development runbook](docs/runbooks/local-dev-setup.md) to build from source. + +## Why Relavium + +| | | +|---|---| +| **Conversation becomes infrastructure** | A useful session can graduate into a durable workflow instead of disappearing into chat history. | +| **Git-native by construction** | Workflows are diffable `.relavium.yaml` files — reviewable in pull requests and owned by the team that runs them. | +| **Multi-model without framework lock-in** | Relavium owns its `LLMProvider` seam and routes across Anthropic, OpenAI-compatible providers, and Gemini without LangChain or the Vercel AI SDK. | +| **Local-first control** | Local BYOK is the default, no Relavium account is required, and provider keys are stored in the OS keychain rather than workflow files. | +| **Execution you can inspect** | Typed event streams, local run history, human gates, checkpoints, retries, fallback chains, and cost controls make a run observable. | +| **One engine, multiple surfaces** | `AgentSession` and `WorkflowEngine` share one platform-pure core designed for the CLI, desktop, VS Code, and future cloud workers. | ## Architecture -```mermaid -flowchart TD - subgraph Surfaces - D[Desktop · Tauri] - C[CLI] - V[VS Code extension] - P[Web portal · planned] - end - subgraph Engine["@relavium/core — one pure-TypeScript engine"] - WE[WorkflowEngine] - AS[AgentSession] - BUS[(RunEventBus · ToolRegistry)] - WE --- BUS - AS --- BUS - end - SEAM["@relavium/llm seam"] - PROV[Anthropic · OpenAI/DeepSeek · Gemini] - D --> Engine - C --> Engine - V --> Engine - P --> Engine - Engine --> SEAM --> PROV -``` +

+ Relavium architecture: multiple surfaces over one engine with AgentSession, WorkflowEngine, the LLM seam, MCP, and durable storage +

+ +The center of Relavium is `@relavium/core`, a strict TypeScript engine with **zero +platform-specific imports**. It exposes two co-equal entry points: -One engine, **two co-equal entry points** — `WorkflowEngine` (runs YAML pipelines) and -`AgentSession` (runs conversational chat) — sharing the same tool registry, the same -`@relavium/llm` multi-provider seam, and the same event bus. The engine has **zero -platform-specific imports**, so the same source runs in the Tauri WebView, the VS Code -host, the Node CLI, and (planned) a Bun server. Supporting packages: `@relavium/shared` -(Zod contracts), `@relavium/db` (Drizzle — SQLite locally, PostgreSQL planned), and -`@relavium/ui` (ReactFlow canvas + shadcn). See [docs/architecture/](docs/architecture/). +- **`AgentSession`** for conversational, multi-turn work. +- **`WorkflowEngine`** for declarative `.relavium.yaml` execution. -## Execution modes +Both reuse the same `ToolRegistry`, typed event substrate, and Relavium-owned +`@relavium/llm` abstraction. Official provider SDKs are confined to thin adapters; no +vendor SDK type crosses the seam. Host packages supply persistence, MCP connections, +keychain access, files, processes, and network I/O without making the engine +platform-specific. Read the [architecture overview](docs/architecture/) or the +[decision records](docs/decisions/) for the reasoning behind those boundaries. -One engine, three modes behind the one `LLMProvider` seam: +> The CLI is the currently published product surface. Desktop and VS Code integrations, +> plus managed inference and cloud execution, are under development or planned. The +> diagram shows the shared-engine topology, not equal release availability. -- **Local (BYOK)** — the default. Your keys, your machine, zero Relavium data. -- **Managed inference** — planned. Relavium's metered keys; the engine still runs locally. -- **Cloud execution** — planned. Run workflows on cloud workers for 24/7 automation and - team sharing. +## What ships today -## Status +| Capability | Available in `relavium@0.1.1` | +|---|---| +| Conversational agents | Streaming multi-turn chat, persisted sessions, resume, model reseat, context compaction, and workflow export | +| Workflow runtime | YAML parse and validation, DAG execution, parallel branches, retries, model fallback, checkpoints, and typed live events | +| Human control | Per-tool approval modes plus durable workflow gates that can pause and resume out of process | +| Operations | Interactive Home, run status and history, event-log replay, deterministic exit codes, and NDJSON output for CI | +| Providers and tools | Anthropic, OpenAI-compatible, Gemini, an inbound MCP client, built-in tools, and a live/offline model catalog | +| Local ownership | BYOK, OS-keychain storage, project-local git artifacts, and local run/session history | -The engine is complete and the CLI is feature-complete (cut as **v0.1.1**, npm publish pending). -What's shipped: +For exact command behavior and contracts, use the [reference documentation](docs/reference/) +rather than this overview. -- **Agent sessions** — `relavium chat` with persistent, resumable, exportable multi-turn sessions. -- **Workflow engine** — `relavium run` executes `.relavium.yaml` pipelines with live streaming, - checkpoint/resume, multi-model fallback, cost governance, and human gates. -- **Interactive Home** — the bare `relavium` invocation opens a management center with a - slash-command system, per-tool approval modes (ask/plan/accept-edits/auto), and context compaction. -- **MCP client** — agents consume tools from external MCP servers over stdio + network - transports, with secrets in the OS keychain. -- **Live model catalog** — onboard with a wizard, browse models, switch mid-session, track - per-model cost. -- **YAML authoring** — `relavium create` (wizard), `import`, and share-safe `export`. +## Local-first, precisely + +- **No account is required for local BYOK.** The CLI runs the engine and stores history on + your machine. +- **Provider keys do not belong in workflows or committed configuration.** Interactive + setup stores them in the OS keychain; a documented environment fallback exists for + automation. +- **Workflows remain ordinary files.** They can be reviewed, branched, reverted, and moved + without exporting from a proprietary database. +- **Network use is explicit.** LLM requests go to the provider you configure; optional + catalog refreshes and future managed/cloud modes are not hidden prerequisites for local + execution. + +The binding guarantees live in the [product constraints](docs/product-constraints.md) and +[security standard](docs/standards/security-review.md). + +## Project status + +Relavium is under active development. The **CLI is published as v0.1.1**; the pure engine, +agent-session entry point, workflow runtime, inbound MCP client, and CLI management surface +are implemented. The current engineering focus is **Phase 2.6.5 — Core Reliability +Remediation**, which hardens the execution core before the next product wave opens. + +Status changes quickly, so this README intentionally stays high-level. The canonical source +for the exact active wave, completed work, and open reliability obligations is +[docs/roadmap/current.md](docs/roadmap/current.md). + +## Repository map + +| Path | Responsibility | +|---|---| +| [`packages/core`](packages/core/) | Platform-pure agent-session and workflow engine | +| [`packages/llm`](packages/llm/) | Relavium model seam, adapters, fallback, usage, and cost logic | +| [`packages/shared`](packages/shared/) | Zod schemas and inferred types — the contract source of truth | +| [`packages/db`](packages/db/) | Local SQLite persistence with a Postgres-compatible schema and migrations | +| [`packages/mcp`](packages/mcp/) | SDK-confined inbound MCP client and schema validation | +| [`apps/cli`](apps/cli/) | Published terminal product and integration harness | +| [`apps/desktop`](apps/desktop/) | Tauri desktop surface under development | +| [`apps/vscode-extension`](apps/vscode-extension/) | VS Code surface under development | + +The full dependency graph and ownership rules live in +[docs/project-structure.md](docs/project-structure.md). + +## Development + +Relavium is a pnpm + Turborepo monorepo. For a first local verification: + +```bash +corepack enable +pnpm install --frozen-lockfile +pnpm run ci +``` -**Next: Phase 2.6 (Conversational Authoring and the First-Class CLI)** — a full-screen -Home-managed CLI with conversational workflow authoring, management browsers, competitor-breadth -tools, settings/theming, and `en`/`tr` localization. For live status and the full roadmap, see -[docs/roadmap/current.md](docs/roadmap/current.md) and the [roadmap](docs/roadmap/README.md). +Use pnpm only — never npm or yarn for workspace development. Start with the +[local development setup](docs/runbooks/local-dev-setup.md), then read +[`CLAUDE.md`](CLAUDE.md) or [`AGENTS.md`](AGENTS.md) before making changes. ## Documentation -The canonical documentation lives in [`docs/`](docs/) — start at -[docs/README.md](docs/README.md), which is organized by _the kind of question each -section answers_. +The canonical documentation is organized by the question you are trying to answer: -| Start here | | -|------------|---| -| [Vision](docs/vision.md) · [Product constraints](docs/product-constraints.md) · [UVP](docs/uvp.md) | What and why | -| [Tech stack](docs/tech-stack.md) · [Project structure](docs/project-structure.md) | What it's built with | -| [Architecture](docs/architecture/) · [Decisions (ADRs)](docs/decisions/) · [Reference](docs/reference/) | How it works | -| [Roadmap](docs/roadmap/README.md) · [Standards](docs/standards/) | Where it's going, and the rules | +| Start here | Answers | +|---|---| +| [Vision](docs/vision.md) · [Product constraints](docs/product-constraints.md) · [UVP](docs/uvp.md) | What is Relavium, and why does it exist? | +| [Architecture](docs/architecture/) · [ADRs](docs/decisions/) | How is it built, and why these boundaries? | +| [Reference](docs/reference/) | What are the exact YAML, event, CLI, database, and integration contracts? | +| [Roadmap](docs/roadmap/README.md) · [Current state](docs/roadmap/current.md) | What is shipped, active, and next? | +| [Standards](docs/standards/) · [Runbooks](docs/runbooks/) | How should the project be changed and operated? | ## License Relavium is **proprietary software** — © 2026 HodeTech, all rights reserved. It is -**not** open source and grants no rights except as expressly stated. See -[LICENSE](LICENSE) for the full terms. For licensing inquiries, written permission, or -commercial-use agreements, contact [HodeTech](https://github.com/HodeTech). +not open source and grants no rights except as expressly stated. See [LICENSE](LICENSE) +for the full terms. For licensing inquiries or commercial-use agreements, contact +[HodeTech](https://github.com/HodeTech). diff --git a/apps/cli/src/chat/persister.test.ts b/apps/cli/src/chat/persister.test.ts index e07febc5..8936f0e4 100644 --- a/apps/cli/src/chat/persister.test.ts +++ b/apps/cli/src/chat/persister.test.ts @@ -48,6 +48,11 @@ describe('createSessionPersister', () => { }); afterEach(() => { client.sqlite.close(); + // Suite-level, not per-test. Several tests below spy `updateSession`/`recordSessionCost` into a throwing + // SQLITE_BUSY stub; a test-local `vi.restoreAllMocks()` at the END of the test is skipped the moment an + // assertion above it fails, and the stub then leaks into every later test in the file — turning one real + // failure into a cascade whose reported causes are all fictional. + vi.restoreAllMocks(); }); /** @@ -250,6 +255,58 @@ describe('createSessionPersister', () => { persister.close(); }); + it('LATCHES a failed session:cancelled write, and still detaches the listener (CR-01)', async () => { + // The one arm that still called the store bare. `RunEventBus` isolates a listener throw from the + // producer, so a failing terminal write let the cancel path report success while the latch never set — + // and the latch is what refuses the next egress. A session whose terminal row never landed would resume + // believing it had ended cleanly. + const { built } = await setup(scriptedResolver([textTurn('hi')])); + const persister = createSessionPersister({ + governor: undefined, + attachDurabilityProbe: built.attachDurabilityProbe, + store, + handle: built.handle, + sessionId: built.sessionId, + agent: built.agent, + context: built.context, + now: () => Date.parse('2026-06-25T00:00:00.000Z'), + uuid: () => 'msg-x', + }); + // Observe the DETACH directly by wrapping the unsubscribe the handle hands back. Nothing else can see + // it: a second `cancel()` emits no event on an already-cancelled session, and `close()` calls the same + // idempotent unsubscribe, so both of those pass whether or not the listener leaked. + let unsubscribed = 0; + const realSubscribe = built.handle.subscribe.bind(built.handle); + vi.spyOn(built.handle, 'subscribe').mockImplementation((listener) => { + const off = realSubscribe(listener); + return () => { + unsubscribed += 1; + off(); + }; + }); + built.session.start(); + persister.start(); + const updateSpy = vi.spyOn(store, 'updateSession').mockImplementation(() => { + throw Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' }); + }); + + built.session.cancel(); + + expect(persister.durabilityFailure?.message).toContain('database is locked'); + // …and the user was still told: latching must not remove the listener-error notice. + expect(listenerNotes.join('\n')).toMatch(/database is locked/); + + // **The half that actually matters, asserted directly.** The self-detach is in a `finally`, so a throwing + // write cannot jump over it and leave this persister attached to the bus. An earlier version of this test + // asserted `close()` does not throw — which is true whether or not the listener leaked, because `close()` + // calls the same idempotent unsubscribe. The real evidence is that a SUBSEQUENT event does not re-enter a + // persister that cannot write, so `updateSession` is never called again. Deliberately WITHOUT `close()`. + expect(updateSpy).toHaveBeenCalledTimes(1); // the failing write happened exactly once + // THE assertion: the throw did not jump over the unsubscribe. Without the `finally` this is 0, and the + // persister stays on the bus — so every later event re-enters one that cannot write. + expect(unsubscribed).toBe(1); + }); + it('does not let a failed cost write advance the in-memory total (#W15-4)', async () => { // `mutableSessionColumns` makes `recordSessionCost` the SINGLE writer of the durable total, so a total // advanced past a write that did not land can never be repaired by a later flush — and a resume then diff --git a/apps/cli/src/chat/persister.ts b/apps/cli/src/chat/persister.ts index f20950da..08e8f5ff 100644 --- a/apps/cli/src/chat/persister.ts +++ b/apps/cli/src/chat/persister.ts @@ -412,9 +412,27 @@ export function createSessionPersister(deps: SessionPersisterDeps): SessionPersi case 'session:cancelled': // The session's sole terminal — mark it ended (still resumable from the persisted transcript), then // self-detach so the bus listener does not leak if the REPL's close() is skipped on an early exit. - deps.store.updateSession(record('ended')); - unsubscribe?.(); - unsubscribe = undefined; + // + // Through `persistDurably`, like every other write here (`CR-01`) — but be precise about WHAT that + // buys, because the obvious reading is wrong. The latch half is inert on this arm: `session:cancelled` + // is emitted only by `AgentSession.cancel()`, which sets `#status = 'cancelled'`, and every later + // egress entry point is already refused by `#assertSendable()` with `not_active`. Nothing reads + // `durabilityFailure` again. It is wrapped for SYMMETRY with its siblings — one arm reaching the store + // bare is how the next reader concludes the wrapper is optional. + // + // **The `finally` is the load-bearing half.** Before it, a throwing write jumped straight over the two + // unsubscribe lines and left this persister attached to the bus — so every later event re-entered a + // persister that could not write. (The user was told either way: the throw always escaped `onEvent` + // into `deliver`'s listener-error sink.) Cleanup must happen on both paths, and a failed write is + // exactly when a leak is worst. + try { + persistDurably(() => { + deps.store.updateSession(record('ended')); + }); + } finally { + unsubscribe?.(); + unsubscribe = undefined; + } return; default: return; diff --git a/apps/cli/src/commands/agent-run.ts b/apps/cli/src/commands/agent-run.ts index 21ccf674..db6abd0e 100644 --- a/apps/cli/src/commands/agent-run.ts +++ b/apps/cli/src/commands/agent-run.ts @@ -18,6 +18,7 @@ import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; import { createMcpSecretResolver, type McpSecretResolver } from '../secrets/mcp-secret.js'; import { makePlainPrinter } from './chat.js'; +import { stringifyJsonLine } from '../render/sanitize.js'; /** * `relavium agent run ` (2.Q) — invoke a single agent **one-shot** (non-interactive) on the same @@ -158,7 +159,7 @@ async function runOneShotTurn( ): Promise { let turnErrorCode: string | undefined; const renderer: (event: SessionStreamHandleEvent) => void = deps.global.json - ? (event) => deps.io.writeOut(`${JSON.stringify(event)}\n`) + ? (event) => deps.io.writeOut(`${stringifyJsonLine(event)}\n`) : // A ONE-SHOT: the session is cancelled in `finally` right after, so suppress the session-continuity recovery // hint (2.5.H) — "the session is still active / resend / `/compact`" would be false with no live REPL. makePlainPrinter(deps.io, false); diff --git a/apps/cli/src/commands/chat-export.ts b/apps/cli/src/commands/chat-export.ts index 17bf4fea..672fa2ca 100644 --- a/apps/cli/src/commands/chat-export.ts +++ b/apps/cli/src/commands/chat-export.ts @@ -6,6 +6,7 @@ import { openSessionStore, type OpenedSessionStore } from '../history/session-op import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; +import { stringifyJsonLine } from '../render/sanitize.js'; /** * `relavium chat-export ` (2.P) — export a persisted session to a `.relavium.yaml` **scaffold** for @@ -87,7 +88,7 @@ export function chatExportCommand( sequenceNumber: result.sequenceNumber, workflowPath: result.path, }); - deps.io.writeOut(`${JSON.stringify(event)}\n`); + deps.io.writeOut(`${stringifyJsonLine(event)}\n`); } else { deps.io.writeOut(`Exported session ${args.sessionId} to ${result.path}\n`); } diff --git a/apps/cli/src/commands/chat.ts b/apps/cli/src/commands/chat.ts index 5abd305c..c06f1b19 100644 --- a/apps/cli/src/commands/chat.ts +++ b/apps/cli/src/commands/chat.ts @@ -109,6 +109,7 @@ import { import { createChatStore, type ChatStoreController } from '../render/tui/chat-store.js'; import { createMentionReader, type MentionReader } from '../render/tui/mention.js'; import { createMcpSecretResolver, type McpSecretResolver } from '../secrets/mcp-secret.js'; +import { stringifyJsonLine } from '../render/sanitize.js'; /** * `relavium chat` (2.M) — the agent-first interactive REPL over `@relavium/core`'s `AgentSession`. It binds @@ -2389,7 +2390,7 @@ export async function drivePlain(ctx: ChatDriveContext): Promise { const unsubscribe = ctx.handle.subscribe((event) => - ctx.io.writeOut(`${JSON.stringify(event)}\n`), + ctx.io.writeOut(`${stringifyJsonLine(event)}\n`), ); const rl = createInterface({ input: ctx.io.stdin, terminal: false }); const onSigint = (): void => rl.close(); diff --git a/apps/cli/src/commands/export.ts b/apps/cli/src/commands/export.ts index 97164a33..62d9bade 100644 --- a/apps/cli/src/commands/export.ts +++ b/apps/cli/src/commands/export.ts @@ -12,6 +12,7 @@ import { import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; +import { stringifyJsonLine, stripTerminalControls } from '../render/sanitize.js'; export interface ExportCommandArgs { /** The in-file `id` of the workflow/agent to export (resolved across both project catalogs). */ @@ -59,11 +60,18 @@ export function exportCommand(args: ExportCommandArgs, deps: ExportCommandDeps): // Emit the cwd-relative path in both modes — the same contract `import` uses, so a script consuming either // command's `--json` gets the same path shape (and no absolute filesystem tree leaks into the output). if (deps.global.json) { + // `stringifyJsonLine`, not a bare `JSON.stringify` (CR-03). `targetDisplay` derives from the user-typed + // `--out`, so unlike `import --json`'s payload — whose `slug` is kebab-schema-validated and cannot carry + // one — it really can hold a C1 control or a bidi override, which `JSON.stringify` leaves raw. deps.io.writeOut( - `${JSON.stringify({ id: parsed.slug, kind: parsed.kind, path: targetDisplay })}\n`, + `${stringifyJsonLine({ id: parsed.slug, kind: parsed.kind, path: targetDisplay })}\n`, ); } else { - deps.io.writeOut(`Exported ${parsed.kind} '${parsed.slug}' to ${targetDisplay}\n`); + // The human line needs the terminal floor for the same reason: every dynamic string written to a terminal + // passes `stripTerminalControls` (security-review.md), and this one is user-supplied. + deps.io.writeOut( + `Exported ${parsed.kind} '${parsed.slug}' to ${stripTerminalControls(targetDisplay)}\n`, + ); } return EXIT_CODES.success; } diff --git a/apps/cli/src/commands/import.ts b/apps/cli/src/commands/import.ts index a71bf5c1..a8d57350 100644 --- a/apps/cli/src/commands/import.ts +++ b/apps/cli/src/commands/import.ts @@ -12,6 +12,7 @@ import { import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; +import { stringifyJsonLine } from '../render/sanitize.js'; export interface ImportCommandArgs { /** The external workflow/agent YAML to copy into the project (absolute, or relative to cwd). */ @@ -58,7 +59,7 @@ export function importCommand(args: ImportCommandArgs, deps: ImportCommandDeps): writeAuthoredFile(target, rel, serializeAuthored(parsed), args.force); if (deps.global.json) { - deps.io.writeOut(`${JSON.stringify({ id: parsed.slug, kind: parsed.kind, path: rel })}\n`); + deps.io.writeOut(`${stringifyJsonLine({ id: parsed.slug, kind: parsed.kind, path: rel })}\n`); } else { deps.io.writeOut(`Imported ${parsed.kind} '${parsed.slug}' to ${rel}\n`); } diff --git a/apps/cli/src/commands/models-pricing.ts b/apps/cli/src/commands/models-pricing.ts index d861ba3b..f9fddde8 100644 --- a/apps/cli/src/commands/models-pricing.ts +++ b/apps/cli/src/commands/models-pricing.ts @@ -217,9 +217,10 @@ export function modelsPricingCommand( ? '' : `\n Overrides the catalog price for this model: input $${microcentsToUsd(shipped.inputPerMtokMicrocents)}/Mtok, output $${microcentsToUsd(shipped.outputPerMtokMicrocents)}/Mtok. Yours wins. Run \`relavium models pricing ${stripTerminalControls(args.model)} --clear\` to go back to the catalog's.`; // Strip any terminal-control byte from the (user-typed) model id before echo — parity with `renderModelList`'s - // FIX 2. `ModelListingSchema` only requires min(1), so an id can carry a control byte; the JSON path is safe on - // its own (JSON.stringify escapes them). The provider is a validated (kebab) ProviderId, and the prices are - // numbers — both already safe. + // FIX 2. `ModelListingSchema` only requires min(1), so an id can carry a control byte; the JSON path is safe + // because it goes through `stringifyJsonLine` — NOT because `JSON.stringify` escapes them, which it does not + // (see security-review.md). The provider is a validated (kebab) ProviderId, and the prices are numbers — + // both already safe. deps.io.writeOut( `Set user pricing for ${stripTerminalControls(args.model)} (${args.provider}): input $${args.inputUsdPerMtok}/Mtok, output $${args.outputUsdPerMtok}/Mtok${cachedNote}. It applies to your next run/chat and survives \`models refresh\`.${divergence}\n`, ); diff --git a/apps/cli/src/commands/models.ts b/apps/cli/src/commands/models.ts index e0b6356e..0ecaa7bc 100644 --- a/apps/cli/src/commands/models.ts +++ b/apps/cli/src/commands/models.ts @@ -247,7 +247,7 @@ function toModelJson(m: ModelCatalogListing, providerSlug: (uuid: string) => str return { // The SLUG (e.g. `anthropic`), not the internal `llm_providers` UUID the catalog row carries — matching the // `models refresh` report + the documented `{ provider }` contract. `--json` is unchanged otherwise - // (JSON.stringify escapes any control byte on its own, so the slug is not terminal-sanitized here). + // (the record goes out through `writeRecordLines` → `stringifyJsonLine`, which is what escapes the C1/bidi bytes `JSON.stringify` leaves raw — see security-review.md). provider: providerSlug(m.providerId), modelId: m.modelId, displayName: m.displayName, diff --git a/apps/cli/src/render/json-line-surfaces.test.ts b/apps/cli/src/render/json-line-surfaces.test.ts new file mode 100644 index 00000000..4321ef87 --- /dev/null +++ b/apps/cli/src/render/json-line-surfaces.test.ts @@ -0,0 +1,34 @@ +/** + * `CR-03` — the behavioural half of the machine-output floor. + * + * `#W15-10` built `stringifyJsonLine` and wired two call sites. `CR-03`'s finding named three more, and + * closing it routed FIVE — `commands/import.ts` and `commands/export.ts` write the same record shape and were + * not in the finding. `export.ts` was the last one found, and only because a source-scanning regex had missed + * it purely because prettier wrapped the argument onto its own line. A guard that only checks the places someone remembered + * catches nothing new, so the CALL-SITE half is now an ESLint `no-restricted-syntax` selector in + * `eslint.config.mjs` — it fires on the shape, anywhere in `apps/cli/src`, the first time a new surface is + * written. That is the mechanism; this file keeps the reason it exists visible as executable behaviour. + */ +import { describe, expect, it } from 'vitest'; + +import { stringifyJsonLine } from './sanitize.js'; + +describe('CR-03 — the --json machine-output floor', () => { + it('the shared serializer neutralizes what a bare JSON.stringify leaves raw', () => { + // Built from code points at RUNTIME, never typed into this file: a raw C1/bidi byte in source is a + // Trojan-Source hazard in its own right, and an editor or a tool in the chain can silently rewrite it. + const CSI_8BIT = String.fromCharCode(0x9b); // the 8-bit CSI — `JSON.stringify` leaves it raw + const DEL = String.fromCharCode(0x7f); + const RLO = String.fromCharCode(0x202e); // right-to-left override + const hostile = { id: 'n1', text: `ok${CSI_8BIT}2J${RLO}evil${DEL}` }; + const safe = stringifyJsonLine(hostile); + const bare = JSON.stringify(hostile); + + for (const raw of [CSI_8BIT, RLO, DEL]) { + expect(bare, 'the bare form is the hazard this item is about').toContain(raw); + expect(safe).not.toContain(raw); + } + // Lossless: `--json` promises to reproduce the data, which is why this escapes rather than strips. + expect(JSON.parse(safe)).toEqual(hostile); + }); +}); diff --git a/assets/readme/agent-to-workflow.svg b/assets/readme/agent-to-workflow.svg new file mode 100644 index 00000000..9fc0fa6d --- /dev/null +++ b/assets/readme/agent-to-workflow.svg @@ -0,0 +1,70 @@ + + From agent session to owned workflow runs + Four stages show an agent session becoming a proven flow, a committed Relavium YAML workflow, and observable runs. + + + + + + + + + + + + + + + + + + + + + + + THE RELAVIUM CONTINUUM + A conversation is the beginning, not the dead end. + + + + + + + 1 + AGENT SESSION + explore in conversation + + persistent + tool-aware + + + + + 2 + PROVEN FLOW + keep what actually works + + decisions + structure + + + + + 3 + .RELAVIUM.YAML + review, diff, commit + + portable + git-native + + + + + 4 + OWN EVERY RUN + observe and operate + + events + history + gates + + + ONE ENGINE · TWO ENTRY POINTS · REPEATABLE EXECUTION + + diff --git a/assets/readme/relavium-architecture.svg b/assets/readme/relavium-architecture.svg new file mode 100644 index 00000000..8a8ffc85 --- /dev/null +++ b/assets/readme/relavium-architecture.svg @@ -0,0 +1,100 @@ + + Relavium shared-engine architecture + CLI, desktop, VS Code, and cloud surfaces connect to one pure TypeScript engine with AgentSession and WorkflowEngine entry points, backed by shared contracts, an LLM seam, MCP, and durable storage. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + SHARED-ENGINE ARCHITECTURE + + SURFACES + + + + + + + + CLI + terminal + CI + + Desktop + Tauri management center + + VS Code + authoring + execution + + Cloud host + future workers + API + + + + + + + + @RELAVIUM/CORE + PURE TYPESCRIPT · ZERO PLATFORM IMPORTS + + + + + AgentSession + conversational, multi-turn entry point + + + + + WorkflowEngine + declarative DAG execution entry point + + + TOOL REGISTRY · TYPED EVENT BUS · CHECKPOINT / RESUME · POLICY + + + + + HOST-INJECTED CAPABILITIES + SHARED CONTRACTS + + + + + + + @relavium/llm + provider-neutral model seam + @relavium/shared + Zod contracts + types + @relavium/mcp + inbound tools + SDK fence + @relavium/db + durable local / cloud stores + + POLICY IN CORE · MECHANISM AT THE HOST · PROVIDERS BEHIND THE SEAM + + diff --git a/assets/readme/relavium-hero.svg b/assets/readme/relavium-hero.svg new file mode 100644 index 00000000..4335142a --- /dev/null +++ b/assets/readme/relavium-hero.svg @@ -0,0 +1,99 @@ + + Relavium + A local-first agent platform: start as an agent, ship the workflow, own every run. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + LOCAL-FIRST · MULTI-MODEL · GIT-NATIVE + + + Start as an agent. + Ship the workflow. + Own every run. + Conversation becomes version-controlled automation + on one pure-TypeScript engine. + + + + + + AGENT SESSION + explore · refine · persist + + + + + + + + + .RELAVIUM.YAML + diff · review · commit + + GIT-NATIVE + + + + + + + + + + RUN 01 + local · completed + + + + RUN 02 + CI · streaming + + + RELAVIUM · A HODETECH PRODUCT + + diff --git a/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md b/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md index 26b87cf4..5f03bd8b 100644 --- a/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md +++ b/docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md @@ -4,6 +4,13 @@ - **Date**: 2026-08-09 - **Related**: [ADR-0074](0074-durable-conservative-budget-commitments.md) (the conservative half of the same problem), [ADR-0075](0075-fail-closed-resume-on-an-unreadable-event-log.md) (how this event degrades on an older binary), [ADR-0070](0070-durable-per-model-session-cost-attribution.md) (`SUM(run_costs) == runs.total_cost_microcents`), [ADR-0028](0028-workflow-resource-governance.md) (the pre-egress cap), [ADR-0036](0036-run-loop-substrate-event-bus-and-execution-host.md) (durable run events), and [sse-event-schema.md](../reference/contracts/sse-event-schema.md) (the canonical event contract). +> **Amended 2026-08-10** by [ADR-0077](0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md): §1's +> MECHANISM only. The guarantee — a realized charge is durable before the run spends or mutates again — stands +> unchanged; the "awaited inline emit at the attempt boundary" shape does not, because the seam's attempt +> observer is synchronous and there is no `await` to place there. ADR-0077 substitutes ADR-0074 §2's +> chain-and-join and adds a third barrier before tool dispatch. Not a reversal, so not a supersession. The +> in-place markers in §1 below stay where the superseded sentences are. + ## Context ADR-0074 made the **conservative** commitment durable — money a provider may already have billed for a call that returned no trustworthy usage. It deliberately did not touch the other half. A call that *did* return trustworthy usage has no equivalent barrier, and that gap is larger than it looks. @@ -30,9 +37,17 @@ The event carries the attempt's identity (node, model, the within-chain attempt Three properties make it a ledger rather than another observation: -1. **Written before the next thing that can spend or mutate.** It goes through the engine's `#emitDurable` choke point, and the attempt boundary AWAITS that call — before the next tool side effect, before the next egress, and before the node/turn terminal. A durability failure fails the active owner loudly, exactly as ADR-0074 §2 decided for its estimate twin; surfacing it on a later unrelated node is the misattribution that barrier exists to prevent. - - **The mechanism is the awaited emit plus an explicit check, NOT a §2-style queue-and-flush.** §2 needs +1. **Written before the next thing that can spend or mutate.** *(The MECHANISM below is **amended by + [ADR-0077](0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)** — the guarantee stands, + the "awaited emit" shape does not: `onAttempt` is synchronous, so there is no `await` to place there.)* It + goes through the engine's `#emitDurable` choke point, and the attempt boundary AWAITS that call *(← the + bare inline await is the part ADR-0077 supersedes: the write is STARTED at the settle instant and JOINED at + each barrier)* — before the next tool side effect, before the next egress, and before the node/turn + terminal. A durability failure fails the active owner loudly, exactly as ADR-0074 §2 decided for its estimate twin; surfacing it on a later unrelated node is the misattribution that barrier exists to prevent. + + **The mechanism is the awaited emit plus an explicit check, NOT a §2-style queue-and-flush.** + **← AMENDED BY [ADR-0077](0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md); the + paragraph below is superseded and kept only because the corpus is append-only.** §2 needs `flushBudgetCommitments` because a conservative commitment is emitted fire-and-forget from inside the governor, so something has to join the outstanding writes at the turn boundary. A settled attempt is emitted by the engine at the point it settles, on the path that is about to continue, so no queue is @@ -98,7 +113,8 @@ A new durable event `type` is precisely the input ADR-0075 governs. Before it, a ### Negative -- **More durable writes on the hot path**, one per settled provider attempt, each awaited. That is the cost of the barrier and it is the same cost ADR-0074 §2 accepted for estimates; the volume is per-attempt, not per-token, so a long run adds rows in the hundreds, not the millions. Mitigation: the write shares the single `BEGIN IMMEDIATE` its event already takes, so it is not an additional transaction. +- **More durable writes on the hot path**, one per settled provider attempt, each awaited *(→ per ADR-0077: + each STARTED at its settle instant and JOINED at the next barrier, not awaited inline)*. That is the cost of the barrier and it is the same cost ADR-0074 §2 accepted for estimates; the volume is per-attempt, not per-token, so a long run adds rows in the hundreds, not the millions. Mitigation: the write shares the single `BEGIN IMMEDIATE` its event already takes, so it is not an additional transaction. - **A durability failure now fails a turn that would previously have completed.** Deliberate, and the same posture as §2: a run that cannot record what it spent must not keep spending. Mitigation: the failure is classified and names the attempt, rather than surfacing later as an unexplained cap refusal. - **An older binary cannot replay a log containing this event.** Inherited from ADR-0075 and stated there; the remedy is an upgrade, and every read-only surface still shows the run. - **`cost:updated` remains, and now has a durable sibling.** Two events describing one charge is a real risk of drift. Mitigation: the streamed one keeps its documented role as the live observation and the durable one is the record; the spec says which is authoritative, and the engine emits both from one place so they cannot disagree about the amount. diff --git a/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md b/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md new file mode 100644 index 00000000..961c4c50 --- /dev/null +++ b/docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md @@ -0,0 +1,214 @@ +# ADR-0077: The realized-cost ledger uses ADR-0074 §2's barrier mechanism (amends ADR-0076 §1) + +- **Status**: Accepted +- **Date**: 2026-08-10 +- **Related**: [ADR-0076](0076-durable-per-attempt-realized-cost-ledger.md) §1 (the mechanism this corrects; its decision, event and properties stand), [ADR-0074](0074-durable-conservative-budget-commitments.md) §2 (the mechanism this adopts), [ADR-0038](0038-agentrunner-llm-call-boundary.md) (the one-chain-per-node-execution boundary the barriers sit on), [ADR-0011](0011-internal-llm-abstraction.md) (the `LLMProvider` seam the rejected alternative would have widened), and [sse-event-schema.md](../reference/contracts/sse-event-schema.md) (the canonical event contract). + +## Context + +[ADR-0076](0076-durable-per-attempt-realized-cost-ledger.md) decided that a settled provider attempt's realized +charge becomes a durable run event, `cost:attempt_settled`. That decision is right and is not reopened here. + +Its §1 also decided **how** the write becomes a barrier, and drew a line against its own sibling: + +> **The mechanism is the awaited emit plus an explicit check, NOT a §2-style queue-and-flush.** §2 needs +> `flushBudgetCommitments` because a conservative commitment is emitted fire-and-forget from inside the +> governor, so something has to join the outstanding writes at the turn boundary. A settled attempt is emitted +> by the engine at the point it settles, on the path that is about to continue, so no queue is needed — and +> adding one would introduce the very concurrency the await removes. + +**The asymmetry that paragraph rests on does not exist**, and the code says so in three places: + +1. The seam's attempt observer is synchronous and returns nothing — + `onAttempt?: (record: AttemptRecord) => void` ([fallback-chain.ts](../../packages/llm/src/fallback-chain.ts)), + invoked as a bare synchronous call inside the chain's attempt loop. +2. Both money events are emitted from **that same callback, a few lines apart** — the conservative + `settleAtReservedEstimate()` and the realized `cost:updated` sit in one `onAttempt` body + ([agent-turn.ts](../../packages/core/src/engine/agent-turn.ts)). There is no "engine path that is about to + continue" for one of them and not the other; there is one synchronous observer for both. +3. The governor's own field documentation already states the constraint in plain words: *"The ledger mutation + is SYNCHRONOUS — `settleAtReservedEstimate` is called from inside the fallback chain's `onAttempt` callback, + **which cannot await**"* ([budget-governor.ts](../../packages/core/src/engine/budget-governor.ts)). + +So §1 asked for an `await` at a point where no `await` can be placed. Worse, the codebase had already written +the warning for exactly this mistake: [node-executor.ts](../../packages/core/src/engine/node-executor.ts)'s +comment on why `budget:estimate_committed` is absent from the streamed in-node event union ends with *"If you +came here to add it, add it to the governor's emit type instead."* + +Two things §1 got **right** are worth keeping explicit, because this ADR narrows one paragraph and not the +section: + +- `#emitDurable` is **total for store faults** — a `persistEvent` rejection is absorbed into the run's failure + state and the promise RESOLVES. Awaiting it alone is therefore never a barrier. §1 named this trap and the + trap is real; it survives unchanged. +- The guarantee §1 wanted — *written before the next thing that can spend or mutate* — is the right guarantee. + Only its shape was wrong. + +## Decision + +**`cost:attempt_settled` uses the same mechanism ADR-0074 §2 built for its estimate twin: the emit is started +synchronously at the settle point onto a chained in-flight promise, and JOINED at every barrier that precedes +spending or mutating. It is not a bare awaited emit.** + +Concretely, five things: + +1. **Start the write at the settle instant.** `onAttempt` cannot await, but it can *begin*. The emit is chained + onto a per-owner in-flight promise, copying the shape of §2's `#commitmentsInFlight` — which also serializes + the writes, so two settles in the same tick cannot interleave their persists. **The shape is copied; the + owner is not** — see §5. +2. **Join at three barriers, not two.** §2 joins at the pre-egress check and at the turn/node boundary. + ADR-0076's guarantee names a third thing to precede — the next **tool side effect** — so the ledger's + barrier set is, and these three names are used throughout this ADR: + + - **B1** — before the next egress admission (§2 has this one); + - **B2** — **before tool dispatch** (new here); + - **B3** — at the turn/node terminal (§2 has this one). + + **B2** is what this ADR adds to the §2 shape; without it the ledger would repeat §2's coverage rather than + extend it, and the duplicate-effect window ADR-0076 exists to narrow would stay open on the path that + mutates the world. +3. **Every barrier awaits AND observes.** Because `#emitDurable` is total, a barrier that only awaits proceeds + on a run whose ledger write did not land. Each barrier must therefore also read the failure state and refuse + to continue, and §2's retained-failure pattern (`#commitmentFailure`, re-thrown at the next barrier) is the + shape that carries it — a rejection nobody awaits at the call site would otherwise be unhandled. +4. **One join, not two.** After this ADR there are two chained in-flight promises — the conservative one and + the realized one — and three barriers that must join both. Joining them individually is a correctness bug + waiting for the next barrier someone adds. So the barriers call **one** joining entry point that owns both + chains and both retained failures; there is no supported way to await half the money. This is what turns + the "a future barrier joins only one chain" hazard from a review item into something a caller cannot + express. +5. **The ledger and its join are owned by the ENGINE, unconditionally — never by `BudgetGovernor`.** Reading + §1's "as §2's `#commitmentsInFlight`" as "put it on the governor" would silently omit most runs, and every + signpost in the codebase points that way (`node-executor.ts`'s comment even says "add it to the governor's + emit type instead"). It is wrong here, because the governor is **conditional and the ledger is not**: + + - `engine.ts` constructs a `BudgetGovernor` only `if (params.plan.budget !== undefined)`, and `budget` is + optional in the workflow schema. An unbudgeted run spends real money and would get no ledger at all. + - `#flushBudgetCommitments` returns immediately when there is no governor, so **barrier B3 becomes a + no-op**. + - `#makePreEgressHook()` returns `undefined` without a governor, so the turn core never installs the + `preAttempt` wrapper and **barrier B1 does not exist**. + - Even on a BUDGETED run, `const preEgress = budgetApproved ? undefined : this.#makePreEgressHook()` drops + the hook for an approved `pause_for_approval` re-dispatch — so B1 disappears on the exact path the user + just authorised more money on. + + Implemented literally, an unbudgeted run would get one barrier of three, and an unbudgeted run with no tool + calls would get none — the fire-and-forget state this ADR exists to remove, passing every test written + against a budgeted fixture. + + **The correct shape already exists one surface over.** `session-host.ts` installs an `preEgress` hook that + is *"ALWAYS present, cap or no cap"*, reads the durability probe FIRST, and only then delegates to + `governor?.preEgress(info)` — *"It composes with the governor rather than replacing it, and it runs + FIRST."* The run path never got that. So this ADR's own "the session path is stronger here" section had the + answer to this flaw in it; the run path adopts the same composition. + +### Two implementation traps this decision creates + +Named because both are silent, and both would look correct in review: + +- **The ledger emit must follow the cumulative fold, not precede it.** The engine advances its run-wide total + in `#nodeEmit`'s `cost:updated` arm. Emitting the ledger draft *before* that leaves the cumulative stale, and + `refineCostAttemptSettled` rejects `cumulative < cost` at the producer gate — which runs in `#bus.next`, + **outside** `#emitDurable`'s `try`. So the wrong order does not degrade: `#emitDurable` **rejects** instead + of resolving, in the one place the whole design assumes it cannot. Nothing pins that order today. +- **The durable emit must not be routed through `#nodeEmit`.** It returns `void`, so the promise would be + unawaitable — the exact unbarriered shape this ADR exists to prevent, reached through the door + `node-executor.ts`'s in-node-event-union comment already warns about. + +The hook that carries the emit into the turn core is an OPTIONAL `AgentTurnParams` field, modelled on +`preEgress`, because `agent-turn.ts` is the boundary `AgentSession` shares. Leaving it unset on the session +path is what makes this event run-only as a runtime fact rather than a comment. + +### Why the tool-dispatch barrier matters beyond this ADR + +It is the first place the engine is required to reach a durability checkpoint *before dispatching a side +effect*, and that checkpoint is the seam the durable **effect journal** (`CR-12` in the 2.6.5 phase) needs: +`prepared → dispatched → committed | ambiguous` has to be written at exactly this point in exactly this path. +Landing it here means the effect journal extends an existing barrier rather than threading a new one through +`ToolRegistry.dispatch`. Named so the two are not built twice, and so a reviewer of either can check the other. + +Considered and rejected: + +- **Make the seam's `onAttempt` awaitable** (`void | Promise`, awaited by the chain), so §1's sentence + becomes literally true. Rejected on three counts. It widens `@relavium/llm`'s public API for a concern that + is not the seam's — durability belongs to the engine, and [ADR-0011](0011-internal-llm-abstraction.md) keeps + the seam a provider contract, not an execution-host one. It would let a durable store write block a live + provider stream, and an observer that hangs would stall the chain with no timeout of our own. And it would + give the two money events emitted from one callback two different durability mechanisms, which is precisely + the drift that made this correction necessary. +- **Emit from the turn loop after the chain call returns**, where an `await` is genuinely available. Rejected + because it is strictly worse than the chosen mechanism on the failure this ADR is about: the write would not + even *start* until the whole chain call finished, so a crash mid-chain loses every failover attempt inside + it. Chaining starts the write at the settle instant and only *joins* later, which is a narrower window, not a + wider one. +- **Leave §1 as written and implement something else.** Rejected outright. An ADR whose mechanism paragraph + describes something the code does not do is the corpus drift this project has repeatedly paid for; the + remedy is an append-only correction, not a quiet reinterpretation. + +### The session path needs no barrier, and the reason is structural + +ADR-0076 scoped its event to the run path on the ground that "the session path already has this ledger". A +review of this ADR asked whether that leaves a session resume losing realized cost the same way. It does not, +and the reason is worth recording once so the question stops recurring: the session write is **synchronous and +already committed** when the handler returns. + +`persister.ts` handles `cost:updated` through `persistDurably(() => store.recordSessionCost(...))`, and +`recordSessionCost` is declared `(entry: SessionCostEntry) => void` — a synchronous `better-sqlite3` +transaction, not a promise. `persistDurably` calls it inline and re-throws, so the money is on disk before the +next line of the handler runs, and the handler itself runs on the delivery of an event emitted from the +synchronous `onAttempt`. There is no in-flight window for a crash to land in. + +So the asymmetry this ADR institutionalises is not "run path protected, session path forgotten". It is that +the run path's store is `persistEvent: (event) => Promise` — asynchronous by seam design, because a +cloud store must plug in — and an asynchronous write is exactly what needs a barrier. The session path is +stronger here, not weaker, and stays so until its store becomes asynchronous; at that point it inherits this +decision rather than needing a new one. + +### What ADR-0076 keeps + +Unchanged and not reopened: the event exists, its name, its meaning, its shape's canonical home, the +idempotency boundary and the crash-after-commit case it explicitly does NOT cover, the no-double-count +arithmetic against `node:completed`'s telescoping delta, the run-path-only scope and why the session path needs +no arm, the four rejected alternatives, and the reason it lands after +[ADR-0075](0075-fail-closed-resume-on-an-unreadable-event-log.md). This ADR replaces one paragraph. + +## Consequences + +### Positive + +- **One mechanism for both money events, emitted from one callback.** They cannot drift apart, and a future + reader who finds one finds the other. The alternative left two durability shapes three lines apart in the + same function. +- **The guarantee ADR-0076 wanted is actually obtainable**, and the tool-dispatch barrier makes it stronger + than §2's: realized spend is durable before the run mutates the world, not merely before it spends again. +- **No seam change.** `@relavium/llm` keeps its provider contract, and a slow disk cannot stall a provider + stream. +- The write starts earlier than any barrier-only design would allow, so the crash window is the narrowest of + the three mechanisms considered. + +### Negative + +- **A third barrier on the hot path.** Mitigation: §2's outstanding-count guard applies unchanged — the barrier + costs nothing when nothing is in flight, which is the common case, and an unconditional await would change + observable interleaving that existing tests legitimately pin. +- **Two chained in-flight promises now exist** (the governor's conservative chain and the ledger's realized + chain), so a barrier that joined only one would be silently half-safe. Mitigation is structural rather than + advisory — Decision §4: the barriers join through a single entry point that owns both chains and both + retained failures, so "await the wrong one" is not an expressible mistake. What a reviewer still has to + catch is a new barrier that calls neither. +- **Three barrier holes have to close for this to be true at all**, and each is a line that reads as correct + today: the governor-conditional `#flushBudgetCommitments` early return, the `undefined` from + `#makePreEgressHook`, and `budgetApproved ? undefined`. Mitigation: Decision §5 names all three, and the + regression that proves them is an **unbudgeted** run — a fixture with a budget passes even when every + barrier is missing. +- **The tool-dispatch barrier is new engine surface**, on the hottest path a run has, and it lands before the + effect journal that will build on it. If it is placed wrong, both this ledger and `CR-12` inherit the same + crash window. Mitigation: it goes in at `agent-turn.ts`'s single `await dispatchToolCalls(...)` — one call + site, before any tool runs — rather than inside `ToolRegistry.dispatch`, where per-call placement would have + to be re-proven for every dispatch path. +- **A durability failure now fails a turn at the tool boundary too**, where previously it would have surfaced + only at the next egress or the terminal. Deliberate, and the same posture §2 accepted: a run that cannot + record what it spent must not go on to mutate anything. +- **ADR-0076 §1 must be read together with this ADR**, since the corpus is append-only and that paragraph + stays on the page. Mitigation: the title, the `Related` line and the "What ADR-0076 keeps" section above make + the scope of the correction unambiguous — one paragraph, not a section. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 4b1b1143..e224c685 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -120,6 +120,7 @@ flowchart TD | 0074 | [Durable conservative budget commitments across run and session resume (amends ADR-0028, ADR-0036, ADR-0045, ADR-0070)](0074-durable-conservative-budget-commitments.md) | Accepted | 2026-07-30 | | 0075 | [A resume fails closed on an unreadable event log (amends ADR-0074 §5)](0075-fail-closed-resume-on-an-unreadable-event-log.md) | Accepted | 2026-08-09 | | 0076 | [A durable per-attempt realized-cost ledger (amends ADR-0070, extends ADR-0074)](0076-durable-per-attempt-realized-cost-ledger.md) | Accepted | 2026-08-09 | +| 0077 | [The realized-cost ledger uses ADR-0074 §2's barrier mechanism (amends ADR-0076 §1)](0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md) | Accepted | 2026-08-10 | ## Creating a new ADR diff --git a/docs/reference/cli/commands.md b/docs/reference/cli/commands.md index f5e91ee1..bd93c37f 100644 --- a/docs/reference/cli/commands.md +++ b/docs/reference/cli/commands.md @@ -50,6 +50,14 @@ who opts out of color wins over a tool/CI that opts in. (A `--json`/CI/no-TTY st ### The `--json` machine-output contract +**Every record is serialized with `stringifyJsonLine`, not a bare `JSON.stringify`.** The wire contract is +unchanged — `JSON.parse` round-trips to the identical string, which is exactly why the escape was chosen over +a strip — but what reaches a terminal differs: `JSON.stringify` escapes `ESC` and leaves `DEL`, the C1 block +(including `U+009B`, a working escape-sequence introducer) and the Trojan-Source bidi family RAW in content +the model, a tool, or an imported artifact controls. Enforced by lint rather than review; the reasoning and +the one allowlisted exception live in +[security-review.md](../../standards/security-review.md#cli-terminal-render-safety--interactive-approval). + Under `relavium run --json`, the CLI emits a stable machine contract a CI job can pipe and assert on ([ADR-0049](../../decisions/0049-cli-machine-output-contract.md)). The contract covers a workflow **run**; `--help`, `--version`, and a bare no-command invocation are exit-`0` meta-operations that diff --git a/docs/reference/contracts/sse-event-schema.md b/docs/reference/contracts/sse-event-schema.md index b88baf35..3d35c3b7 100644 --- a/docs/reference/contracts/sse-event-schema.md +++ b/docs/reference/contracts/sse-event-schema.md @@ -64,7 +64,8 @@ export type RunEvent = | RunTimeoutEvent | BudgetWarningEvent | BudgetPausedEvent - | BudgetEstimateCommittedEvent; // dual-envelope (ADR-0074 §2) — a conservative ESTIMATE, never realized spend + | BudgetEstimateCommittedEvent // dual-envelope (ADR-0074 §2) — a conservative ESTIMATE, never realized spend + | CostAttemptSettledEvent; // RUN-ONLY (ADR-0076) — the REALIZED twin of the line above; the only DURABLE cost: event ``` > `RunPausedEvent` is the multi-gate aggregate (below); `RunTimeoutEvent` / `BudgetWarningEvent` / `BudgetPausedEvent` / `BudgetEstimateCommittedEvent` are the resource-governance events defined in [Workflow governance and reserved events](#workflow-governance-and-reserved-events). @@ -80,7 +81,7 @@ export type RunEvent = | `agent:approval_requested` | A side-effecting tool dispatch is awaiting an **interactive per-tool approval** decision (ADR-0057 EA3/EA5). The engine's `confirmDispatch` emits it — for **every** governed dispatch reaching the gate, whether the host then prompts a human or auto-decides — just before invoking the host's `ConfirmActionHook`; the registry then awaits the verdict (approve ⇒ dispatch, reject ⇒ a fatal `tool_denied`). A **dual-envelope** event (`runId`/`sessionId`), like `agent:tool_call` — in Phase 2.5 emitted only on the chat session path (the approval regime), and **carried on the session stream** (not run-only — it is **not** dropped like `agent:file_patch_proposed`). | `nodeId`, `toolId`, `action: 'fs_write' \| 'process' \| 'egress' \| 'os'` (the governed side-effect class — [tool-registry.md](../shared-core/tool-registry.md)), `preview` (**secret-free, display-only**: `{ path? }` for a write, `{ command? }` for a process, `{ host? }` for egress, `{}` for an `os` action like `read_clipboard`/`notify` — never a full URL/query, never a secret. Secret-freedom is **enforced** by the registry's redaction, not asserted, so a field **may contain the literal `[redacted]` marker and is not guaranteed to be a resolvable path/command/host — a machine consumer must never parse it**; see [tool-registry.md](../shared-core/tool-registry.md) §Preview redaction), `attemptNumber?` | | `agent:file_patch_proposed` | An agent proposed a file change (**gated — no write until the user accepts**; e.g. the VS Code inline-diff review). | `nodeId`, `patches: [{ uri, unifiedDiff }]` (≥1 — an empty proposal is meaningless), `attemptNumber?` | | `cost:updated` | A node's token cost was tallied (drives the cost waterfall). | `nodeId`, `model`, `inputTokens`, `outputTokens`, `costMicrocents`, `cumulativeCostMicrocents` (integer micro-cents — canonical unit in [llm-provider-seam.md](../shared-core/llm-provider-seam.md#6-usage); **includes realized media spend**, folded as a disjoint addend per [ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §3 — the per-unit `Usage.mediaUnits` axis is **not yet a field on this event**, deferred, see [deferred-tasks.md](../../roadmap/deferred-tasks.md)), `attemptNumber?` (1-based **within-chain** FallbackChain attempt — resets per node-retry re-dispatch; **distinct** from `node:*.attemptNumber`, see the [two attemptNumber families](#two-attemptnumber-families) note), `priced?` (**ADR-0070** — additive + optional, so an older reader ignores it: `false` when the egress could **not be priced**. An unpriced model still emits this event with its **real tokens** and `costMicrocents: 0`, which makes "cost 0 + tokens > 0" ambiguous between *could not price* and *genuinely free* — an ambiguity nothing else in the event resolves. The durable `session_costs` row records it as an `unpriced_calls` **counter**, not a boolean, because a model can be priced **mid-session**). **Generative-node variant (1.AG Section C, [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5):** a `media_surface: 'generative'` agent node emits **exactly one** `cost:updated` with `inputTokens` / `outputTokens` **= 0** (no token billing — the spend rides entirely in `costMicrocents` as the per-modality media addend) and **no `attemptNumber`** (no FallbackChain on the generative path — one provider, no failover). | -| `node:completed` | A node finished successfully. | `nodeId`, `output`, `tokensUsed: {input, output, model?}` (`model` only for LLM nodes), `durationMs`, `selected?` (a `condition`'s chosen target ids — the authoritative branch record checkpoint/resume restores from, 1.R; **may be an empty array** when the condition routes to no branch, dimming all downstream), `attemptNumber?` (1-based **node-retry** dispatch attempt — 1.S; absent ⇒ attempt 1), `cumulativeCostMicrocents?` (the run-wide running total snapshotted at this node boundary — the durable cost source checkpoint/resume restores from, since `cost:updated` is streamed-only; the engine always populates it. `node:failed` mirrors this field, 2.S/D-GC) | +| `node:completed` | A node finished successfully. | `nodeId`, `output`, `tokensUsed: {input, output, model?}` (`model` only for LLM nodes), `durationMs`, `selected?` (a `condition`'s chosen target ids — the authoritative branch record checkpoint/resume restores from, 1.R; **may be an empty array** when the condition routes to no branch, dimming all downstream), `attemptNumber?` (1-based **node-retry** dispatch attempt — 1.S; absent ⇒ attempt 1), `cumulativeCostMicrocents?` (the run-wide running total snapshotted at this node boundary — **one of the durable absolute totals** checkpoint/resume maxes over, alongside `node:failed`'s, `budget:paused.spentMicrocents`, and `cost:attempt_settled.cumulativeCostMicrocents` since [ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md); see the fold rule below for why it is a max and never a sum. This snapshot is still the ONLY carrier of media spend, which emits no attempt row. `cost:updated` is streamed-only and is not a durable source at all. The engine always populates this field. `node:failed` mirrors it, 2.S/D-GC) | | `node:failed` | A node failed (TERMINAL — exactly one per node; emitted when the node-retry budget is exhausted, on a fatal / `retry_on`-excluded failure, **or** when a pending retry is abandoned by a cancel or a sibling abort — see 1.S). | `nodeId`, `error: {code, message, retryable, correlationId?}` (`code` is an [`ErrorCode`](#error-code-taxonomy); `correlationId` is a secret-free id joined to the internal log — ADR-0036), `attemptNumber?` (the last attempt, when a retry budget was spent — 1.S), `cumulativeCostMicrocents?` (the run-wide running total snapshotted AT this node boundary — the durable fail-cost so a billed-but-failed **paid media job**'s realized spend survives the transient `cost:updated`, 2.S/D-GC [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5; mirrors `node:completed`) | | `node:retrying` | A retryable node attempt failed and the engine will re-dispatch the whole node (1.S, [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md)) — **non-terminal** (the node continues; `node:failed` is the terminal). | `nodeId`, `attemptNumber` (the attempt that just failed, 1-based), `error: {code, message, retryable}` (the `NodeFailure` shape — **no** `correlationId`; that anchors the terminal failure), `delayMs` (backoff before the next attempt) | | `node:skipped` | A node was skip-propagated (never ran). | `nodeId`, `reason: 'branch_not_taken' \| 'upstream_unreachable'` (`branch_not_taken` = a `condition` routed away from it; `upstream_unreachable` = every in-edge is dead because an upstream was skipped/failed). Emitted so the event log is a **complete, replayable** record — checkpoint/resume reconstructs a skipped vertex from it ([run-plan.md](../shared-core/run-plan.md)) and a surface can render the dimmed path instead of the node silently vanishing. | @@ -91,6 +92,9 @@ export type RunEvent = | `run:completed` | The run finished. | `outputs` (a record **keyed by each terminal `output` vertex's node id**, the value being that vertex's captured output — see [run-plan.md §output capture](../shared-core/run-plan.md)), `totalTokensUsed`, `totalCostMicrocents` (integer micro-cents closing total for the whole run), `durationMs` | | `run:failed` | The run failed. | `error: {code, message, retryable, nodeId?, correlationId?}` (`code` is an [`ErrorCode`](#error-code-taxonomy); `nodeId` is the root-cause node; `correlationId` joins to the internal log — ADR-0036), `partialOutputs`, `cumulativeCostMicrocents?` (the run-wide running total at failure — the durable fail-cost for a **paid media job** a sibling node's failure abandoned, whose lone estimate addend is folded just before this terminal _after_ the root-cause `node:failed` snapshot, 2.S/D-GC [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5; mirrors `run:cancelled` and the `run:completed` counterpart `totalCostMicrocents`) | | `run:cancelled` | The run was cancelled. | `cumulativeCostMicrocents?` (the run-wide running total at cancellation — the durable fail-cost for a **paid media job** pending at the cancel, whose lone estimate addend is folded just before this terminal, 2.S/D-GC [ADR-0045](../../decisions/0045-async-media-job-loop-poll-checkpoint-resume-cancel.md) §5; the `run:completed` counterpart is `totalCostMicrocents`) | +| `cost:attempt_settled` | **One settled provider attempt's REALIZED charge, made durable** ([ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md)) — the realized twin of `budget:estimate_committed`, emitted from the same callback and joined at the same barriers ([ADR-0077](../../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)). It exists because `cost:updated` is streamed and never persisted, so before this event a crash mid-agent-loop discarded every charge since the last node boundary — and the resumed run spent it AGAIN, against a cap understated by exactly that amount. **RUN-ONLY** (`runId` always, never `sessionId`), unlike `cost:updated` and unlike its estimate twin: the session path already records the same per-attempt increment into `session_costs` synchronously (ADR-0070 + `#W15-4`), so it needs no arm here. **Scope:** it covers the provider attempts of an AGENT TURN. A media job's realized cost (ADR-0045 §5) does **not** emit one and does not need to — it is already durable through the `node:completed` / `node:failed` / `run:*` cumulative snapshots. And no cost event of any kind makes a TOOL EFFECT idempotent; that is a separate decision about a separate failure. | `nodeId`, `model`, `attemptNumber` (**required**, 1-based within-chain — optional on `cost:updated` only because that event predates this one and has historical rows; a per-attempt ledger row that cannot say which attempt it is, is not a ledger), `inputTokens`, `outputTokens`, `costMicrocents` (**this attempt's delta**, not a cumulative — `0` is reachable for an unpriced or genuinely free model; a delta a restoring reader must **never sum** — see the fold rule below), `cumulativeCostMicrocents` (the run-wide total **after** this attempt — a durable ABSOLUTE total, and one of the inputs a restoring reader takes `Math.max` over; the fold rule below is the canonical statement), `priced` (**required**, same reason as `attemptNumber`: without it `costMicrocents: 0` with real tokens cannot distinguish *could not price* from *genuinely free*, which is the ambiguity a ledger exists to remove) | + +> **Which cost event is authoritative for what — the three-way split, stated once.** `cost:updated` is the **live observation**: streamed, never persisted, last-wins, and it is what a surface renders while a run is moving. `cost:attempt_settled` is the **durable record of what was charged**: one row per settled attempt, and the only cost event a reader can trust after a restart. **Restoring** a run's realized total is `Math.max` over every durable ABSOLUTE total in the log — the node-boundary snapshots (`node:completed` / `node:failed` / `run:*` `cumulativeCostMicrocents`), `budget:paused.spentMicrocents`, and `cost:attempt_settled.cumulativeCostMicrocents`. Each is read immediately after its own increment, so each is a true run-wide total at that instant and the largest is the engine's real total whatever order the rows landed in. **Do not sum the attempt deltas**, by either obvious route: adding `costMicrocents` into the same total double-counts against a node snapshot that already includes those attempts, and summing them into a *separate* accumulator to max against the snapshots **under-counts** — the two sources cover different money, since a media node writes a snapshot and emits no attempt row at all, so every attempt made after the last node boundary vanishes whenever earlier media spend is the larger figure. (`Math.max` is right here and wrong for the conservative twin below for one reason only: realized spend is monotonic, while a conservative commitment can be deliberately released.) The derived `run_costs` / `runs` rows this folds into are specified in [database-schema.md](../shared-core/database-schema.md), not here. > **The money basis is FROZEN at submit time** ([ADR-0074](../../decisions/0074-durable-conservative-budget-commitments.md) §3). `units` is the authored billed volume the submission was priced on, and `acceptedCostMicrocents` is what the admission actually reserved. A resume restores the reservation from `acceptedCostMicrocents` with **no pricing lookup**, so neither a workflow edit nor a user-price/catalog change between submission and resume can move a commitment the provider has already accepted, or rewrite the job's historical cost. `acceptedCostMicrocents: 0` is meaningful and distinct from absent — it says the submission was priced and reserved nothing, which is what an unpriced model's allow-degrade path does. **`acceptedCostMicrocents` present ⇒ `units` required** (schema-enforced): a frozen cost on an unfrozen basis would restore the old reservation while re-deriving the volume, which is the drift §3 exists to prevent. The converse does NOT hold — `units` alone is legitimate, and is exactly what the approved-bypass path writes (it freezes the volume and omits the cost, because no pricing hook ran and `0` would freeze "priced at zero" for a job that was never priced). **Both absent means the row is LEGACY** (written before §3): resume must re-derive the volume from the workflow definition and re-price from today's catalog, which may under-reserve if the price has fallen, so with a cap configured the governor **fails closed**: new egress is **held** — awaited, not refused — until that job settles and its realized charge replaces the guess. Holding rather than throwing is deliberate: `budget_exceeded` is not in `RETRYABLE_ERROR_CODES` and `retry_on` cannot widen it, so a thrown refusal would kill a sibling node and abort the run, abandoning the very job it was waiting for. The hold is bounded by the job's own `deadlineAt` and is always broken by an abort, and it announces itself once through the governor's hold notice (routed to stderr by `relavium gate`) — that notice is the observability §3 requires, so a resume is never a silent stall. @@ -99,9 +103,9 @@ export type RunEvent = `attemptNumber` appears on two **independent** counter families that must not be conflated (1.S, [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md)): - **Node-retry dispatch attempt** — on `node:started` / `node:completed` / `node:failed` / `node:retrying`. The engine's **above-chain** whole-node re-dispatch index. Absent ⇒ attempt 1; present + >1 ⇒ a re-dispatch (distinguishes "attempt N starting" from a replay). -- **Within-chain attempt** — on `cost:updated` / `agent:tool_call` / `agent:tool_result` / `agent:file_patch_proposed`. The **within-chain** `FallbackChain` attempt index inside a *single* node dispatch; it **resets to 1 on every node-retry re-dispatch** (a fresh chain runs each time). +- **Within-chain attempt** — on `cost:updated` / `cost:attempt_settled` / `agent:tool_call` / `agent:tool_result` / `agent:file_patch_proposed` (and `budget:estimate_committed`). The **within-chain** `FallbackChain` attempt index inside a *single* node dispatch; it **resets to 1 on every node-retry re-dispatch** (a fresh chain runs each time). It is optional on all of them except `cost:attempt_settled`, where it is required — that event has no historical rows to accommodate, and it is the one whose identity *is* the attempt. -The two do **not** join: on a node the budget retried, `node:completed.attemptNumber` may be `2` while the accompanying `cost:updated.attemptNumber` is `1`. To attribute cost to a node-retry attempt, **partition the `sequenceNumber`-ordered stream at each `node:started` / `node:retrying` boundary** — do not key by `(nodeId, attemptNumber)` across families. (Run totals are unaffected: `cost:updated.cumulativeCostMicrocents` is the engine's authoritative running total.) +The two do **not** join: on a node the budget retried, `node:completed.attemptNumber` may be `2` while the accompanying `cost:updated.attemptNumber` is `1`. To attribute cost to a node-retry attempt, **partition the `sequenceNumber`-ordered stream at each `node:started` / `node:retrying` boundary** — do not key by `(nodeId, attemptNumber)` across families. (Run totals are unaffected: `cost:updated.cumulativeCostMicrocents` is the engine's authoritative running total **for a LIVE reader**. A reader reconstructing a run's total from the durable log uses the three-way split above instead — `cost:updated` is never persisted, so it is not available to one.) ### Selected definitions @@ -223,6 +227,19 @@ export interface RunTimeoutEvent extends BaseEvent { elapsedMs: number; timeoutMs: number; } + +export interface CostAttemptSettledEvent extends BaseEvent { + type: 'cost:attempt_settled'; + // RUN-ONLY: `runId` always, `sessionId` never — unlike its estimate twin and unlike cost:updated. + nodeId: string; // required — on the run path every attempt has an owning vertex + model: string; // canonical model id this attempt actually ran on + attemptNumber: number; // 1-based WITHIN-CHAIN attempt. REQUIRED here, optional on cost:updated — see below + inputTokens: number; + outputTokens: number; + costMicrocents: number; // THIS attempt's realized charge. A restoring reader must NOT sum these — see the fold rule above. 0 is reachable (unpriced/free) + cumulativeCostMicrocents: number; // the run-wide total AFTER it — a durable ABSOLUTE total; restore is Math.max over the absolutes, never a sum of the deltas + priced: boolean; // REQUIRED here, optional on cost:updated — see below +} ``` ### Security: event payloads never carry secrets @@ -299,7 +316,7 @@ A turn that **fails** (a provider error, a rate limit, an exhausted budget cap) Within a turn, the conversational work reuses the **same** `agent:token` / `agent:reasoning` / `agent:tool_call` / `agent:tool_result` / `cost:updated` event shapes the `AgentRunner` already emits — carried on the session envelope (`sessionId`). The per-turn append of user/assistant/tool messages is persisted as `session_messages` (see [database-schema.md](../shared-core/database-schema.md)); the contract is owned by [agent-session-spec.md](agent-session-spec.md). On every surface session events are produced and consumed **in-process** exactly like run events — only `llm_stream` crosses IPC on the desktop ([ipc-contract.md](ipc-contract.md#run-events-are-webview-side)). So the **complete typed event stream for a session** is the eight `session:*` lifecycle/side events (the `SessionEvent` union above — started / turn_started / turn_completed / cancelled / exported / compacting / compacted / trimmed) **plus** `agent:token` / `agent:reasoning` / `agent:tool_call` / `agent:tool_result` / `cost:updated` (and, on the chat approval path, `agent:approval_requested` — ADR-0057) carrying `sessionId` — plus `budget:estimate_committed` once [ADR-0074](../../decisions/0074-durable-conservative-budget-commitments.md) §4 wires the session's durable budget write; the schema and the sink already carry it, but no session producer reaches it yet — this full set is exactly what `relavium chat --json` emits (`agent:reasoning` included: a `--json` consumer that does not want it filters on `type`). -**The session stream (`SessionHandle`, 1.W).** A session is **long-lived across turns**, so — unlike a run's exactly-one-terminal `RunHandle` — the `SessionHandle.events` async-iterable stays **open across turns**: `session:turn_completed` is a per-turn boundary, **not** a stream terminal. The stream closes **only** on `session:cancelled` (the session's sole terminal); `session:exported` is a side event (1.Z), never a terminal. The bus assigns the **per-session** `sequenceNumber` — a monotonic counter keyed on `sessionId`, independent of any run's `runId` counter on the same shared bus (ADR-0036 "one bus, two namespaces") — with the **same** gap-detection / resync rule as a run. `AgentSession` (1.V) emits *envelope-free* drafts through its injected `SessionEventSink`; 1.W's `createSessionEventSink` attaches the `sessionId` and the bus stamps the `sequenceNumber` + `timestamp` at the one authoritative translation point. The bus's validation gate accepts both families via the combined `RunOrSessionEventSchema` (`@relavium/shared`). `agent:file_patch_proposed` is **run-only** (it carries `runId`, emitted by the `AgentRunner` workflow adapter — not the shared turn core), so it is **not** part of a session stream; `createSessionEventSink` drops it defensively at the seam. By contrast, `agent:token` / `agent:reasoning` (EA6), `agent:approval_requested` (ADR-0057 EA3/EA5) are dual-envelope events the chat session path emits, so the sink **carries** them (they are not run-only). `budget:estimate_committed` (ADR-0074 §2) is dual-envelope and the sink carries it too, but the session producer is §4's work — today only the run path emits it. +**The session stream (`SessionHandle`, 1.W).** A session is **long-lived across turns**, so — unlike a run's exactly-one-terminal `RunHandle` — the `SessionHandle.events` async-iterable stays **open across turns**: `session:turn_completed` is a per-turn boundary, **not** a stream terminal. The stream closes **only** on `session:cancelled` (the session's sole terminal); `session:exported` is a side event (1.Z), never a terminal. The bus assigns the **per-session** `sequenceNumber` — a monotonic counter keyed on `sessionId`, independent of any run's `runId` counter on the same shared bus (ADR-0036 "one bus, two namespaces") — with the **same** gap-detection / resync rule as a run. `AgentSession` (1.V) emits *envelope-free* drafts through its injected `SessionEventSink`; 1.W's `createSessionEventSink` attaches the `sessionId` and the bus stamps the `sequenceNumber` + `timestamp` at the one authoritative translation point. The bus's validation gate accepts both families via the combined `RunOrSessionEventSchema` (`@relavium/shared`). `agent:file_patch_proposed` is **run-only** (it carries `runId`, emitted by the `AgentRunner` workflow adapter — not the shared turn core), so it is **not** part of a session stream; `createSessionEventSink` drops it defensively at the seam. By contrast, `agent:token` / `agent:reasoning` (EA6), `agent:approval_requested` (ADR-0057 EA3/EA5) are dual-envelope events the chat session path emits, so the sink **carries** them (they are not run-only). `budget:estimate_committed` (ADR-0074 §2) is dual-envelope and the sink carries it too, but the session producer is §4's work — today only the run path emits it. `cost:attempt_settled` ([ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md)) is **run-only like `agent:file_patch_proposed`**, and for a stronger reason than a sink-level drop: the hook that produces it is an optional turn-core parameter the session path never sets ([ADR-0077](../../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)), so nothing is emitted there to drop. The session's realized spend is recorded per attempt into `session_costs` instead (ADR-0070), synchronously and before the next egress (`#W15-4`). ## Workflow governance and reserved events @@ -343,6 +360,8 @@ Removing or repurposing an existing field/type is a breaking change and is not d **Where the promise does NOT apply: a REPLAY.** "Ignore unknown `type`s" is addressed to consumers that RENDER the stream. A caller that reconstructs authoritative state in order to *do* something — `checkpointer.ts` seeding `engine.resumeFromCheckpoint` — cannot safely ignore a row it does not understand: it has no way to know whether that row was a node terminal, an async job submission, a gate decision or a cost commitment, so tolerating the hole means re-running completed work or re-submitting an already-billed job, silently. [ADR-0075](../../decisions/0075-fail-closed-resume-on-an-unreadable-event-log.md) narrows ADR-0074 §5 accordingly: the replay read (`loadRunEventLogForReplay`) **refuses** when any row was skipped, and every display read stays tolerant. There is no session counterpart because no session resume reads a stored event log — a session's durable state is typed rows, not events. +**`cost:attempt_settled` is the first event to exercise that carve-out, and it is why ADR-0075 landed first.** Adding it is additive and v1.0-legal for every RENDERING consumer — an older `relavium logs` drops the row and shows the rest of the run. It is deliberately **not** additive for a REPLAY: an older binary resuming a log that contains it would re-run paid work against a cap missing the very charges the event exists to record, which is the failure that makes the fix self-defeating on a downgrade. The refusal is the correct behaviour, and the remedy is an upgrade. Read the two halves together before adding the next durable type — "adding a new event `type` is never a breaking change" is true of the stream and false of the resume. + ## Transport notes ### Phase 1 — local (in-process on every surface) diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index f2d6d8d4..ddeda7b3 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -2,7 +2,7 @@ > Status: Living > -> Last updated: 2026-07-29 +> Last updated: 2026-08-11 - **Related**: [README.md](README.md), [phases/phase-2.5-cli-consolidation.md](phases/phase-2.5-cli-consolidation.md), [phases/phase-2.5.5-hardening-and-remediation.md](phases/phase-2.5.5-hardening-and-remediation.md), [phases/phase-2-cli.md](phases/phase-2-cli.md), [deferred-tasks.md](deferred-tasks.md), [../project-structure.md](../project-structure.md), [../tech-stack.md](../tech-stack.md) @@ -89,7 +89,9 @@ any order"* — never as headcount. flowchart TD W0["Wave 0 — One true baseline
baseline ✅ · CI truth · numbers"] W1["Wave 1 — Stop the bleeding ✅
3 CRITICALs · cost cap · ADR-0074"] - W2["Wave 2 — Shut the doors
MCP · fs jail · secrets
certifies 2.5.5 EXIT 1–3"] + LEDGER["#W15-1 — realized-cost ledger
ADR-0076 implementation"] + P265["Phase 2.6.5 — Core reliability
46 CR items · 8 P0 ADRs
absorbs the hostile-MCP class"] + W2["Wave 2 — Shut the doors
fs jail · secrets · config trust
certifies 2.5.5 EXIT 1–3"] W3["Wave 3 — Clear the ground
god-file decomposition · CLI net"] W4a["Wave 4a — The spine
2.6.A/D/H/K + 2 ADRs"] W4b["Wave 4b — Money floor & close-out
2.5.5.B/C/F/H drain · v0.2.0"] @@ -97,7 +99,7 @@ flowchart TD W5b["Wave 5b — Home becomes the product
2.6.G/I/J · then i18n sweep"] W6["Wave 6 — Hands, voice, lineage
2.6.M/B/E + 2.6.N foundation"] W7["Wave 7 — Orchestration & the gate
2.6.O/P · go/no-go → Phase 3"] - W0 --> W1 --> W2 --> W3 --> W4a --> W5a --> W5b --> W6 --> W7 + W0 --> W1 --> LEDGER --> P265 --> W2 --> W3 --> W4a --> W5a --> W5b --> W6 --> W7 W3 --> W4b W4b -.->|v0.2.0 release| W7 W4a -.->|file-disjoint lanes| W4b @@ -243,10 +245,17 @@ Ordered by whether the repo currently states something untrue, then by blast rad **Every item names the check that would close it, not just the defect.** -> **Status, 2026-08-09 — 23 of 24 closed.** Everything marked ✅ below is fixed, -> break-verified with the mutation confirmed applied, and committed on `development`. §E's six coverage gaps -> are all closed — `#W15-16`'s composition test fails by TIMING OUT when the abort listener is removed, which -> is the unkillable run reproduced exactly rather than an assertion standing in for it. +> **Status, 2026-08-10 — 24 of 24 closed, across TWO PRs.** Everything marked ✅ below is fixed and +> break-verified with the mutation confirmed applied; the two PRs are recorded separately because the closure +> dates and the merge states differ: +> +> - **23 items merged to `main` via PR #81** (2026-08-09). +> - **`#W15-1`, the last one, landed 2026-08-10** behind ADR-0076 + ADR-0077 (see §A) and rides **PR #82** +> (`development` → `main`), which is open at the time of writing. It is closed as work, not yet merged. +> +> §E's six coverage gaps are all closed — `#W15-16`'s composition test fails by TIMING OUT when the abort +> listener is removed, which is the unkillable run reproduced exactly rather than an assertion standing in +> for it. > > **`#W15-2` closed 2026-08-09** as [ADR-0075](../decisions/0075-fail-closed-resume-on-an-unreadable-event-log.md), > amending ADR-0074 §5: a read that feeds a REPLAY refuses when any row was skipped, a read that feeds a @@ -277,14 +286,34 @@ Ordered by whether the repo currently states something untrue, then by blast rad > transaction as its event. The `node:completed` delta then telescopes to zero on its own; a test must pin > that `SUM(run_costs) == runs.total_cost_microcents` still holds, since that is the invariant carrying the > no-double-count claim. -> 4. `packages/core/src/engine/engine.ts` — emit through `#emitDurable` at the attempt boundary, AWAITED before -> the next tool side effect, the next egress and the node terminal. This is the barrier; without the await -> the event is just another observation. -> 5. `packages/core/src/engine/checkpoint.ts` — fold it as a SUM of deltas, never a last-wins snapshot (ADR-0074 -> §2's reasoning: concurrent events under a `fan_out` have no canonical `seq` order). +> 4. The attempt boundary and its barriers — **corrected by +> [ADR-0077](../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)**. ADR-0076 +> §1 asked for an `await` at the settle point; there is no `await` to be had there. The seam's observer is +> `onAttempt?: (record) => void` — synchronous, void — and BOTH money events are emitted from that one +> callback a few lines apart, which `budget-governor.ts` already states outright ("which cannot await"). So +> the ledger takes ADR-0074 §2's shape: **start** the write at the settle instant on a chained in-flight +> promise, **join** it at three barriers — before the next egress admission, **before tool dispatch** +> (`agent-turn.ts`'s `await dispatchToolCalls(...)`, the barrier this adds beyond §2), and at the turn/node +> terminal. Every barrier awaits AND observes the failure, because `#emitDurable` is total for store faults +> and resolves. +> 5. `packages/core/src/engine/checkpoint.ts` — **corrected during implementation**. The instruction here was +> "fold it as a SUM of deltas", borrowing ADR-0074 §2's reasoning, and the wrong half of that cost two +> rewrites. Summing into the realized total double-counts (a node terminal's snapshot already contains its +> attempts); summing into a separate accumulator and maxing the two families UNDER-counts (a media node +> writes a snapshot and emits no attempt row, so every attempt after the last boundary vanishes). What is +> correct is the fold already two arms above it: `Math.max` over every durable ABSOLUTE total. §2's +> rejection of `Math.max` does not carry — it was rejected there only because a release can DECREASE a +> conservative total, and realized spend is monotonic. > -> The break-verify that matters most is step 4's ordering: deleting the `await` must redden a test, or the -> ledger records the charge without the guarantee that makes it one. +> **STATUS: all five steps are landed** (`8d7ffcf` … `50c60bd`), each with an Opus and a Sonnet round folded. +> The break-verify that matters most was step 4's ordering, and it now holds: with the tests draining to +> quiescence before asserting, deleting B2 reddens the tool-dispatch test and deleting B3 reddens the node- +> terminal test. B1 is masked by B2 on the current fixture (the second egress only comes after the tool), so +> it is covered by construction rather than by a red — named rather than implied. +> +> **This is now the prerequisite of Phase 2.6.5, not a parallel track.** Its five steps touch the same four +> files `CR-10` (ordered append tail) and `CR-12` (effect journal) restructure, so it lands first — see the +> 2.6.5 section below. > > **What ADR-0076 explicitly does NOT close**, so it is not read as more than it is: the cost of a TOOL EFFECT. > A resumed run can still re-execute an `http_request` POST, a `run_command` or an MCP mutation — the @@ -301,7 +330,10 @@ Ordered by whether the repo currently states something untrue, then by blast rad #### A. Needs an ADR first — the only part that is NOT PR #81's to close -- **`#W15-1` (blocker) — a workflow's REALIZED cost is not durable at the provider-attempt boundary.** +- ✅ **`#W15-1` (blocker) — a workflow's REALIZED cost is not durable at the provider-attempt boundary.** + *Closed 2026-08-10 by ADR-0076 + ADR-0077 and the five staged steps; the barrier is a chain-and-join, not + the inline await ADR-0076 §1 first specified. See the staged plan below for what each step landed and the + three carried-forward gaps.* `engine.ts` folds `cost:updated` into memory and streams it; the store documents that it is never persisted, and the checkpoint can only recover cost from a LATER `node:completed`/gate snapshot. So: a paid call succeeds → the model asks for a tool → the process dies during the tool → the realized cost is gone and the @@ -425,6 +457,50 @@ the review of `#W15-16`, fixed with it, and now the signal that test drives. `llm-provider-seam.md` still claims model-discovery/media-poll keep a small SDK retry; ADR-0028's amendment note still says ADR-0074 §2–§5 have not landed. Each could send a future reader back toward a #91-class hole. +### Phase 2.6.5 — Core reliability remediation (between Wave 1 and Wave 2) + +Three independent core reviews of the post-Wave-1 tree converged — independently — on the **same seven blocking +gaps**: effect journal, stdio MCP consent-before-spawn, run lease, compaction trust elevation, stream grammar, +input admission, event-log ordering. Three separate reviews landing on the same seven points is not opinion. + +The full, self-contained work list is +[phase-2.6.5-core-reliability-remediation.md](phases/phase-2.6.5-core-reliability-remediation.md) — **46 items** +(`CR-01`…`CR-95`) with evidence, fix, acceptance criteria and a decision/ADR/gate register, written so the work +can be done from that document alone. An adversarial plan review on 2026-08-10 corrected the phase boundary, +the exit rule and the execution order, and added two items (`CR-17` resume identity, `CR-63` `input_schema` +docs-only). + +**This is the corrected execution order, and it is what the graph above shows:** + +1. **`#W15-1` first.** Its five staged steps touch `run-event.ts`, `engine.ts`, `checkpoint.ts` and + `run-history-store.ts` — the same four files `CR-10` and `CR-12` restructure. Landing it after the durability + spine means writing its barrier against a persistence path that is about to change. +2. **Then Phase 2.6.5, closing before Wave 2 opens.** Which is only true if the work Wave 2 would block on + lives there — so **the hostile-MCP threat class moves into 2.6.5**: `CR-16` (consent before a stdio spawn) + plus `CR-40`–`CR-42` (transport cancellation, DNS/redirect SSRF, ingress bounds). Wave 2's queue item 2 is + **executed as 2.6.5's `W4`**; the 2.5.5 finding ids keep their home in the 2.5.5 phase doc and certify from + there. Wave 2 keeps the MCP name-collision/tool-poisoning trust items, the fs jail, the secrets layer, + persistence-security and the certification. +3. **Within 2.6.5, the oracle comes before the spine.** `CR-90`/`CR-91` (crash + durable-truth harness) land + first because they are the instrument `CR-10 → CR-11 → CR-92 → CR-12` is proven with; `CR-92` moved out of + the harness group into the spine, since an API returning `completed` while history says `failed` is a + runtime defect, not a test concern. + +Two rules that changed with the plan review: + +- **Fourteen items are non-deferrable** — every W0/W1 item plus `CR-50`, `CR-55`, `CR-73`, `CR-80`, `CR-92` and + `CR-95`'s short-term fix. Each has a cheap fail-closed option (refuse, remove, narrow the claim), so "too big + to fix now" argues for the cheap option, never for deferral. The previous exit criterion permitted deferring + all 43 items and declaring the phase complete. +- **The gate is `pnpm run ci` AND `pnpm coverage`.** `coverage` is a separate required CI check and is not + inside the `ci` script; this phase edits `packages/core` heavily, and the CI coverage job does not block on + `core`. + +`CR-01`–`CR-03` were half-closed by Wave 1 and finished right after the oracle — **all three are closed as of +2026-08-11** (PR #82). `CR-03` was a propagation gap from `#W15-10`'s own fix: the finding named three `--json` +paths that never got the safe serializer, and closing it found **five**, which is why the call-site half is now +an ESLint selector rather than a list. + ### Wave 2 — Shut the doors The two named live security holes, one unified filesystem jail, the secrets layer under test — then @@ -432,10 +508,14 @@ certify while the reviews are still booked. 1. 2.5.5.D · **project-layer MCP name collision redirecting a global secret** (G19), then **MCP tool-definition poisoning** (#202). -2. **MCP queue, strict** — connect-phase timeouts on every transport (#35, G32, #205) → transport/discovery/result - size bounds against a hostile server (G33, #201, #209, #288) → structured `serverId`/`reason` discriminants - (#203, #204) → the two fail-loud violations (#206, #207) → per-file transport-adapter tests (#297) → - `CLIENT_INFO.version` (#208). +2. **MCP queue, strict — EXECUTED AS PHASE 2.6.5 `W4`, not here.** Connect-phase timeouts on every transport + (#35, G32, #205) → transport/discovery/result size bounds against a hostile server (G33, #201, #209, #288) → + structured `serverId`/`reason` discriminants (#203, #204) → the two fail-loud violations (#206, #207) → + per-file transport-adapter tests (#297) → `CLIENT_INFO.version` (#208). These finding ids keep their home in + [phases/phase-2.5.5-hardening-and-remediation.md](phases/phase-2.5.5-hardening-and-remediation.md) and + certify from there in step 6 below; the *work* happens in 2.6.5 alongside `CR-16` (consent before a stdio + spawn) and `CR-41` (the DNS/redirect SSRF hole), because they are the same code and the same security + sitting. Splitting them across two phases would book the hostile-MCP reviewer twice. 3. **The fs jail, ONE reviewed change set** — 2.5.5.D · sensitive-credential floor for `.kube`/`.azure`/gcloud (#36, #39) → extract `deepestExistingReal` to `packages/shared` (#235) → **2.6.M's `project`-tier `extraRoots`** and **2.6.N's fs-floor home-anchoring**, both *pulled forward*. `~/.relavium/tmp/` (G21) @@ -449,9 +529,9 @@ certify while the reviews are still booked. context sites (G22). 2.5.5.D · `default_headers` (G20, after D13). 6. **Certify 2.5.5 exit criteria 1, 2 and 3** + the sub-stream D acceptance sign-off. **Closes M2.5.5-2 (D half).** -> Book **five security-review sittings by threat class** (hostile-MCP · fs/path-jail · secrets-at-rest · -> secrets-input · config-trust), not ~30 per-PR passes. Reviewer availability, not code, is this wave's -> critical path. +> Book **four security-review sittings by threat class** (fs/path-jail · secrets-at-rest · secrets-input · +> config-trust), not ~30 per-PR passes. The **hostile-MCP** sitting moved to Phase 2.6.5 with the queue-2 work; +> it covers `CR-16` and `CR-40`–`CR-42` there. Reviewer availability, not code, is this wave's critical path. ### Wave 3 — Clear the ground diff --git a/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md b/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md index e2e801d5..868b48e2 100644 --- a/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md +++ b/docs/roadmap/phases/phase-2.5.5-hardening-and-remediation.md @@ -446,7 +446,7 @@ This substream is the development-process substrate itself — `.github/workflow | In-phase | Completed by | Outcome | |----------|--------------|---------| -| ✅ M2.5.5-1 Safety-critical propagation fixes *(met 2026-07-30, PR #81 — its two CRITICALs, #91 and #228, are closed; ADR-0074 §1 remains open and is tracked as Wave 1.5 in [current.md](../current.md), which this milestone does not gate)* | 2.5.5.A + 2.5.5.C | Two of the three CRITICAL findings closed (`#91` the unredacted approval-preview secret leak; `#228` the chat-persister unhandled-rejection crash — the third, `#56`, lands with 2.5.5.I under M2.5.5-2); the budget governor's concurrency gap and the two-process migration race are fixed | +| ✅ M2.5.5-1 Safety-critical propagation fixes *(met 2026-07-30, PR #81 — its two CRITICALs, #91 and #228, are closed. ADR-0074 §1 was open at that date and CLOSED on 2026-08-09 with `/cost --release`; the "Wave 1.5" framing this note used was retracted — that work was PR #81's own closing list, not a later wave. See [current.md](../current.md).)* | 2.5.5.A + 2.5.5.C | Two of the three CRITICAL findings closed (`#91` the unredacted approval-preview secret leak; `#228` the chat-persister unhandled-rejection crash — the third, `#56`, lands with 2.5.5.I under M2.5.5-2); the budget governor's concurrency gap and the two-process migration race are fixed | | M2.5.5-2 Security-surface hardening | 2.5.5.D + 2.5.5.I | MCP tool-poisoning/config-secret-hijack gaps closed; the third CRITICAL (`#56`) closed as terminal-control sanitization reaches `relavium run`, the error boundary, and the approval card; `chat.ts`/`home-controller.ts` decomposed ahead of 2.6.E/2.6.G | | M2.5.5-3 CLI operator-safety net | 2.5.5.E | The outermost crash/signal/error boundary behaves uniformly across every command; `--version`/`--help` startup tax closed | | M2.5.5-4 Money-safety & process substrate | 2.5.5.B + 2.5.5.H | The three LLM adapters share guard/classification parity and `FallbackChain` is the sole retry authority; CI actually executes the compiled binary and the coverage/dependency/logging gates match what they claim to enforce | diff --git a/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md new file mode 100644 index 00000000..f3e0405e --- /dev/null +++ b/docs/roadmap/phases/phase-2.6.5-core-reliability-remediation.md @@ -0,0 +1,1072 @@ +# Phase 2.6.5 — Core reliability remediation (interlude) + +- **Status**: planned +- **Opened**: 2026-08-09 · **Plan corrected**: 2026-08-10 +- **Predecessor**: Wave 1 of the 2.5.5 remediation (complete — PR #81), then the `#W15-1` realized-cost + ledger implementation (**complete 2026-08-10**, ADR-0076 + ADR-0077 — see [Prerequisite](#prerequisite)) +- **Successor**: Wave 2 of the 2.5.5 remediation, **reduced** — this phase absorbs Wave 2's hostile-MCP + threat class (see [Phase boundary](#phase-boundary)); Wave 2 keeps the filesystem jail, the secrets + layer, persistence-security and the 2.5.5 certification +- **Kind**: interlude, like [phase 2.5.5](phase-2.5.5-hardening-and-remediation.md) — no new product surface, + only the invariants an existing surface already claims + +## Why this phase exists + +Three independent core reviews of the engine were run against the tree as it stood immediately after Wave 1. +They converged — independently, without seeing each other — on the **same seven blocking gaps**. That +convergence is the reason this phase exists: three separate reviews landing on the same seven points is not a +matter of opinion. + +The verdict all three reached is worth stating, because it shapes every item below: + +> The architecture is right. Nothing here calls for a rewrite. The gaps are in the **last ten percent of +> reliable execution** — durable ownership, effect identity, trust provenance, stream grammar, admission and +> ordering — and the existing seams are good enough to carry the fixes. + +The second reason is sharper. Several of our own canonical documents currently promise behaviour the code does +not deliver: *crash-safe resumable workflow*, *duplicate-free side effect*, *bounded stream*, *first completed +branch wins*, *schema-validated input/output*, *full session-to-workflow graduation*. A claim the code does not +keep is worse than a missing feature, because it is the claim a user plans around. + +### Provenance of the findings + +Recorded so the severities below can be audited rather than trusted: + +| Source | Ran against | What it was | +|--------|-------------|-------------| +| Review A | post-Wave-1 tree, 2026-08-08/09 | Full-engine reliability read, findings with file/line evidence | +| Review B | post-Wave-1 tree, 2026-08-09 | Independent full-engine read, no sight of A | +| Review C | post-Wave-1 tree, 2026-08-09 | Independent read scoped to durability, MCP and cost | +| Plan review | this document + tree, 2026-08-10 | Adversarial read of the plan itself; produced `CR-17`, `CR-63`, the phase-boundary correction and the non-deferrable exit rule | + +"Consensus" severity below means at least two of A/B/C rated it independently. Every evidence line was +re-verified against the working tree before it was written here; where a review's claim did not survive that +check it was dropped rather than restated. + +## What must NOT change + +Recorded so a remediation pass does not quietly undo a decision that is correct: + +- The pure-TypeScript engine with zero platform imports. +- The `LLMProvider` seam and the rule that no vendor SDK type crosses it. +- Deny-by-default tool policy, the approval chain, and the QuickJS sandbox. +- Strict authored YAML with Zod at every boundary. +- Event-sourced checkpoint/resume as the recovery model. +- Keys in the OS keychain only. + +## Phase boundary + +This phase runs **between Wave 1 and Wave 2**, and it closes before Wave 2 opens. That is only true if the +work Wave 2 would otherwise have blocked on lives here, so: + +- **The hostile-MCP threat class moves into this phase.** `CR-16` (consent before a stdio spawn) plus `W4` + (`CR-40`–`CR-42`) are the same code area and the same security-review sitting as Wave 2's MCP queue. Wave 2's + queue item 2 is **executed as this phase's `W4`**; its 2.5.5 finding ids keep their home in + [phase-2.5.5](phase-2.5.5-hardening-and-remediation.md) and are certified from there. +- **Wave 2 keeps** the project-layer MCP name collision and tool-definition poisoning (a trust/config concern, + not the transport boundary), the unified filesystem jail, the secrets layer, persistence-security, and the + certification of 2.5.5 exit criteria 1–3. +- **`current.md` is canonical for execution order** and carries the corrected graph. This section states the + scope line; the ordering statement lives there. + +The alternative — leaving `CR-16`/`W4` in Wave 2 and declaring this phase closed *during* Wave 2 — was +rejected: a phase whose exit criteria cannot be evaluated until a later phase is half-done is not a gate. + +## Prerequisite + +**`#W15-1` — the durable per-attempt realized-cost ledger — lands before `W1` starts. ✅ SATISFIED +(2026-08-10).** All five steps are implemented and landed on `development`, each with an Opus and a Sonnet +round folded; they ride **PR #82** and are **pending merge to `main`**. The decision acquired a correction on +the way: +[ADR-0077](../../decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md) amends +ADR-0076 §1, whose stated mechanism (an inline `await` at the attempt boundary) is unimplementable — the +seam's attempt observer is synchronous. The ledger uses ADR-0074 §2's chain-and-join shape instead, with a +third barrier before tool dispatch that `CR-12`'s effect journal is expected to extend rather than re-thread. + +Two things `W1` inherits from it. **`CR-10` gains a concrete adversary:** the ledger's derived `run_costs` row +had to be written as a telescoping delta rather than the event's raw charge, precisely because +`#emitDurable` starts every persist immediately and serializes only delivery — out-of-order commit is real +today, not a cloud-store hypothetical. And **`CR-11`/`CR-92` gain a known-open seam:** the ledger's observe +half reads the run's `#failure` because `#emitDurable` discards the store error, which under-triggers during +a cancel and on an already-failing run. A per-event durable outcome would close it, and that is a change to +the same choke point `CR-10` restructures. + +The original reasoning for the ordering, kept because it is what made it right: + +[ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md) is Accepted and its implementation +was staged in five steps in [current.md](../current.md). Those five steps touch +`packages/shared/src/run-event.ts`, `packages/core/src/engine/engine.ts`, +`packages/core/src/engine/checkpoint.ts` and `packages/db/src/run-history-store.ts` — the same four files +`CR-10` and `CR-12` restructure. Landing it after the durability spine means writing its barrier against a +persistence path that is about to change, then rebasing it onto the moved floor. Landing it first means its +awaited emit inherits `CR-10`'s ordered tail for free. + +There is a second reason, and it is the one this project keeps re-learning: an accepted ADR with no +implementation is a decision that reads as shipped. Wave 1's completion claim was wrong twice for exactly that +shape. + +## Working discipline for this phase + +This phase inherits the discipline Wave 1 arrived at the hard way. It is not optional ceremony; every clause +below exists because its absence produced a real defect during Wave 1. + +1. **A decision gets an ADR before it gets code.** All eight `W1` items change what a contract *means* — what + "exactly once" means, what `system` authority means, what "the stream ended" means, what a resume is allowed + to assume. Wave 1 proved that treating a decision as a task item produces false completion claims. Write the + ADR, get it accepted, then implement. +2. **Break-verify every regression test, and verify the mutation actually applied.** Wave 1 produced three + tests that passed with the production code mutated. Each was found only by checking that the mutation had + landed (e.g. `grep -c` the mutated symbol → 0) before trusting a red. +3. **Never ship a hollow test.** If the honest test cannot be built, leave the gap open, name it, and name the + mutation that would close it. A test that implies coverage it does not have is worse than none. +4. **Record rather than assert.** Any claim that cannot be demonstrated goes in as a limitation, not as a + property. +5. **Fix the canonical doc in the same change.** One canonical home per artifact; a fix that leaves the spec + describing the old behaviour has not landed. Where a *standard* states a guarantee the code does not keep, + the correction lands with the ADR, not with the last line of implementation. +6. **Never assert on model behaviour.** A prompt-security test proves a *structural* property — which role a + byte can reach, which type the builder accepts — never "the model did not comply". A test that depends on a + model's judgement is a hollow test under clause 3. +7. **Security review is booked by threat class, not per PR.** Four sittings cover this phase: + **prompt/trust provenance** (`CR-13`), **hostile MCP** (`CR-16`, `CR-40`–`CR-42`), **media bytes** + (`CR-50`, `CR-53`, `CR-54`), **provider/config trust** (`CR-80`). Each sitting produces a recorded + [security-review](../../standards/security-review.md) checklist entry with a direct adversarial test — not a + reviewer's assurance. +8. **The gate is `pnpm run ci` AND `pnpm coverage`, both exit 0, checked by exit code.** `pnpm run ci` does not + include coverage; `coverage` is a separate required check. CI's coverage job enforces the floor for + `@relavium/llm` and `@relavium/mcp` only — `packages/core` is measured but not blocking, a dated carve-out + whose promotion trigger is Wave 3. This phase edits `core` heavily, so the local `pnpm coverage` (which + enforces all three) is the honest gate here. + +## How the work is grouped + +`CR-##` ids below are this phase's own; they are **stable** and never renumbered. The thematic groups `W0`…`W9` +are the reference layout. The **execution order is separate from the grouping** and is stated once, below. + +Severity is the consensus of the three reviews, except where an item is marked *(plan review)*. + +### Execution order + +The groups are not the schedule. Test honesty comes first because it is the oracle everything after it is +proven with, and the durability spine is a chain. + +```mermaid +flowchart TD + PRE["Prerequisite
#W15-1 realized-cost ledger"] + ORACLE["Oracle first
CR-90 · CR-91"] + HALVES["W0 — finish the halves
CR-01 · CR-02 · CR-03"] + SPINE["Durability spine
CR-10 → CR-11 → CR-92 → CR-12"] + IND["Independent P0 lines
CR-13 · CR-14+CR-21 · CR-15+CR-17"] + MCP["Hostile MCP
CR-16 · CR-40 · CR-41 · CR-42"] + CLAIM["False-claim blockers
CR-50 · CR-55 · CR-73 · CR-80 · CR-94 · CR-95"] + REST["Remaining P1 themes
W2 · W3 · W5 · W6 · W7 · W8 · W9 residual"] + EXIT["Exit certification"] + PRE --> ORACLE --> HALVES --> SPINE --> CLAIM --> REST --> EXIT + HALVES --> IND --> REST + HALVES --> MCP --> REST +``` + +`CR-92` sits inside the spine, not in `W9`: an API that returns `completed` while durable history says `failed` +is a runtime correctness defect, not a harness concern. `CR-90`/`CR-91` come before the spine because a crash +and durable-truth oracle is what proves `CR-92` landed, and half of `CR-10` — building it last would mean +asserting the spine with the harness the reviews already found insufficient. It does **not** instrument +`CR-11` or `CR-12`; that was an early overclaim, corrected once the oracle was built and measured (see +`CR-91`). Budget those two their own predicates. + +- **W0 — finish the halves.** Cheap, already half-done. Leaving them half-done is the exact failure this phase + is correcting. +- **W1 — the P0 blockers.** ADR-first. Seven are the convergent set; `CR-17` was added by the plan review. +- **W2…W9 — P1 themes.** Grouped so each theme is one reviewable PR, not sixty scattered tickets. + +### Decision, ADR and gate register + +Every row states whether the decision is already made. **An item whose decision column says "open" does not +start until the maintainer settles it** — deciding during implementation is the failure mode this phase exists +to correct. + +| Item | Decision | ADR | Non-deferrable | Security sitting | +|------|----------|-----|----------------|------------------| +| `CR-01` | made (route through the latch) | — | yes | — | +| `CR-02` | made (counter stays; rule = *engaged*) | — | yes | — | +| `CR-03` | made (route through `stringifyJsonLine`) | — | yes | — | +| `CR-10` | made (one ordered append tail per run) | new | yes | — | +| `CR-11` | made (lease + fencing token) | new | yes | — | +| `CR-12` | made (three-tier effect contract) | new, amends [ADR-0041](../../decisions/0041-external-action-governance-seam.md) + [ADR-0037](../../decisions/0037-engine-tool-execution-boundary.md) | yes | — | +| `CR-13` | made (summary is untrusted, never `system`) | new, supersedes [ADR-0062](../../decisions/0062-context-compaction-and-cli-history-commands.md) §1 | yes | prompt/trust | +| `CR-14` | made (exactly one terminal, grammar pinned) | new | yes | — | +| `CR-15` | made (engine-side admission) | new | yes | — | +| `CR-16` | made (consent before spawn, lazy connect) | new | yes | hostile MCP | +| `CR-17` | made (persist and verify resume identity) | with `CR-15`'s | yes | — | +| `CR-20` | made (honour `timeout_ms`) | — | no | — | +| `CR-21` | made (per-attempt deadline) | with `CR-14`'s | no | — | +| `CR-22` | made (absolute deadlines) | — | no | — | +| `CR-23` | **open** — grace period length, quarantine policy | new | no | — | +| `CR-30` | made — implement [ADR-0036](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md)'s accepted no-drop producer-await | none (already decided) | no | — | +| `CR-31` | **open** — the cap values | — | no | — | +| `CR-32` | **open** — the bound values | — | no | — | +| `CR-33` | **open** — retention policy shape | — | no | — | +| `CR-40` | made (forward signal + deadline) | with `CR-16`'s | no | hostile MCP | +| `CR-41` | made (apply the built-in egress floor) | with `CR-16`'s | no | hostile MCP | +| `CR-42` | **open** — the ingress bounds | with `CR-16`'s | no | hostile MCP | +| `CR-50` | **open** — complete the handle path, or remove the tool | — | yes | media bytes | +| `CR-51` | made (gate on model-level capability) | — | no | — | +| `CR-52` | made (pull [ADR-0039](../../decisions/0039-same-provider-reasoning-replay.md)'s deferral forward) | ADR-0039 follow-up | no | — | +| `CR-53` | made (bounded stream to the store) | — | no | media bytes | +| `CR-54` | made (content-hashed handle at first resolution) | — | no | media bytes | +| `CR-55` | made (missing rate ⇒ unpriced) | — | yes | — | +| `CR-60` | **open** — race semantics, or rename + correct the table | — | no | — | +| `CR-61` | **open** — needs a JSON-Schema validator dependency | new (dependency ADR) | no | — | +| `CR-62` | **open** — extract dependencies, or fail loudly | — | no | — | +| `CR-63` | made (verify the docs; no runtime change) | — | no | — | +| `CR-64` | made (enforce at parse / `$ref` resolution) | — | no | — | +| `CR-70` | **open** — persist bounded tool pairs, or narrow the spec | — | no | — | +| `CR-71` | with `CR-70` | with `CR-70`'s | no | — | +| `CR-72` | **open** — implement, or reject at parse | — | no | — | +| `CR-73` | made (wire it, or stop advertising it) | — | yes | — | +| `CR-80` | made (fail closed) | — | yes | provider/config | +| `CR-81` | made (match the canonical request) | — | no | — | +| `CR-82` | made (missing usage is unknown) | — | no | — | +| `CR-90` | made (exclude repo-local checkouts) | — | no | — | +| `CR-91` | made (build the durable-truth oracle) | — | no | — | +| `CR-92` | made (durable outbox, distinct typed result) | with `CR-10`'s | yes | — | +| `CR-93` | made (scope per run/session/tenant; may defer the work, not the decision) | — | no | — | +| `CR-94` | made (numeric approval lease) | — | no | — | +| `CR-95` | made (short term: fail closed) | — | yes (short term) | — | + +--- + +## W0 — Finish what is already half-done + +These are partially closed. Each has a verified open half. + +### CR-01 — `session:cancelled` bypasses the session durability latch · High · ✅ CLOSED 2026-08-11 + +**Evidence.** The CLI chat persister wraps its cost and turn writes in the `persistDurably` failure latch, and +since Wave 1 the `session:compacted` and `session:trimmed` arms go through it too. The `session:cancelled` arm +still calls `deps.store.writeTurn(...)` directly (`apps/cli/src/chat/persister.ts`). + +**Why it matters.** The event bus isolates listener errors from the producer, so a failing DB write lets the +cancel path report success. The durability latch never sets, so nothing gates the next provider egress. + +**Fix.** Route the `session:cancelled` write through `persistDurably`, exactly as its two siblings now are. + +**Acceptance.** A failing `writeTurn` on cancel sets `durabilityFailure`, and the next egress is refused. +Break-verify by removing the wrapper. + +**Closed — but the acceptance above is half unsatisfiable, and the real defect was the other half.** + +- The arm writes through `updateSession`, not `writeTurn`: the terminal marks the session `ended` rather than + appending a turn. +- **"the next egress is refused" cannot hold for a terminal event.** `session:cancelled` is emitted only by + `AgentSession.cancel()`, which sets `#status = 'cancelled'`, and every later egress entry point is already + refused by `#assertSendable()` with `not_active`. Nothing reads `durabilityFailure` again, and each + construction site builds a fresh persister with a fresh latch. The latch half is wrapped for SYMMETRY — + one arm reaching the store bare is how the next reader concludes the wrapper is optional. +- **The load-bearing half is the `finally`.** A throwing write jumped straight over the two unsubscribe lines + and left the persister attached to the bus, so every later event re-entered one that could not write. (The + user was told either way — the throw always escaped into `deliver`'s listener-error sink, so "let the cancel + path report success" was also not quite right.) Pinned by wrapping the handle's unsubscribe and asserting it + ran; the first two attempts at that assertion were vacuous — `close()` calls the same idempotent + unsubscribe, and a second `cancel()` emits no event — and both passed with the `finally` deleted. + +### CR-02 — A failed turn flush leaves the turn counter incremented · Medium · ✅ CLOSED 2026-08-11 + +**Evidence.** `packages/core/src/engine/agent-session.ts` increments `#turnCount` on the success path *before* +it awaits `flushBudgetCommitments()`. Wave 1 fixed the transcript half of this (the flush now precedes the +assistant append, so the rollback's one-message `pop()` is correct again). The counter half was not examined. + +**The decision, made.** The counter **stays where it is.** The rule the code already applies on both catch +paths is *count a turn against the hard cap only when a provider actually engaged* — and by the time the +success path reaches the increment, `#runTurn` has resolved, so a provider did engage. A flush rejection after +that point is a durability failure, not evidence the turn never happened. Moving the increment after the flush +would make a billed turn free. + +**Fix.** Keep the increment; state the rule in a comment at the success-path site the way the two catch paths +already state it, so the next reader does not re-derive it. Pin it with a test. + +**Acceptance.** A rejecting `flushBudgetCommitments` on an otherwise-successful turn still consumes a turn +against the cap. `#turnCount` is a JS private field and **must not** be given a production accessor to test — +assert through the observable turn-cap behaviour (`maxTurns: 1`, one flushing-failing turn, the next +`sendMessage` refused by the cap). Break-verify by moving the increment below the flush. + +**Closed — the counter stays, and the code now says why.** The decision was made from the code rather than +chosen: both catch paths already apply *count a turn only when a provider ENGAGED*, and by the increment +`#runTurn` has resolved. A flush rejection past that point is a durability failure, not evidence the turn +never happened — moving the increment below the flush hands back a turn the provider billed. One detail worth +recording for the test: the rejection SURFACES out of `sendMessage` (ADR-0074 §2 fails the active owner +loudly), so the regression awaits a rejection and then drives the cap. + +**And the decision had a mirror-image hole one line further on.** A flush rejection settles through the +unclassified branch, which emitted its terminal with a hardcoded `{0,0}` — so the turn the decision insists +the provider billed consumed its cap slot AND silently dropped its real tokens from every total. That is the +opposite of the error the counter decision refuses to make, and it contradicts EA2/ADR-0055's rule to report +real usage whenever a provider engaged. The success path now captures the usage just before the flush and the +unclassified terminal reports it, falling back to zero when nothing engaged. + +### CR-03 — Three `--json` paths still bypass the safe serializer · High · ✅ CLOSED 2026-08-11 + +**Evidence.** Wave 1 introduced `stringifyJsonLine` (`apps/cli/src/render/sanitize.ts`), which losslessly +escapes C1 controls and Trojan-Source bidi characters, and routed the workflow renderer and the record writer +through it. Three sibling paths still use bare `JSON.stringify`: +`apps/cli/src/commands/agent-run.ts:161`, `apps/cli/src/commands/chat-export.ts:90`, +`apps/cli/src/commands/chat.ts:2392`. + +**Why it matters.** Same class as the gap Wave 1 closed: `JSON.stringify` escapes `ESC` but leaves `U+009B` +(8-bit CSI) and the bidi family raw, and these streams carry model-, tool- and persisted content. This is a +propagation gap — the fix landed, the siblings did not get it. + +**Fix.** Route all **five** through `stringifyJsonLine` — the three above plus `commands/import.ts` and +`commands/export.ts`, which write the same record shape and were not in the finding (see the closing note). +The escape is lossless, so no machine contract changes. + +**Acceptance.** One shared adversarial test drives a C1 + bidi payload through the serializer and asserts the +raw code points do not survive while `JSON.parse` round-trips to the identical string; the CALL-SITE half — +that all five surfaces, and any sixth written later, actually go through it — is an ESLint +`no-restricted-syntax` selector, because a list only ever proves the places someone remembered. + +**Closed — and it was FIVE paths, not three.** The finding named three; `import.ts` and `export.ts` write the +same record shape and were not in it. `docs/reference/cli/commands.md` pairs the last two by that shape in one +sentence, which is how the miss was findable at all. + +Two corrections to my own reasoning while closing it, both recorded because they were stated as fact first: + +- **`import --json` is the LEAST attacker-reachable of the set, not the most.** I justified adding it by + saying `parsed.slug` "comes straight out of an imported artifact". It does — and `kebabIdSchema` rejects the + document before that value exists, so it cannot carry a C1 or bidi code point at all. The genuinely + reachable ones are the session-event streams the original finding already named, plus `export --json`, whose + `path` derives from the user-typed `--out`. Fixing `import` is cheap hygiene; the ranking was wrong. +- **A source-scanning test is the wrong mechanism, and it proved so immediately.** The first regression + matched `writeOut(\`${JSON.stringify(` and missed `export.ts` purely because prettier had wrapped the + argument onto its own line. A guard that only checks the places someone remembered catches nothing new by + construction — which is how this propagation gap reopened twice already. The call-site half is now an + ESLint `no-restricted-syntax` selector (`eslint.config.mjs`) that fires on the SHAPE anywhere in + `apps/cli/src`, so a sixth surface is caught the first time it is written rather than at review. + `json-line-surfaces.test.ts` keeps only the behavioural assertion, with its hostile payload built from code + points at runtime — a raw C1/bidi byte in source is a Trojan-Source hazard in its own right. + +`process/render-error.ts` is the one deliberate exception and is allowlisted: it pre-sanitizes with the +STRIPPING sanitizer, so the `--json` error envelope is lossy on purpose where the NDJSON stream is lossless. + +--- + +## W1 — The P0 blockers + +`CR-10 → CR-11 → CR-92 → CR-12` is a chain: an effect journal is meaningless while two processes can own the +same run, run ownership rests on the log being an ordered prefix, and a terminal that can diverge from durable +truth undermines both. `CR-13`, `CR-14` and `CR-15`+`CR-17` are independent lines that can run in parallel. +`CR-16` opens the hostile-MCP line. + +Seven of the eight are the set all three reviews converged on. `CR-17` was added by the plan review of this +document and verified against the engine's own interface contract. + +### CR-10 — The durable event log is not a gap-free ordered prefix · Blocker · needs an ADR + +**Evidence.** The engine assigns sequence numbers centrally but starts each event's persistence independently +for concurrency; the code comment states the split explicitly ("persistence concurrent, delivery serialized"). +The SQLite store writes each event in its own transaction, so a higher sequence can commit while a lower one is +still in flight (`packages/core/src/engine/engine.ts`, `packages/db/src/run-history-store.ts`). + +**Failure.** Event `N`'s write is slow; `N+1` commits; the process dies before `N`. The disk holds a pseudo- +prefix with a hole in its sequence, and the causal predecessor the checkpoint fold depends on is missing. + +**Why it is first.** This is the root cause behind the sum-vs-max decision already recorded for conservative +commitments: concurrent events under a `fan_out` have no canonical `seq` order. Every durability property below +assumes the log is an ordered prefix. + +**Fix.** One ordered append tail per run. `N+1` must not reach the store until `N` is durable. Store-side +compare-and-append against the expected last sequence. + +**Acceptance.** A store harness that delays event `N` and commits `N+1` must be rejected, not accepted. A crash +injected between the two must leave a prefix with no hole. Existing gap-free assertions must still pass. +Break-verify by restoring the concurrent start. + +### CR-11 — No cross-process run ownership or fencing · Blocker · needs an ADR + +**Evidence.** The engine states that its cross-process guarantee rests on store uniqueness, but the only DB +uniqueness is `(run_id, seq)`. `resumeFromCheckpoint` performs an in-memory check, then loads the checkpoint and +builds an independent `RunExecution`. There is no run lease, fencing token, or checkpoint-cursor CAS. + +**Failure.** Two CLI or desktop processes read the same paused run at the same time and become two independent +side-effect producers for one run. + +**The decision, made.** A **run lease with a monotonic fencing token**, checked on every durable write — not a +bare `last_seq` CAS. A CAS on `last_seq` stops two processes writing the same row; it does **not** stop two +processes performing the same external effect before either of them writes anything. The fence is what makes a +stale owner harmless after it loses the lease. + +**Fix.** `acquireRunLease(runId, ownerId, expectedGeneration, ttl)` returning a fencing token; every durable +write carries the token and is rejected if it is stale. The process that loses becomes a read-only observer. + +**Acceptance.** Two concurrent resumes of one paused run: exactly one proceeds, the other degrades to observer +with a typed, actionable error. A lease that expires mid-run is fenced out of further durable writes — proven +by driving a write with a stale token and asserting the rejection, not by asserting the lease table's contents. + +### CR-12 — No durable effect/idempotency journal on the hot path · Blocker · needs an ADR + +**Evidence.** `NodeExecContext` carries an attempt number but no run/effect idempotency key. +`ToolDispatchContext` carries a node id but no run correlation or semantic key. The registry calls +`def.dispatch` directly with no durable prepare step. + +**Failure.** `node:started` persists → an `http_request` POST or MCP mutation completes at the target → the +process dies before the tool result or `node:completed` persists → resume re-runs the node → the target sees +the same effect twice. + +**Impact.** Duplicate ticket, deploy, payment, message or commit. The *duplicate-free* product claim is false +until this closes, and audit/compensation cannot be built on top of it. + +**The contract, stated precisely — three tiers, not one.** A general HTTP POST, MCP mutation or shell command +cannot be made exactly-once by the engine alone. The interval *"the target completed the effect → the process +died → no receipt reached the store"* is irreducibly ambiguous when the target offers no idempotency key and no +way to ask. Promising exactly-once across that gap would be a claim of exactly the kind this phase exists to +remove. The contract is therefore tiered: + +1. **Target accepts an idempotency key** → safe retry under the same key. Effectively exactly-once, and the + only tier that may say so. +2. **Target's outcome is queryable** → reconcile from a receipt lookup before deciding. Exactly-once after + reconciliation. +3. **Opaque, non-idempotent effect** → `dispatched → ambiguous → needs_attention`. **Never auto-retried.** This + is *at-most-once dispatch attempt*, and the docs must say that, not "exactly once". + +**Fix.** Add `EffectAttemptId = runId + nodeId + nodeAttempt + toolCallId + effectiveArgsHash` to +`ToolDispatchContext`. Side-effectful host ports take it as a required argument. A durable state machine in the +run store: `prepared → dispatched → committed | ambiguous → needs_attention`, with the tier recorded per effect. + +**Canonical docs this must correct, in the ADR's own PR:** + +- [architectural-principles.md](../../standards/architectural-principles.md) §11 currently states that a stable + key derived from `runId + nodeId + retryCount` means *"a retry never double-applies a side effect"*. That is + false today and remains false for tier 3 after this lands. It must be rewritten to the tiered contract. +- [ADR-0041](../../decisions/0041-external-action-governance-seam.md) places transactional/idempotent wrapping + in the optional enterprise `ActionGuard`. This ADR moves a baseline effect-identity floor below that seam, so + it **amends** ADR-0041 rather than sitting beside it. +- [ADR-0037](../../decisions/0037-engine-tool-execution-boundary.md) owns the `ToolHost` boundary this changes. + +**Acceptance.** Kill the process after the effect completes at the target — resume produces no duplicate for +tier 1 and tier 2, and produces `needs_attention` with no retry for tier 3. The same effect key arriving from +two processes commits once. A tier-3 effect is never auto-retried, proven by a test that would go red if the +retry path were reachable. + +**Relationship to the realized-cost ledger.** [ADR-0076](../../decisions/0076-durable-per-attempt-realized-cost-ledger.md) +explicitly scopes effect duplication out and names this as the decision that owns it. That ADR's implementation +is this phase's [prerequisite](#prerequisite); this item is the larger of the two. + +### CR-13 — Compaction summary is elevated to `system` authority · Blocker (untrusted content) · needs an ADR + +**Evidence.** The prior conversation is handed to the summarizer as user-role data; the summarizer's model +output is taken as a plain string, becomes the context preamble, and is concatenated directly into the authored +`agent.system_prompt` on the next turn. Resume and model reseat restore the persisted summary into the same +preamble (`packages/core/src/engine/agent-session.ts`). + +**Attack.** A user message or a read document contains a closing tag for the summary fence plus "ignore previous +instructions". The summarizer preserves it as a task. On the next tool-capable turn those bytes sit in the +`system` role — and survive a restart, because the summary is persisted. An XML fence is not a trust boundary. + +**Conflict.** Our binding security standard treats model output and tool results as untrusted and forbids +concatenating untrusted content into `system`. [ADR-0062](../../decisions/0062-context-compaction-and-cli-history-commands.md) +§1 fixed exactly that concatenation as its decision, so this needs an append-only superseding ADR, not a patch. + +**What the superseding ADR must answer.** ADR-0062 §1 did not choose the preamble by accident — it explicitly +rejected injecting the summary as a transcript message because *"a summary message either forces an +`assistant`-first array Anthropic rejects"*. The new ADR has to answer that, not ignore it. The likely shape is +a separate **untrusted content part inside the first user-role turn**, which is neither a standalone message nor +a `system` concatenation; whatever is chosen, the rejected alternative is engaged on its own terms. + +**Fix.** Carry the summary as untrusted. Dynamic summary bytes must never reach the `system` builder. Make +`AgentTurnParams.system` accept only a branded type the authored builder can produce, so the compiler — not a +convention — enforces it. + +**Acceptance.** Structural and type-level only, per working-discipline clause 6: + +- A type-level test proves a dynamic `string` cannot be passed as `system`; the branded type is constructible + only by the authored-prompt builder. +- Given a summary containing a fence-closing sequence plus instructions, the assembled request has those bytes + in a user-role content part and **zero** occurrences in the `system` field — asserted on the built request, + before any provider call, and again after a restore from persistence and after a model reseat. +- Tool authorization for the following turn is computed from policy alone; a test mutates the summary text + arbitrarily and asserts the resolved tool set is byte-identical. +- **No assertion of the form "the model did not obey the injected instruction."** + +### CR-14 — A stream that ends without a terminal `stop` counts as success · Blocker · needs an ADR + +**Evidence.** The fallback chain emits a success attempt when the provider iterator ends cleanly with no usage +(`packages/llm/src/fallback-chain.ts`). The agent turn starts with a default `stopReason` of `stop` and returns +a normal result without ever seeing a terminal chunk. The canonical `StreamChunkSchema` +(`packages/llm/src/types.ts`) carries both `stop` and `error` as terminal arms. **An existing test pins the +no-terminal case as success** — so the fix changes that test, deliberately, rather than deleting it. + +**Failure.** A transport, proxy or provider closes after `text_delta: "partial"` with no terminal chunk. +Relavium treats the partial text as a completed assistant answer and passes it downstream as a successful node +output. + +**Fix — the grammar, stated in full.** The ADR pins all of it, and names which layer enforces it (adapter vs. +`FallbackChain`) so the rule has one owner: + +1. Exactly one terminal per stream: `stop` **xor** `error`. +2. The terminal is the last chunk. +3. No `*_delta` or content chunk after the terminal. +4. Two `stop`, `stop` + `error`, or two `error` are protocol violations. +5. A clean EOF with no terminal is a `transport`/`protocol` error, never a success. +6. An empty stream that reaches EOF with no chunks at all is an error. +7. A pre-content failure may fail over; a **content-committed** failure must not fail over or retry — it is + surfaced. + +**Acceptance.** One test per numbered rule above, driven through a fake provider. A stream ending without a +terminal produces a classified error, not a node success. A content-committed truncation does not trigger +failover. The superseded test is rewritten to assert the new rule **and keeps a note recording the reasoning for +the behaviour it replaces**, the same way `checkpointer.test.ts` was handled in Wave 1. + +### CR-15 — The engine does not enforce the authored input contract · Blocker · needs an ADR + +**Evidence.** The shared contract states the engine validates inputs before a run starts. `WorkflowEngine.start()` +passes the caller's `inputs` object straight to the run execution, which stores it without cloning, applying +defaults, or validating against the schema. The input node reads values straight out of that map. The CLI's own +validator does unknown/required checks and coarse coercion only, and explicitly leaves deep validation and +default resolution to the engine. + +**Failure.** An input with a declared default that the caller omitted arrives as `undefined`. A wrong type or a +value outside an enum reaches deep execution. CLI, desktop, extension and the future API behave differently — +which breaks the "one engine, every surface" guarantee directly. + +**Fix.** A pure `resolveAndValidateWorkflowInputs` in core, running before id generation and before the first +event. + +**Acceptance — the whole authored contract, not a sample.** `InputValidationSchema` +(`packages/shared/src/workflow.ts`) carries `format`, `pattern`, `enum`, `min`, `max`, `min_length`, +`max_length`, alongside `required`, `default` and `type`. Each gets a case: + +- A missing `required` input fails admission. +- An unknown input key fails admission. +- Every validation field above rejects a violating value and accepts a conforming one. +- A declared `default` is itself validated — an authored default that violates its own rules fails at parse, + not at run. +- The coercion/strictness split between surface and engine is stated in the ADR and pinned: the CLI's coarse + coercion happens *before* the engine, and the engine is strict about what it receives. +- The engine **clones** the caller's `inputs`; mutating the caller's object after `start()` does not change the + run. +- `start` and `resume` apply the same admission. +- An admission failure produces a typed error and **no `runId`, no `run:started`, no row** — asserted by + checking the store is untouched. + +### CR-16 — Stdio MCP spawns a local process before consent · Blocker · needs an ADR + +**Evidence.** The MCP connection is opened while the chat session is being built, before a mode or turn starts; +the agent-run path prepares MCP first and applies mode policy afterwards. An agent or workflow declaration +produces the `command`, `args` and `cwd` spawn spec directly, and the SDK adapter spawns it. + +**Failure.** No tool call is required. `ask` mode does not protect the spawn. `shell: false` does not stop the +declaration from naming `sh`, `bash` or `node`. An imported community artifact executes arbitrary local code at +load time. + +**Fix.** Consent before the first spawn, showing executable, args, cwd, artifact source and hash. Unknown +fingerprint fails closed. In non-interactive mode, fail closed unless an explicit `--allow-mcp-stdio ` +is supplied. Replace auto-start with lazy connect on first actual MCP tool need. + +**Acceptance.** Importing and opening an artifact with a stdio MCP server spawns nothing until consent — proven +with a **real spawn counter** injected at the process boundary, asserted at zero, not by inspecting a state +flag. A non-interactive run without the digest flag fails closed with an actionable message. A changed +fingerprint re-prompts. + +### CR-17 — A resume trusts caller-supplied identity it never verifies · Blocker *(plan review)* · ADR with `CR-15` + +**Evidence.** `ResumeFromCheckpointInput` (`packages/core/src/engine/engine.ts`) documents its own gap: the +checkpoint verifies workflow identity, but `inputs`, `executionMode` and `planOptions` are **not** checkpoint- +derived, and the docblock states that passing different ones "would silently diverge the rehydrated execution +from its `run:started` state". The remedy is written there as "a future revision". + +**Failure.** A host resumes with different inputs — a different file path, a different flag, a different +execution mode — and the run continues under a different contract than the one its durable log records. Nothing +errors. This falsifies the *crash-safe resumable workflow* claim independently of `CR-10`/`CR-11`: even with a +perfect log and a single owner, the resumed run can be a different run. + +**Fix.** + +- Persist a canonical, resolved input **snapshot or digest** at admission (after `CR-15` resolves defaults, so + the digest covers what the run actually used). +- Persist `executionMode` and the execution-plan identity; verify or restore them on resume rather than trusting + the caller. +- Verify the workflow by content/plan digest, not only slug/id. +- **Do not persist secret-typed values.** A `secret` input is carried as a key reference plus version, with a + defined re-supply contract on resume; a mismatch is a typed error, never a silent substitution. +- A divergence produces a typed error, never a silent continue. + +**Acceptance.** Resuming with a changed input, a changed `executionMode`, or a changed plan each fails with a +distinct typed error. Resuming with a `secret` input re-supplied at the same version succeeds; at a different +version it fails. The digest of a run started with defaults omitted equals the digest of the same run started +with those defaults written out explicitly. No secret value appears in any persisted row — asserted by scanning +the written rows. + +--- + +## W2 — Liveness and deadlines + +### CR-20 — Agent-node `timeout_ms` is completely inert · High +The node schema accepts the field, but the runner passes only the run-level signal to the agent turn; no timer, +controller or deadline consumes `node.timeout_ms`. An authored liveness bound is silently ignored. +**Fix + acceptance.** Honour it as a real deadline; a node exceeding it fails with a classified timeout, proven +with a fake clock. + +### CR-21 — No Relavium-owned deadline for a normal provider attempt · Medium-High +The chain awaits generate/stream with the caller's signal only; unlike list-models and key validation, it sets +no per-attempt timeout. Without a node or run timeout, the vendor SDK default becomes the product's liveness +semantics — which our security standard forbids for outbound requests. +**Fix + acceptance.** An injected controller/timer factory; merge caller-abort with deadline-abort. A timeout is +`kind: 'timeout'`, a user cancel stays `cancelled`. A pre-content timeout may fail over; a content-committed one +must not. Prove timer cleanup with a fake clock. This is a different timer from `CR-20` — both are needed, and +the pre/post-content split must agree with `CR-14`'s rule 7, so it lands on that ADR. + +### CR-22 — Gate and run deadlines are not preserved across resume · High +The checkpoint's pending gate carries gate/node/budget data but not an absolute deadline, and the whole-run +timeout is re-armed for its full duration on every resume — so a crash extends the cap. +**Fix + acceptance.** Persist absolute deadlines; resume computes the remaining time. A run crashed and resumed +repeatedly still times out at its original absolute deadline. + +### CR-23 — Exactly-one-terminal is a safety property, not a liveness one · High · **decision open** +Cancel only fires the abort signal, and the terminal waits for the running-node count to reach zero. The node +executor seam accepts an arbitrary promise; honouring the signal is not guaranteed at the type level. An +executor that ignores abort and never settles leaves the run without a terminal forever. +**Open decision.** The grace-period length and whether a quarantined executor is disabled process-wide or per +run. Settle before starting. +**Fix + acceptance.** A bounded grace period after cancel/timeout, then a generation token that fences the run +state from late outcomes, plus executor quarantine. Tests: a never-settling executor and a late-success executor +both produce exactly one terminal in bounded time, and the late output is not applied. + +--- + +## W3 — Resource governance and bounds + +### CR-30 — The accepted no-drop bounded stream is not implemented · High +`push()` appends without a capacity check; `whenDrained` is advisory and only awaited at node boundaries, while +token deltas are emitted synchronously. A single provider stream can grow memory without a hard bound. + +**This is not an open choice.** [ADR-0036](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md) +already decided it: *"buffering is bounded per consumer with a producer-await (no-drop) policy"*, because the +stream must stay gap-free — a drop would force a `sequenceNumber` resync, and reconnect/resync plus event-sourced +resume are both built on that. Implement the accepted decision. A drop policy would need an ADR that supersedes +ADR-0036, and nothing found here argues for one. +**Fix + acceptance.** Real producer-await backpressure with a hard per-consumer ceiling. A test drives a fast +producer against a slow consumer and asserts the buffer never exceeds the ceiling **and** that no sequence +number is skipped. + +### CR-31 — No safe default concurrency and no absolute graph/retry caps · High · **cap values open** +An omitted `max_parallel_nodes` means `Infinity`. There is no absolute cap on authored nodes, edges, fan-out +width, fallback-chain length, retry counts, parallel tools or total attempts; the parser's text-size limit does +not stop many small ones. +**Fix + acceptance.** A small capacity-derived default concurrency that an authored value may only narrow, plus +explicit absolute caps enforced at parse/compile admission with typed errors. The numbers are a maintainer call. + +### CR-32 — Workflow output, state and durable event size are unbounded · High · **bound values open** +**Fix + acceptance.** Bound them at the durable boundary with a typed rejection, the same way tool output is +already bounded. The numbers are a maintainer call. + +### CR-33 — Finished runs are retained forever in memory · Medium-High · **policy shape open** +`WorkflowEngine` keeps completed runs indefinitely. +**Fix + acceptance.** A retention policy with an explicit bound; a long-lived process running many workflows +does not grow without limit. Whether retention is count-, age- or memory-based is a maintainer call. + +--- + +## W4 — MCP hostile boundary + +Executes Wave 2's MCP queue (see [Phase boundary](#phase-boundary)). One security sitting covers this group +plus `CR-16`. + +### CR-40 — Cancellation and deadlines never reach the MCP transport · High +The dispatch context can carry a signal, but the manager's `callTool` chain does not forward it, and neither the +connection nor the stdio SDK call accepts an abort or deadline. After a user cancels, the remote or child effect +continues and child processes can survive. + +### CR-41 — The MCP SSRF floor does not cover DNS or redirects · High +The network config checks the authored hostname and scheme, then hands the raw opener to the SDK. The code +itself documents the DNS-to-private and redirect-to-private holes. A public-looking domain can resolve to +loopback, RFC1918 or a metadata IP. +**Fix.** Apply the built-in egress mechanism — resolve-all, range-block, pinned IP — to the MCP HTTP, SSE and +WebSocket transports. + +### CR-42 — MCP discovery and result ingress are unbounded · High · **bound values open** +Page count is capped for `tools/list`, but total tool count, byte size, description and schema size are not, and +a result is fully materialized before core bounding applies. A hostile server can exhaust startup memory, inflate +prompt token cost, or bloat a workflow output. + +**Acceptance for W4.** A hostile-server harness, with adversarial cases rather than a smoke test: a server that +never answers is cancelled within its deadline and leaves **no orphan process** (asserted on the child-process +table, not on the promise); a DNS record resolving to loopback, to an RFC1918 range and to the cloud metadata +address are each refused; a redirect to any of those is refused; oversized discovery and results are rejected +with typed errors **before** they reach the prompt. + +--- + +## W5 — Media correctness + +### CR-50 — `read_media` does not work end to end · High · **decision open** +The builtin returns a base64 media part, the registry places generic output into the tool result, and the LLM +message schema explicitly forbids raw media bytes there. The canonical alternative is a handle-only attachment, +but all three adapters deliberately drop that field, and production chat does not wire the media-read scope. +There is no registry → turn → chain → adapter path. +**Open decision.** Complete the handle path through the adapters, or remove the tool from the catalog until it +works. Either is acceptable; shipping an advertised tool that cannot work is not. +**Fix + acceptance.** An end-to-end test proves the chosen answer. If removal is chosen, a test asserts the tool +is absent from the catalog the model sees. + +### CR-51 — Model-level tool/attachment capability is not enforced · High +The catalog carries `toolCall` and `attachment` metadata, but runtime gates on the provider-wide flags and the +adapters attach tools unconditionally. A model the catalog marks as tool-incapable can still be sent tools, and +the chain can treat it as capable instead of pre-skipping it — turning a free skip into a paid 400. + +### CR-52 — Gemini reasoning/tool continuation metadata is lost · Medium-High +Part-level thought signatures on function calls are not captured, and replay drops reasoning after a tool +result. Multi-turn thinking-plus-function-calling continuations can 400 or break semantically. +**Governance note.** This is **not** newly discovered drift. [ADR-0039](../../decisions/0039-same-provider-reasoning-replay.md) +deliberately deferred it and recorded it in [deferred-tasks.md](../deferred-tasks.md). This phase pulls that +deferral forward; the item closes by landing the work **and** checking off the deferred-tasks entry, so the two +records do not disagree. + +### CR-53 — Large media travels fully buffered and base64-encoded · High +Audio and video responses are read into a full buffer and converted to base64; raw buffer, typed view, base64 +string and the store's decoded copy can all exist at once. +**Fix + acceptance.** A bounded stream or download lease from the adapter; the host writes straight to the media +store with a `Content-Length` and streamed-byte ceiling. A large-media test asserts peak memory stays bounded. + +### CR-54 — URL media can produce different bytes on resume · High +Remote URL media is resolved again on resume, and the same URL can return different content, a different +redirect or a different DNS answer. The run appears to have the same inputs while the model sees different bytes. +**Fix + acceptance.** Convert to a content-hashed handle at first resolution; resume must not re-fetch. Pairs +with `CR-17`: the input digest is only meaningful if a URL input's bytes are pinned. + +### CR-55 — Missing media rates can bypass a strict cost cap · Blocker for the strict-cap claim +Where a media rate is absent the estimator can fall to zero, the generated catalog projection produces no media +rate, and the DB's media-rate columns are not carried into the listing/overlay path. The governor can then treat +a missing rate as *priced at zero* rather than *unpriced*, admitting paid image/audio/video generation under a +strict cap and reporting `cost:updated = 0`. +**Fix + acceptance.** Missing media rate ⇒ unpriced, never zero. Under a strict cap an unpriced media generation +is refused. Carry the media rates through the overlay path. Break-verify by restoring the zero fallback. + +--- + +## W6 — Authoring correctness + +### CR-60 — `merge_strategy: first` is not implemented as specified · High · **decision open** +The canonical table says the first resolved branch wins and the others are ignored. The engine waits for every +branch to settle and the handler takes the first surviving output in declaration order. A slow loser adds +latency and cost; a failing loser can fail the run even when the winner is ready. +**Open decision.** Implement race semantics, or rename the strategy and correct the canonical table. The name +and the behaviour must agree; which one moves is a maintainer call. +**Fix + acceptance.** Whichever is chosen, a test pins latency and cost behaviour — a slow loser must not extend +the run's wall clock if race semantics are chosen, and must be documented as doing so if they are not. + +### CR-61 — Agent and transform `output_schema` do not deep-validate · Medium · **needs a dependency ADR** +An agent's `output_schema` becomes a response-format hint and is then only `JSON.parse`d; the transform node's +`output_schema` explicitly does not deep-validate. + +**The governance drift, stated precisely.** [ADR-0038](../../decisions/0038-agentrunner-llm-call-boundary.md) +required node-side `output_schema` validation to land **with 1.O in the same PR or behind a hard acceptance +gate**, "never silently deferred". What shipped is +[error-handling.md](../../standards/error-handling.md)'s narrower rule: Phase-1 scope is parse-as-JSON only, and +deep JSON-Schema conformance is a deferred follow-up **because it needs a JSON-Schema validator dependency +behind an ADR** — Zod cannot consume an arbitrary JSON Schema. So the two documents disagree about what was +required, and the narrowing was never recorded as a decision. +**Fix + acceptance.** Either land deep validation (which requires a new-dependency ADR under CLAUDE.md rule 2, +and that ADR is the gate), or record the narrowed contract as a decision that supersedes ADR-0038's clause. +Silence is not an option. A typed `validation` failure on a schema-violating but valid-JSON output is the test +either way — passing if validation lands, documented-as-absent if the narrowing is chosen. + +### CR-62 — Expression dependencies are invisible, so silent mis-routing is possible · High · **decision open** +The DAG derives edges from template references only and deliberately does not order JS-expression reads of run +outputs. A condition that runs before its producer compares `undefined` and silently takes the false branch — a +test currently pins that silence. +**Open decision.** Extract dependencies from expressions, or fail loudly when an expression reads an unresolved +output. Not a silent false, either way. +**Fix + acceptance.** The pinning test is rewritten to the chosen rule and keeps a note recording the reasoning +for the behaviour it replaces. + +### CR-63 — Agent `input_schema` runtime enforcement: verify the docs, do not implement · Low *(plan review)* +Split out of `CR-61` because it is a different contract with a different answer. +[agent-yaml-spec.md](../../reference/contracts/agent-yaml-spec.md) states that `input_schema` is *"purely +additive metadata — it drives type-safe node chaining and editor (VS Code) completion; it does not change +run-time execution"*. Under that spec, the absence of runtime enforcement is **correct**, not a defect. +**Fix + acceptance.** No runtime change. Grep every canonical document and the product surface for a claim that +an agent's `input_schema` is validated at run time; if one exists, correct it to match the spec. If none exists, +close the item with that finding recorded — do not implement enforcement to satisfy a claim nobody made. + +### CR-64 — Node-`tools` narrowing is enforced at RUN time, and two ADRs say "parser-enforced" · Medium + +**Evidence.** `resolveGrant` in `packages/core/src/engine/agent-runner.ts` is where a node's `tools:` is +checked against the agent's grant — the node executor, reached only after the run has started. +[ADR-0038](../../decisions/0038-agentrunner-llm-call-boundary.md) states the opposite in parentheses: +*"`tools:` **narrows** the agent's grant and never widens ([ADR-0029], parser-enforced — a node listing a tool +the agent lacks fails validation)"*. Nothing in `parser.ts`, `dag.ts` or `run-plan.ts` touches `tools` at all. + +**Why it matters.** The security boundary itself HOLDS — a widening node is refused at dispatch, so no +ungranted tool ever runs; ADR-0029(b) is not violated. What is wrong is WHERE and WHEN. A workflow that widens +passes `relavium validate`, looks correct in review, and fails partway through a run — after upstream nodes +have already spent real money. And a canonical ADR tells the next reader the parser caught it, which is how +someone later "simplifies" the runtime check as redundant. + +**Fix.** Enforce it at parse for an INLINE agent, where both sides are present in one document, and at `$ref` +resolution for an external one — the point at which the agent registry has been read. The runtime check stays +as the last line of defence (a host can construct a node executor directly), but it stops being the first. + +**Acceptance.** A workflow whose node lists a tool its inline agent lacks fails `relavium validate` with a +typed error naming the node, the tool and the agent — before any run id exists. The same for a `$ref`'d agent +once resolved. ADR-0038's parenthetical is corrected in the same change, or the enforcement moves to match it; +the two must agree. + +--- + +## W7 — Agent product correctness + +### CR-70 — Cross-turn tool-call memory does not exist · High (product) · **decision open** +Only the final assistant text enters the cross-turn transcript; within-turn tool call/result pairs are dropped. A +coding agent cannot remember a file it read in the previous turn and calls the tool again. +**Open decision.** Persist the tool pairs (bounded), or narrow the canonical session spec. The bound is part of +the decision. + +### CR-71 — Session transcript and export lose tool history · High +The persister stores the same text-only model, so the exporter — written to derive tool names from assistant +tool-call parts — has nothing to read. The canonical session spec promises a full transcript with a tool union. +**Fix + acceptance for CR-70/71.** One decision, one PR. A graduated workflow must represent the flow that +actually happened, or the spec must stop promising it. + +### CR-72 — Authored agent `memory` policy is inert · Medium · **decision open** +`none | window | summary` is accepted by the schema and consumed by nothing. +**Fix + acceptance.** Implement it or reject it at parse time with a typed error. A schema that accepts a field +nothing reads is a claim the code does not keep. + +### CR-73 — `invoke_agent` has no production delegate · High (2.6 go/no-go) +The builtin exists in the catalog and errors with tool-unavailable when no delegate is wired; nothing wires one +in production, and the availability filter keeps the delegate-backed tool in the catalog. +**Fix + acceptance.** Wire the delegate, or stop advertising the tool to the model when no delegate exists. The +cheap option is always available, which is why this is non-deferrable. + +--- + +## W8 — Provider and conformance + +### CR-80 — An invalid custom base URL fails OPEN to the official endpoint · High (security/compliance) +When the custom provider factory rejects a private, malformed or credential-bearing URL, the error is caught and +the default adapter is left standing — and a test pins that fail-open as correct. A user expecting an internal +gateway silently sends prompts and keys to the official API after a config drift. +**Fix + acceptance.** Fail closed with an explicit message. The error must name the rejected URL's *shape* +without echoing an embedded credential — asserted directly. Rewrite the test that pins the old behaviour, +keeping a note with the reasoning it replaces. + +### CR-81 — Conformance cassettes do not verify the request wire contract · High +A recorded response carries status, content type and body only; replay checks that the request body is parseable +JSON and matches nothing else — not method, path, headers, or a canonical body. A dropped `response_format`, a +lost tool result on continuation, or a broken system-instruction mapper leaves the suite green. +**Fix + acceptance.** Match the canonical request. Mutating the request lowering must redden the suite — proven +by actually applying at least three distinct mutations, not by assertion count. + +### CR-82 — Missing or partial usage can be read as zero · High +Carried from our own backlog and re-confirmed by the reviews. +**Fix + acceptance.** Missing usage is *unknown*, never zero, on every path that reaches a cost total. + +--- + +## W9 — Harness honesty and isolation + +`CR-90` and `CR-91` execute **first** (see [Execution order](#execution-order)); `CR-92` executes inside the +durability spine. They are grouped here by theme only. + +### CR-90 — Root coverage collects in-repo worktrees · Medium · ✅ CLOSED 2026-08-10 +The root Vitest include patterns have no exclude for repo-local checkout directories, so a stale worktree adds +test files and 0%-covered sources to the run. In one working tree this produced 471 discovered test files and 23 +suite failures; excluding the worktree path produced 237 files and a clean run. +**Fix + acceptance.** Exclude repo-local checkout areas from both `test.exclude` and `coverage.exclude`. The +self-test uses a **synthetic repo-local checkout fixture** — a directory tree containing a test file and a +source file — not a real `git worktree`; a real worktree makes the test depend on git state and the working +directory, which is exactly the non-determinism this item is about. + +**Closed by `5f69ddc` + its review fold.** `REPO_LOCAL_CHECKOUTS` in `vitest.config.ts` (spread into both +excludes), the guard at `tools/test-isolation/check.mjs`, wired into `pnpm run ci` and `.github/workflows/ci.yml`, +plus the tracked `.gitignore` entries the local `.git/info/exclude` was standing in for. + +Three corrections to the record above, all measured rather than recalled: + +- **Re-measured 2026-08-10: 472 → 238**, not 471 → 237; the tree grew one test file. The foreign 234 were a + full agent-tooling checkout under `.claude/worktrees//`. +- **It did NOT threaten the coverage floor**, which the original evidence implied and an earlier version of + this fix asserted. Vitest matches threshold globs ROOT-RELATIVE with no implicit leading `**`, so the foreign + sources match none of the three per-package globs and land in the unset `global` group. The counterfactual + exits 0 at 48.85% lines. What the leak destroys is the reported number and the lcov/html artifact — and it + becomes floor-load-bearing the moment a global threshold is added. +- **CI never saw any of it.** `.claude/worktrees/` was hidden by a LOCAL `.git/info/exclude` entry and CI checks + out clean, so both excludes are no-ops there. The exclusions defend a developer's tree; the guard is the part + that defends CI, by failing if they are ever weakened. + +The acceptance criterion above names a fixture, and a fixture alone is not sufficient — recorded because the +first attempt shipped exactly that gap. Deriving fixtures from the exclusion list proves every LISTED location +is excluded, and cannot prove the list is COMPLETE: removing the one entry that mattered also removes its +probe, and the guard reports green while re-collecting all 234 foreign suites. The guard's primary assertion is +therefore structural and consults no list — any collected file whose ancestor carries its own +`pnpm-workspace.yaml` is in a second checkout. + +**One recorded limitation, not a property.** "A second checkout always carries its own `pnpm-workspace.yaml`" +holds for `git worktree add`, `git clone` and a full copy — every case seen here — but not for a sparse +checkout, a `--no-checkout` worktree, or a partial rsync of `packages/**`. Such a tree would be collected with +nothing in its ancestry to detect, and the primary assertion would pass in silence, leaving only the +list-based checks. Nothing in this repo produces one today; if that changes, the predicate needs a second +marker. Stated here and in the guard's own docblock because the first version asserted it as an absolute. + +### CR-91 — The workflow E2E harness is not a crash or durability oracle · Medium · ✅ CLOSED 2026-08-10 +It pins that the live stream reported success; it does not prove durable truth after a restart. +**Fix + acceptance.** An oracle that asserts live result, DB history, resume and reconcile agree on the same +terminal type and payload. It lands before the spine because a spine asserted with the harness the reviews +already found insufficient is not asserted. *(The original wording here called it "the instrument `CR-10`, +`CR-11`, `CR-12` and `CR-92` are proven with". That was written before the oracle existed and it is not true +of `CR-11` or `CR-12` — see the table below.)* + +**Closed by `packages/core/src/engine/durable-truth.ts`** (exported from the package index — the surfaces that +most need it, notably `apps/cli`'s harness against the real `history.db`, are outside this package). +`checkDurableTruth` compares the live terminal, the durable history, the history a FRESH engine leaves after +`reconcile()`, the checkpoint status, and the log's own order, returning a structured verdict rather than +throwing so callers can assert on the specific disagreement and the oracle can have its own tests. + +**What it instruments, corrected — the first version of this note claimed all four spine items and that was +false:** + +| item | expressible | why | +|---|---|---| +| `CR-92` | yes | terminal agreement across live / history / restart is exactly what it compares | +| `CR-10` | **partly** | see the limitation below | +| `CR-11` | **no** | it has no concept of run ownership or a fencing token | +| `CR-12` | **no** | external effects need an effect-journal view and a side-effect counter it does not model | + +`CR-11` and `CR-12` need their own predicates. Budget for that rather than assuming this covers them. + +**`CR-10`'s property is only half-checkable from the log, and finding that out cost a wrong assertion.** The +durable log's sequence numbers are a strictly increasing SUBSEQUENCE of the run's, never `0..n-1` — streamed +events (`agent:token`, `cost:updated`, …) take numbers and are deliberately never persisted. A real completed +run reads `[0,1,2,3,5,10,11,12,13,14]`, and asserting `0..n-1` failed a perfectly healthy engine. So "no +persisted event is missing from the middle" is **not** expressible from the log alone: a streamed event's +absence is indistinguishable from a lost one. **`CR-10`'s acceptance needs a store harness that records which +events it was ASKED to persist.** What the log proves unaided is that it starts at seq 0 and only moves +forward — which does catch out-of-order commit, the thing `#emitDurable` currently permits. + +Three design points worth carrying into the spine work: + +- **Envelope fields are excluded from the comparison on purpose.** `timestamp` and `sequenceNumber` move on + every restart; comparing them would make every run look like a disagreement while catching nothing. +- **`expect: 'repaired'` exists because a CORRECT crash repair was a false failure.** A run that died without + a terminal and was reconciled on restart verdicted `a restart CHANGED the terminal: before=none + after=run:failed` — so no crash test could use the oracle at all, which is most of what it is for. +- **The resume view the acceptance criterion names is still not the checkpoint fold.** A real resume goes + through the host's `Checkpointer`, and the CLI's uses ADR-0075's STRICT read that REFUSES a log with an + uninterpretable row; a local fold reads it happily. `loadCheckpoint` takes the real port, and the verdict + reports `checkpointSource` so a reader can tell which was used. Supplying it is `CR-92`'s job. + +Twenty-seven unit tests cover the detection logic (the headline live-`completed`/history-`failed` case, a +same-type terminal whose payload differs, each compared field pinned independently, a cross-run log, +out-of-order and headless logs, a correct repair, key-order / circular / shared-reference payloads), and three +e2e tests apply the oracle to real engine runs on all three terminals. Break-verified: stopping the engine +from persisting terminals reddens all three e2e tests. + +**Three things `CR-92` must still add before it can certify itself with this** — measured, not guessed: + +1. `DurableTruthInput.live` is `RunEvent | undefined`, so it structurally cannot carry "a distinct typed + result that is not a `RunEvent`" — which is exactly what `CR-92`'s acceptance says the API must return when + durability is uncertain. That input has to widen before the test can express its own scenario. +2. Even with the real `loadCheckpoint` wired, the resume leg can only compare `RunStatus` — a four-value enum. + `CheckpointState` carries no outputs, error code, `correlationId` or tokens, so `CR-92`'s "agree on the same + terminal type **and payload**" is only checkable for the type half on that leg. +3. `expect` has `'settled' | 'repaired'` and no mode for "durability uncertain, outbox retry pending", which + is `CR-92`'s own state and fits neither. + +### CR-92 — Terminal persistence failure lets live and durable truth diverge · High · in the durability spine +The engine can complete the delivery chain for a terminal event even when its persistence failed, and +reconciliation may later produce a different terminal than the original. Media reclaim can run before the +terminal is durable. A caller can receive a success result and outputs while history shows the run failed. +**Fix + acceptance.** Hold the intended terminal payload in a durable outbox and retry it under the same +identity. If durability is uncertain the API must not say `completed` — it returns a distinct typed result. +Terminal asset cleanup and handle resolution happen only after the terminal is durable. The test proves live, +history, resume and reconcile agree. + +### CR-93 — Process-global catalog and parameter-learning state is not tenant-safe · Medium now, High for cloud +Process-global mutable state is fine for a single local user and wrong for the multi-tenant cloud surface. +**Fix + acceptance.** Scope it per run/session/tenant before any multi-tenant surface ships. **The decision is +recorded in this phase even if the work is deferred** — a deferral here moves to +[deferred-tasks.md](../deferred-tasks.md) with its trigger (the first multi-tenant surface) named. + +### CR-94 — One budget approval opens every redispatch without a numeric limit · High +An approved vertex skips pre-egress admission on redispatch, and one agent turn can make many tool rounds plus a +final generation. The user approves a projection they saw, but the approval means "this vertex is exempt", not +"up to this amount". +**Fix + acceptance.** Make the approval an immutable numeric lease bound to money/token/attempt scope, model and +expiry, consumed atomically by each egress. An approved node that exceeds its lease pauses again. + +### CR-95 — A mid-tool-loop budget pause replays the whole loop · High (Blocker while CR-12 is open) +A budget pause becomes a paused outcome; on approval the node is reset to pending and dispatched from the start. +The code already acknowledges that this repeats earlier provider and tool calls. +**Fix + acceptance.** **Short term (non-deferrable): forbid a mid-loop budget pause — fail closed.** Long term: +checkpoint the continuation — provider messages, tool call/result pairs, round index — and resume from that +point. A mid-loop pause plus approval must not repeat a mutation. The short-term fix closes the claim; the long +term may be deferred with its trigger named. + +--- + +## Exit criteria + +This phase is done when **all** of the following hold: + +1. **Every non-deferrable item is closed with a break-verified test.** Non-deferrable means the register above + says so: `CR-01`–`CR-03`, `CR-10`–`CR-17`, `CR-50`, `CR-55`, `CR-73`, `CR-80`, `CR-92`, and `CR-95`'s + short-term fail-closed fix. **None of these may be deferred**, for one reason: each is either a security or + correctness boundary, or a shipped claim the code does not keep — and every one of them has a cheap + fail-closed option (refuse, remove, narrow the claim) available when the full fix is too large. "Too big to + fix now" is an argument for the cheap option, never for deferral. +2. **Every other item is closed, or deferred with a full record.** A deferral is written in + [deferred-tasks.md](../deferred-tasks.md) with its severity, its trigger, and — this is the part that is + usually skipped — **the product claim it narrows**. A deferred item's behaviour must not remain in any + canonical document as a shipped guarantee. +3. **Each `W1` item has an accepted ADR, and the ADRs land in dependency order.** Eight ADRs (or fewer, where + the register above pairs two items onto one), all before their implementation. +4. **Every canonical document that promised behaviour listed here says what the code now does** — in particular + the claims about crash-safe resume, duplicate-free effects, bounded streams, first-branch-wins, + schema-validated input/output, and session-to-workflow graduation. This explicitly includes + [architectural-principles.md](../../standards/architectural-principles.md) §11's idempotency sentence. +5. **Four security-review sittings are recorded**, each with a checklist entry and its adversarial test: + prompt/trust provenance, hostile MCP, media bytes, provider/config trust. +6. **`pnpm run ci` exits 0 and `pnpm coverage` exits 0**, both checked by exit code. `pnpm run ci` alone is not + the merge gate — `coverage` is a separate required check, and this phase edits `packages/core` heavily. +7. **A closing register in this file states, per item, the code that closes it** — verified by reading the code, + not by trusting the mark. Wave 1's completion claim was wrong twice before this discipline was adopted. + +## What a later architecture review contributed — and what it did not + +Two design reviews of the YAML definition layer and of the git-native posture were read against the tree on +2026-08-11. **Exactly one item from them belongs in this phase** (`CR-64`), and that outcome is worth recording +rather than quietly discarding the rest, because the same documents will be read again. + +**Two of the reviews' own high-priority findings are NOT REAL, verified against the schema:** + +- **"Circular `$ref` has no guard."** It cannot happen. `AgentSchema` is `.strict()` and declares neither + `agents:` nor `$ref`, so an `.agent.yaml` cannot reference another agent — the resolution depth is exactly + one hop, workflow → agent. A `$ref` inside an agent file is rejected by its own parse. +- **"A workflow at `schema_version: '1.0'` could `$ref` an agent at `'0.9'`."** It cannot. The field is + `z.literal(SCHEMA_VERSION)`, not a free string, so any file carrying another version fails its own parse + before cross-file skew is reachable. + +**The rest is real but belongs elsewhere**, and putting it here would dilute a phase whose whole scope line is +"no new product surface, only the invariants an existing surface already claims": + +| finding | where it belongs | +|---|---| +| Zod → JSON Schema emit for IDE autocomplete/validation | tooling, Phase 2.6 or later — the reviews' own strongest UX point | +| `expression_type: 'js'` mandatory with no alternative | authoring ergonomics | +| Edge `condition` vs `condition` node — two mechanisms | authoring; the DANGEROUS half is already `CR-62` | +| `parallel_of` duplicating edges | authoring ergonomics; adjacent to `CR-60` | +| "did you mean" on Zod errors, property-based schema tests | DX / test tooling | +| `merge_strategy` widening (`zip`/`union`/`intersect`) | a feature, not a repair | +| Session export: linear-only, one node per turn | ADR-0026 scoped this deliberately; Phase 2.6 | +| `reasoning_effort` unvalidated against provider | no false claim — ADR-0066 has adapters withhold it; a nicety | +| Auto-commit on save, `.relavium/` discoverability, merge-conflict culture | product/desktop, Phase 3 | + +Also confirmed by those reviews and left alone: the QuickJS sandbox as a single point of failure is already +under "What must NOT change", and MCP command injection is already `CR-16`. + +## Scope note + +One item already in flight is NOT part of this phase and keeps its own tracking: the transaction-handle cleanup +across the remaining database stores. It is listed here only so nobody schedules it twice. + +The realized-cost ledger is **no longer scoped out** — it is this phase's [prerequisite](#prerequisite), for the +file-overlap reason recorded there. diff --git a/docs/standards/security-review.md b/docs/standards/security-review.md index d7902596..f3886de7 100644 --- a/docs/standards/security-review.md +++ b/docs/standards/security-review.md @@ -389,6 +389,17 @@ security invariants** a review must confirm are: [tool-registry.md §error taxonomy](../reference/shared-core/tool-registry.md#error-taxonomy) — not restated here. +**Machine output has its own arm of that floor, and it is the opposite choice.** Every `--json` record is +serialized with `stringifyJsonLine` (`apps/cli/src/render/sanitize.ts`), never a bare `JSON.stringify`: +`JSON.stringify` escapes `ESC` and stops, leaving `DEL`, the whole C1 block — including `U+009B`, a working +escape-sequence introducer on real terminals — and the Trojan-Source bidi family RAW in content the model, a +tool, or an imported artifact controls. It **escapes** where the human-display path **strips**, because +`--json` is a machine contract: escaping changes the JSON *text* but `JSON.parse` returns the identical value, +so a consumer loses nothing — while stripping would silently hand it different data. The rule is enforced by an ESLint +`no-restricted-syntax` selector rather than by review, because the gap reopened twice when it was not +(`#W15-10`, then `CR-03`). `apps/cli/src/process/render-error.ts` is the single allowlisted exception: it +pre-sanitizes with the stripping sanitizer, so the `--json` ERROR envelope is deliberately lossy. + ## Never hand-roll crypto - We **never implement cryptography, TLS, or keychain primitives ourselves**. We use vetted diff --git a/docs/standards/testing.md b/docs/standards/testing.md index f1daf09b..42b06364 100644 --- a/docs/standards/testing.md +++ b/docs/standards/testing.md @@ -135,3 +135,12 @@ journeys, not exhaustive logic — exhaustive logic belongs in engine unit tests PRs must pass: typecheck, lint ([code-style-typescript.md](code-style-typescript.md)), all unit tests, and the fixture-mode conformance suite. The live conformance suite and the desktop e2e suite run nightly and on release branches. A red required check blocks merge. + +**Test isolation is part of that gate** (`pnpm lint:test-isolation`, +[tools/test-isolation/check.mjs](../../tools/test-isolation/check.mjs)). A root run must +collect this repo's tests and nothing else. A second checkout of the repo inside the working tree — an agent +worktree, a scratch clone — is otherwise collected in full: measured once at 234 foreign test files out of 472, +with its sources in the coverage denominator. The guard fails on any collected file whose ancestor carries its +own `pnpm-workspace.yaml`, so it stays true even when the exclusion list in +[vitest.config.ts](../../vitest.config.ts) is the thing that is wrong. It also asserts every workspace still yields tests, because the cheapest way to pass an +isolation check is an over-broad exclude that goes green by not running. diff --git a/eslint.config.mjs b/eslint.config.mjs index 13fd9c7c..99b97348 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -60,6 +60,44 @@ const seamSyntaxRules = /** @type {const} */ ([ }, ]); +/** + * The machine-output fence (`CR-03`) — every `--json` record leaves through `stringifyJsonLine`. + * + * `JSON.stringify` escapes `ESC` and stops there: `U+007F` (DEL), the whole C1 block including `U+009B` + * (the 8-bit CSI, a working escape sequence introducer on real terminals), and the Trojan-Source bidi family + * all survive into the emitted line. A `--json` stream carries model output, tool results and imported-artifact + * fields straight to a terminal, so a bare stringify is a terminal-injection surface. + * + * **A lint rule rather than a test, because the propagation gap kept reopening.** `#W15-10` fixed two call + * sites; `CR-03` found three more; a source-scanning regex written for `CR-03` then missed a sixth purely + * because prettier had wrapped the argument onto its own line. A guard that only checks the places someone + * remembered catches nothing new by construction. This selector fires on the SHAPE — a `JSON.stringify` call + * anywhere inside a `writeOut`/`writeErr` argument — so a new surface is caught the first time it is written. + * + * `stringifyJsonLine` ESCAPES rather than strips, deliberately: `--json` is a machine contract and must + * reproduce its data, so `JSON.parse` round-trips to the identical string. That is the opposite choice from + * the human-display floor (`stripTerminalControls`), and `process/render-error.ts` is the one deliberate + * exception — it pre-sanitizes its fields with the STRIPPING sanitizer and is therefore lossy on purpose. + * + * **Three residual blind spots, named so the guarantee is not over-trusted.** The selector is syntactic, so it + * sees a `JSON.stringify` anywhere inside a `writeOut`/`writeErr` argument — through a template literal, a + * ternary, an object literal, a callback — but NOT: (1) a destructured writer, `const { writeOut } = io`, + * whose callee is an `Identifier` rather than a `MemberExpression`; (2) indirection through a helper that + * stringifies and returns, which no syntactic rule can follow; (3) a writer under a different name. None of + * the three exists in the tree today, and each is caught by review the same way the pre-rule call sites were — + * but "caught the first time it is written" is true of the inline shape only. + */ +const JSON_LINE_MESSAGE = + 'A --json record must be serialized with stringifyJsonLine (apps/cli/src/render/sanitize.ts), not a bare ' + + 'JSON.stringify: JSON.stringify leaves DEL, the C1 block (incl. the 8-bit CSI) and the Trojan-Source bidi ' + + 'family RAW in content the model, a tool, or an imported artifact controls (CR-03). The escape is lossless, ' + + 'so no machine contract changes.'; +const jsonLineSyntaxRule = { + selector: + "CallExpression[callee.property.name=/^write(Out|Err)$/] CallExpression[callee.object.name='JSON'][callee.property.name='stringify']", + message: JSON_LINE_MESSAGE, +}; + /** * The pure-engine purity fence (CLAUDE.md rule 5). `packages/core` SHIPPING source must import no * platform-specific module and reach no Node global, so the engine runs identically in Node, the Tauri @@ -205,6 +243,21 @@ export default tseslint.config( 'no-restricted-syntax': seamSyntaxRules, }, }, + { + // The machine-output fence (CR-03). Scoped to the CLI, which is the only surface emitting NDJSON to a + // terminal today; `render-error.ts` is the documented exception (it pre-strips, and is lossy on purpose). + // `.tsx` too: the TUI is where a debug/export affordance is most likely to grow one, and a `.ts`-only + // glob would leave nineteen files outside the fence. + files: ['apps/cli/src/**/*.{ts,tsx}'], + // Only `render-error.ts` is load-bearing — verified: removing it produces a real diagnostic. It is the + // documented exception (it pre-strips, so the error envelope is lossy on purpose). `sanitize.ts` is + // listed defensively: it DEFINES `stringifyJsonLine` and today has no writer call of its own, so the + // entry is currently inert rather than wrong. + ignores: ['apps/cli/src/process/render-error.ts', 'apps/cli/src/render/sanitize.ts'], + rules: { + 'no-restricted-syntax': [...seamSyntaxRules, jsonLineSyntaxRule], + }, + }, { // Plain JS (and any stray non-ignored JS): never attach type-aware rules — they // require a TS program and would otherwise crash ESLint on a file with no project. diff --git a/package.json b/package.json index dd881b40..370fda27 100644 --- a/package.json +++ b/package.json @@ -18,7 +18,8 @@ "test": "turbo run test", "coverage": "vitest run --coverage", "coverage:enforced": "node tools/coverage-gate/run.mjs", - "ci": "turbo run lint typecheck test && pnpm db:sync-check && pnpm typecheck:tools && pnpm lint:tools && turbo run build format:check && pnpm lint:fence-check && pnpm lint:engine-deps && pnpm lint:bundle-closure && pnpm smoke:cli", + "ci": "pnpm lint:test-isolation && turbo run lint typecheck test && pnpm db:sync-check && pnpm typecheck:tools && pnpm lint:tools && turbo run build format:check && pnpm lint:fence-check && pnpm lint:engine-deps && pnpm lint:bundle-closure && pnpm smoke:cli", + "lint:test-isolation": "node tools/test-isolation/check.mjs", "lint:fence-check": "node tools/lint-fixtures/assert-fence.mjs", "lint:engine-deps": "node tools/engine-deps/check.mjs", "lint:bundle-closure": "node tools/bundle-closure/check.mjs", diff --git a/packages/core/src/engine/agent-runner.ts b/packages/core/src/engine/agent-runner.ts index 99a3e992..2fc2be8c 100644 --- a/packages/core/src/engine/agent-runner.ts +++ b/packages/core/src/engine/agent-runner.ts @@ -396,6 +396,10 @@ async function executeAgent( dispatchContext, limits: deps.limits ?? DEFAULT_AGENT_TURN_LIMITS, ...(preEgress === undefined ? {} : { preEgress }), + // Straight from the ctx, with no `deps` fallback — the ledger belongs to a RUN and only the run loop can + // supply it. A host wiring a runner directly gets no ledger, which is correct: there is no run to + // record against (ADR-0076 / ADR-0077). + ...(ctx.money === undefined ? {} : { money: ctx.money }), ...(deps.resolvePrice === undefined ? {} : { resolvePrice: deps.resolvePrice }), // user-pricing overlay (S10) // Media cost governance (1.AF/D17): forward the node's requested output modalities + a per-modality // unit estimate so the budget governor prices a media-output turn pre-egress. Both omitted for a diff --git a/packages/core/src/engine/agent-session.test.ts b/packages/core/src/engine/agent-session.test.ts index cee01442..163b2d54 100644 --- a/packages/core/src/engine/agent-session.test.ts +++ b/packages/core/src/engine/agent-session.test.ts @@ -366,6 +366,47 @@ describe('AgentSession (1.V) — multi-turn entry point over the shared turn cor expect(completed.tokensUsed).toEqual({ input: 4, output: 2 }); }); + it('a turn whose durability flush REJECTS still consumes its slot against the cap (CR-02)', async () => { + // `CR-02`'s decision, pinned. The counter is incremented BEFORE `flushBudgetCommitments` is awaited, and + // that ordering is deliberate: the rule both catch paths apply is "count a turn only when a provider + // ENGAGED", and by the increment `#runTurn` has resolved. A flush rejection past that point is a + // durability failure, not evidence the turn never happened — moving the increment below the flush would + // hand back a turn the provider billed. + // + // Asserted through the CAP, not through `#turnCount`. It is a JS private field, and giving it a + // production accessor to satisfy a test would weaken the encapsulation to observe it. + const { deps, events } = harness([textTurn('billed, then unrecordable')], { + maxTurns: 1, + flushBudgetCommitments: () => Promise.reject(new Error('disk full')), + }); + const s = session(deps); + s.start(); + // Turn 1 engages a provider and then fails on the flush. The rejection surfaces (ADR-0074 §2: a + // durability failure fails the ACTIVE owner loudly) — and the turn still counts. + await expect(s.sendMessage('first')).rejects.toThrow('disk full'); + + // The failed turn's terminal reports the REAL usage, not a hardcoded zero (`CR-02`). Consuming the cap + // slot while dropping the tokens is the mirror of the error the counter decision refuses to make: the + // provider engaged and billed either way. + const failedTurn = events.find( + (e) => e.type === 'session:turn_completed' && e.error?.code === 'internal', + ); + expect(failedTurn, `saw: ${JSON.stringify(events.map((e) => e.type))}`).toBeDefined(); + expect( + failedTurn?.type === 'session:turn_completed' ? failedTurn.tokensUsed : undefined, + ).not.toEqual({ input: 0, output: 0 }); + + events.length = 0; + // Resolves — the cap refusal is a settled terminal, not a throw. Asserted so the mutation that moves the + // increment below the flush dies on the CAP assertion below rather than on a stray 'disk full' rejection. + await expect(s.sendMessage('second — must be over the cap')).resolves.toBeUndefined(); + + const completed = events.find((e) => e.type === 'session:turn_completed'); + expect(completed?.type === 'session:turn_completed' ? completed.error?.code : undefined).toBe( + 'turn_limit', + ); + }); + it('emits session:turn_completed{turn_limit} — loudly, no egress — when driven past the hard cap', async () => { // MANDATORY regression: a turn-loop refactor must not be able to silently drop the cap signal. // maxTurns 1: turn 1 runs; turn 2 is blocked with turn_limit and never reaches the provider. diff --git a/packages/core/src/engine/agent-session.ts b/packages/core/src/engine/agent-session.ts index 8e0d5c13..bf31a01a 100644 --- a/packages/core/src/engine/agent-session.ts +++ b/packages/core/src/engine/agent-session.ts @@ -422,6 +422,15 @@ export class AgentSession { * entries, a budget refusal, a pre-flight cancel) never burns a turn the model never took. */ #turnCount = 0; + + /** + * The last turn's real accumulated usage, captured on the success path just before the durability flush. + * + * Exists so an unclassified failure AFTER a provider engaged — in practice a `flushBudgetCommitments` + * rejection — can report what was actually billed instead of `{0,0}` (`CR-02`). Cleared at the top of each + * turn so a later failure can never inherit an earlier turn's numbers. + */ + #lastEngagedUsage: { input: number; output: number } | undefined; /** Session-wide running cost total, authoritatively stamped onto every `cost:updated`. */ #cumulativeCostMicrocents = 0; #status: SessionStatus = 'created'; @@ -612,6 +621,7 @@ export class AgentSession { // only on the NEXT turn, so the advertise-filter + approval regime stay consistent within this turn. const turnPolicy = this.#turnPolicy; try { + this.#lastEngagedUsage = undefined; // never inherit an earlier turn's numbers const result = await this.#runTurn(abort.signal, turnPolicy); // A cancel landed mid-turn — the cancel path owns the terminal session:cancelled; stay quiet, but // roll the user message back so a cancelled turn leaves no dangling user turn in the transcript @@ -625,6 +635,12 @@ export class AgentSession { // (the reply is kept, the turn is counted). `abort()` interrupts an IN-FLIGHT turn only; a turn the // model already finished is not discarded. This success path **never reads `#abortingTurn`** — that is // precisely what makes a late abort structurally invisible here; the `finally` still clears the marker. + // Counted HERE, BEFORE the durability flush below, and that ordering is a decision rather than an + // accident (`CR-02`). The rule both catch paths already apply is *count a turn against the hard cap + // only when a provider actually ENGAGED* — and by this line `#runTurn` has resolved, so one did. A + // `flushBudgetCommitments` rejection past this point is a durability failure, not evidence the turn + // never happened; moving the increment below the flush would hand back a turn the provider billed. + // Pinned by "a turn whose durability flush REJECTS still consumes its slot against the cap". this.#turnCount += 1; // Append the assistant reply to the cross-turn transcript as TEXT-ONLY. The turn core keeps the // within-turn tool_use/tool_result pairs internal (they never leave runAgentTurn — it returns only the @@ -648,6 +664,8 @@ export class AgentSession { // and its comment relies on "nothing is pushed after the user message on a throw" — which stopped being // true when this await landed between them: a flush rejection popped the ASSISTANT message and left the // user turn dangling, the exact shape that rollback exists to prevent. Awaiting first restores it. + // Captured BEFORE the flush, so a rejection below still knows what the provider billed (`CR-02`). + this.#lastEngagedUsage = { input: result.usage.input, output: result.usage.output }; await this.#deps.flushBudgetCommitments?.(); if (result.text.length > 0) { this.#messages.push({ role: 'assistant', content: [{ type: 'text', text: result.text }] }); @@ -756,15 +774,18 @@ export class AgentSession { } // An unexpected (non-classified) error — settle the turn LOUDLY first so the stream stays balanced (every // session:turn_started gets a terminal), then re-raise so the caller still sees the bug. - this.#emitTurnCompleted( - 'error', - { input: 0, output: 0 }, - { - code: 'internal', - message: 'the session turn failed with an unexpected error', - retryable: false, - }, - ); + // + // `#lastEngagedUsage`, not a hardcoded zero (`CR-02`). The only unclassified error that reaches here in + // practice is a `flushBudgetCommitments` rejection — a turn whose provider ALREADY engaged and billed, + // which is exactly why the same decision counts it against the cap. Reporting `{0,0}` here consumed the + // cap slot and silently dropped the turn's real tokens from every total, which is the mirror of the error + // the counter decision refuses to make, and contradicts EA2/ADR-0055's rule to report real usage whenever + // a provider engaged. Falls back to zero when nothing engaged, which stays truthful. + this.#emitTurnCompleted('error', this.#lastEngagedUsage ?? { input: 0, output: 0 }, { + code: 'internal', + message: 'the session turn failed with an unexpected error', + retryable: false, + }); throw err; } diff --git a/packages/core/src/engine/agent-turn.ts b/packages/core/src/engine/agent-turn.ts index e9c78db1..30398ba0 100644 --- a/packages/core/src/engine/agent-turn.ts +++ b/packages/core/src/engine/agent-turn.ts @@ -62,6 +62,7 @@ import { CommitmentDurabilityError, type BudgetAdmission, } from './budget-governor.js'; +import { LedgerDurabilityError, type TurnMoneyPort } from './money-durability.js'; import type { NodeStreamEvent } from './node-executor.js'; /** @@ -163,6 +164,16 @@ export interface AgentTurnParams { readonly limits: AgentTurnLimits; /** Pre-egress budget hook (default no-op; 1.AC fills it). */ readonly preEgress?: PreEgressHook; + /** + * The run's money-durability port (ADR-0076 / ADR-0077) — RUN-PATH ONLY, and that is what makes + * `cost:attempt_settled` a run-only event as a runtime fact rather than a comment: `AgentSession` never sets + * this, so nothing is recorded on the session path and there is nothing for a sink to drop. (The session's + * realized spend is already recorded synchronously into `session_costs`, ADR-0070 + `#W15-4`.) + * + * Optional exactly like {@link preEgress}, and threaded the same way, because this is the boundary the + * session shares. + */ + readonly money?: TurnMoneyPort; /** * The non-text output the node requested (1.AF/D17) — forwarded to {@link PreEgressHook} so the budget * governor knows this is a media-output turn. The AgentRunner lowers it from the node's `output_modalities`. @@ -449,6 +460,14 @@ function throwMappedChainError(error: LlmError): never { if (error.cause instanceof CommitmentDurabilityError) { throw error.cause; } + // ADR-0076/ADR-0077's realized twin, for the SAME two reasons — barrier B1 joins the money chain inside + // `preAttempt`, so a ledger failure arrives here wrapped exactly like a commitment failure. Without this + // arm the class is flattened away: `isLedgerDurabilityError` stops narrowing at the engine's B3 catch, and + // the `nodeId` identifying WHOSE write broke is replaced by whichever node's turn happened to observe the + // barrier — the fan-out misattribution `CommitmentDurabilityError` got this arm to prevent. + if (error.cause instanceof LedgerDurabilityError) { + throw error.cause; + } throw new AgentTurnError(codeForLlmError(error), error.message, error.retryable); } @@ -654,6 +673,13 @@ async function dispatchToolUseTurn( false, ); } + // **Barrier B2 (ADR-0077)** — the one this ADR adds beyond ADR-0074 §2's pair, and the reason the ledger + // extends the guarantee rather than repeating it: realized spend must be durable before the run MUTATES THE + // WORLD, not merely before it spends again. Before the loop, not per call — one join per tool turn. + // + // It awaits AND observes: `join()` throws the retained failure, which propagates as a turn failure. Awaiting + // alone would not be a barrier, since `#emitDurable` absorbs a store fault and resolves. + await params.money?.join(); // A reached `tool_use` stop always followed a successful (non-skipped) attempt, so `nonSkippedAttempts >= 1`. const dispatched = await dispatchToolCalls(toolCalls, params, activeModel, nonSkippedAttempts); let next = corrections; @@ -844,6 +870,24 @@ async function driveAgentTurn( // between "unpriced" and "genuinely free" — and a free-LOOKING row in the /cost breakdown would be a lie. priced: record.cost !== undefined, }); + // ADR-0076's durable ledger row, STARTED here and joined at the next barrier (ADR-0077) — this callback + // cannot await, which is the whole reason the mechanism is a chain plus barriers rather than an inline + // await. + // + // **Strictly AFTER `params.emit` above, and the order is load-bearing.** The engine advances its run-wide + // `#cumulativeCostMicrocents` inside `#nodeEmit`'s `cost:updated` arm, and stamps that counter onto this + // draft. Recording FIRST would stamp a stale total, which `refineCostAttemptSettled` rejects at the + // producer gate — and that gate runs in `#bus.next`, OUTSIDE `#emitDurable`'s try, so the wrong order does + // not degrade quietly: it makes `#emitDurable` REJECT in the one place the design assumes it cannot. + params.money?.record({ + nodeId: params.nodeId, + model: record.model, + attemptNumber: nonSkippedAttempts, + inputTokens: record.usage.inputTokens, + outputTokens: record.usage.outputTokens, + costMicrocents: record.cost?.costMicrocents ?? 0, + priced: record.cost !== undefined, + }); }; const chainCapabilities: ChainCapabilities = @@ -873,7 +917,11 @@ async function driveAgentTurn( // maxTokens }` — `provider` is THIS attempt's routing provider (review M2) — so wrap the hook to also carry // the turn-static media estimate (1.AF/D17); otherwise the failover-attempt check would silently drop the // media addend (ADR-0044 §3). `...info` forwards `provider` to the governor's endpoint estimate unchanged. - ...(preEgress === undefined + // Installed when EITHER a budget hook or the money port is present. Gating it on `preEgress` alone — as it + // was — is one of the three barrier holes ADR-0077 §5 names: `#makePreEgressHook()` returns `undefined` + // without a governor, and `budgetApproved` drops it even WITH one, so an unbudgeted run (or an approved + // re-dispatch) would have got no B1 at all while still spending real money. + ...(preEgress === undefined && params.money === undefined ? {} : { preAttempt: async (info: { @@ -881,6 +929,22 @@ async function driveAgentTurn( readonly provider: ProviderId; readonly maxTokens?: number; }) => { + // **Barrier B1 (ADR-0077)** — before the next egress admission, and before the governor call, so a + // run whose ledger write did not land admits nothing further. It awaits AND observes: `join()` + // throws the retained failure rather than returning, which is the only way a caller here can see + // it (`#emitDurable` absorbs a store fault and resolves). + // + // **UNTESTED, and the reason is worth knowing rather than guessing.** Deleting this line leaves + // the entire core suite green, and no fixture reddens it because on today's engine every path + // that reaches a SECOND egress after a settled attempt passes through B2 or B3 first: within one + // chain, a settled attempt ends it (a post-content failure surfaces rather than failing over), so + // a second egress means either another tool round (B2) or another node dispatch (B3). B1 is + // therefore defence in depth against a future path that reaches egress without crossing either — + // a chain that continues past a settled attempt, or a turn core that stops routing tools through + // `dispatchToolUseTurn`. Kept deliberately; if a later reader finds it genuinely unreachable, the + // honest move is to delete it and say so, not to leave an untestable line with a hopeful comment. + await params.money?.join(); + if (preEgress === undefined) return; // This is the only admitting boundary. Check cancellation on BOTH sides of the awaited governor call: // a cancellation landing while warning durability/admission is pending must not reach key resolution // or provider egress, and any just-acquired lease is released before the cancellation propagates. diff --git a/packages/core/src/engine/checkpoint.test.ts b/packages/core/src/engine/checkpoint.test.ts index 017f2a6c..60a6622b 100644 --- a/packages/core/src/engine/checkpoint.test.ts +++ b/packages/core/src/engine/checkpoint.test.ts @@ -616,6 +616,127 @@ describe('reconstructCheckpointState', () => { error: { code: 'tool_failed', message: 'boom', retryable: false }, }); }); + + describe('the realized-cost ledger (ADR-0076)', () => { + /** One settled attempt. `cost` is THIS attempt's delta; `cumulative` is the producer's snapshot after it. */ + const settled = (seq: number, nodeId: string, cost: number, cumulative: number): RunEvent => ({ + type: 'cost:attempt_settled', + ...base(seq), + nodeId, + model: 'claude-opus-4-8', + attemptNumber: 1, + inputTokens: 10, + outputTokens: 5, + costMicrocents: cost, + cumulativeCostMicrocents: cumulative, + priced: true, + }); + + /** A node terminal carrying the durable run-wide snapshot — the OTHER family feeding the same field. */ + const completedAt = (seq: number, nodeId: string, cumulative: number): RunEvent => ({ + type: 'node:completed', + ...base(seq), + nodeId, + output: null, + tokensUsed: { input: 0, output: 0 }, + durationMs: 1, + cumulativeCostMicrocents: cumulative, + }); + + it('restores from the attempt events, taking the highest absolute cumulative', () => { + const state = reconstructCheckpointState([ + started, + settled(1, 'a', 400, 400), + settled(2, 'a', 600, 1_000), + ]); + expect(state?.cumulativeCostMicrocents).toBe(1_000); + }); + + it('does NOT double-count a node snapshot against the attempts it already covers', () => { + // A completed node's snapshot ALREADY contains its attempts. This pins that the snapshot is MAXED in, + // never added: `acc.cumulativeCostMicrocents += event.cumulativeCostMicrocents` on the `node:completed` + // arm restores 2_000 for a run that spent 1_000, and this test reddens on it. + // + // **What it does NOT pin, stated rather than implied.** Replacing the attempt arm's `Math.max` with + // `+= event.costMicrocents` leaves this — and every other test here — green, because on a WELL-FORMED + // log the two folds are equivalent: a node's attempts always carry a lower `sequenceNumber` than its + // own terminal, so the delta is summed exactly once before the snapshot maxes over it. `Math.max` is + // chosen for robustness (it needs no assumption about ordering or about an event being folded once) and + // for matching the sibling arms, not because a fixture can tell them apart. No honest test can, so none + // is written; the mutation that WOULD close the gap is a log with a node terminal ordered before the + // attempts it covers, which the engine does not produce. + const state = reconstructCheckpointState([ + started, + settled(1, 'a', 400, 400), + settled(2, 'a', 600, 1_000), + completedAt(3, 'a', 1_000), + ]); + expect(state?.cumulativeCostMicrocents).toBe(1_000); + }); + + it('is ORDER-INDEPENDENT, which a last-wins read of the event snapshot is not', () => { + // Concurrent events under a `fan_out` have no canonical `seq` order, so the LOWER seq can carry the + // HIGHER cumulative. Last-wins would restore 400 here and hand 600 back to the cap as headroom. + const state = reconstructCheckpointState([ + started, + settled(1, 'a', 600, 1_000), + settled(2, 'b', 400, 400), + ]); + expect(state?.cumulativeCostMicrocents).toBe(1_000); + }); + + it('restores the HIGHER total when a crash landed mid-loop, past the last node terminal', () => { + // The motivating scenario, and the one that killed the max-of-two-families design. Node `a` is a MEDIA + // node: it writes a 1_000 snapshot and emits no attempt row at all. Node `b` then makes two paid calls + // and the process dies before its terminal. Summing attempts into a separate accumulator gives 800, + // `max(1_000, 800)` gives 1_000, and node b's 800 vanishes — a resume would spend it again, which is + // the exact bug the ledger exists to close. Maxing over ABSOLUTE cumulatives gives 1_800. + const state = reconstructCheckpointState([ + started, + completedAt(1, 'a', 1_000), + settled(2, 'b', 500, 1_500), + settled(3, 'b', 300, 1_800), + ]); + expect(state?.cumulativeCostMicrocents).toBe(1_800); + }); + + it('a stale budget:paused cannot CLOBBER a higher ledger total', () => { + // `budget:paused.spentMicrocents` is captured at the pre-egress check and the event is emitted much + // later, after the outcome propagates. Under a `fan_out` a sibling attempt settles a HIGHER cumulative + // in that window — and this arm used to ASSIGN, handing the resumed cap headroom for money already + // spent. Latent before the ledger, because node boundaries are rare; the ledger writes per attempt. + const state = reconstructCheckpointState([ + started, + settled(1, 'a', 5_000, 5_000), + { + type: 'budget:paused', + ...base(2), + nodeId: 'b', + spentMicrocents: 3_000, + limitMicrocents: 10_000, + gateId: 'g1', + }, + ]); + expect(state?.cumulativeCostMicrocents).toBe(5_000); + }); + + it('keeps realized and conservative money apart', () => { + const state = reconstructCheckpointState([ + started, + settled(1, 'a', 400, 400), + { + type: 'budget:estimate_committed', + ...base(2), + nodeId: 'a', + model: 'claude-opus-4-8', + estimateMicrocents: 900, + cumulativeConservativeMicrocents: 900, + }, + ]); + expect(state?.cumulativeCostMicrocents).toBe(400); + expect(state?.conservativeCostMicrocents).toBe(900); + }); + }); }); describe('createInMemoryCheckpointer', () => { diff --git a/packages/core/src/engine/checkpoint.ts b/packages/core/src/engine/checkpoint.ts index b7d1fa10..95aa79da 100644 --- a/packages/core/src/engine/checkpoint.ts +++ b/packages/core/src/engine/checkpoint.ts @@ -94,10 +94,39 @@ export interface CheckpointState { /** Running token totals (summed from `node:completed`), restored so a resumed run's `run:completed` totals stay correct. */ readonly totalInputTokens: number; readonly totalOutputTokens: number; - /** The run-wide cumulative cost (integer micro-cents), restored on resume from the durable - * `node:completed.cumulativeCostMicrocents` snapshot and/or `budget:paused.spentMicrocents` — the higher - * wins (`Math.max`, order-independent). (`cost:updated` is also folded when present, but it is streamed, - * not persisted, so it never appears in a real durable log.) Keeps post-resume cost cumulative. */ + /** + * The run-wide realized cost (integer micro-cents), restored on resume so post-resume spend keeps + * accumulating against a cap that remembers what already went out. + * + * **The fold is `Math.max` over every durable ABSOLUTE total that is folded here**, and there are exactly + * three: `node:completed.cumulativeCostMicrocents`, `node:failed.cumulativeCostMicrocents`, and — since + * [ADR-0076](../../../../docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md) — + * `cost:attempt_settled.cumulativeCostMicrocents`. Plus `budget:paused.spentMicrocents` in + * `applyGateEvent`, maxed the same way. Each is read immediately after its own increment, so each is a true + * run-wide total at that instant and the largest is the engine's real total, whatever order the rows landed + * in. + * + * **`run:failed` / `run:cancelled` carry the field but are NOT folded here**, and that is deliberate rather + * than an oversight — the engine states it at the emit site. A run terminal is the last event of the run, so + * a restored total that omitted it would only ever be read by a resume that cannot happen. Named because the + * field list above reads like it should include them. + * + * **Neither obvious alternative works, and one of them under-counts silently:** + * + * - SUMMING `cost:attempt_settled.costMicrocents` into this field double-counts, because a node terminal's + * snapshot already contains the attempts it covers. + * - Summing the attempts into a SEPARATE accumulator and taking the max of the two families under-counts. + * The families cover different money — a media node writes a snapshot and emits no attempt row at all — + * so whenever earlier media spend is the larger figure, every attempt made after the last node boundary + * is silently dropped. That is exactly the crash-mid-agent-loop case the ledger exists for. + * + * The `Math.max` reasoning {@link conservativeCostMicrocents} rejects does not carry here: it was rejected + * there only because a future deliberate release would DECREASE the conservative total. Realized spend has + * no release and is monotonic by construction. + * + * (`cost:updated` is also folded when present, but it is streamed, not persisted, so it never appears in a + * real durable log.) + */ readonly cumulativeCostMicrocents: number; /** * The run-wide durable **conservative** total (integer micro-cents) — money a provider may already have billed @@ -290,8 +319,16 @@ function applyGateEvent(acc: ReconAccumulator, event: RunEvent): void { // keeps its spend and the re-seeded governor blocks correctly (H2). A plain-human-gate / crash resume now // recovers the same total from the durable `node:completed.cumulativeCostMicrocents` (applyNodeEvent above) — // the two durable sources reconcile via that `Math.max` fold (cost-event persistence is no longer deferred). + // + // `Math.max`, not an ASSIGN — which is what this was, while the comment above already claimed otherwise. + // `spentMicrocents` is captured at `checkPreEgress` and the event is emitted much later, after the + // outcome propagates through `#onOutcome`/`#settlePaused`. Under a `fan_out` a sibling attempt can settle + // a HIGHER cumulative in that window, and an assign then clobbers it — handing the resumed cap headroom + // for money already spent, which is the exact bypass ADR-0074/ADR-0076 exist to close. Latent before the + // realized ledger, because `node:completed` was the only other writer and node boundaries are rarer; + // `cost:attempt_settled` multiplies the high-water marks available to clobber. if (event.type === 'budget:paused') { - acc.cumulativeCostMicrocents = event.spentMicrocents; + acc.cumulativeCostMicrocents = Math.max(acc.cumulativeCostMicrocents, event.spentMicrocents); } return; } @@ -350,6 +387,30 @@ export function reconstructCheckpointState( // §1's future release. See the field's doc for the full comparison.) acc.conservativeCostMicrocents += event.estimateMicrocents; } + if (event.type === 'cost:attempt_settled') { + // `Math.max` over the event's ABSOLUTE cumulative — the same fold `node:completed` / `node:failed` use + // two arms above, and deliberately NOT the sum-of-deltas its conservative sibling uses. Three things + // make that the right choice, and two plausible alternatives are wrong: + // + // - SUMMING `costMicrocents` into this field DOUBLE-COUNTS: a node terminal's snapshot already contains + // the attempts it covers, and both write here. + // - Summing into a SEPARATE accumulator and taking the max at the end UNDER-counts, which is subtler. + // The two families cover different money: a media node writes a snapshot and emits no attempt row at + // all, so `max(snapshots, sum(attempts))` silently drops every attempt made after the last boundary + // whenever earlier media spend is the larger number. + // - `Math.max` over absolute totals has neither failure. The producer reads this counter AFTER folding + // this attempt into it, so the value is a true run-wide total at that instant, exactly like a node + // boundary's — and the largest one seen is the engine's real total no matter what order the rows land + // in. + // + // ADR-0074 §2 rejected `Math.max` for the CONSERVATIVE total, and that reasoning does not carry: it was + // rejected only because a future deliberate release would DECREASE that total. Realized spend has no + // release and is monotonic by construction. + acc.cumulativeCostMicrocents = Math.max( + acc.cumulativeCostMicrocents, + event.cumulativeCostMicrocents, + ); + } applyRunEvent(acc, event); applyNodeEvent(acc, event); applyMediaJobEvent(acc, event); diff --git a/packages/core/src/engine/durable-truth.test.ts b/packages/core/src/engine/durable-truth.test.ts new file mode 100644 index 00000000..8e912c69 --- /dev/null +++ b/packages/core/src/engine/durable-truth.test.ts @@ -0,0 +1,525 @@ +import type { RunEvent } from '@relavium/shared'; +import { describe, expect, it } from 'vitest'; + +import { checkDurableTruth, formatDurableTruth } from './durable-truth.js'; + +const TS = '2026-01-01T00:00:00.000Z'; +const base = (sequenceNumber: number) => ({ runId: 'r1', sequenceNumber, timestamp: TS }); + +const started: RunEvent = { + type: 'run:started', + ...base(0), + workflowId: '00000000-0000-4000-8000-000000000001', + inputs: {}, + executionMode: 'local', +}; + +const completed = ( + seq: number, + outputs: Record = { a: 1 }, + cost = 500, +): RunEvent => ({ + type: 'run:completed', + ...base(seq), + outputs, + totalTokensUsed: { input: 1, output: 1 }, + totalCostMicrocents: cost, + durationMs: 10, +}); + +const failed = (seq: number, code: 'internal' | 'tool_failed' = 'internal'): RunEvent => ({ + type: 'run:failed', + ...base(seq), + error: { code, message: 'boom', retryable: false }, + partialOutputs: {}, +}); + +const log = (...events: RunEvent[]) => { + const stored = [started, ...events]; + return { eventsFor: () => stored, stored }; +}; + +describe('checkDurableTruth', () => { + it('AGREES when live, history, reconcile and the checkpoint all say the same thing', async () => { + const terminal = completed(1); + const { eventsFor } = log(terminal); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: terminal, + eventsFor, + reconcile: () => Promise.resolve([]), + }); + + expect(verdict.agrees).toBe(true); + expect(verdict.disagreements).toEqual([]); + expect(verdict.durableTerminalCount).toBe(1); + expect(verdict.checkpointStatus).toBe('completed'); + }); + + it('catches the headline defect: a caller told `completed` while history says `failed`', async () => { + // CR-92's exact shape. The live stream is the only thing an e2e assertion used to read, so this shipped + // invisible: the API returns success and outputs, and the durable record disagrees. + const { eventsFor } = log(failed(1)); + const verdict = await checkDurableTruth({ runId: 'r1', live: completed(1), eventsFor }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements[0]).toContain('live and durable history disagree'); + expect(formatDurableTruth(verdict)).toContain('run:completed'); + expect(formatDurableTruth(verdict)).toContain('run:failed'); + }); + + it('catches a SAME-TYPE terminal whose payload differs', async () => { + // Type-only comparison would pass this. Two runs both `completed`, different outputs and different money + // — which is the reconciliation-rewrote-the-answer case, not a crash. + const { eventsFor } = log(completed(1, { a: 1 }, 500)); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1, { a: 2 }, 900), + eventsFor, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements[0]).toContain('live and durable history disagree'); + }); + + it('IGNORES envelope drift — a restart moves the clock, and that is not a disagreement', async () => { + const stored: RunEvent[] = [ + started, + { ...completed(1), timestamp: '2026-02-02T00:00:00.000Z' }, + ]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(7), // a different sequenceNumber too + eventsFor: () => stored, + }); + + expect(verdict.agrees).toBe(true); + }); + + it('catches a restart that CHANGES the terminal', async () => { + const stored: RunEvent[] = [started, completed(1)]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor: () => stored, + reconcile: () => { + // A reconcile that rewrites an already-settled run — the "may later produce a different terminal" + // half of CR-92. + stored.splice(1, 1, failed(2)); + return Promise.resolve([failed(2)]); + }, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('a restart CHANGED the terminal'))).toBe( + true, + ); + }); + + it('catches a reconcile that adds a SECOND terminal to an already-closed run', async () => { + const stored: RunEvent[] = [started, completed(1)]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor: () => stored, + reconcile: () => { + stored.push(failed(2)); + return Promise.resolve([failed(2)]); + }, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.durableTerminalCount).toBe(2); + expect(verdict.disagreements.some((d) => d.includes('exactly one closes a run'))).toBe(true); + expect(verdict.disagreements.some((d) => d.includes('never one that landed'))).toBe(true); + // And the terminal-changed check fires too, now that `terminalIn` reads the LAST terminal — with `find` + // it read the first, so a reconcile that APPENDED a second was invisible to this check. + expect(verdict.disagreements.some((d) => d.includes('a restart CHANGED the terminal'))).toBe( + true, + ); + }); + + it('catches a checkpoint fold that disagrees with the durable terminal', async () => { + // The fold is what a resume seeds itself from, so a disagreement here means a resumed run does the wrong + // work even though every event on disk is intact. Constructed by omitting `run:started`, which makes the + // fold return `undefined` while the terminal is plainly there. + const stored: RunEvent[] = [completed(1)]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor: () => stored, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.checkpointStatus).toBeUndefined(); + expect(verdict.disagreements.some((d) => d.includes('checkpoint fold'))).toBe(true); + }); + + it('catches a run that streamed a terminal and persisted NONE — the crash window', async () => { + const { eventsFor } = log(); + const verdict = await checkDurableTruth({ runId: 'r1', live: completed(1), eventsFor }); + + expect(verdict.agrees).toBe(false); + expect(verdict.durableTerminalCount).toBe(0); + expect(verdict.history).toBeUndefined(); + }); + + it('accepts an async eventsFor — the real store returns a promise', async () => { + const terminal = completed(1); + const stored = [started, terminal]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: terminal, + eventsFor: () => Promise.resolve(stored), + }); + + expect(verdict.agrees).toBe(true); + }); + + it('catches a log belonging to ANOTHER run', async () => { + // The store ignored its id argument, or the caller crossed two runs. `durableTerminalCount === 1` does + // not catch it — the other run has a terminal too — so the oracle used to report agreement on the wrong + // log entirely. CR-11's scenario is two owners over one store, which is exactly this shape. + const other: RunEvent[] = [ + { ...started, runId: 'rOTHER' }, + { ...completed(1), runId: 'rOTHER' }, + ]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor: () => other, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('belonging to another run'))).toBe(true); + // …and every DOWNSTREAM number describes this run, not the foreign one. Naming the right cause while + // counting the other run's terminal was the defect: `history` read `rOTHER`'s `run:completed` and this + // came back as 1, so a reader who trusted the counts over the message saw a healthy log. + expect(verdict.durableTerminalCount).toBe(0); + expect(verdict.history).toBeUndefined(); + }); + + it('catches a live terminal carrying a different runId', async () => { + const { eventsFor } = log(completed(1)); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: { ...completed(1), runId: 'rOTHER' }, + eventsFor, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('not r1'))).toBe(true); + }); + + it('accepts a durable log whose seqs SKIP — streamed events take numbers and are never persisted', async () => { + // A real completed run reads [0,1,2,3,5,10,11,12,13,14]. Asserting `0..n-1` here failed a healthy engine, + // which is how this limitation was found: "no persisted event is missing from the middle" is not + // expressible from the log alone, because a streamed event's absence looks identical to a lost one. + const stored: RunEvent[] = [started, { ...completed(1), sequenceNumber: 7 }]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: { ...completed(1), sequenceNumber: 7 }, + eventsFor: () => stored, + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(true); + }); + + it('catches an OUT-OF-ORDER durable log — CR-10s checkable half', async () => { + const stored: RunEvent[] = [ + { ...started, sequenceNumber: 0 }, + { ...failed(9) }, + { ...completed(1), sequenceNumber: 4 }, + ]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: { ...completed(1), sequenceNumber: 4 }, + eventsFor: () => stored, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('is not ordered'))).toBe(true); + }); + + it('catches a durable log missing its head', async () => { + const stored: RunEvent[] = [{ ...completed(1), sequenceNumber: 4 }]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: { ...completed(1), sequenceNumber: 4 }, + eventsFor: () => stored, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('not 0'))).toBe(true); + }); + + it('catches an event persisted AFTER the terminal', async () => { + const stored: RunEvent[] = [ + started, + completed(1), + { type: 'node:skipped', ...base(2), nodeId: 'x', reason: 'branch_not_taken' }, + ]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor: () => stored, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('continues past its terminal'))).toBe(true); + }); + + it('does NOT blame this run for events reconcile() wrote for ANOTHER one', async () => { + // `reconcile()` repairs every interrupted run in the store and returns all of their events. Attributing + // the raw count here produced a factually wrong message and a false failure. + const { eventsFor } = log(completed(1)); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor, + reconcile: () => Promise.resolve([{ ...failed(9), runId: 'rOTHER' }]), + }); + + expect(verdict.agrees).toBe(true); + expect(verdict.reconciledCount).toBe(0); + }); + + it('accepts a CORRECT crash repair under expect:"repaired" — the case this phase exists for', async () => { + // Before the mode existed, a run that died without a terminal and was repaired on restart verdicted + // `a restart CHANGED the terminal: before=none after=run:failed` — so no crash test could use the oracle + // at all, which is most of what CR-91 is for. + const stored: RunEvent[] = [started]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: undefined, + expect: 'repaired', + eventsFor: () => stored, + reconcile: () => { + stored.push(failed(1)); + return Promise.resolve([failed(1)]); + }, + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(true); + expect(verdict.reconciledCount).toBe(1); + expect(verdict.afterReconcile?.type).toBe('run:failed'); + }); + + it('under expect:"repaired", a restart that repairs NOTHING is the failure', async () => { + const stored: RunEvent[] = [started]; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: undefined, + expect: 'repaired', + eventsFor: () => stored, + reconcile: () => Promise.resolve([]), + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('NO durable terminal'))).toBe(true); + }); + + // The other two `expect:'repaired'` arms. Without them the mode was only ever exercised where it PASSES or + // where the restart did nothing — so a caller that asked for `'repaired'` on a run that had, in fact, + // settled would have been told it agreed, which is the mode's whole failure direction. + it('under expect:"repaired", a log that ALREADY holds a terminal is the disagreement', async () => { + const { eventsFor } = log(completed(1)); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: undefined, + expect: 'repaired', + eventsFor, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('already holds a terminal'))).toBe(true); + }); + + it('under expect:"repaired", a LIVE terminal means the run did not die — also a disagreement', async () => { + const { eventsFor } = log(); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + expect: 'repaired', + eventsFor, + }); + + expect(verdict.agrees).toBe(false); + expect(verdict.disagreements.some((d) => d.includes('the live stream produced'))).toBe(true); + }); + + /** + * A `run:failed` differing from the baseline in exactly one error field. The override type is DERIVED from + * the contract rather than asserted with `as never`, so a renamed field or an invalid `code` fails to + * compile instead of silently building a fixture the oracle can never disagree about. + */ + type FailureError = Extract['error']; + const failedWith = (error: Partial): RunEvent => ({ + type: 'run:failed', + ...base(1), + error: { code: 'tool_failed', message: 'boom', retryable: false, ...error }, + partialOutputs: {}, + }); + + // One test per field, each flipping exactly ONE. A single fixture flipping two proved only "at least one + // of them is compared" — dropping either from `sameView` left it green. + it.each([ + ['retryable', failedWith({ nodeId: 'A', correlationId: 'c-1', retryable: true })], + ['nodeId', failedWith({ nodeId: 'B', correlationId: 'c-1' })], + ['correlationId', failedWith({ nodeId: 'A', correlationId: 'c-2' })], + ])('catches a failure whose %s differs while the CODE matches', async (_field, durable) => { + const liveEvent = failedWith({ nodeId: 'A', correlationId: 'c-1' }); + const verdict = await checkDurableTruth({ + runId: 'r1', + live: liveEvent, + eventsFor: () => [started, durable], + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(false); + }); + + it('catches a completed run whose TOKEN totals differ', async () => { + const durable: RunEvent = { + type: 'run:completed', + ...base(1), + outputs: { a: 1 }, + totalTokensUsed: { input: 9, output: 9 }, + totalCostMicrocents: 500, + durationMs: 10, + }; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1), + eventsFor: () => [started, durable], + }); + + expect(verdict.agrees).toBe(false); + }); + + it('RENDERS every compared field, so a DISAGREES verdict shows why', async () => { + // The rendered diff used to print type/code/cost/outputs only — so a pair differing solely in + // `retryable` or `correlationId` printed byte-identically on both sides and told the reader nothing. + const verdict = await checkDurableTruth({ + runId: 'r1', + live: failedWith({ retryable: false, nodeId: 'A', correlationId: 'c-1' }), + eventsFor: () => [ + started, + failedWith({ retryable: true, nodeId: 'B', correlationId: 'c-2' }), + ], + }); + + const rendered = formatDurableTruth(verdict); + expect(rendered).toContain('retryable=false'); + expect(rendered).toContain('retryable=true'); + expect(rendered).toContain('correlationId=c-1'); + expect(rendered).toContain('correlationId=c-2'); + expect(rendered).toContain('node=A'); + expect(rendered).toContain('node=B'); + }); + + it('compares outputs key-order-insensitively, and never throws on an uncomparable payload', async () => { + const reordered = await checkDurableTruth({ + runId: 'r1', + live: completed(1, { a: 1, b: 2 }), + eventsFor: () => [started, completed(1, { b: 2, a: 1 })], + }); + expect(reordered.agrees, 'key order is not a disagreement').toBe(true); + + // A circular payload used to throw OUT of the oracle, so the instrument became the failure. + const circular: Record = {}; + circular['self'] = circular; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1, circular), + eventsFor: () => [started, completed(1, circular)], + }); + expect(verdict.agrees).toBe(true); + }); + + it('tells two DIFFERENT circular payloads apart — not one marker for every cycle', async () => { + // Comparing a cyclic payload with ITSELF (above) passes under any implementation, including the broken + // one. Sorting the keys makes the replacer hand `JSON.stringify` a fresh object at every level, which + // defeats its native cycle detector: the walk recursed until the stack blew and BOTH sides came back as + // `[uncomparable: RangeError]`. Two structurally different cyclic outputs compared EQUAL and the oracle + // verdicted `agrees` — the false agreement it exists to catch, produced by the instrument itself. + const live: Record = { note: 'live' }; + live['self'] = live; + const durable: Record = { note: 'durable' }; + durable['self'] = durable; + + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1, live), + eventsFor: () => [started, completed(1, durable)], + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(false); + expect(formatDurableTruth(verdict)).toContain('[circular]'); + }); + + it('canonicalizes a Date through toJSON — walking the tree by hand must not lose it', async () => { + // Native `JSON.stringify` applies `toJSON` before the replacer; the hand-rolled cycle-aware walk has to + // apply it too. Without that a `Date` reduces to `{}` (it has no own enumerable entries) and ANY two dates + // compare equal — a false agreement introduced by the cycle fix rather than one it removes. + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1, { at: new Date('2026-01-01T00:00:00.000Z') }), + eventsFor: () => [started, completed(1, { at: new Date('2026-09-09T00:00:00.000Z') })], + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(false); + }); + + it('treats a SHARED reference as shared, not circular — a repeated object is not a cycle', async () => { + // The first cycle guard was a flat `WeakSet` that was never un-marked on the way back up, so it flagged + // every REPEATED reference. `{ a: shared, b: shared }` serialized as `{"a":{…},"b":"[circular]"}` while a + // structurally identical payload built from two separate objects serialized normally — two equal outputs + // disagreeing, which is the exact false failure `canonical` exists to prevent. A `fan_in` echoing one + // value into two keys is enough to hit it. + const shared = { x: 1 }; + const verdict = await checkDurableTruth({ + runId: 'r1', + live: completed(1, { a: shared, b: shared }), + eventsFor: () => [started, completed(1, { a: { x: 1 }, b: { x: 1 } })], + }); + + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(true); + }); + + it('reports WHERE the checkpoint status came from — a local fold is not the resume path', async () => { + const terminal = completed(1); + const { eventsFor } = log(terminal); + + const local = await checkDurableTruth({ runId: 'r1', live: terminal, eventsFor }); + expect(local.checkpointSource).toBe('local-fold'); + + const viaPort = await checkDurableTruth({ + runId: 'r1', + live: terminal, + eventsFor, + loadCheckpoint: () => Promise.resolve({ runStatus: 'completed' as const }), + }); + expect(viaPort.checkpointSource).toBe('checkpointer'); + + // And the port is what is BELIEVED when supplied: a checkpointer that refuses (undefined) disagrees with + // a durable terminal, which a local fold would have read happily. + const refusing = await checkDurableTruth({ + runId: 'r1', + live: terminal, + eventsFor, + loadCheckpoint: () => Promise.resolve(undefined), + }); + expect(refusing.agrees).toBe(false); + expect(refusing.disagreements.some((d) => d.includes('checkpoint port'))).toBe(true); + }); + + it('skips view 3 when no reconcile is supplied, and says so in the counts', async () => { + const terminal = failed(1, 'tool_failed'); + const { eventsFor } = log(terminal); + const verdict = await checkDurableTruth({ runId: 'r1', live: terminal, eventsFor }); + + expect(verdict.agrees).toBe(true); + expect(verdict.reconciledCount).toBe(0); + expect(verdict.afterReconcile).toEqual(verdict.history); + }); +}); diff --git a/packages/core/src/engine/durable-truth.ts b/packages/core/src/engine/durable-truth.ts new file mode 100644 index 00000000..96438884 --- /dev/null +++ b/packages/core/src/engine/durable-truth.ts @@ -0,0 +1,500 @@ +/** + * The durable-truth oracle (`CR-91`) — does a run's LIVE answer survive a restart? + * + * The e2e harness could already prove that a run streamed a terminal and that the terminal said what the test + * expected. That is one view, taken in-process, while the engine that produced it is still alive. It cannot + * catch the failure class phase 2.6.5 exists for: a caller receives `run:completed` with outputs while the + * durable log says the run failed, or a restart reconciles the same run to a DIFFERENT terminal, or a resume + * re-runs work the log already records as done. Every one of those is invisible to a live assertion. + * + * So this compares several views of the same run and reports where they disagree: + * + * 1. **live** — the terminal the caller drained from `RunHandle.events`. + * 2. **history** — the terminal as it exists in the store, read back. + * 3. **after reconcile** — the history a *fresh* engine leaves behind after `reconcile()` on restart. + * 4. **checkpoint** — the status `reconstructCheckpointState` derives from the durable log alone. A resume + * seeds itself from this, so if it disagrees with the terminal, a resumed run does the wrong work. + * 5. **the durable prefix itself** — sequence numbers gap-free and ordered, exactly one terminal, and the + * terminal last. Not a "view" of the terminal, but the property every other view silently assumes. + * + * They are four PROJECTIONS of two sources (the live stream and the store), not four independent + * observations — worth saying because an earlier version of this docblock called them independent, and in an + * in-memory harness views 1 and 2 are literally the same object. + * + * **What it CAN and CANNOT instrument, stated because the phase doc once claimed more.** It expresses + * `CR-92` (terminal durable truth) and `CR-10` (the durable log is an ordered, gap-free prefix). It does NOT + * express `CR-11` — it has no concept of run ownership or a fencing token — and it does NOT express `CR-12`, + * which is about external effects and needs an effect-journal view plus a side-effect counter this module + * does not model. Those two need their own predicates; this one is not the instrument for them. + * + * It also does not yet cover the RESUME view that `CR-91`'s acceptance criterion names. View 4 folds the log + * locally, which is close but not the same thing: a real resume goes through the host's `Checkpointer`, and + * the CLI's implementation uses ADR-0075's STRICT read that refuses a log with an uninterpretable row. A + * resume that would refuse therefore reads here as a clean fold. `loadCheckpoint` exists so a caller can + * supply the real port; when it is absent the verdict says the fold was local. + * + * **It returns a verdict rather than throwing.** A thrower is hard to test and forces every caller into the + * same message; a structured verdict lets callers assert on the specific disagreement, and lets this module + * have its own tests. {@link formatDurableTruth} renders it when a caller just wants a readable diff. + */ + +import type { RunEvent, RunStatus } from '@relavium/shared'; + +import { reconstructCheckpointState } from './checkpoint.js'; + +/** The run terminals. Exactly one closes a run (ADR-0036); the oracle's whole job is to check they agree. */ +const TERMINALS = new Set(['run:completed', 'run:failed', 'run:cancelled']); + +/** The run status each terminal implies — the checkpoint fold must arrive at the same one. */ +const STATUS_FOR: Partial> = { + 'run:completed': 'completed', + 'run:failed': 'failed', + 'run:cancelled': 'cancelled', +}; + +/** + * A terminal reduced to what must agree across views: its type, and the payload a consumer acts on. + * + * Deliberately NOT the whole event. `timestamp` and `sequenceNumber` are envelope fields the bus stamps, and + * comparing them would make every restart look like a disagreement while catching nothing — the failure this + * oracle is for is a terminal whose TYPE or OUTPUT changed, not one whose clock moved. + */ +export interface TerminalView { + readonly type: RunEvent['type']; + /** `run:completed.outputs` / `run:failed.partialOutputs`, canonically stringified. `run:cancelled` has none. */ + readonly outputs?: string; + /** `run:failed.error.code`, when the terminal carries one. */ + readonly errorCode?: string; + /** + * The rest of `run:failed.error`'s identity. Dropped in the first version, and the omission let a real + * defect through: a live `{tool_failed, nodeId:'A', retryable:false}` compared equal to a durable + * `{tool_failed, nodeId:'B', retryable:true}`. A flipped `retryable` changes whether a surface offers a + * retry, and `nodeId` names the root cause. + */ + readonly errorRetryable?: boolean; + readonly errorNodeId?: string; + /** + * The secret-free id joined to the internal log (ADR-0036) — the single best discriminator here, because it + * identifies THIS terminal event. It differs across views only when the terminal was genuinely rewritten, + * which is exactly the defect. + */ + readonly correlationId?: string; + /** The run-wide realized total the terminal reports, when it carries one. */ + readonly costMicrocents?: number; + /** `run:completed.totalTokensUsed`, restored by the same fold as cost — comparing one and not the other + * was an inconsistency, not a decision. */ + readonly tokens?: string; +} + +export interface DurableTruthVerdict { + readonly agrees: boolean; + readonly runId: string; + readonly live: TerminalView | undefined; + readonly history: TerminalView | undefined; + readonly afterReconcile: TerminalView | undefined; + readonly checkpointStatus: RunStatus | undefined; + /** + * Where {@link checkpointStatus} came from. `'local-fold'` means this module folded the log itself, which + * is NOT the resume path — a real resume goes through the host's `Checkpointer` and may refuse a log this + * fold reads happily (ADR-0075). Supply `loadCheckpoint` to get `'checkpointer'`. + */ + readonly checkpointSource: 'local-fold' | 'checkpointer'; + /** How many terminals the durable log holds. Anything but 1 breaks exactly-one-terminal (ADR-0036). */ + readonly durableTerminalCount: number; + /** Events `reconcile()` produced FOR THIS RUN — it repairs every interrupted run and returns them all. */ + readonly reconciledCount: number; + /** One sentence per disagreement, in the order checked. Empty iff `agrees`. */ + readonly disagreements: readonly string[]; +} + +/** + * Codepoint order, NOT `localeCompare` — the comparison must be identical on every machine that runs this + * oracle. `localeCompare` honours the host locale (and its ICU build), so two runs of the same payload can + * canonicalize to different strings under different `LC_ALL`s, which is a disagreement manufactured by the + * instrument. The order itself is arbitrary; only its stability matters. + */ +function byCodepoint([a]: readonly [string, unknown], [b]: readonly [string, unknown]): number { + if (a < b) return -1; + if (a > b) return 1; + return 0; +} + +/** + * Stringify a payload for comparison, key-order-insensitively and WITHOUT ever throwing out of the oracle. + * + * A bare `JSON.stringify` fails twice: `{a:1,b:2}` and `{b:2,a:1}` compare unequal, which is a false failure + * the moment either side has been through a store round-trip — the whole point of this module; and a circular + * or `bigint` payload throws, turning the instrument itself into the failure with no verdict at all. + * + * **Cycles are detected HERE rather than left to `JSON.stringify`, and the reason is measured.** Sorting the + * keys means the replacer returns a NEW object at every level, which defeats the native cycle detector — it + * compares against the objects it is currently serializing, and it never sees the same one twice. So a cyclic + * payload did not raise the `TypeError` the catch below was written for; it recursed until the stack blew, and + * BOTH sides came back as `[uncomparable: RangeError]`. Two structurally different cyclic outputs therefore + * compared EQUAL and the oracle verdicted `agrees` — the exact false agreement it exists to catch. + * + * The ancestor stack is popped on the way back up, which is what separates a cycle from a shared reference. + * The first attempt at this used a flat `WeakSet` that was never un-marked, so it flagged every REPEATED + * reference: `{ a: shared, b: shared }` serialized as `{"a":{…},"b":"[circular]"}` while a structurally + * identical payload built from two separate objects serialized normally — two equal outputs disagreeing, and a + * `fan_in` echoing one value into two keys is enough to hit it. Ancestry is the correct predicate; the pop is + * the whole difference. + */ +function canonical(value: unknown): string { + const ancestors: object[] = []; + const walk = (val: unknown): unknown => { + if (typeof val === 'bigint') return `${val.toString()}n`; + if (typeof val !== 'object' || val === null) return val; + // A path-distinguishing marker, not a whole-payload one: the REST of the object still serializes, so two + // different cyclic payloads still differ everywhere they actually differ. + if (ancestors.includes(val)) return '[circular]'; + ancestors.push(val); + try { + // `toJSON` first, or a `Date` canonicalizes to `{}` — and then any two Dates would compare equal, which + // is a false agreement introduced by the fix. Native `JSON.stringify` applies it before the replacer; + // walking the tree by hand means applying it by hand. + const toJson: unknown = (val as { toJSON?: unknown }).toJSON; + if (typeof toJson === 'function') return walk((toJson as () => unknown).call(val)); + if (Array.isArray(val)) return val.map((entry: unknown) => walk(entry)); + return Object.fromEntries( + Object.entries(val) + .sort(byCodepoint) + .map(([key, entry]: [string, unknown]) => [key, walk(entry)]), + ); + } finally { + ancestors.pop(); + } + }; + try { + // `?? '[undefined]'`: `JSON.stringify(undefined)` returns `undefined`, and the return type says string. + return JSON.stringify(walk(value)) ?? '[undefined]'; + } catch (error) { + // The backstop is still here — a hostile getter, a `toJSON` that throws, anything unforeseen — but it is + // no longer the cycle path, so it no longer collapses distinct payloads onto one marker. + return `[uncomparable: ${error instanceof Error ? error.name : 'unknown'}]`; + } +} + +function viewOf(event: RunEvent | undefined): TerminalView | undefined { + if (event === undefined) return undefined; + switch (event.type) { + case 'run:completed': + return { + type: event.type, + outputs: canonical(event.outputs), + costMicrocents: event.totalCostMicrocents, + tokens: canonical(event.totalTokensUsed), + }; + case 'run:failed': + return { + type: event.type, + outputs: canonical(event.partialOutputs), + errorCode: event.error.code, + errorRetryable: event.error.retryable, + ...(event.error.nodeId === undefined ? {} : { errorNodeId: event.error.nodeId }), + ...(event.error.correlationId === undefined + ? {} + : { correlationId: event.error.correlationId }), + ...(event.cumulativeCostMicrocents === undefined + ? {} + : { costMicrocents: event.cumulativeCostMicrocents }), + }; + case 'run:cancelled': + return { + type: event.type, + ...(event.cumulativeCostMicrocents === undefined + ? {} + : { costMicrocents: event.cumulativeCostMicrocents }), + }; + default: + return undefined; + } +} + +/** + * The LAST terminal, not the first. + * + * A duplicated log is caught by the count check either way, but `find` made two views disagree by + * construction: `reconstructCheckpointState` folds run status last-wins, so view 4 read the last terminal + * while views 2 and 3 read the first. Taking the last aligns them and leaves duplication to the count. + */ +function terminalIn(events: readonly RunEvent[]): RunEvent | undefined { + return [...events].reverse().find((event) => TERMINALS.has(event.type)); +} + +function describe(view: TerminalView | undefined): string { + if (view === undefined) return 'none'; + // EVERY compared field, not a subset. The first version rendered only type/code/cost/outputs — so a pair + // differing solely in `errorRetryable` or `correlationId` printed BYTE-IDENTICALLY on both sides of a + // `DISAGREES` verdict, leaving a reader with no visible reason for exactly the fields that matter most. + const parts: string[] = [view.type]; + if (view.errorCode !== undefined) parts.push(`code=${view.errorCode}`); + if (view.errorRetryable !== undefined) parts.push(`retryable=${String(view.errorRetryable)}`); + if (view.errorNodeId !== undefined) parts.push(`node=${view.errorNodeId}`); + if (view.correlationId !== undefined) parts.push(`correlationId=${view.correlationId}`); + if (view.costMicrocents !== undefined) parts.push(`cost=${view.costMicrocents}`); + if (view.tokens !== undefined) parts.push(`tokens=${view.tokens}`); + if (view.outputs !== undefined) parts.push(`outputs=${view.outputs}`); + return parts.join(' '); +} + +function sameView(a: TerminalView | undefined, b: TerminalView | undefined): boolean { + if (a === undefined || b === undefined) return a === b; + return ( + a.type === b.type && + a.outputs === b.outputs && + a.errorCode === b.errorCode && + a.errorRetryable === b.errorRetryable && + a.errorNodeId === b.errorNodeId && + a.correlationId === b.correlationId && + a.costMicrocents === b.costMicrocents && + a.tokens === b.tokens + ); +} + +export interface DurableTruthInput { + readonly runId: string; + /** The terminal the caller drained from the live stream, or `undefined` if the stream produced none. */ + readonly live: RunEvent | undefined; + /** Read the run's persisted events — the same store the run was executed against. */ + readonly eventsFor: (runId: string) => readonly RunEvent[] | Promise; + /** + * Run `reconcile()` as a FRESH engine over that same store, exactly as a restarted process would. Omit only + * when the caller has already restarted for its own reasons; then view 3 is skipped and the verdict says so. + */ + readonly reconcile?: () => Promise; + /** + * The host's real `Checkpointer.load` — what a resume actually goes through. Supply it and view 4 observes + * the resume path (including ADR-0075's strict read, which REFUSES a log with an uninterpretable row); + * omit it and the verdict folds the log locally and records that it did. + */ + readonly loadCheckpoint?: (runId: string) => Promise<{ runStatus: RunStatus } | undefined>; + /** + * What the run is expected to look like on disk, and it changes what counts as a disagreement. + * + * - `'settled'` (default): the run closed under its own power. `reconcile()` must find nothing to do, and a + * changed terminal is a defect. + * - `'repaired'`: the run DIED without a terminal and a restart is expected to write one. Then a durable + * terminal appearing where there was none is the PASS condition, not a failure. + * + * Without this, a correct crash repair — the exact scenario this phase exists for — reported as + * `a restart CHANGED the terminal: before=none after=run:failed`, so no crash test could use the oracle. + */ + readonly expect?: 'settled' | 'repaired'; +} + +/** + * Bind everything to THIS run before comparing anything. Without it the oracle happily reports agreement on + * another run's log — measured — and `durableTerminalCount === 1` does not catch it, because the other run + * has a terminal too. A store that ignores its id argument, or a caller that crosses two runs, both land + * here; and `CR-11`'s whole scenario is two owners over one store. + */ +function checkRunBinding( + runId: string, + live: RunEvent | undefined, + before: readonly RunEvent[], +): readonly string[] { + const out: string[] = []; + const foreign = before.filter((event) => event.runId !== runId); + if (foreign.length > 0) { + out.push( + `eventsFor(${runId}) returned ${foreign.length} event(s) belonging to another run ` + + `(e.g. ${String(foreign[0]?.runId)}) — every view below would be comparing the wrong log`, + ); + } + if (live !== undefined && live.runId !== runId) { + out.push(`the live terminal carries runId ${live.runId}, not ${runId}`); + } + return out; +} + +function checkLiveVsHistory( + expectRepaired: boolean, + live: TerminalView | undefined, + history: TerminalView | undefined, +): readonly string[] { + const out: string[] = []; + if (expectRepaired) { + // A run that died mid-flight: nothing durable yet, and nothing live either. + if (history !== undefined) { + out.push( + `expected a run needing repair, but the log already holds a terminal (${describe(history)})`, + ); + } + if (live !== undefined) { + out.push( + `expected a run that died without a terminal, but the live stream produced ${describe(live)}`, + ); + } + } else if (!sameView(live, history)) { + out.push( + `live and durable history disagree: live=${describe(live)} history=${describe(history)}`, + ); + } + return out; +} + +function checkReconcileOutcome( + expectRepaired: boolean, + history: TerminalView | undefined, + afterReconcile: TerminalView | undefined, + reconciledCount: number, +): readonly string[] { + const out: string[] = []; + if (expectRepaired) { + if (afterReconcile === undefined) { + out.push( + 'a restart left the run with NO durable terminal — a crashed run must reconcile to one', + ); + } + if (reconciledCount === 0) { + out.push('reconcile() produced no event for this run, so nothing repaired it'); + } + return out; + } + if (!sameView(history, afterReconcile)) { + out.push( + `a restart CHANGED the terminal: before=${describe(history)} after=${describe(afterReconcile)}`, + ); + } + if (history !== undefined && reconciledCount > 0) { + out.push( + `reconcile() produced ${reconciledCount} event(s) for a run that already had a durable terminal ` + + `(${describe(history)}) — reconciliation repairs a run that died WITHOUT one, never one that landed`, + ); + } + return out; +} + +/** + * Exactly-one-terminal (ADR-0036) plus the durable ORDER property (`CR-10`). Every view above assumes the + * latter and none of them checked it: a log committed out of order, or with an event after its terminal, used + * to verdict `agrees`. + * + * **What is checkable here, and what is NOT** — the distinction cost a wrong assertion, so it is written + * down. The durable log's sequence numbers are a strictly increasing SUBSEQUENCE of the run's, never + * `0..n-1`: streamed events (`agent:token`, `cost:updated`, …) take numbers and are deliberately never + * persisted. A real completed run reads `[0,1,2,3,5,10,11,12,13,14]`, and asserting `0..n-1` failed it. + * + * So "no persisted event is missing from the middle" — CR-10's actual property — is NOT expressible from + * the log alone: a streamed event's absence is indistinguishable from a lost one. CR-10's acceptance needs + * a store harness that knows which events it was ASKED to persist. What the log can prove on its own is + * that it starts at `run:started` (always seq 0, always durable) and only ever moves forward. + */ +function checkLogShape(ours: readonly RunEvent[], terminalCount: number): readonly string[] { + const out: string[] = []; + if (terminalCount !== 1) { + out.push( + `the durable log holds ${terminalCount} terminal(s); exactly one closes a run (ADR-0036)`, + ); + } + + const seqs = ours.map((event) => event.sequenceNumber); + if (seqs.some((seq, index) => index > 0 && seq <= (seqs[index - 1] ?? -1))) { + out.push( + `the durable log is not ordered: sequenceNumbers ${JSON.stringify(seqs)} — a higher seq committed ` + + `before a lower one, so the checkpoint fold can read a state its causal predecessor never reached`, + ); + } + if (seqs.length > 0 && seqs[0] !== 0) { + out.push( + `the durable log starts at sequenceNumber ${String(seqs[0])}, not 0 — \`run:started\` is always ` + + `persisted and always first, so the head of the log is missing`, + ); + } + const last = ours.at(-1); + if (last !== undefined && terminalCount === 1 && !TERMINALS.has(last.type)) { + out.push( + `the durable log continues past its terminal (last event is '${last.type}') — exactly one terminal ` + + `CLOSES a run, so nothing may follow it`, + ); + } + return out; +} + +function checkCheckpoint( + viaPort: boolean, + checkpointStatus: RunStatus | undefined, + afterReconcile: TerminalView | undefined, +): readonly string[] { + const expectedStatus = afterReconcile === undefined ? undefined : STATUS_FOR[afterReconcile.type]; + if (expectedStatus === undefined || checkpointStatus === expectedStatus) return []; + return [ + `the checkpoint ${viaPort ? 'port' : 'fold'} says ` + + `'${String(checkpointStatus)}' while the durable terminal says '${expectedStatus}' — a resume seeds ` + + `itself from it, so it would do the wrong work`, + ]; +} + +/** + * Compare the four views. Cheap enough to call at the end of every e2e run; it reads the log twice and runs + * one reconcile. + */ +export async function checkDurableTruth(input: DurableTruthInput): Promise { + const disagreements: string[] = []; + const expectRepaired = input.expect === 'repaired'; + + const before = [...(await input.eventsFor(input.runId))]; + disagreements.push(...checkRunBinding(input.runId, input.live, before)); + + // EVERY view below reads the filtered log, not the raw one. Reporting the foreign events above and then + // comparing against them anyway was the bug in the first version of this binding: a store that ignored its + // id argument had `history` read ANOTHER run's terminal, and `durableTerminalCount` counted it — so the + // verdict named the right cause while every downstream number described the wrong run. + let ours = before.filter((event) => event.runId === input.runId); + + const live = viewOf(input.live); + const history = viewOf(terminalIn(ours)); + disagreements.push(...checkLiveVsHistory(expectRepaired, live, history)); + + let reconciledCount = 0; + let afterReconcile = history; + if (input.reconcile !== undefined) { + const reconciled = await input.reconcile(); + // Per-RUN, not the whole array. `reconcile()` repairs every interrupted run in the store and returns all + // of their events, so attributing the raw count to this run produced a factually wrong message and a + // false failure the moment a second run existed — which is, again, `CR-11`'s exact shape. + reconciledCount = reconciled.filter((event) => event.runId === input.runId).length; + ours = [...(await input.eventsFor(input.runId))].filter((event) => event.runId === input.runId); + afterReconcile = viewOf(terminalIn(ours)); + disagreements.push( + ...checkReconcileOutcome(expectRepaired, history, afterReconcile, reconciledCount), + ); + } + + const terminalCount = ours.filter((event) => TERMINALS.has(event.type)).length; + disagreements.push(...checkLogShape(ours, terminalCount)); + + const viaPort = input.loadCheckpoint !== undefined; + const loaded = await input.loadCheckpoint?.(input.runId); + const checkpointStatus = viaPort + ? loaded?.runStatus + : reconstructCheckpointState(ours)?.runStatus; + disagreements.push(...checkCheckpoint(viaPort, checkpointStatus, afterReconcile)); + + return { + agrees: disagreements.length === 0, + runId: input.runId, + live, + history, + afterReconcile, + checkpointStatus, + checkpointSource: input.loadCheckpoint === undefined ? 'local-fold' : 'checkpointer', + durableTerminalCount: terminalCount, + reconciledCount, + disagreements, + }; +} + +/** Render a verdict for an assertion message — every view on its own line, so the diff reads at a glance. */ +export function formatDurableTruth(verdict: DurableTruthVerdict): string { + return [ + `durable truth for run ${verdict.runId}: ${verdict.agrees ? 'AGREES' : 'DISAGREES'}`, + ` live : ${describe(verdict.live)}`, + ` history : ${describe(verdict.history)}`, + ` after reconcile : ${describe(verdict.afterReconcile)}`, + ` checkpoint : ${String(verdict.checkpointStatus)} (${verdict.checkpointSource})`, + ` terminals in log: ${verdict.durableTerminalCount}`, + ...verdict.disagreements.map((d) => ` ✖ ${d}`), + ].join('\n'); +} diff --git a/packages/core/src/engine/engine.ts b/packages/core/src/engine/engine.ts index bed05d9d..f5062506 100644 --- a/packages/core/src/engine/engine.ts +++ b/packages/core/src/engine/engine.ts @@ -68,6 +68,11 @@ import { type BudgetAdmission, } from './budget-governor.js'; import type { CheckpointPendingMediaJob, CheckpointState } from './checkpoint.js'; +import { + LedgerDurabilityError, + MoneyDurability, + isLedgerDurabilityError, +} from './money-durability.js'; import type { AbortControllerLike, ExecutionHost } from './execution-host.js'; import type { GateRequest, @@ -347,6 +352,8 @@ class RunExecution { #runTimeoutDisarm: (() => void) | undefined; /** The pre-egress budget governor, when a workflow `budget` is configured (ADR-0028, 1.AC). */ readonly #budgetGovernor: BudgetGovernor | undefined; + /** The money-durability barrier for BOTH chains — always present, cap or no cap (ADR-0077 §5). */ + readonly #money: MoneyDurability; /** Vertices whose budget gate was APPROVED — their next re-dispatch (and all its node-retry attempts) skips * the pre-egress check so the deferred LLM call actually issues (H3). Consumed once per dispatch in * `#dispatch` and cleared on `#settle`. */ @@ -409,6 +416,48 @@ class RunExecution { this.#maskedInputs = maskInputs(params.inputs, secretNames); this.#maxTokensEstimate = params.maxTokensEstimate ?? DEFAULT_MAX_TOKENS_ESTIMATE; this.#resolvePrice = params.resolvePrice; + // UNCONDITIONAL, unlike the governor below (ADR-0077 §5). The conservative half is inherently + // budget-scoped — no cap, nothing to reserve — but a run without a budget still spends real money, so a + // ledger that only existed alongside a governor would silently skip every unbudgeted run. It fronts the + // join for BOTH chains, so `flushConservative` is wired to the governor once that exists. + this.#money = new MoneyDurability({ + emit: async (draft, cumulativeCostMicrocents) => { + // **The observe half, and it has to be here rather than in `MoneyDurability`.** `#emitDurable` is + // TOTAL for store faults: it absorbs a `persistEvent` rejection into `#failure` and RESOLVES. So the + // ledger's own `.catch` never fires for the failure mode it exists to catch, and a barrier that only + // awaited would sail straight past a run whose money write did not land — exactly the trap ADR-0076 + // §1 named and ADR-0077 kept. Comparing `#failure` across the await is what turns the absorbed fault + // back into something the barrier can throw. It can over-trigger when a SIBLING fails in the same + // window; that direction is fail-closed and correct at a money barrier. + const failureBefore = this.#failure; + await this.#emitDurable({ + ...draft, + type: 'cost:attempt_settled', + runId: this.runId, + // The total CAPTURED AT `record()` TIME, passed in — deliberately not a fresh read of + // `#cumulativeCostMicrocents` here. `#nodeEmit`'s `cost:updated` arm folds the charge into the + // counter and the turn core records strictly after that, so the captured value satisfies + // `refineCostAttemptSettled`'s "cumulative already includes this charge" by construction. Reading it + // HERE would not: this callback is chained behind the previous write's `persistEvent`, so under a + // `fan_out` — concurrent nodes sharing one chain — it can run after several more attempts have + // settled and report their money as this attempt's running total. + cumulativeCostMicrocents, + }); + if (this.#failure !== failureBefore) { + // Typed, not a bare `Error` (error-handling.md). `#emitDurable` discards the store error in its own + // catch, so there is no `cause` left to preserve — the run's `#failure` carries the user-facing + // reason instead, and this class exists to keep the node attribution that would otherwise be lost + // when the chain flattens a `preAttempt` throw. + throw new LedgerDurabilityError( + new Error('the run failed while this realized charge was being made durable'), + draft.nodeId, + ); + } + }, + ...(params.plan.budget === undefined + ? {} + : { flushConservative: async () => this.#budgetGovernor?.flushCommitments() }), + }); if (params.plan.budget !== undefined) { this.#budgetGovernor = new BudgetGovernor({ budget: params.plan.budget, @@ -1125,7 +1174,7 @@ class RunExecution { // the next pre-egress check; this covers the node boundary, which a crash could otherwise land inside with // a possibly-billed call recorded nowhere. Awaited (not fire-and-forget) so a failed write fails the node // loudly; the conservative amount keeps consuming capacity either way. - await this.#flushBudgetCommitments(vertex.id); + await this.#joinMoneyDurability(vertex.id); const willRetry = outcome.kind === 'failed' && !this.#settled && @@ -1225,11 +1274,41 @@ class RunExecution { signal: this.#abort.signal, attemptNumber, ...(preEgress === undefined ? {} : { preEgress }), + // Unconditional, unlike `preEgress` above — which `budgetApproved` deliberately drops for an approved + // re-dispatch. The ledger must not be dropped with it: an approved node is the one the user just + // authorised MORE money on, so it is the last place to stop recording what that money was. + money: this.#money.turnPort(() => this.#cumulativeCostMicrocents), }; // After the executor completes, an `output` node with `save_to` writes its produced media to the // host (1.AF/D16). A write failure FAILS the node (→ run:failed) — save_to is a real deliverable. return await this.#applySaveTo(vertex, await this.#executor.execute(ctx)); - } catch { + } catch (error) { + // A money-durability failure is NOT an anonymous handler throw. Barriers B1 and B2 (ADR-0077) both sit + // INSIDE the turn, and `throwMappedChainError` has two arms whose only job is to keep the class and its + // owning `nodeId` intact on the way out — under a `fan_out` the broken write may be a sibling's, and + // the message that names the remedy differs between the estimate and realized halves. Then + // `turnOutcomeForError` re-throws anything it cannot classify and the generic arm below reported `the + // node handler threw an unexpected error`: preserved for exactly one frame, then discarded. B3 two + // lines later cannot repair it either — `join()` surfaces a retained failure exactly ONCE, and the + // in-turn barrier already consumed it. + // + // **When this arm is actually reached, stated because measuring it corrected a claim.** On the ordinary + // path it is NOT: `#emitDurable` absorbs the store fault, sets `#failure` and aborts, so the turn ends + // at `throwIfAborted` and the node classifies as cancelled before any barrier throws. It is reached + // where the abort does not fire — `#emitDurable` skips it when `#failure` is already set by a sibling — + // which is precisely ADR-0077's stated required regression, still unbuilt. The arm is here because the + // flattening is wrong whenever it does happen, not because a test drives it today. + if (isLedgerDurabilityError(error) || error instanceof CommitmentDurabilityError) { + this.#failMoneyDurability(error, vertex.id); + return { + kind: 'failed', + error: { + code: 'internal', + message: this.#failure?.error.message ?? 'a money write could not be made durable', + retryable: false, + }, + }; + } // The catch-all: any uncaught throw from a node handler maps to a single internal failure // (a tool handler classifies its own failures as tool_failed; a sandbox throw as sandbox_error). return { @@ -2093,7 +2172,12 @@ class RunExecution { } /** - * Await the budget governor's conservative-commitment barrier at a node boundary (ADR-0074 §2). + * Await BOTH money chains at a node boundary — the conservative commitments (ADR-0074 §2) and the realized + * ledger (ADR-0077 §4). Named for the join rather than for the commitments it once awaited alone: since + * ADR-0077 this is the single barrier fronting `MoneyDurability`, and `#flushBudgetCommitments` read as + * though the ledger were still someone else's to await. (The host-supplied + * `AgentSessionDeps.flushBudgetCommitments` keeps that name and is correct — the SESSION path has no + * realized ledger to join, because its per-attempt increment is already written synchronously.) * * **The await is the substance.** `#emitDurable` resolves only after `persistEvent` has settled (its `await * settled` at the end), so waiting here means a commitment made inside the attempt is on disk — or has already @@ -2106,31 +2190,55 @@ class RunExecution { * write. Kept here so the two surfaces cannot diverge in what a durability failure means: never a released * reservation, always a loud failure. */ - async #flushBudgetCommitments(nodeId: string): Promise { - const governor = this.#budgetGovernor; - if (governor === undefined) return; + async #joinMoneyDurability(nodeId: string): Promise { + // **Barrier B3 (ADR-0077)** — and it is now the SINGLE join for both money chains. The old + // `if (governor === undefined) return;` is gone: it was one of §5's three barrier holes, because a run + // without a budget has no conservative commitments but does have a realized ledger, and returning early + // left that ledger started and never joined — exactly the fire-and-forget state ADR-0076 exists to remove. + // `MoneyDurability.join()` fronts both, so there is no supported way to await half the money. try { - await governor.flushCommitments(); + await this.#money.join(); } catch (error) { - // Attribute it to the node whose commitment actually failed, not to whichever node reached this barrier - // first — under a `fan_out` both branches await the same chain link, so the first to flush may have made no - // commitment at all. Falls back to this node when the error carries no owner. - const owner = error instanceof CommitmentDurabilityError ? (error.nodeId ?? nodeId) : nodeId; - this.#failure ??= { - nodeId: owner, - error: { - code: 'internal', - // The cause is deliberately NOT in the message: a durable-write failure can carry a filesystem path, and - // a user-facing `run:failed` message must not. It survives on `CommitmentDurabilityError.cause` for a - // host that narrows on the class — which is the only carrier, since this path has no store to log it. - message: 'a conservative budget commitment could not be made durable', - retryable: false, - }, - }; - this.#abort.abort(); + this.#failMoneyDurability(error, nodeId); } } + /** + * Record a money-durability failure against the node whose write actually broke, and abort. + * + * Shared by B3's own catch and by `#runAttempt`'s catch-all, and the second caller is why it is a method: + * barriers B1 and B2 live INSIDE the turn, so their throw arrives as a bare `LedgerDurabilityError` / + * `CommitmentDurabilityError` that the catch-all used to flatten. See the note at that call site for when + * that path is actually reached — it is narrower than it looks. + * + * `??=` throughout: a sibling's already-recorded root cause always wins, which is also why this is usually + * a no-op on the ordinary path (`#emitDurable` has already set `#failure` from the same fault). + */ + #failMoneyDurability(error: unknown, nodeId: string): void { + // Attribute it to the node whose write actually failed, not to whichever node reached this barrier + // first — under a `fan_out` both branches await the same chain link, so the first to flush may have made no + // commitment at all. Falls back to this node when the error carries no owner. + const ledger = isLedgerDurabilityError(error); + const owner = + error instanceof CommitmentDurabilityError || ledger ? (error.nodeId ?? nodeId) : nodeId; + this.#failure ??= { + nodeId: owner, + error: { + code: 'internal', + // The cause is deliberately NOT in the message: a durable-write failure can carry a filesystem path, and + // a user-facing `run:failed` message must not. It survives on the error's `cause` for a host that + // narrows on the class — which is the only carrier, since this path has no store to log it. The two + // messages are distinct because the remedies differ: an estimate that could not be recorded leaves the + // cap conservative, while a realized charge that could not be recorded leaves it UNDERSTATED. + message: ledger + ? 'a realized provider charge could not be made durable' + : 'a conservative budget commitment could not be made durable', + retryable: false, + }, + }; + this.#abort.abort(); + } + async #emitDurable(draft: RunEventDraft): Promise { // Persist the boundary/terminal event, then deliver (ADR-0036 persist-before-deliver, so a crash // can never re-run a completed node or lose its output). This method is **total for store faults** (the diff --git a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts index 7d71ba40..722058a1 100644 --- a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts +++ b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts @@ -50,6 +50,7 @@ import type { ToolDef as CoreToolDef, ToolRegistry, ToolResultPart } from '../to import { markUntrusted } from '../tools/untrusted.js'; import { reconstructCheckpointState } from './checkpoint.js'; import { WorkflowEngine } from './engine.js'; +import { checkDurableTruth, formatDurableTruth } from './durable-truth.js'; import { createInMemoryHost, InMemoryRunStore } from './execution-host.js'; import { createStandardNodeExecutor } from './node-handlers/dispatcher.js'; import type { RunHandle } from './run-handle.js'; @@ -653,6 +654,10 @@ function buildEngine( resolveProvider: (id: ProviderId) => LlmProvider | undefined, resolveMediaSurface?: (model: string) => 'chat' | 'generative' | undefined, resolvePrice?: PricingOverlay, + // Overridable ONLY so the ADR-0077 barrier tests can observe tool dispatch directly. B2's whole claim is + // "the ledger write lands before the run mutates the world", and the only honest witness to that is the + // dispatch itself — not an event, which would just be re-reading the thing under test. + registry: ToolRegistry = echoRegistry, ): WorkflowEngine { return new WorkflowEngine({ host, @@ -663,7 +668,7 @@ function buildEngine( resolveProvider, ...(resolveMediaSurface === undefined ? {} : { resolveMediaSurface }), ...(resolvePrice === undefined ? {} : { resolvePrice }), - registry: echoRegistry, + registry, tools: [echoToolDef], keyFor: () => 'k', sleep: () => Promise.resolve(), @@ -1059,6 +1064,309 @@ describe('M2 — end-to-end Node harness (1.U)', () => { ); }); + // --- The durable-truth oracle (CR-91) -------------------------------------------------------------- + // + // Everything above this point asserts on the LIVE stream. That is one view, taken in-process while the + // engine that produced it is still alive, and it cannot see the failure class this phase exists for: a + // caller told `completed` while the log says `failed`, a restart reconciling to a different terminal, or a + // checkpoint fold that disagrees with the terminal and would make a resume do the wrong work. + // + // `checkDurableTruth` compares four views. These tests apply it to the three terminals a run can reach, on + // real engine runs rather than synthetic logs — `durable-truth.test.ts` covers the oracle's own detection + // logic, this covers the engine actually satisfying it. + + /** Run the oracle over a finished run, restarting a FRESH engine over the same store to reconcile. */ + async function assertDurableTruth( + host: Host, + store: InMemoryRunStore, + events: readonly RunEvent[], + resolveProvider: (id: ProviderId) => LlmProvider | undefined, + ) { + const runId = events[0]?.runId; + if (runId === undefined) expect.unreachable('no run:started'); + const verdict = await checkDurableTruth({ + runId, + live: events.at(-1), + eventsFor: (id) => store.eventsFor(id), + // A fresh engine over the SAME store, which is what a restarted process is. + reconcile: () => buildEngine(host, resolveProvider).reconcile(), + }); + expect(verdict.agrees, formatDurableTruth(verdict)).toBe(true); + return verdict; + } + + it('durable truth: a COMPLETED run survives a restart unchanged (CR-91)', async () => { + const store = new InMemoryRunStore(); + const host = createInMemoryHost({ store }); + const provider = scriptedProvider([toolUseTurn('c1'), textTurn('a summary')]); + const { events } = await drive( + buildEngine(host, () => provider).start({ workflow: HAPPY_PATH, inputs: INPUTS }), + host, + ); + + expect(events.at(-1)?.type).toBe('run:completed'); + const verdict = await assertDurableTruth(host, store, events, () => provider); + // A run that already closed must give reconcile() nothing to do — repairing a run that DIED without a + // terminal is its job; touching one that landed is the defect. + expect(verdict.reconciledCount).toBe(0); + expect(verdict.durableTerminalCount).toBe(1); + }); + + it('durable truth: a FAILED run agrees across all four views (CR-91)', async () => { + const store = new InMemoryRunStore(); + const host = createInMemoryHost({ store }); + // A fatal provider error — no retry budget on HAPPY_PATH, so the node fails and the run follows. + const provider = scriptedProvider([ + [ + { + type: 'error', + error: { kind: 'auth', retryable: false, provider: 'anthropic', message: 'nope' }, + }, + ], + ]); + const { events } = await drive( + buildEngine(host, () => provider).start({ workflow: HAPPY_PATH, inputs: INPUTS }), + host, + ); + + expect(events.at(-1)?.type).toBe('run:failed'); + const verdict = await assertDurableTruth(host, store, events, () => provider); + expect(verdict.checkpointStatus).toBe('failed'); + // The concrete code FIRST. `history === live` alone is satisfied by `undefined === undefined`, so it holds + // just as well for a verdict that read no error at all from either side — including one where a refactor + // stopped populating `errorCode`. Pinning the value is what makes the equality mean something. + expect(verdict.history?.errorCode).toBe('provider_auth'); + expect(verdict.history?.errorCode).toBe(verdict.live?.errorCode); + }); + + it('durable truth: a CANCELLED run agrees across all four views (CR-91)', async () => { + const store = new InMemoryRunStore(); + const host = createInMemoryHost({ store }); + const provider = scriptedProvider([toolUseTurn('c1'), textTurn('a summary')]); + const engine = buildEngine(host, () => provider); + const handle = engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }); + const events: RunEvent[] = []; + for await (const event of handle.events) { + events.push(event); + // `handle.cancel()`, not `engine.cancel(runId)`: the latter THROWS `run_already_terminal` once the run + // settled, and this stream is a buffered push/pull adapter — the consumer's position relative to engine + // progress is a microtask-interleaving property, not a guarantee. `work` is a real node that finishes on + // its own, so a lagging consumer would throw inside this `for await` instead of failing an assertion. + if (event.type === 'node:started' && event.nodeId === 'work') handle.cancel(); + } + + expect(events.at(-1)?.type).toBe('run:cancelled'); + await assertDurableTruth(host, store, events, () => provider); + }); + + // --- The realized-cost ledger's barriers (ADR-0076 / ADR-0077) ------------------------------------ + // + // Every test here runs on HAPPY_PATH, which declares NO `budget` — deliberately. ADR-0077 §5's whole + // finding is that the barriers used to live behind a governor that only exists when a workflow declares a + // cap, so a budgeted fixture would pass even with all three holes open. An unbudgeted run is the regression. + + /** A registry that records every dispatch, so a barrier can be witnessed by the side effect it gates. */ + function spyingRegistry(): { registry: ToolRegistry; calls: string[] } { + const calls: string[] = []; + return { + calls, + registry: { + has: (id) => echoRegistry.has(id), + list: () => echoRegistry.list(), + dispatch: (call, ctx) => { + calls.push(call.name); + return echoRegistry.dispatch(call, ctx); + }, + }, + }; + } + + /** + * Drain the queue to QUIESCENCE — microtasks and a macrotask turn, repeatedly. + * + * Load-bearing, and its absence made the first version of every test below hollow. `ledger.blocked()` flips + * INSIDE `persistEvent`, several turns before the run would have reached the action a barrier guards. Assert + * at that instant and the dispatch is still sitting in the queue, so the assertion passes whether or not the + * barrier exists. Draining first is what makes deleting a join go red. + */ + async function settle(): Promise { + for (let i = 0; i < 50; i += 1) { + for (let j = 0; j < 200; j += 1) await Promise.resolve(); + await new Promise((resolve) => setTimeout(resolve, 0)); + } + } + + /** A store that blocks `cost:attempt_settled`'s write until released, recording persist order. */ + function blockingLedgerStore() { + const inner = new InMemoryRunStore(); + const persistOrder: string[] = []; + let resolveWrite: (() => void) | undefined; + return { + persistOrder, + blocked: () => resolveWrite !== undefined, + release: () => resolveWrite?.(), + store: { + resolveWorkflowId: (slug: string) => inner.resolveWorkflowId(slug), + listInterruptedRuns: () => inner.listInterruptedRuns(), + eventsFor: (runId: string) => inner.eventsFor(runId), + persistEvent: async (event: RunEvent): Promise => { + if (event.type === 'cost:attempt_settled' && resolveWrite === undefined) { + await new Promise((resolve) => { + resolveWrite = resolve; + }); + } + // `type:nodeId`, not just the type. HAPPY_PATH's `input` node completes long before the agent + // spends anything, so a bare `node:completed` check would fire on the wrong node and pass for the + // wrong reason — which it did, on the first run of this test. + persistOrder.push( + 'nodeId' in event && typeof event.nodeId === 'string' + ? `${event.type}:${event.nodeId}` + : event.type, + ); + await inner.persistEvent(event); + }, + }, + }; + } + + it('ledger B2: a TOOL is not dispatched until the attempt`s charge is durable (ADR-0077)', async () => { + // The barrier ADR-0077 adds beyond ADR-0074 §2's pair, and the one that makes this ledger EXTEND the + // guarantee rather than repeat it: realized spend must be durable before the run mutates the world, not + // merely before it spends again. Witnessed on the dispatch itself. + const { registry, calls } = spyingRegistry(); + const ledger = blockingLedgerStore(); + const provider = scriptedProvider([toolUseTurn('t1'), textTurn('done')]); + const host = createInMemoryHost({ store: ledger.store }); + const engine = buildEngine(host, () => provider, undefined, undefined, registry); + const handle = engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }); + + // Spin until the ledger write blocks. HAPPY_PATH runs input → agent → output and the charge only settles + // after a full successful attempt, so this needs more turns of the microtask queue than §2's fixture, + // whose commitment fires on a usage-less stream almost immediately. + for (let i = 0; i < 500 && !ledger.blocked(); i += 1) await Promise.resolve(); + expect(ledger.blocked()).toBe(true); + await settle(); // give the guarded action every chance to run before asserting it did not + // THE assertion: the model asked for a tool, the charge is not durable, so nothing ran. + expect(calls).toEqual([]); + + ledger.release(); + const events: RunEvent[] = []; + for await (const event of handle.events) events.push(event); + expect(calls).toEqual(['echo']); // it did run, once the write landed + expect(events.at(-1)?.type).toBe('run:completed'); + }); + + it('ledger B3: the node terminal is not durable until the attempt`s charge is (ADR-0077)', async () => { + const ledger = blockingLedgerStore(); + const provider = scriptedProvider([textTurn('done')]); + const host = createInMemoryHost({ store: ledger.store }); + const engine = buildEngine(host, () => provider); + const handle = engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }); + + // Spin until the ledger write blocks. HAPPY_PATH runs input → agent → output and the charge only settles + // after a full successful attempt, so this needs more turns of the microtask queue than §2's fixture, + // whose commitment fires on a usage-less stream almost immediately. + for (let i = 0; i < 500 && !ledger.blocked(); i += 1) await Promise.resolve(); + expect(ledger.blocked()).toBe(true); + await settle(); // give the guarded action every chance to run before asserting it did not + // The AGENT node's own terminal must not be durable — a crash here records progress the money log lacks. + // (`in`'s terminal is legitimately already there; it spent nothing.) + expect(ledger.persistOrder).not.toContain('node:completed:work'); + + ledger.release(); + const events: RunEvent[] = []; + for await (const event of handle.events) events.push(event); + // PRESENCE first, then order. A bare `indexOf(a) < indexOf(b)` passes vacuously when `a` is missing — + // `-1` is less than everything — so a run that never settled a charge at all would satisfy the ordering + // it is supposed to prove. + expect(ledger.persistOrder).toContain('cost:attempt_settled:work'); + expect(ledger.persistOrder).toContain('node:completed:work'); + expect(ledger.persistOrder.indexOf('cost:attempt_settled:work')).toBeLessThan( + ledger.persistOrder.indexOf('node:completed:work'), + ); + }); + + it('ledger: a REJECTED write dispatches no tool — the observe half, not just the await (ADR-0077)', async () => { + // **What stops the dispatch here is the ENGINE'S ABORT, not the barrier — stated because the obvious + // reading of this test is wrong.** `#emitDurable` absorbs the rejection, sets `#failure` and calls + // `#abort.abort()`, and `dispatchToolUseTurn`'s `throwIfAborted` ends the turn. Deleting the observe half + // OR B2 leaves this test green, so it does NOT pin either. + // + // What it does pin is still worth having: a rejected ledger write ends the run without mutating the + // world, and the terminal message stays secret-free. ADR-0077's stated required regression — the barrier + // itself refusing — needs a case that DEFEATS the abort: reject the ledger write while a concurrent + // sibling has already set `#failure`, so `#emitDurable`'s `this.#failure === undefined` guard skips the + // abort and only the barrier is left to stop the dispatch. Not built; named so it is not lost. + const { registry, calls } = spyingRegistry(); + const inner = new InMemoryRunStore(); + const store = { + resolveWorkflowId: (slug: string) => inner.resolveWorkflowId(slug), + listInterruptedRuns: () => inner.listInterruptedRuns(), + eventsFor: (runId: string) => inner.eventsFor(runId), + persistEvent: async (event: RunEvent): Promise => { + // Reject ASYNCHRONOUSLY, after a tick — a synchronous throw would let the engine's own abort win the + // race and make this pass without any barrier at all. + if (event.type === 'cost:attempt_settled') { + await Promise.resolve(); + throw new Error('ledger write failed'); + } + await inner.persistEvent(event); + }, + }; + const provider = scriptedProvider([toolUseTurn('t1'), textTurn('done')]); + const host = createInMemoryHost({ store }); + const engine = buildEngine(host, () => provider, undefined, undefined, registry); + const handle = engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }); + + const events: RunEvent[] = []; + for await (const event of handle.events) events.push(event); + + expect(calls).toEqual([]); // the world was not mutated on an unrecorded charge + const terminal = events.at(-1); + expect(terminal?.type).toBe('run:failed'); + if (terminal?.type === 'run:failed') { + // `#emitDurable`'s own message, not the barrier's — and that is correct, not a miss. It absorbs the + // rejection and sets `#failure` FIRST, and `#failure ??=` keeps the first cause. The barrier's job here + // is to stop the dispatch, not to rename the failure; its own message reaches the terminal only when a + // ledger write fails somewhere `#emitDurable` does not already absorb. + expect(terminal.error.message).toBe('a durable run-event write failed'); + // Secret-free either way: a store error can carry a filesystem path and must never reach the user. + expect(terminal.error.message).not.toContain('ledger write failed'); + } + // And the NODE terminal says `the run was cancelled`, NOT the barrier's diagnosis — measured, after an + // assertion here claimed otherwise. It is the same fact as the paragraph above: the abort wins, the turn + // ends at `throwIfAborted`, and B2's `LedgerDurabilityError` is never the error that classifies this node. + // `#runAttempt` DOES have an arm that keeps that class intact (it used to flatten it to `the node handler + // threw an unexpected error`), but reaching it needs the same abort-defeating fixture named above. + const nodeFailed = events.find((e) => e.type === 'node:failed'); + expect(nodeFailed?.type === 'node:failed' ? nodeFailed.error.message : undefined).toBe( + 'the run was cancelled', + ); + }); + + it('ledger: an UNBUDGETED run records its realized spend at all (ADR-0077 §5)', async () => { + // The plainest regression for §5's finding. Before it, the ledger's barriers hung off `BudgetGovernor`, + // which the engine builds only when a workflow declares a `budget` — so this run, which spends real + // money, would have started a write nobody ever joined. + const inner = new InMemoryRunStore(); + const provider = scriptedProvider([textTurn('done')]); + const host = createInMemoryHost({ store: inner }); + const engine = buildEngine(host, () => provider); + const handle = engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }); + const { events } = await drive(handle, host); + + expect(events.at(-1)?.type).toBe('run:completed'); + const persisted = inner.eventsFor(events[0]?.runId ?? ''); + const ledgerRows = persisted.filter((e) => e.type === 'cost:attempt_settled'); + expect(ledgerRows).toHaveLength(1); + const row = ledgerRows[0]; + if (row?.type !== 'cost:attempt_settled') expect.unreachable('missing ledger row'); + expect(row.attemptNumber).toBe(1); + expect(row.priced).toBe(true); // the built-in pricing table knows this model, so the charge is real + // The cumulative must already include this charge, which is what the schema refinement pins. + expect(row.cumulativeCostMicrocents).toBeGreaterThanOrEqual(row.costMicrocents); + }); + it('conservative commitment: a RESUMED run still cannot spend past the cap it committed against (ADR-0074 §2)', async () => { // The half the previous test cannot reach: `#seedFromCheckpoint` must hand the folded total to the resumed // governor. Remove that one call and the fold still computes the right number, the log still contains the diff --git a/packages/core/src/engine/money-durability.test.ts b/packages/core/src/engine/money-durability.test.ts new file mode 100644 index 00000000..6ec3f4e7 --- /dev/null +++ b/packages/core/src/engine/money-durability.test.ts @@ -0,0 +1,213 @@ +import { describe, expect, it, vi } from 'vitest'; + +import { + MoneyDurability, + isLedgerDurabilityError, + type SettledAttemptDraft, +} from './money-durability.js'; + +const draft = (nodeId = 'n1', costMicrocents = 400): SettledAttemptDraft => ({ + nodeId, + model: 'claude-opus-4-8', + attemptNumber: 1, + inputTokens: 10, + outputTokens: 5, + costMicrocents, + priced: true, +}); + +describe('MoneyDurability', () => { + it('SERIALIZES writes chained in the same tick', async () => { + // Chaining rather than a set is what stops two attempts settling in one tick from interleaving their + // persists — the same reason ADR-0074 §2 chains its estimates. + const order: string[] = []; + let releaseFirst: (() => void) | undefined; + const money = new MoneyDurability({ + emit: async (d) => { + order.push(`start:${d.nodeId}`); + if (d.nodeId === 'a') { + await new Promise((resolve) => { + releaseFirst = resolve; + }); + } + order.push(`end:${d.nodeId}`); + }, + }); + + money.record(draft('a'), 400); + money.record(draft('b'), 400); + await vi.waitFor(() => { + expect(releaseFirst).toBeDefined(); + }); + expect(order).toEqual(['start:a']); // b has not started while a is in flight + releaseFirst?.(); + await money.join(); + expect(order).toEqual(['start:a', 'end:a', 'start:b', 'end:b']); + }); + + it('re-throws a failed write at the barrier, ONCE, and stays sticky', async () => { + // Nobody awaits the write at its call site, so the rejection has to be retained and surfaced here or it + // is unhandled. Surfaced once — a later join must not re-report the same broken write as new — while the + // sticky flag keeps the barrier meaningful for the rest of the run. + const money = new MoneyDurability({ + emit: () => Promise.reject(new Error('disk full')), + }); + money.record(draft('a'), 400); + + await expect(money.join()).rejects.toSatisfy(isLedgerDurabilityError); + expect(money.durabilityBroken).toBe(true); + await expect(money.join()).resolves.toBeUndefined(); // not re-reported + expect(money.durabilityBroken).toBe(true); // but still broken + }); + + it('names the owning node and keeps the cause OFF the message', async () => { + // A durable-write failure can carry a filesystem path, and this message reaches a user-facing + // `run:failed`. The cause rides `cause` for a host that narrows on the class. + const cause = new Error('ENOENT: /Users/someone/.relavium/history.db'); + const money = new MoneyDurability({ emit: () => Promise.reject(cause) }); + money.record(draft('the-node'), 400); + + await money.join().then( + () => expect.unreachable('join must throw'), + (error: unknown) => { + if (!isLedgerDurabilityError(error)) expect.unreachable('wrong error class'); + expect(error.nodeId).toBe('the-node'); + expect(error.cause).toBe(cause); + expect(error.message).not.toContain('.relavium'); + }, + ); + }); + + it('survives a SYNCHRONOUSLY throwing sink instead of bricking the chain', async () => { + // A `better-sqlite3`-backed store throws synchronously. A bare `emit(draft)` inside the `.then` would + // escape before `.catch`/`.finally` attach: the pending count leaks at >= 1 forever and the chain is left + // permanently rejected, so every later join throws with no way to clear it. + const seen: string[] = []; + let boom = true; + const money = new MoneyDurability({ + emit: (d) => { + if (boom) throw new Error('sync boom'); + seen.push(d.nodeId); + }, + }); + money.record(draft('a'), 400); + await expect(money.join()).rejects.toSatisfy(isLedgerDurabilityError); + + // The recovery half, on the SAME instance — and it has to be. What a synchronous throw could brick is + // THIS chain's tail and THIS pending count; a second, healthy instance shares neither, so asserting on + // one proved only that a fresh object works. Here the later write must reach the sink and the join must + // resolve, which is false if `#inFlight` was left permanently rejected or `#pending` leaked at >= 1. + boom = false; + money.record(draft('b'), 400); + await expect(money.join()).resolves.toBeUndefined(); + expect(seen).toEqual(['b']); + expect(money.durabilityBroken).toBe(true); // sticky, as designed — usable is not the same as healthy + }); + + it('joins the CONSERVATIVE chain too — one join, not two (ADR-0077 §4)', async () => { + const flushConservative = vi.fn(async () => {}); + const money = new MoneyDurability({ emit: () => {}, flushConservative }); + + await money.join(); + expect(flushConservative).toHaveBeenCalledTimes(1); + }); + + it('surfaces a CONSERVATIVE failure through the same join', async () => { + // The estimate half throws its own retained failure from `flushCommitments`. A caller awaiting the single + // join must see it — otherwise "one join" would silently swallow half the money's failures. + const money = new MoneyDurability({ + emit: () => {}, + flushConservative: () => Promise.reject(new Error('commitment write failed')), + }); + money.record(draft('a'), 400); + + await expect(money.join()).rejects.toThrow('commitment write failed'); + }); + + it('is a no-op join when nothing was ever recorded — an unbudgeted, zero-egress run', async () => { + const money = new MoneyDurability({ emit: () => expect.unreachable('nothing to emit') }); + await expect(money.join()).resolves.toBeUndefined(); + expect(money.durabilityBroken).toBe(false); + }); + + it('exposes a turn port that records and joins without leaking the class', async () => { + const seen: SettledAttemptDraft[] = []; + const money = new MoneyDurability({ + emit: (d) => { + seen.push(d); + }, + }); + // Destructured, because the turn core passes `params.money` around by value. + const { record, join } = money.turnPort(() => 0); + record(draft('a', 700)); + await join(); + expect(seen).toEqual([expect.objectContaining({ nodeId: 'a', costMicrocents: 700 })]); + }); + + it('captures the run-wide total at RECORD time, not when the chained write runs', async () => { + // The write is chained behind the previous one's `persistEvent` — real I/O. Reading the engine's live + // counter inside `emit` therefore yields the total including every attempt that settled DURING that wait, + // which under a `fan_out` (concurrent nodes sharing one chain) is a different number. Here the counter + // advances while `a` is in flight, and `b` must still report the total as of its own settle. + const stamped: number[] = []; + let releaseFirst: (() => void) | undefined; + const money = new MoneyDurability({ + emit: async (d, cumulative) => { + stamped.push(cumulative); + if (d.nodeId === 'a') { + await new Promise((resolve) => { + releaseFirst = resolve; + }); + } + }, + }); + let counter = 100; + const { record, join } = money.turnPort(() => counter); + + record(draft('a')); // settles at 100 + await vi.waitFor(() => { + expect(releaseFirst).toBeDefined(); + }); + counter = 250; + record(draft('b')); // settles at 250 — queued behind `a`, which is still blocked + counter = 900; // a third node settles while `a`'s write is still in flight + releaseFirst?.(); + await join(); + + // Reading the counter inside `emit` would make this `[100, 900]` — `b`'s row claiming a total that + // includes money `b` never spent, from an attempt that had not settled when `b` did. + expect(stamped).toEqual([100, 250]); + }); + + it('reports the CONSERVATIVE failure first, then the retained ledger failure on the next join', async () => { + // Both halves can be broken at once, and `join()` surfaces one failure per call. The order is not + // arbitrary: the conservative flush is awaited BEFORE the retained ledger failure is thrown, so a caller + // sees the estimate error first — and the ledger error must still be there afterwards rather than being + // dropped by the throw that overtook it. + // + // The fixture rejects ONCE on purpose, because that is `BudgetGovernor.flushCommitments`'s own contract + // (a retained failure is surfaced once, then cleared) — and the coupling is load-bearing rather than + // incidental: a conservative half that rejected on EVERY call would mask the retained ledger failure for + // the rest of the run, since `join()` never reaches the throw below. + let conservativeBroken = true; + const money = new MoneyDurability({ + emit: () => Promise.reject(new Error('ledger disk full')), + flushConservative: () => { + if (!conservativeBroken) return Promise.resolve(); + conservativeBroken = false; + return Promise.reject(new Error('commitment write failed')); + }, + }); + money.record(draft('the-node'), 400); + + await expect(money.join()).rejects.toThrow('commitment write failed'); + await money.join().then( + () => expect.unreachable('the retained ledger failure must still surface'), + (error: unknown) => { + if (!isLedgerDurabilityError(error)) expect.unreachable('wrong error class'); + expect(error.nodeId).toBe('the-node'); + }, + ); + expect(money.durabilityBroken).toBe(true); + }); +}); diff --git a/packages/core/src/engine/money-durability.ts b/packages/core/src/engine/money-durability.ts new file mode 100644 index 00000000..313ad67f --- /dev/null +++ b/packages/core/src/engine/money-durability.ts @@ -0,0 +1,186 @@ +/** + * The run's money-durability barrier — the single place a caller waits for BOTH kinds of money write + * ([ADR-0077](../../../../docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md)). + * + * Two events record money and both are emitted from the SAME synchronous `onAttempt` callback, a few lines + * apart in `agent-turn.ts`: `budget:estimate_committed` (a conservative ESTIMATE, ADR-0074 §2) and + * `cost:attempt_settled` (the REALIZED charge, ADR-0076). Neither can be awaited at its emit site — the seam's + * observer is `(record: AttemptRecord) => void` — so each STARTS its durable write there and something joins it + * later. This owns the realized chain and fronts the join for both. + * + * **Why it is not part of `BudgetGovernor`, which is where every signpost points.** The governor owns the + * conservative chain and `node-executor.ts`'s comment literally says "add it to the governor's emit type + * instead". But the engine constructs a governor only when the workflow declares a `budget`, and the realized + * ledger is not optional: an unbudgeted run spends real money and must still record it. Hosting it there would + * have silently skipped every unbudgeted run while passing any test written against a budgeted fixture. + * + * **One join, not two** (ADR-0077 §4). `join()` awaits the realized chain AND the conservative one, and + * reports whichever failure it finds. There is deliberately no public way to await half the money. + */ + +/** A realized-cost ledger write that did not reach the store. Carries the owning node for attribution. */ +export class LedgerDurabilityError extends Error { + readonly nodeId: string | undefined; + + constructor(cause: unknown, nodeId?: string) { + // Secret-free by construction: the cause can carry a filesystem path, so it rides `cause` and never the + // message — the same posture `CommitmentDurabilityError` takes for its estimate twin. + super('a realized-cost ledger write could not be made durable', { cause }); + this.name = 'LedgerDurabilityError'; + this.nodeId = nodeId; + } +} + +export function isLedgerDurabilityError(error: unknown): error is LedgerDurabilityError { + return error instanceof LedgerDurabilityError; +} + +/** The per-attempt ledger draft, minus the run-wide total the ENGINE stamps (it owns that counter). */ +export interface SettledAttemptDraft { + readonly nodeId: string; + readonly model: string; + readonly attemptNumber: number; + readonly inputTokens: number; + readonly outputTokens: number; + readonly costMicrocents: number; + readonly priced: boolean; +} + +export interface MoneyDurabilityOptions { + /** + * Start the durable write for one settled attempt. The engine wires this to `#emitDurable`. **May throw + * synchronously** — a `better-sqlite3` store does — which is why {@link MoneyDurability.record} never calls + * it bare. + * + * `cumulativeCostMicrocents` is passed IN rather than read here, and that is the whole point of the + * parameter: the writes are chained, so this callback runs an unbounded time after the attempt it describes + * — behind the previous write's `persistEvent`, which is real I/O. Reading the engine's live counter at that + * moment yields the total including every attempt that settled DURING the wait, which under a `fan_out` + * (concurrent nodes sharing one chain) is a different number. The contract says each absolute is "a true + * run-wide total at that instant"; only a value captured at `record()` time is. + */ + readonly emit: ( + draft: SettledAttemptDraft, + cumulativeCostMicrocents: number, + ) => Promise | void; + /** + * Join the CONSERVATIVE chain, when a budget governor exists. Wired to `BudgetGovernor.flushCommitments`, + * which throws its own retained failure. Absent on an unbudgeted run — the realized half still applies. + */ + readonly flushConservative?: () => Promise; +} + +/** + * The port `agent-turn.ts` sees. Narrower than the class on purpose: the turn core may START a write and JOIN + * the barrier, and has no business inspecting or clearing durability state. + */ +export interface TurnMoneyPort { + readonly record: (draft: SettledAttemptDraft) => void; + readonly join: () => Promise; +} + +export class MoneyDurability { + readonly #options: MoneyDurabilityOptions; + + /** + * The realized chain. Chaining (rather than a set) serializes the writes, so two attempts settling in the + * same tick cannot interleave their persists — the same reason ADR-0074 §2 chains its estimates. + */ + #inFlight: Promise = Promise.resolve(); + + /** + * How many ledger writes are outstanding, so the barrier costs NOTHING when there is nothing to wait for. + * Awaiting an already-resolved promise still burns a microtask, and the barriers sit on the hot path; worse + * than the cost, an unconditional await changes observable interleaving that existing tests legitimately + * pin. Copied from `#pendingCommitments` for exactly those reasons. + */ + #pending = 0; + + /** + * A ledger write that FAILED, retained until someone joins the barrier. + * + * Nobody awaits the write at its call site, so without this the rejection would be unhandled. It is + * surfaced ONCE — a later join must not re-report the same broken write as if it were new — while + * {@link #broken} stays set, so the barrier is still entered and a subsequent failure is still reported. + */ + #failure: LedgerDurabilityError | undefined; + + /** Sticky: a run whose ledger has ever failed keeps paying the barrier for the rest of the process. */ + #broken = false; + + constructor(options: MoneyDurabilityOptions) { + this.#options = options; + } + + /** Whether a ledger write has ever failed on this run — the state a caller checks alongside the await. */ + get durabilityBroken(): boolean { + return this.#broken; + } + + /** + * START one settled attempt's durable write. Synchronous by necessity (the caller is a chain observer that + * cannot await) and never awaited here, so a rejection is captured and re-thrown at the next barrier. + */ + record(draft: SettledAttemptDraft, cumulativeCostMicrocents: number): void { + this.#pending += 1; + this.#inFlight = this.#inFlight.then(() => + // `Promise.resolve().then(…)`, NOT a bare `emit(draft)`. A sink that throws SYNCHRONOUSLY — which a + // `better-sqlite3`-backed store does — would escape before `.catch`/`.finally` attach: no typed error, + // `#pending` leaked at >= 1 forever, and `#inFlight` left permanently REJECTED so every later join + // throws with no way to clear it. The governor's estimate twin documents the same brick at length. + // The inner chain therefore ALWAYS resolves, which is why there is no onRejected arm here. + Promise.resolve() + // The captured total, not a fresh read — see `MoneyDurabilityOptions.emit`. It rides the closure so + // the write describes the instant it was recorded, not the instant the chain got round to it. + .then(() => this.#options.emit(draft, cumulativeCostMicrocents)) + .catch((error: unknown) => { + // Keep the FIRST failure: it is the one that broke durability, and later writes may well fail for + // the same reason. + this.#failure ??= new LedgerDurabilityError(error, draft.nodeId); + this.#broken = true; + }) + .finally(() => { + this.#pending -= 1; + }), + ); + } + + /** + * The barrier — ADR-0077's B1 / B2 / B3, all three of them this one call. + * + * Awaits the realized chain and the conservative one, then throws the first retained failure. Both halves + * matter: awaiting alone is NOT a barrier, because the engine's `#emitDurable` is total for store faults — + * it absorbs a `persistEvent` rejection into the run's failure state and RESOLVES — so a caller that only + * awaits proceeds on a run whose write did not land. The throw is how a caller in the turn core, which has + * no access to the engine's own failure state, observes it. + */ + async join(): Promise { + if (this.#pending > 0 || this.#failure !== undefined) { + await this.#inFlight; + } + // The conservative half is joined unconditionally when a governor exists — its own barrier is cheap when + // nothing is outstanding, and skipping it here is how "await the wrong one" would creep back in. + await this.#options.flushConservative?.(); + const failure = this.#failure; + if (failure !== undefined) { + this.#failure = undefined; + throw failure; + } + } + + /** + * The narrow port handed to the turn core. Bound so a destructured `record` still works. + * + * `snapshotCumulative` is read SYNCHRONOUSLY here, inside `record`, because that is the only moment the + * run-wide total is the one this attempt produced: the turn core calls `record` immediately after the + * engine folded this charge into its counter, and the chained write runs later. + */ + turnPort(snapshotCumulative: () => number): TurnMoneyPort { + return { + record: (draft) => { + this.record(draft, snapshotCumulative()); + }, + join: () => this.join(), + }; + } +} diff --git a/packages/core/src/engine/node-executor.ts b/packages/core/src/engine/node-executor.ts index 843354b1..d18cea34 100644 --- a/packages/core/src/engine/node-executor.ts +++ b/packages/core/src/engine/node-executor.ts @@ -191,6 +191,15 @@ export interface NodeExecContext { * attempt is gated before egress. */ readonly preEgress?: import('./agent-turn.js').PreEgressHook; + /** + * The run's money-durability port (ADR-0076 / ADR-0077) — the realized-cost ledger's START hook and the + * single barrier the turn core joins at. + * + * Optional on this seam so a stub executor and the session path stay unchanged, but the run loop supplies it + * UNCONDITIONALLY — unlike {@link preEgress}, which is budget-scoped and is deliberately dropped for an + * approved re-dispatch. A run with no `budget` still spends real money. + */ + readonly money?: import('./money-durability.js').TurnMoneyPort; } /** The injected per-vertex executor. 1.O (`AgentRunner`) and 1.P (node handlers) implement it. */ diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 09cc287a..04866875 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -126,6 +126,16 @@ export type { CheckpointPendingGate, CheckpointPendingMediaJob, } from './engine/checkpoint.js'; +// The durable-truth oracle (CR-91) — a SUPPORTED testing API, exported deliberately. It is engine-pure and +// ships in `dist` either way (the build excludes only `*.test.ts`), and the surfaces that most need it are +// outside this package: `apps/cli`'s regression harness runs against the real `history.db` store, which is +// where CR-92's acceptance ("live, history, resume and reconcile agree") actually has to be certified. +export { checkDurableTruth, formatDurableTruth } from './engine/durable-truth.js'; +export type { + DurableTruthInput, + DurableTruthVerdict, + TerminalView, +} from './engine/durable-truth.js'; export type { ExecutionHost, RunStore, diff --git a/packages/db/src/run-history-store.test.ts b/packages/db/src/run-history-store.test.ts index 56c0d274..a28e55ed 100644 --- a/packages/db/src/run-history-store.test.ts +++ b/packages/db/src/run-history-store.test.ts @@ -1367,4 +1367,244 @@ describe('createRunHistoryReader', () => { expect(reader.loadRun('run-z')?.totalCostMicrocents).toBe(0); }); }); + + describe('the realized-cost ledger (ADR-0076 / #W15-1)', () => { + const ts = TS; + + const sumRunCosts = (runId: string): number => + client.db + .select({ c: runCosts.costMicrocents }) + .from(runCosts) + .where(eq(runCosts.runId, runId)) + .all() + .reduce((total, row) => total + row.c, 0); + + /** One settled attempt. `cumulative` is the run-wide total AFTER it (the schema refinement pins that). */ + const settled = ( + runId: string, + seq: number, + nodeId: string, + cost: number, + cumulative: number, + attemptNumber = 1, + tokens = { input: 10, output: 5 }, + ): RunEvent => + evRun( + runId, + 'cost:attempt_settled', + seq, + { + nodeId, + model: 'claude-opus-4-8', + attemptNumber, + inputTokens: tokens.input, + outputTokens: tokens.output, + costMicrocents: cost, + cumulativeCostMicrocents: cumulative, + priced: true, + }, + ts, + ); + + const startRun = async (runId: string): Promise> => { + const store = storeFor('wf'); + const workflowId = await store.resolveWorkflowId('wf'); + await store.persistEvent( + evRun(runId, 'run:started', 0, { workflowId, inputs: {}, executionMode: 'local' }, ts), + ); + await store.persistEvent( + evRun(runId, 'node:started', 1, { nodeId: 'n1', nodeType: 'agent' }, ts), + ); + return store; + }; + + it('writes a run_costs row AND advances the run total — the pair the invariant rests on', async () => { + const store = await startRun('run-l1'); + await store.persistEvent(settled('run-l1', 2, 'n1', 400, 400)); + + expect(reader.loadRun('run-l1')?.totalCostMicrocents).toBe(400); + expect(sumRunCosts('run-l1')).toBe(400); + const rows = client.db.select().from(runCosts).where(eq(runCosts.runId, 'run-l1')).all(); + expect(rows).toHaveLength(1); + // The REAL per-attempt tokens, unlike `foldCumulative`'s money-only addend. + expect(rows[0]).toMatchObject({ nodeId: 'n1', inputTokens: 10, outputTokens: 5 }); + }); + + it('is the INVERSE of budget:estimate_committed — the realized twin DOES touch the money of record', async () => { + // The estimate falls through `applyDerived`'s default and must never reach `runs`/`run_costs` + // (ADR-0074's central negative guarantee). Its realized twin must, or the ledger records nothing. + const store = await startRun('run-l2'); + await store.persistEvent( + evRun( + 'run-l2', + 'budget:estimate_committed', + 2, + { + nodeId: 'n1', + attemptNumber: 1, + model: 'claude-opus-4-8', + estimateMicrocents: 999, + cumulativeConservativeMicrocents: 999, + }, + ts, + ), + ); + expect(reader.loadRun('run-l2')?.totalCostMicrocents).toBe(0); + + await store.persistEvent(settled('run-l2', 3, 'n1', 400, 400)); + expect(reader.loadRun('run-l2')?.totalCostMicrocents).toBe(400); + expect(sumRunCosts('run-l2')).toBe(400); + }); + + it('telescopes: attempt rows advance the total, so node:completed contributes ZERO', async () => { + // ADR-0076 property 3, the load-bearing one. The terminal row is still written (it carries the node's + // token totals); its COST is zero by arithmetic, and the SUM invariant holds with no special case. + const store = await startRun('run-l3'); + await store.persistEvent(settled('run-l3', 2, 'n1', 400, 400, 1)); + await store.persistEvent(settled('run-l3', 3, 'n1', 600, 1_000, 2)); + await store.persistEvent( + evRun( + 'run-l3', + 'node:completed', + 4, + { + nodeId: 'n1', + output: { ok: true }, + tokensUsed: { input: 20, output: 10 }, + durationMs: 5, + cumulativeCostMicrocents: 1_000, + }, + ts, + ), + ); + + const rows = client.db.select().from(runCosts).where(eq(runCosts.runId, 'run-l3')).all(); + expect(rows).toHaveLength(3); // two attempts + the terminal + expect(rows[2]?.costMicrocents).toBe(0); // the terminal's delta telescoped away + expect(reader.loadRun('run-l3')?.totalCostMicrocents).toBe(1_000); + expect(sumRunCosts('run-l3')).toBe(1_000); // ADR-0070's invariant, unchanged + }); + + it('does NOT double-count when a higher cumulative committed first (persists are concurrent)', async () => { + // `#emitDurable` starts every `persistEvent` immediately and serializes only DELIVERY — "persists stay + // concurrent", in its own words. So a sibling's terminal can land BEFORE an attempt row whose charge it + // already includes. Writing the event's raw `costMicrocents` would add that money twice; the telescoping + // fold cannot, because every money event carries an ABSOLUTE cumulative. + const store = await startRun('run-l7'); + await store.persistEvent( + evRun('run-l7', 'node:started', 2, { nodeId: 'n2', nodeType: 'agent' }, ts), + ); + // The sibling commits first with a cumulative that ALREADY contains n1's 400. + await store.persistEvent( + evRun( + 'run-l7', + 'node:completed', + 3, + { + nodeId: 'n2', + output: {}, + tokensUsed: { input: 1, output: 1 }, + durationMs: 1, + cumulativeCostMicrocents: 1_000, + }, + ts, + ), + ); + expect(reader.loadRun('run-l7')?.totalCostMicrocents).toBe(1_000); + + // n1's attempt row lands late, carrying its own true charge of 400 and a cumulative of 400. + await store.persistEvent(settled('run-l7', 4, 'n1', 400, 400)); + + // 1_000, not 1_400. The money was already banked; this row contributes nothing and says so. + expect(reader.loadRun('run-l7')?.totalCostMicrocents).toBe(1_000); + expect(sumRunCosts('run-l7')).toBe(1_000); + }); + + it('keeps the step row at the node TRUE cost after telescoping, not the zeroed delta', async () => { + // The regression the telescoping would otherwise ship silently: `node:completed` used its own delta for + // `step_executions.costMicrocents`, and that delta is now routinely 0. The column is user-visible + // (`relavium status --json`), and nothing pinned it before this test. + const store = await startRun('run-l4'); + await store.persistEvent(settled('run-l4', 2, 'n1', 400, 400, 1)); + await store.persistEvent(settled('run-l4', 3, 'n1', 600, 1_000, 2)); + await store.persistEvent( + evRun( + 'run-l4', + 'node:completed', + 4, + { + nodeId: 'n1', + output: { ok: true }, + tokensUsed: { input: 20, output: 10 }, + durationMs: 5, + cumulativeCostMicrocents: 1_000, + }, + ts, + ), + ); + + const step = client.db + .select() + .from(stepExecutions) + .where(and(eq(stepExecutions.runId, 'run-l4'), eq(stepExecutions.nodeId, 'n1'))) + .get(); + expect(step?.costMicrocents).toBe(1_000); + }); + + it('keeps the step row at the node TRUE cost on a FAILED node too', async () => { + // `node:failed` shared the same `nodeCost`-for-two-purposes bug, plus an `if (nodeCost > 0)` guard that + // would have skipped the write entirely on exactly the failures whose cost matters most. + const store = await startRun('run-l5'); + await store.persistEvent(settled('run-l5', 2, 'n1', 700, 700, 1)); + await store.persistEvent( + evRun( + 'run-l5', + 'node:failed', + 3, + { + nodeId: 'n1', + error: { code: 'provider_unavailable', message: 'boom', retryable: false }, + cumulativeCostMicrocents: 700, + }, + ts, + ), + ); + + const step = client.db + .select() + .from(stepExecutions) + .where(and(eq(stepExecutions.runId, 'run-l5'), eq(stepExecutions.nodeId, 'n1'))) + .get(); + expect(step?.costMicrocents).toBe(700); + expect(reader.loadRun('run-l5')?.totalCostMicrocents).toBe(700); + expect(sumRunCosts('run-l5')).toBe(700); + }); + + it('counts run-level TOKENS once — the attempt row must not bump the run totals', async () => { + // `node:completed.tokensUsed` is already the sum across this node's attempts, so an attempt arm that + // also bumped `runs.total_*_tokens` would double them. The `run_costs` row still carries per-attempt + // tokens for attribution; nothing sums that column. + const store = await startRun('run-l6'); + await store.persistEvent(settled('run-l6', 2, 'n1', 400, 400, 1, { input: 12, output: 6 })); + await store.persistEvent( + evRun( + 'run-l6', + 'node:completed', + 3, + { + nodeId: 'n1', + output: {}, + tokensUsed: { input: 12, output: 6 }, + durationMs: 5, + cumulativeCostMicrocents: 400, + }, + ts, + ), + ); + + const run = reader.loadRun('run-l6'); + expect(run?.totalInputTokens).toBe(12); + expect(run?.totalOutputTokens).toBe(6); + }); + }); }); diff --git a/packages/db/src/run-history-store.ts b/packages/db/src/run-history-store.ts index 28f0d478..a6842c21 100644 --- a/packages/db/src/run-history-store.ts +++ b/packages/db/src/run-history-store.ts @@ -525,6 +525,29 @@ export function createRunHistoryStore(db: Db, deps: RunHistoryStoreDeps): RunHis return delta; }; + /** + * Every micro-cent `run_costs` holds for one node — the step row's TRUE cost once a node's money can arrive + * in more than one row (ADR-0076). + * + * Before the realized-cost ledger, a node's whole charge landed in exactly one `run_costs` row written at its + * terminal, so the terminal's telescoping delta and the node's cost were the same number and the terminal arms + * used one variable for both. They are no longer the same number: `cost:attempt_settled` rows advance + * `runs.total_cost_microcents` as each attempt settles, so by the time the terminal folds its snapshot the + * delta is **zero by arithmetic** — which is correct for the run total and wrong for `step_executions`, a + * user-visible column (`relavium status --json`). Read the node's rows instead. + * + * **Attribution caveat, stated rather than discovered.** Under a node-RETRY there are several + * `step_executions` rows for one `nodeId` (keyed by attempt), while `run_costs` has no attempt column — so an + * earlier attempt's money lands on the final step row. That is the same class of approximation the per-node + * fan-out delta already carries (see the `node:completed` arm); the run-level SUM stays exact either way. + */ + const nodeSettledCost = (tx: TxDb, runId: string, nodeId: string): number => + tx + .select({ c: sql`coalesce(sum(${runCosts.costMicrocents}), 0)` }) + .from(runCosts) + .where(and(eq(runCosts.runId, runId), eq(runCosts.nodeId, nodeId))) + .get()?.c ?? 0; + /** Apply an event's derived `runs`/`step_executions`/`run_costs` writes (`run:started` inserts the runs row). */ const applyDerived = (tx: TxDb, event: RunEvent, runId: string, ts: number): void => { switch (event.type) { @@ -588,7 +611,11 @@ export function createRunHistoryStore(db: Db, deps: RunHistoryStoreDeps): RunHis outputJson: JSON.stringify(event.output), inputTokens: event.tokensUsed.input, outputTokens: event.tokensUsed.output, - costMicrocents: nodeCost, + // The node's TRUE cost, NOT `nodeCost` (ADR-0076). `nodeCost` is the terminal's telescoping delta, + // and once `cost:attempt_settled` rows have advanced the run total it is zero by arithmetic — + // correct for `runs`, and a silent regression here, because this column is user-visible. Read after + // the insert above so this node's terminal row is included. See `nodeSettledCost`. + costMicrocents: nodeSettledCost(tx, runId, event.nodeId), durationMs: event.durationMs, completedAt: ts, updatedAt: ts, @@ -623,19 +650,74 @@ export function createRunHistoryStore(db: Db, deps: RunHistoryStoreDeps): RunHis // persisted, so this snapshot is the only durable carrier, and dropping it left the run total (and // `sum(run_costs)`) short of money that was really charged. failStepRow(tx, event, runId, ts); - const nodeCost = foldCumulative( - tx, - runId, - event.cumulativeCostMicrocents, - ts, - event.nodeId, - ); - if (nodeCost > 0) { - tx.update(stepExecutions) - .set({ costMicrocents: nodeCost, updatedAt: ts }) - .where(stepMatch(runId, event.nodeId, event.attemptNumber)) - .run(); - } + foldCumulative(tx, runId, event.cumulativeCostMicrocents, ts, event.nodeId); + // Same correction as `node:completed` (ADR-0076): the step row carries the node's TRUE cost, not the + // terminal's telescoping delta. The old `if (nodeCost > 0)` guard went with it — with attempt rows in + // play the delta is routinely zero on a node that really spent, so the guard would skip the write on + // exactly the failures whose cost matters most. `nodeSettledCost` returns 0 for a node that spent + // nothing, which writes a truthful 0 rather than leaving a stale value. + tx.update(stepExecutions) + .set({ costMicrocents: nodeSettledCost(tx, runId, event.nodeId), updatedAt: ts }) + .where(stepMatch(runId, event.nodeId, event.attemptNumber)) + .run(); + return; + } + case 'cost:attempt_settled': { + // ADR-0076: one settled provider attempt's REALIZED charge becomes a `run_costs` row in the SAME + // transaction as its event, which is what makes the ledger idempotent without a second uniqueness key — + // `UNIQUE(run_id, seq)` already bars a duplicate event, and the derived row cannot outlive it. + // + // **The row is a TELESCOPING delta off the event's cumulative — NOT the raw `costMicrocents`.** The + // raw value is this attempt's true charge and the event keeps carrying it (that is what a reader and + // the checkpoint fold sum); what goes in `run_costs` is `max(0, cumulative - currentRunCost)`, the same + // arithmetic every other money arm here uses. The reason is ordering, and it is not hypothetical: + // `#emitDurable` starts each `persistEvent` immediately and only serializes DELIVERY — its own comment + // says "persists stay concurrent". So a later event's write can commit FIRST. Writing the raw delta + // then DOUBLE-COUNTS: a sibling's `node:completed` lands with a cumulative that already includes this + // attempt, and this row adds it a second time. Telescoping cannot — every money event carries an + // ABSOLUTE cumulative, so the running total converges on the largest one seen regardless of the order + // the rows commit in, and `SUM(run_costs) == runs.total_cost_microcents` holds by construction. + // + // In the ordered case (the normal one) the two are identical: `cumulative - prev == costMicrocents` + // exactly. The divergence is a `fan_out`/out-of-order case, where per-ATTEMPT attribution degrades to + // an approximation — precisely the caveat `node:completed`'s per-node delta already carries and + // documents. The event remains the exact per-attempt record; this row is the money of record. + // + // **The two writes are a pair.** ADR-0076 property 3 says the terminal's fold telescopes to zero + // because the attempt rows advanced `sum(run_costs)`. What the fold actually subtracts is + // `runs.total_cost_microcents` (`currentRunCost`), and the two are equal only because every writer + // bumps `runs` alongside its `run_costs` insert. Dropping either write here would keep compiling, keep + // a test that checks only one table green, and corrupt every later delta. + const prev = currentRunCost(tx, runId); + const attemptCost = Math.max(0, event.cumulativeCostMicrocents - prev); + tx.insert(runCosts) + .values({ + id: deps.uuid(), + runId, + nodeId: event.nodeId, + // The REAL per-attempt tokens, unlike `foldCumulative`'s money-only addend — per-attempt token + // attribution is half of why this ledger exists. The row is written even when `attemptCost` is 0 + // (an unpriced or free attempt, or one whose money a concurrent sibling already banked), because + // the tokens are still real; `foldCumulative` skips a zero row only because it has none to carry. + // Only the COST column carries ADR-0070's `SUM(run_costs) == runs.total_cost_microcents` + // invariant; nothing sums these token columns, and `node:completed` remains the single writer of + // the run-level token totals (see below). + inputTokens: event.inputTokens, + outputTokens: event.outputTokens, + costMicrocents: attemptCost, + createdAt: ts, + // `modelId` is deliberately NOT written. It is an FK to `model_catalog` (a UUID) and `schema.ts` + // documents it as a dead column; `event.model` is a raw provider string this store cannot resolve. + // Per-model attribution on the run path comes from the durable EVENT, not from this row. + } satisfies NewRunCostRow) + .run(); + tx.update(runs) + // Money only. `runs.total_input_tokens` / `total_output_tokens` are NOT bumped here: `node:completed` + // already adds the turn's full `tokensUsed`, which is the sum across these attempts, so adding them + // here would double the run's token totals. + .set({ totalCostMicrocents: prev + attemptCost, updatedAt: ts }) + .where(eq(runs.id, runId)) + .run(); return; } case 'human_gate:paused': @@ -702,6 +784,12 @@ export function createRunHistoryStore(db: Db, deps: RunHistoryStoreDeps): RunHis // must never reach `runs.total_cost_microcents` or `run_costs`. Folding it into either would present an // upper bound as an invoice and break ADR-0070's `SUM(run_costs) == runs.total_cost_microcents`. Falling // through here is what keeps that true — asserted, not assumed, in this package's tests. + // + // Its realized twin `cost:attempt_settled` (ADR-0076) does the OPPOSITE and has its own arm above. Note + // what that means for anyone adding the next durable type: this `default` is not an exhaustiveness + // guard. `RunEvent` has no `assertNever` anywhere in the repo, so a new money event added without an + // arm lands here silently — a `run_events` row, no derived write, no compile error, no failing test. + // Decide deliberately which side of this line a new type belongs on. return; } }; diff --git a/packages/shared/src/constants.ts b/packages/shared/src/constants.ts index 454d0432..4b4ce4d5 100644 --- a/packages/shared/src/constants.ts +++ b/packages/shared/src/constants.ts @@ -18,8 +18,10 @@ export type SchemaVersion = typeof SCHEMA_VERSION; * and the per-event ordinal is always `sequenceNumber`, never `seqNo`. Order mirrors the * `RunEvent` union in the spec: `agent:reasoning` sits immediately after `agent:token` (the reasoning * host-emit, EA6/2.5.H amending [ADR-0036]); `agent:approval_requested` + `agent:file_patch_proposed` - * sit after `agent:tool_result`, and the five governance events close the list: `run:paused`, `run:timeout`, + * sit after `agent:tool_result`, then the five governance events: `run:paused`, `run:timeout`, * `budget:warning`, `budget:paused` (ADR-0028) and `budget:estimate_committed` ([ADR-0074], dual-envelope). + * `cost:attempt_settled` ([ADR-0076]) closes the list — the realized twin of that last one, and the only + * durable member of the `cost:` namespace (`cost:updated` is streamed). */ export const RUN_EVENT_TYPES = [ 'run:started', @@ -56,6 +58,11 @@ export const RUN_EVENT_TYPES = [ // rides 'runId' on a run and 'sessionId' on a session, so it is listed here (with the run types) and reused // on the session envelope rather than duplicated into SESSION_EVENT_TYPES. NEVER realized spend. 'budget:estimate_committed', + // One settled provider attempt's REALIZED charge, made durable (ADR-0076) — the realized twin of the + // estimate above, emitted from the same `onAttempt` callback and joined at the same barriers (ADR-0077). + // RUN-ONLY, deliberately: the session path already records per-attempt realized spend into `session_costs` + // on every 'cost:updated' (ADR-0070), so it needs no session arm and is NOT in SESSION_EVENT_TYPES. + 'cost:attempt_settled', ] as const; export type RunEventType = (typeof RUN_EVENT_TYPES)[number]; diff --git a/packages/shared/src/run-event.test.ts b/packages/shared/src/run-event.test.ts index ffc37563..c0fd6dfb 100644 --- a/packages/shared/src/run-event.test.ts +++ b/packages/shared/src/run-event.test.ts @@ -175,6 +175,18 @@ const valid: Record> = { estimateMicrocents: 400, cumulativeConservativeMicrocents: 400, }, + 'cost:attempt_settled': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'claude-opus-4-8', + attemptNumber: 1, + inputTokens: 10, + outputTokens: 5, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, }; /** One targeted invalid payload per variant (a missing/invalid required field). */ @@ -219,6 +231,140 @@ const reject: Record> = { estimateMicrocents: 400, cumulativeConservativeMicrocents: 400, }, + // ADR-0076. The first two pin the DIVERGENCES from this event's two siblings: `attemptNumber` and `priced` are + // optional on `cost:updated` / `budget:estimate_committed` (they had historical rows) and REQUIRED here (it has + // none). Loosening later stays additive; tightening later would be the one-way door `parseStoredRunEvent` + // describes — so a test that lets either slip through optional is what makes the door close silently. + 'cost:attempt_settled (no attemptNumber)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, + 'cost:attempt_settled (no priced)': { + // Without the flag, `costMicrocents: 0` with real tokens is ambiguous between "unpriced" and "free" — the + // exact ambiguity a ledger row exists to resolve. + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + }, + 'cost:attempt_settled (cumulative below this attempt)': { + // The same producer bug the conservative twin pins: reading the run-wide counter BEFORE folding this + // attempt into it. The cumulative IS the restore path (reconstruction maxes it), so such a row restores a + // total missing this charge — nothing else anywhere would complain, which is why this fixture exists. + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 500, + cumulativeCostMicrocents: 0, + // `priced` is REQUIRED on this event, so omitting it here would reject the fixture on the MISSING FIELD and + // never reach the refinement — a test that passes with the refinement deleted. It did, until a break-verify + // caught it. Every fixture below carries a complete payload for the same reason: exactly one thing wrong. + priced: true, + }, + // `nodeId` and `model` are required too, and the file's own convention is to defend a required field with + // a fixture — `budget:estimate_committed` has `(no model)`, `budget:paused` has `(missing/empty nodeId)`. + // Without these, loosening either to `.optional()` reddens nothing, which is the same silent one-way door + // the `attemptNumber`/`priced` fixtures exist to hold shut. + 'cost:attempt_settled (no nodeId)': { + type: 'cost:attempt_settled', + ...env, + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, + 'cost:attempt_settled (empty nodeId)': { + type: 'cost:attempt_settled', + ...env, + nodeId: '', + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, + 'cost:attempt_settled (no model)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, + 'cost:attempt_settled (empty model)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: '', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, + 'cost:attempt_settled (attemptNumber 0)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + attemptNumber: 0, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 400, + cumulativeCostMicrocents: 400, + priced: true, + }, + 'cost:attempt_settled (fractional cost)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: 12.5, + // 13, not 12.5 — a fractional cumulative would reject this fixture on the CUMULATIVE field and leave + // `costMicrocents`'s integer bound untested. One thing wrong per fixture, and it has to be the named one. + cumulativeCostMicrocents: 13, + priced: true, + }, + 'cost:attempt_settled (negative cost)': { + type: 'cost:attempt_settled', + ...env, + nodeId: 'n', + model: 'm', + attemptNumber: 1, + inputTokens: 1, + outputTokens: 1, + costMicrocents: -1, + cumulativeCostMicrocents: 0, + priced: true, + }, 'run:started (bad executionMode)': { type: 'run:started', ...env, @@ -556,7 +702,7 @@ describe('RunEvent union — every variant', () => { } }); - it('covers exactly the 24 canonical colon-namespaced names, pinned to a literal list', () => { + it('covers exactly the 25 canonical colon-namespaced names, pinned to a literal list', () => { // A hardcoded contract list — independent of RUN_EVENT_TYPES — so the union and the // constant cannot silently drift together. const CONTRACT_NAMES = [ @@ -584,6 +730,7 @@ describe('RunEvent union — every variant', () => { 'budget:warning', 'budget:paused', 'budget:estimate_committed', // ADR-0074 §2 — a durable conservative commitment; an ESTIMATE, not spend + 'cost:attempt_settled', // ADR-0076 — the realized twin of the line above; the only DURABLE cost: event ]; // The matrix above proves each canonical name's valid payload parses (so a // renamed/missing variant fails there); the union member count catches an *extra* @@ -591,7 +738,7 @@ describe('RunEvent union — every variant', () => { // RunEventSchema wraps the union in the correlation-key refinement; reach the raw union. expect(RunEventSchema.innerType().options).toHaveLength(CONTRACT_NAMES.length); expect(new Set(RUN_EVENT_TYPES)).toEqual(new Set(CONTRACT_NAMES)); - expect(Object.keys(valid)).toEqual(CONTRACT_NAMES); // the matrix covers all 24 + expect(Object.keys(valid)).toEqual(CONTRACT_NAMES); // the matrix covers all 25 // STRUCTURAL, not a comment: the §5 forward-compat fixtures stand in for "a type a newer binary wrote" using // the `test:` prefix. Step B first used `budget:estimate_committed` for that and ADR-0074 §2 then made it // real, silently inverting three fixtures. A `test:`-prefixed name must never become a canonical event. diff --git a/packages/shared/src/run-event.ts b/packages/shared/src/run-event.ts index 03ba4368..db14cb19 100644 --- a/packages/shared/src/run-event.ts +++ b/packages/shared/src/run-event.ts @@ -590,6 +590,107 @@ export const BudgetEstimateCommittedEventSchema = z.object({ }); export type BudgetEstimateCommittedEvent = z.infer; +/** + * One SETTLED provider attempt's REALIZED charge, made durable + * ([ADR-0076](../../../docs/decisions/0076-durable-per-attempt-realized-cost-ledger.md)). + * + * The realized twin of {@link BudgetEstimateCommittedEventSchema}, and the pairing is the point: + * `budget:estimate_committed` records money that MIGHT have been billed, this records money that WAS. Both are + * emitted from the SAME synchronous `onAttempt` callback a few lines apart (`agent-turn.ts`), which is why + * [ADR-0077](../../../docs/decisions/0077-realized-cost-ledger-uses-the-conservative-commitment-barrier.md) + * gives them the same barrier mechanism rather than two. + * + * **Why it exists at all.** `cost:updated` is the only other carrier of a realized per-attempt charge, and it + * is streamed, never persisted. The durable record is otherwise reconstructed from a LATER boundary — + * `node:completed.cumulativeCostMicrocents`. An agent turn is a loop, so a node can make many paid calls + * before it completes, and a crash mid-loop discarded every one of them. Worse than forgetting: the resumed + * run SPENT IT AGAIN, because the cap it re-evaluated against was understated by exactly the amount already + * charged. + * + * **RUN PATH ONLY — `...runBase`, deliberately not `...dualBase`.** `cost:updated` is dual-envelope, so the + * absence of a session arm is a choice, not an oversight: the session path ALREADY has this ledger. + * `persister.ts` writes `recordSessionCost` on every `cost:updated`, carrying the PER-ATTEMPT increment (not a + * cumulative snapshot) into `session_costs`, and since `#W15-4` that write is latched so a failure gates + * further egress. This event closes the RUN path's equivalent gap. + * + * **Scope, so the omission is not read as a gap.** This covers the provider attempts of an AGENT TURN. A media + * job's realized cost (ADR-0045 §5) does NOT emit one and does not need to: it is already durable through the + * `node:completed` / `node:failed` / `run:*` cumulative snapshots `#W15-6` added for exactly that reason. And + * no cost event of any kind makes a TOOL EFFECT idempotent — that is the durable effect journal, a different + * decision about a different failure (ADR-0076 scopes it out by name). + */ +export const CostAttemptSettledEventSchema = z.object({ + type: z.literal('cost:attempt_settled'), + ...runBase, + /** The agent node that owned the attempt. Required — on the run path every attempt has an owning vertex. */ + nodeId: nonEmptyString, + /** The canonical model id this attempt actually ran on — the per-model attribution key. */ + model: nonEmptyString, + /** + * 1-based WITHIN-CHAIN (`FallbackChain`) attempt, matching `cost:updated` — it resets to 1 on each node-retry + * re-dispatch, and does NOT join `node:*.attemptNumber` (the node-retry dispatch index). To attribute a + * charge to a node-retry attempt, partition the `sequenceNumber`-ordered stream at the + * `node:started` / `node:retrying` boundaries. + * + * **REQUIRED here, unlike on `cost:updated` and `budget:estimate_committed`, and the divergence is + * deliberate.** Those two carry it optionally because they predate this event and their historical rows may + * lack it. This type is new, so it has no historical rows — and a per-ATTEMPT ledger whose row cannot say + * which attempt it belongs to is not a ledger. The emitter always has it (`onAttempt`'s `nonSkippedAttempts`). + * Requiring it now is free; requiring it LATER would be the one-way door {@link parseStoredRunEvent} + * describes — every historical row without it becomes a known type with an invalid body, which must throw. + */ + attemptNumber: positiveInt, + inputTokens: nonNegativeInt, + outputTokens: nonNegativeInt, + /** + * THIS attempt's realized charge — the exact per-attempt record, and what the derived `run_costs` row and + * any "why did this run cost that" answer are built from. **It is NOT the restore path**: reconstruction + * maxes {@link cumulativeCostMicrocents} instead, and that field's doc explains why summing this one both + * double-counts and under-counts depending on which route you take. + * + * `nonNegativeInt`, NOT the `positiveInt` its estimate twin uses, and the asymmetry is real rather than an + * oversight. A conservative commitment of zero is unreachable by construction (`BudgetGovernor#admit` + * refuses `estimateMicrocents <= 0`). A realized charge of zero is routinely reachable: an UNPRICED model + * (the chain swallows the cost tracker's `UnknownModelError`) and a genuinely FREE one both settle here at + * zero with real tokens. {@link priced} is what separates those two — which is precisely why it is required. + */ + costMicrocents: nonNegativeInt, + /** + * The run-wide realized running total after this attempt — an ABSOLUTE total, read immediately after this + * attempt was folded into the counter, and **this is the field reconstruction restores from**, via + * `Math.max` against every other absolute total in the log (the node-boundary snapshots and + * `budget:paused.spentMicrocents`). + * + * `Math.max`, never LAST-WINS: under a `fan_out` concurrent events have no canonical `seq` order, so the + * lower `seq` can carry the higher total and a last-wins read would hand already-spent money back to the cap + * as headroom. Maxing over absolutes is order-independent because realized spend is monotonic — which is + * exactly the property `budget:estimate_committed` lacks, and the whole reason its fold sums signed deltas + * instead. + * + * **Do not restore by summing {@link costMicrocents}.** Into this same total it double-counts, because a + * node terminal's snapshot already contains the attempts it covers. Into a separate accumulator maxed + * against the snapshots it UNDER-counts, which is the subtler failure: the two sources cover different + * money — a media node writes a snapshot and emits no attempt row at all — so every attempt after the last + * node boundary disappears whenever earlier media spend is the larger figure, which is precisely the + * crash-mid-agent-loop case this event exists for. `costMicrocents` remains the exact per-attempt record for + * accounting and for the derived `run_costs` row; it is not the restore path. + * + * (`cost:updated` carries the same figure but is streamed and never persisted, so it is not a restore source + * at all.) + */ + cumulativeCostMicrocents: nonNegativeInt, + /** + * Whether this egress could be PRICED (ADR-0070 §6). **REQUIRED here, unlike on `cost:updated`.** + * + * On `cost:updated` the flag is additive-and-optional because it was added to an event that already had + * historical rows. This type has none. And without it `costMicrocents: 0` with real tokens is ambiguous + * between "we could not price this" and "this model is genuinely free" — an ambiguity a LEDGER must not + * carry, since the whole reason to write the row is to answer "what did this run cost, and do we know". + */ + priced: z.boolean(), +}); +export type CostAttemptSettledEvent = z.infer; + /** The run-event variants, discriminated on `type` (exposed via `RunEventSchema.innerType()`). */ const RunEventUnionSchema = z.discriminatedUnion('type', [ RunStartedEventSchema, @@ -616,6 +717,7 @@ const RunEventUnionSchema = z.discriminatedUnion('type', [ BudgetWarningEventSchema, BudgetPausedEventSchema, BudgetEstimateCommittedEventSchema, + CostAttemptSettledEventSchema, ]); /** The pre-refinement union value — the input every cross-field refinement helper below receives. */ @@ -663,6 +765,33 @@ function refineBudgetEstimateCommitted(event: RunEventUnion, ctx: z.RefinementCt } } +/** + * A settled attempt's cumulative total must already INCLUDE its own charge (ADR-0076). + * + * The same invariant, and the same single most likely emit-site bug, as {@link refineBudgetEstimateCommitted}: + * reading the run-wide counter BEFORE folding this attempt into it. That mistake emits + * `{ cost: 500, cumulative: 0 }` — and since the cumulative IS the restore path here (reconstruction maxes it; + * see the field's own doc), such a row would restore a total missing this charge and hand the cap headroom for + * money already spent. Pinned here because nothing else anywhere would complain. + * + * (An earlier revision of this comment said reconstruction "sums the deltas, so the RESTORE survives it". + * That described the design ADR-0076 originally staged and step 5 retracted — summing double-counts against a + * node terminal's snapshot. The restore has never summed since the code landed.) + */ +function refineCostAttemptSettled(event: RunEventUnion, ctx: z.RefinementCtx): void { + if ( + event.type === 'cost:attempt_settled' && + event.cumulativeCostMicrocents < event.costMicrocents + ) { + ctx.addIssue({ + code: z.ZodIssueCode.custom, + message: + 'cumulativeCostMicrocents must already include costMicrocents (read the counter AFTER folding this attempt)', + path: ['cumulativeCostMicrocents'], + }); + } +} + /** A gate's on-timeout policy only has meaning when a timeout is configured. */ function refineHumanGateTimeout(event: RunEventUnion, ctx: z.RefinementCtx): void { if ( @@ -795,6 +924,7 @@ export const RunEventSchema = RunEventUnionSchema.superRefine((event, ctx) => { refineMediaJobFrozenBasis(event, ctx); refineApprovalPreview(event, ctx); refineBudgetEstimateCommitted(event, ctx); + refineCostAttemptSettled(event, ctx); }); export type RunEvent = z.infer; diff --git a/tools/test-isolation/check.mjs b/tools/test-isolation/check.mjs new file mode 100644 index 00000000..0c50ef3a --- /dev/null +++ b/tools/test-isolation/check.mjs @@ -0,0 +1,415 @@ +/** + * Test-isolation guard — a repo-local SECOND checkout must never be collected by a root run (`CR-90`). + * + * The defect this closes was live, not theoretical. Measured 2026-08-10: a root `vitest list` discovered + * **472** test files, **234** of them under `.claude/worktrees//packages/...` — a full agent-tooling + * checkout of this same repo. A repo-wide run was therefore executing a foreign tree's tests, and the coverage + * report was counting its sources: 48.85% lines against 96.95% once excluded. + * + * It hid well: `.claude/worktrees/` is excluded through `.git/info/exclude`, which is LOCAL and untracked, so + * `git status` stayed clean and a fresh clone never even carried the rule. That is also why CI never saw it — + * CI checks out clean. The `vitest.config.ts` exclusions defend a developer's tree; **this guard is what + * defends CI**, by failing if those exclusions are ever weakened or removed. + * + * It is a tools check rather than a `*.test.ts` for a structural reason: a test running INSIDE Vitest cannot + * observe which files Vitest chose to collect. Only a subprocess can. + * + * ## What it asserts, and why each one is load-bearing + * + * 1. **PRIMARY — no collected file lives in a second checkout, listed or not.** Consults no list: it walks the + * collected paths and fails on any whose ancestor carries its own `pnpm-workspace.yaml`. This is the only + * assertion that stays true when the exclusion list is itself the thing that is wrong, and it is here + * because assertion 2 alone was not enough — see below. + * 2. **One fixture per `REPO_LOCAL_CHECKOUTS` entry, none collected.** The list is read out of + * `vitest.config.ts` (as text — this file is `.mjs` and the config is TypeScript, so it cannot be + * imported), so a new entry is probed the moment it is added. **But it proves only that every LISTED + * location is excluded, never that the list is COMPLETE**: because the fixtures are derived from the list, + * deleting `'**\/.claude\/**'` also deletes its probe, and the guard re-collects all 234 foreign suites + * while printing a green line. Measured, on the first version of this file. Hence assertion 1. + * 3. **Every workspace still yields tests.** A single canary is not enough: the structural rule this repo + * rejected (`'**\/*\/{packages,apps,tools}\/**'`) drops exactly four real files under + * `packages/core/src/tools/`, and a canary in `packages/shared` never notices. Passing by not running is a + * worse failure than the leak. + * 4. **The list reaches BOTH excludes.** `vitest list` observes `test.exclude` only, so every assertion above + * is blind to `coverage.exclude` — the half that keeps a foreign tree out of the coverage denominator. The + * phase doc's acceptance criterion names both, so the config text is checked directly. + * + * Exits non-zero so CI fails loudly. Run from anywhere: + * node tools/test-isolation/check.mjs + */ +import { spawnSync } from 'node:child_process'; +import { createRequire } from 'node:module'; +import { existsSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { dirname, join, relative, resolve, sep } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..', '..'); +const require = createRequire(join(repoRoot, 'package.json')); + +const CONFIG_PATH = join(repoRoot, 'vitest.config.ts'); +const FIXTURE_DIR = '__test_isolation_fixture__'; + +/** Where the detector's own self-test plants its positive control. Deliberately NOT an excluded location. */ +const PROBE_DIR = '__test_isolation_detector_probe__'; + +/** + * Every workspace that holds at least one `*.test.ts` ON DISK — DERIVED, never hand-listed. + * + * A hardcoded array is the same shape the primary assertion was rebuilt to escape: it can only prove the + * places someone remembered. If `packages/ui` gains its first test tomorrow, a static list silently stops + * checking it. Walking the tree is non-circular — the filesystem says which workspaces have tests, and Vitest + * must then have found them. + */ +function workspacesWithTests() { + const roots = ['packages', 'apps']; + const found = []; + for (const root of roots) { + const rootDir = join(repoRoot, root); + if (!existsSync(rootDir)) continue; + for (const name of readdirSync(rootDir)) { + const src = join(rootDir, name, 'src'); + if (existsSync(src) && hasTestFile(src)) found.push(`${root}/${name}`); + } + } + return found; +} + +function hasTestFile(dir) { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + if (entry.name === 'node_modules' || entry.name === 'dist') continue; + if (entry.isDirectory()) { + if (hasTestFile(join(dir, entry.name))) return true; + } else if (entry.name.endsWith('.test.ts') || entry.name.endsWith('.test.tsx')) { + return true; + } + } + return false; +} + +function fail(message) { + console.error(`✖ ${message}`); + process.exit(1); +} + +/** + * The plantable directory for one exclusion pattern: `**\/.claude\/**` → `.claude`, `worktrees/**` → + * `worktrees`. A pattern that does not reduce to a plain path is rejected rather than skipped — silently + * dropping an entry is exactly how this guard went blind the first time. + */ +function plantableDir(pattern) { + const stripped = pattern.replace(/^\*\*\//, '').replace(/\/\*\*$/, ''); + if (/[*?[\]{}]/.test(stripped) || stripped.length === 0) { + fail( + `REPO_LOCAL_CHECKOUTS entry ${JSON.stringify(pattern)} is not a plantable directory path, so this ` + + `guard cannot prove it works.\n Use a form like '**//**' or '/**', or add an explicit ` + + `fixture location here.`, + ); + } + return stripped; +} + +// --- 1. Read the list from the config itself, so the two cannot drift ----------------------------- + +const configText = readFileSync(CONFIG_PATH, 'utf8'); +const listMatch = /export const REPO_LOCAL_CHECKOUTS = \[([^\]]*)\]/.exec(configText); +if (listMatch === null) { + fail(`cannot find \`export const REPO_LOCAL_CHECKOUTS\` in ${relative(repoRoot, CONFIG_PATH)}`); +} +const patterns = [...listMatch[1].matchAll(/'([^']+)'/g)].map((m) => m[1]); +if (patterns.length === 0) { + fail( + `REPO_LOCAL_CHECKOUTS is EMPTY in ${relative(repoRoot, CONFIG_PATH)}.\n` + + ` A repo-local second checkout would be collected by every root run and counted in coverage.`, + ); +} + +// --- 2. The list must reach BOTH excludes (assertion 3) -------------------------------------------- + +// A GLOBAL count of `...REPO_LOCAL_CHECKOUTS` is the wrong test, and passes the exact defect it is for: two +// spreads inside `test.exclude` and none inside `coverage.exclude` reads as 2 and goes green, while a foreign +// tree's sources sit in the coverage denominator. So each property's own array literal is inspected. The +// arrays here hold no nested brackets, which is what makes the bounded `[^\]]*` match exact rather than +// approximate; a future nested value would end the match early and fail loudly here rather than silently pass. +const coverageAt = configText.indexOf('coverage: {'); +if (coverageAt === -1) { + fail( + `cannot find the \`coverage: {\` block in ${relative(repoRoot, CONFIG_PATH)}, so \`test.exclude\` and ` + + `\`coverage.exclude\` cannot be told apart.`, + ); +} +const excludeArrays = [...configText.matchAll(/\bexclude:\s*\[([^\]]*)\]/g)]; +const spreadsIn = (body) => (body.match(/\.\.\.REPO_LOCAL_CHECKOUTS\b/g) ?? []).length; +// `test.exclude` is the LAST one before the coverage block, `coverage.exclude` the FIRST one after it. +const testExclude = excludeArrays.findLast((m) => m.index < coverageAt); +const coverageExclude = excludeArrays.find((m) => m.index > coverageAt); +for (const [name, match, why] of [ + ['test.exclude', testExclude, "stops a foreign tree's TESTS from running"], + ['coverage.exclude', coverageExclude, 'stops its SOURCES from entering the coverage denominator'], +]) { + const count = match === undefined ? 0 : spreadsIn(match[1]); + if (count !== 1) { + fail( + `\`${name}\` in ${relative(repoRoot, CONFIG_PATH)} spreads \`...REPO_LOCAL_CHECKOUTS\` ${count} ` + + `time(s); it needs exactly 1.\n ${name} ${why}. Both halves are required and they fail ` + + `differently.\n \`vitest list\` observes \`test.exclude\` only, which is why this is checked here ` + + `as text — and why the count is checked PER PROPERTY: a repo-wide total of 2 is also what two ` + + `spreads in one array and none in the other looks like.`, + ); + } +} + +// --- 3. Plant one fixture per entry, collect, assert ------------------------------------------------ + +/** + * Locations probed ALWAYS, whatever `REPO_LOCAL_CHECKOUTS` currently says — the three that exist today. + * + * This is the answer to the hole the header describes: because the pattern-derived fixtures below come FROM + * the list, deleting `'**\/.claude\/**'` deletes its probe too, and the guard goes green while re-collecting a + * foreign tree. The primary structural assertion covers that on a developer's machine, but only if a second + * checkout is actually on disk — and CI checks out clean, so there it covers nothing at all. A planted fixture + * at a known location is a positive control that works in an empty tree. + * + * A hardcoded list is exactly what `workspacesWithTests` was rebuilt to avoid, and the difference matters: + * this one may only ever GROW stale in the safe direction. An entry removed from `REPO_LOCAL_CHECKOUTS` still + * gets probed here and fails loudly; a NEW location nobody added here is still probed by the derived fixtures. + * Neither omission can produce a silent pass. + */ +const ALWAYS_PROBED = ['.claude', '.worktrees', 'worktrees']; + +const fixtureDirs = [...new Set([...patterns.map(plantableDir), ...ALWAYS_PROBED])]; +const fixtures = fixtureDirs.map((dir) => ({ + // The pattern that should be excluding it, for the error message. An always-probed location with no entry + // gets the pattern it OUGHT to have, which is the fix to paste back into `vitest.config.ts`. + pattern: patterns.find((p) => plantableDir(p) === dir) ?? `${dir}/**`, + dir, + root: join(repoRoot, dir, FIXTURE_DIR), + parent: join(repoRoot, dir), + parentExisted: existsSync(join(repoRoot, dir)), +})); + +/** + * A SYNTHETIC checkout, deliberately, not a real `git worktree`: a real one depends on git state and on the + * working directory being a repo at all, which is the exact non-determinism `CR-90` is about. What matters is + * the SHAPE — a nested workspace layout with its own test and source file — and that is reproducible anywhere. + */ +function plant(fixture) { + rmSync(fixture.root, { recursive: true, force: true }); // self-healing after a Ctrl-C'd earlier run + const src = join(fixture.root, 'packages', 'probe', 'src'); + mkdirSync(src, { recursive: true }); + // The workspace marker too, so the fixture shape-matches a real second checkout for BOTH assertions: if an + // exclusion breaks, the structural check fires alongside the leak check and the error names the cause twice. + writeFileSync(join(fixture.root, 'pnpm-workspace.yaml'), "packages:\n - 'packages/*'\n"); + writeFileSync(join(src, 'probe.ts'), 'export const probe = 1;\n'); + writeFileSync( + join(src, 'probe.test.ts'), + [ + "import { describe, expect, it } from 'vitest';", + "import { probe } from './probe.js';", + '', + `describe('${FIXTURE_DIR}', () => {`, + " it('MUST NOT be collected by a root run — see tools/test-isolation', () => {", + ' expect(probe).toBe(1);', + ' });', + '});', + '', + ].join('\n'), + ); +} + +function clear(fixture) { + rmSync(fixture.root, { recursive: true, force: true }); + // Do not leave an empty `worktrees/` or `.worktrees/` behind that this guard itself created. + if ( + !fixture.parentExisted && + existsSync(fixture.parent) && + readdirSync(fixture.parent).length === 0 + ) { + rmSync(fixture.parent, { recursive: true, force: true }); + } +} + +/** + * Resolve vitest through the module graph and spawn it with `process.execPath`, never a bare `npx`. Spawning + * `npx` would search a writeable `PATH`, need `shell: true` for the Windows `.cmd` shim, and can reach for the + * registry when it cannot resolve locally — the convention `tools/coverage-gate/run.mjs` documents. + */ +function collectedFiles() { + let vitestEntry; + try { + const pkgPath = require.resolve('vitest/package.json'); + vitestEntry = join(dirname(pkgPath), require('vitest/package.json').bin.vitest); + } catch (error) { + // THROW, never `process.exit`, from anywhere inside the try/finally below. `process.exit` does not unwind + // a `finally` in Node, so exiting here would leave all three fixtures on disk — and the `.claude/` one is + // a SIBLING of `.claude/worktrees/`, so the `.gitignore` entries do not cover it. The next plain + // `pnpm test` would then collect the leftover fixture: CR-90's own defect, self-inflicted by its guard's + // crash path. Verified: `process.exit` inside a `try` skips the `finally` body. + throw new Error(`cannot resolve the vitest entry point: ${error.message}`); + } + // `--json` returns absolute `file` paths, so nothing depends on the reporter's text layout (a `projects` + // config prefixes `[name] `) or on the platform separator. `--filesOnly` keeps it cheap: the file set is + // resolved without importing a single test module. + const result = spawnSync(process.execPath, [vitestEntry, 'list', '--filesOnly', '--json'], { + cwd: repoRoot, + encoding: 'utf8', + shell: false, + }); + if (result.status !== 0) { + throw new Error(`\`vitest list\` failed (exit ${result.status}):\n${result.stderr ?? ''}`); + } + let parsed; + try { + parsed = JSON.parse(result.stdout); + } catch { + throw new Error(`\`vitest list --json\` did not return JSON:\n${result.stdout.slice(0, 400)}`); + } + return parsed.map((entry) => relative(repoRoot, entry.file).split(sep).join('/')); +} + +/** + * A second checkout of a pnpm monorepo carries its own `pnpm-workspace.yaml` at its root. Any collected file + * with such an ancestor BELOW the repo root is in a tree that is not ours. + * + * `i = 1`, not `0`, and that is load-bearing: `i = 0` yields `candidate = ''`, whose `pnpm-workspace.yaml` is + * the repo's OWN — every file would report as nested inside itself. + * + * **Recorded limitation, not a property.** "A second checkout always carries one" holds for `git worktree + * add`, `git clone` and a full copy — every case seen here — but NOT for a sparse checkout, a + * `--no-checkout` worktree, or a partial rsync of `packages/**`. Those would be collected with nothing in + * their ancestry to detect, and this assertion would pass in silence, leaving only the list-based checks + * below. Nothing in this repo produces such a tree today; if that changes, this predicate needs a second + * marker (a `package.json` whose `name` matches the root's would be the obvious one). + */ +function nestedCheckoutOf(fileRel) { + const parts = fileRel.split('/'); + for (let i = 1; i < parts.length; i += 1) { + const candidate = parts.slice(0, i).join('/'); + if (existsSync(join(repoRoot, candidate, 'pnpm-workspace.yaml'))) return candidate; + } + return undefined; +} + +/** + * Prove the detector above actually fires, BEFORE trusting it to pass. + * + * Without this, `nestedCheckoutOf`'s match branch is never executed by the guard: the listed fixtures are + * (correctly) excluded from collection, so the structural walk only ever sees this repo's own paths and + * returns `undefined` every time. The one demonstration that it works was a manual check against a real + * worktree that happened to be present on one developer's machine — not repeatable, and absent on a fresh + * clone or in CI. A refactor that broke the ancestor walk would have shipped green forever. + * + * A pure-function self-test rather than a second `vitest list`: it exercises the same walk, the same `i = 1` + * boundary and the same marker file, for none of the cost. + */ +function selfTestDetector() { + const probe = join(repoRoot, PROBE_DIR); + try { + mkdirSync(join(probe, 'packages', 'probe', 'src'), { recursive: true }); + writeFileSync(join(probe, 'pnpm-workspace.yaml'), "packages:\n - 'packages/*'\n"); + const inside = `${PROBE_DIR}/packages/probe/src/probe.test.ts`; + if (nestedCheckoutOf(inside) !== PROBE_DIR) { + throw new Error( + `the nested-checkout detector FAILED its own self-test: it did not flag ${inside}, which sits under a` + + `\n directory carrying its own pnpm-workspace.yaml. The primary assertion is not working, so a` + + `\n green result from this guard would mean nothing.`, + ); + } + // And it must not fire on our own tree — an over-eager detector would fail every run for the wrong reason. + if (nestedCheckoutOf('packages/core/src/dag.test.ts') !== undefined) { + throw new Error( + "the nested-checkout detector flags this repo's OWN files. Check the `i = 1` loop bound.", + ); + } + } finally { + rmSync(probe, { recursive: true, force: true }); + } +} + +const cleanupAll = () => { + for (const fixture of fixtures) clear(fixture); +}; + +let files; +try { + selfTestDetector(); + for (const fixture of fixtures) plant(fixture); + files = collectedFiles(); +} catch (error) { + // Explicit cleanup BEFORE reporting, rather than a `finally` plus a `process.exit` inside the try — Node + // does not unwind a `finally` on `process.exit`, and a leaked fixture is CR-90's own defect self-inflicted. + cleanupAll(); + fail(error instanceof Error ? error.message : String(error)); +} +cleanupAll(); + +// --- 4. The PRIMARY assertion: no collected file may live in a second checkout, listed or not ------ +// +// The fixture probes above prove that every LISTED location is excluded. They cannot prove the list is +// COMPLETE — and that distinction is not academic. Because the fixtures are derived from the list, deleting +// `'**/.claude/**'` also deletes its probe: the guard then re-collects all 234 foreign suites and reports +// green. Measured, on the first version of this file. +// +// So the real test is structural and runs against the filesystem rather than a glob (which is what makes it +// safe here — `vitest.config.ts` cannot express this, a script can). `nestedCheckoutOf` is defined above, +// beside the self-test that proves it fires. + +const foreign = new Map(); +for (const file of files) { + const root = nestedCheckoutOf(file); + if (root !== undefined) foreign.set(root, (foreign.get(root) ?? 0) + 1); +} +if (foreign.size > 0) { + fail( + `${[...foreign.values()].reduce((a, b) => a + b, 0)} collected file(s) belong to a SECOND checkout of ` + + `this repo:\n` + + [...foreign] + .map(([root, n]) => ` ${root}/ (${n} file(s), has its own pnpm-workspace.yaml)`) + .join('\n') + + `\n Add it to REPO_LOCAL_CHECKOUTS in vitest.config.ts.` + + `\n This check does not consult that list — it walks the collected paths — so it stays true even when` + + `\n the list is the thing that is wrong.`, + ); +} + +const leaked = files.filter((f) => f.includes(FIXTURE_DIR)); +if (leaked.length > 0) { + // The two causes read very differently, so the message names which one this is. A LISTED location that + // leaked means the pattern does not do what it says; an always-probed one that leaked means the entry that + // used to cover it was removed — the failure the `ALWAYS_PROBED` control exists for, and the one the + // derived fixtures cannot see because deleting an entry deletes its probe. + const missed = fixtures.filter((fx) => leaked.some((f) => f.startsWith(`${fx.dir}/`))); + const listed = missed.filter((fx) => patterns.some((p) => plantableDir(p) === fx.dir)); + const unlisted = missed.filter((fx) => !patterns.some((p) => plantableDir(p) === fx.dir)); + fail( + `a repo-local checkout leaked into the root test run (${leaked.length} file(s)):\n` + + leaked.map((f) => ` ${f}`).join('\n') + + (listed.length === 0 + ? '' + : `\n Listed in REPO_LOCAL_CHECKOUTS but NOT excluded: ` + + listed.map((fx) => JSON.stringify(fx.pattern)).join(', ')) + + (unlisted.length === 0 + ? '' + : `\n MISSING from REPO_LOCAL_CHECKOUTS (vitest.config.ts): ` + + unlisted.map((fx) => JSON.stringify(fx.dir)).join(', ') + + `\n These are probed unconditionally precisely because a derived probe disappears with the entry` + + `\n it came from — removing one would otherwise go green on a clean checkout.`) + + `\n A root run must see this repo's tests and nothing else — a second checkout's suites are not ours` + + `\n to run, and its sources are not ours to count against the coverage floor.`, + ); +} + +const expected = workspacesWithTests(); +const silent = expected.filter((ws) => !files.some((f) => f.startsWith(`${ws}/`))); +if (silent.length > 0) { + fail( + `these workspaces yielded NO test files: ${silent.join(', ')}` + + `\n REPO_LOCAL_CHECKOUTS (vitest.config.ts) is excluding real tests. That is worse than the leak it` + + `\n prevents: the suite goes green by not running. Narrow the pattern.` + + `\n Collected ${files.length} file(s) in total.`, + ); +} + +console.log( + `✓ test isolation holds: ${files.length} file(s) collected, none from a repo-local checkout ` + + `(${fixtures.length} location(s) probed, ${expected.length} workspaces with tests on disk all collected).`, +); diff --git a/vitest.config.ts b/vitest.config.ts index bddb4c17..22d45cf1 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -1,4 +1,33 @@ -import { defineConfig } from 'vitest/config'; +import { defaultExclude, defineConfig } from 'vitest/config'; + +/** + * Directories that hold a SECOND checkout of this repo inside the working tree, and must never be collected + * by a root run (`CR-90`). + * + * This is not hypothetical housekeeping. Measured on 2026-08-10: a root `vitest list` found **472** test + * files, **234** of them under `.claude/worktrees//packages/...` — a full agent-tooling checkout, + * complete with its own sources. So a repo-wide run was executing a foreign tree's tests, and the coverage + * report was counting its sources (48.85% lines, against 96.95% once excluded). + * + * It hides well: `.claude/worktrees/` is excluded through `.git/info/exclude`, which is LOCAL and untracked, + * so `git status` stays clean, a fresh clone does not even carry the rule, and Vitest never consulted git in + * the first place. That also means CI, which checks out clean, never saw any of it — this list defends the + * DEVELOPER's run, and the guard in `tools/test-isolation` is the part that defends CI. + * + * **A list of named locations, deliberately, not a structural predicate.** The obvious general rule — "a + * workspace root nested inside another one", `'**\/*\/{packages,apps,tools}\/**'` — is wrong: it also matches + * this repo's own `packages/core/src/tools/*.test.ts`, because `**` happily eats `packages/core` and leaves + * `src` for the `*`. Verified: it drops exactly four real test files. A rule that silently drops real tests is + * worse than the leak it closes. + * + * **Each entry must be a plantable directory path**, not an arbitrary glob: `tools/test-isolation` imports + * this list and plants one fixture per entry, so a new entry is guarded automatically and the guard cannot + * drift out of step with the list. The two dotted names are matched anywhere (`**\/`); the undotted one is + * ROOT-ANCHORED on purpose — a product whose CLI manages git checkouts could plausibly grow + * `apps/cli/src/worktrees/`, and `**\/worktrees\/**` would silently delete its tests from collection and its + * sources from coverage. + */ +export const REPO_LOCAL_CHECKOUTS = ['**/.claude/**', '**/.worktrees/**', 'worktrees/**']; /** * Root, workspace-aware Vitest config. Per-package `test` scripts run `vitest run` @@ -26,6 +55,9 @@ export default defineConfig({ // SAME `.test.` prefix — an additive extension, not the rejected `.spec.` suffix — and stays confined to // apps/cli's renderer layer (the coverage `exclude` below drops it, and apps are coverage-excluded anyway). include: ['**/*.test.ts', '**/*.test.tsx'], + // `defaultExclude` is SPREAD, not replaced — setting `exclude` overrides Vitest's own list, and dropping + // `**/node_modules/**` from it would be a far bigger collection bug than the one being fixed. + exclude: [...defaultExclude, ...REPO_LOCAL_CHECKOUTS], passWithNoTests: true, coverage: { provider: 'v8', @@ -34,7 +66,20 @@ export default defineConfig({ // cwd-tolerant: `**/src/**` matches whether the run is rooted at the repo or a package, so a // package-scoped `--coverage` run no longer reports a false 0%. Apps stay smoke-only (excluded). include: ['**/src/**/*.ts'], - exclude: ['**/*.test.ts', '**/*.test.tsx', '**/apps/**'], + // Both halves matter and they fail differently: `test.exclude` stops a foreign tree's TESTS from + // running, this stops its SOURCES from being counted. `test-exclude` globs with `dot: true`, so the + // `include` above reaches straight into `.claude/` and a nested checkout's `packages/*/src/**` lands in + // the report as 0%-covered files. Measured: 48.85% lines with the leak, 96.95% without. + // + // **It corrupts the reported number and the html/lcov artifact — it does NOT trip a threshold, and an + // earlier version of this comment wrongly said it did.** Verified against vitest's own matcher: the + // per-package thresholds below are matched ROOT-RELATIVE with no implicit leading `**`, so + // `.claude/worktrees/x/packages/llm/src/a.ts` matches none of them and falls into the `global` group, + // which sets no thresholds and is skipped. The counterfactual run exits 0 at 48.85%. Stated precisely + // because this line becomes genuinely threshold-load-bearing the moment anyone adds a global threshold, + // and a maintainer deciding whether it is still needed should not be reading a mechanism that does not + // exist. + exclude: ['**/*.test.ts', '**/*.test.tsx', '**/apps/**', ...REPO_LOCAL_CHECKOUTS], // The enforced Phase-1 engine floor, scoped per-glob so it targets only the built engine // package(s) and never the not-yet-90% shared/db or the unbuilt core. //