diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 20362b7a..5886deee 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -5,7 +5,9 @@ # # Branch protection (set in the GitHub repo settings, not here): the `ci` job below is the # REQUIRED status check to merge into `main`. The `peer-dep-gate` job is advisory until the -# surface packages land their peers in Phase 1. +# surface packages land their peers in Phase 1. The `coverage` job is advisory too — it enforces +# the testing.md >=90% line+branch engine floor (exit criterion #5) but stays non-required until the +# thin core-branch margin is confirmed stable under CI's Node 22; promote it to a required check then. # # Caching: the always-on layer is the GitHub Actions `.turbo` cache (restored/saved below), # which makes a no-change re-run a Turborepo cache hit — the M0 "demonstrably hitting" @@ -129,6 +131,44 @@ jobs: - name: Install with strict peers run: pnpm install --frozen-lockfile --config.strict-peer-dependencies=true + # Engine coverage floor (testing.md >=90% line+branch, exit criterion #5). Advisory for now — a + # SEPARATE job (not part of the required `ci` job) so it surfaces a regression without blocking merge + # while the core-package branch margin is thin. `pnpm coverage` is a repo-ROOT run, which is what makes + # the root-relative per-glob thresholds (packages/core, packages/llm) authoritative (vitest.config.ts). + # Promote to a required check once the margin is confirmed stable under CI's Node 22. + coverage: + name: engine coverage floor (advisory) + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + persist-credentials: false + - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + with: + node-version-file: .nvmrc + cache: pnpm + - name: Install (frozen lockfile) + run: pnpm install --frozen-lockfile + # Share the same Turborepo local cache the `ci` job writes, so the build below is a warm cache hit + # on a same-SHA / recent-ancestor run rather than a cold rebuild. + - name: Restore Turborepo cache + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: .turbo + key: turbo-${{ runner.os }}-${{ github.sha }} + restore-keys: | + turbo-${{ runner.os }}- + # `pnpm coverage` is a repo-ROOT vitest run (not a turbo task), so it does NOT get the `test` + # task's `^build` — yet the `@relavium/*` package `exports` resolve only to `dist`. Build first so a + # fresh checkout can resolve the cross-package entries (a per-package test gets dist via turbo `^build`; + # each package still covers its OWN src via relative imports, so the floor stays src-accurate). + - name: Build workspaces (so coverage resolves the @relavium/* package entries) + run: pnpm turbo run build + - name: Engine coverage floor (>=90% line+branch) + run: pnpm coverage + # --- Reserved Phase-1 lanes (TODO: enable with the first provider adapter) ------------ # The per-provider conformance suite and the nightly live-API lane land WITH the adapters # in Phase 1 (testing.md); only their CI slots are reserved here so the testing standard diff --git a/AGENTS.md b/AGENTS.md index ff317f3f..4e56f4d3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,11 +24,13 @@ gateway; engine stays local, Phase 2) — split across build phase 5 (managed in phase 6 (cloud execution + portal); the engine is identical across all three (ADR-0012..0015). **Status: Phase 1 in progress — milestone M1 (LLM seam proven) reached (PR #9, 2026-06-07);** `@relavium/llm` (the seam + all three adapters) is landed and green. Phase 0 (M0) landed -the monorepo + `@relavium/shared` + CI + `@relavium/db`. Since then, the ADR-0031 multimodal -seam-shape amendment (1.AD, PR #11) and the `FallbackChain` runner (1.K, PR #13, 2026-06-11) -have landed. Active work continues on the -[`@relavium/core` engine](docs/roadmap/phases/phase-1-engine-and-llm.md) (1.L next); see -[docs/roadmap/current.md](docs/roadmap/current.md). +the monorepo + `@relavium/shared` + CI + `@relavium/db`. The +[`@relavium/core` engine](docs/roadmap/phases/phase-1-engine-and-llm.md) has since landed the full +run-loop + node stack — parser, interpolation, DAG/`RunPlan`, the run loop + `RunEventBus`, the tool +registry, the `AgentRunner`, the six node-type handlers, the human gate, checkpoint/resume, node retry, +the expression sandbox, and the pre-egress budget governor — plus the agent-first `AgentSession` (1.V), +**completing milestone 1.m4** (PRs #13–#26). Next on the critical path is the **1.U** end-to-end Node +harness (milestone **M2**); see [docs/roadmap/current.md](docs/roadmap/current.md) for live status. ## The non-negotiable rules diff --git a/CLAUDE.md b/CLAUDE.md index 3a33f950..23fc094d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,7 +38,7 @@ A run executes in one of **three execution modes** behind the one `LLMProvider` engine is identical across all three. See [ADR-0012](docs/decisions/0012-managed-inference-dual-mode.md) to [ADR-0015](docs/decisions/0015-managed-mode-data-handling-and-compliance.md) and [docs/architecture/managed-inference.md](docs/architecture/managed-inference.md). -**Status: Phase 1 in progress — milestone M1 (LLM seam proven) reached (PR #9, 2026-06-07); the `FallbackChain` runner (1.K) landed, completing 1.m2 with the cost tracker (PR #13, 2026-06-11); the run loop (1.N — `WorkflowEngine` + `RunEventBus`) landed (PR #17, 2026-06-13) **completing 1.m3** (parse → DAG → run loop emits the canonical event stream), with the built-in `ToolRegistry` (1.T, a 1.m4 component) landing alongside it as the other `AgentRunner` (1.O) join prerequisite; the **`AgentRunner` (1.O) — per-node LLM execution behind the seam — landed (PR #18, 2026-06-14)**; and the **node-type handlers (1.P) — the six non-agent `NodeExecutor` arms (condition / transform / fan_out / fan_in / input / output) behind a dispatching executor — landed (PR #20, 2026-06-14)**; and **checkpoint/resume (1.R) + the human gate (1.Q) landed (PR #22, 2026-06-15)** — the derived `Checkpointer` + cross-process `resumeFromCheckpoint`, and the `human_in_the_loop` gate with the one-shot timeout port; and **node retry (1.S) — the above-chain whole-node retry budget ([ADR-0040](docs/decisions/0040-node-retry-budget-above-the-chain.md), amending ADR-0038) — landed (PR #24, 2026-06-15)**, re-dispatching a whole node on a retryable failure up to `retry.max` attempts (with `node:retrying`, abort-aware backoff, and `retry_on` filtering), with retry-from-node (ADR-0040 Part B) deferred to Phase-2. The pre-egress budget governor (1.AC) is next, toward M2.** +**Status:** Phase 1 in progress — milestone M1 (LLM seam proven) reached (PR #9, 2026-06-07); the `FallbackChain` runner (1.K) landed, completing 1.m2 with the cost tracker (PR #13, 2026-06-11); the run loop (1.N — `WorkflowEngine` + `RunEventBus`) landed (PR #17, 2026-06-13) **completing 1.m3** (parse → DAG → run loop emits the canonical event stream), with the built-in `ToolRegistry` (1.T, a 1.m4 component) landing alongside it as the other `AgentRunner` (1.O) join prerequisite; the **`AgentRunner` (1.O) — per-node LLM execution behind the seam — landed (PR #18, 2026-06-14)**; and the **node-type handlers (1.P) — the six non-agent `NodeExecutor` arms (condition / transform / fan_out / fan_in / input / output) behind a dispatching executor — landed (PR #20, 2026-06-14)**; and **checkpoint/resume (1.R) + the human gate (1.Q) landed (PR #22, 2026-06-15)** — the derived `Checkpointer` + cross-process `resumeFromCheckpoint`, and the `human_in_the_loop` gate with the one-shot timeout port; and **node retry (1.S) — the above-chain whole-node retry budget ([ADR-0040](docs/decisions/0040-node-retry-budget-above-the-chain.md), amending ADR-0038) — landed (PR #24, 2026-06-15)**, re-dispatching a whole node on a retryable failure up to `retry.max` attempts (with `node:retrying`, abort-aware backoff, and `retry_on` filtering), with retry-from-node (ADR-0040 Part B) deferred to Phase-2; and the **pre-egress budget governor (1.AC, [ADR-0028](docs/decisions/0028-workflow-resource-governance.md)) + the `AgentSession` agent-first entry point (1.V, [ADR-0024](docs/decisions/0024-agent-first-entry-point-agentsession.md)) landed together (PR #26, 2026-06-16)** — 1.AC was the last 1.m4 component, so **1.m4 is complete** (the full engine stack: node handlers, gate, checkpoint/resume, retry, tools, sandbox, budget governor), and 1.V opens the Lane-C agent-first sub-spine (1.m5). The end-to-end Node harness (1.U) — the **M2 critical-path milestone**, now unblocked — is next, with Lane C continuing at session events (1.W) + persistence (1.X). Phase 0 (M0, 2026-06-04) landed the monorepo, strict toolchain + CI, `@relavium/shared` (the full Zod contract set), the no-vendor-type seam fence, and `@relavium/db`. Phase 1 has since landed `@relavium/llm` — the `LLMProvider` seam + all three adapters (Anthropic, OpenAI/DeepSeek, @@ -60,7 +60,9 @@ executor-only with a `secretInputNames` masking gate on `NodeExecContext`), and `run_events` log, no checkpoint table — ADR-0003) + cross-process `resumeFromCheckpoint` with idempotent re-delivery and a `workflow_mismatch` identity guard, and the `human_in_the_loop` gate's suspend/resume plus the one-shot `setTimer` timeout port — `approve` auto-resolves, `reject` fails with `run_timeout`). -Active work is now the last 1.m4 workstream — the pre-egress budget governor (1.AC) — toward **M2**; see +The last 1.m4 workstream — the pre-egress budget governor (1.AC) — and the agent-first `AgentSession` (1.V) +landed together (PR #26, 2026-06-16), **completing 1.m4**; active work is now the **1.U** end-to-end Node +harness (the **M2** milestone) plus Lane C's session events (1.W) + persistence (1.X); see [docs/roadmap/current.md](docs/roadmap/current.md). See [README.md](README.md) for the public overview. ## Non-negotiable rules for AI agents diff --git a/README.md b/README.md index 428bacec..3e940bc4 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,10 @@ The **run loop** (1.N — `WorkflowEngine` + `RunEventBus`) landed (PR #17, 2026 milestone 1.m3** (parse → DAG → run loop emits the canonical event stream); the **built-in `ToolRegistry`** (1.T, a 1.m4 component) landed alongside it; the **`AgentRunner`** (1.O — per-node LLM execution behind the seam) landed (PR #18, 2026-06-14); and the **node-type handlers** -(1.P — the six non-agent handlers behind a dispatching executor) landed (PR #20, 2026-06-14). Next on -the critical path: the **human gate** (1.Q), checkpoint/resume and retry, plus the **AgentSession** -runtime + export-to-workflow sub-spine. See +(1.P — the six non-agent handlers behind a dispatching executor) landed (PR #20, 2026-06-14), followed by +the **human gate** (1.Q) + **checkpoint/resume** (1.R, PR #22), **node retry** (1.S, PR #24), and the +**pre-egress budget governor** (1.AC) together with the agent-first **`AgentSession`** entry point (1.V) — +both landed in **PR #26 (2026-06-16)**. With the budget governor in, **milestone 1.m4 is complete** (the full +engine stack); next on the critical path is the **end-to-end Node harness** (1.U) — milestone **M2** — with the +agent-first sub-spine (session events 1.W, persistence 1.X) continuing in parallel. See [docs/roadmap/current.md](docs/roadmap/current.md) for live status. diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 470031ed..475851b7 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -2,7 +2,7 @@ > Status: Living -> Last updated: 2026-06-15 +> Last updated: 2026-06-16 - **Related**: [README.md](README.md), [phases/phase-0-foundations.md](phases/phase-0-foundations.md), [phases/phase-1-engine-and-llm.md](phases/phase-1-engine-and-llm.md), [../project-structure.md](../project-structure.md), [../tech-stack.md](../tech-stack.md) @@ -86,7 +86,9 @@ consuming `@relavium/db` for run persistence). Global milestone **M1 — LLM seam proven** is reached (PR #9, 2026-06-07): all three adapters pass the shared conformance suite behind the frozen seam. The next checkpoint is -**M2 — engine end-to-end** (see the [milestone spine](README.md#global-milestone-spine)). +**M2 — engine end-to-end**, now gated **only** by the **1.U** end-to-end Node harness — the rest +of the engine (milestone **1.m4**) completed with the pre-egress budget governor in PR #26 (see the +[milestone spine](README.md#global-milestone-spine)). > **One Phase-0 follow-up lives outside the code:** a maintainer should mark the CI `ci` > job a **required check** in GitHub branch protection (optionally adding `TURBO_TOKEN`/ @@ -155,8 +157,16 @@ auto-resolves, `reject` fails with `run_timeout`). **Node retry (1.S) is ✅ Don ([ADR-0040](../decisions/0040-node-retry-budget-above-the-chain.md) Part A: re-dispatch a whole node on a retryable, `retry_on`-admitted failure up to `retry.max` attempts with abort-aware backoff and the non-terminal `node:retrying`, `node:failed` staying the single terminal; the user-triggered retry-from-node Part B is -deferred to Phase-2). The lane now continues at the last **1.m4** workstream toward **M2** — the **pre-egress -budget governor (1.AC)** — and the agent-first sub-spine (**1.V–1.AA**, Lane C) is open now that 1.O exists. +deferred to Phase-2). The last **1.m4** workstream — the **pre-egress budget governor (1.AC)** ([ADR-0028](../decisions/0028-workflow-resource-governance.md): +the `BudgetGovernor` pre-egress cost gate, `on_exceed` warn/fail/pause_for_approval, `budget:warning`/`budget:paused`/`run:timeout`, +the H3 approve-continues bypass, the per-attempt `FallbackChain` enforcement) — and the agent-first **`AgentSession` +(1.V)** entry point ([ADR-0024](../decisions/0024-agent-first-entry-point-agentsession.md): multi-turn +`start`/`sendMessage`/`cancel` over the shared `runAgentTurn` core, the hard turn cap → `turn_limit`, cost + +emission via an injected `SessionEventSink`) **then landed together — ✅ Done (PR #26, 2026-06-16)**. 1.AC closed +**1.m4** (the full engine stack), so the critical path now reaches **1.U — the end-to-end Node harness (the M2 +milestone)**, now unblocked; in parallel, **Lane C** (the 1.m5 sub-spine) continues from 1.V at **1.W** (wire the +`SessionEventSink` onto the `RunEventBus` + per-session `sequenceNumber`/`SessionHandle`) and **1.X** (session +persistence), with cost-event persistence still a tracked deferral. > **Multimodal I/O — the shape is landed (1.AD ✅ Done, PR #11, 2026-06-10).** First-class > image/audio/video I/O (input **and** output, incl. generate-media-by-rule) was decided on 2026-06-08: @@ -186,8 +196,9 @@ budget governor (1.AC)** — and the agent-first sub-spine (**1.V–1.AA**, Lane > other 1.O join prerequisite; **the `AgentRunner` join (1.O) is ✅ Done (PR #18, 2026-06-14)**; and the > **node-type handlers (1.P) are ✅ Done (PR #20, 2026-06-14)**; **checkpoint/resume (1.R) + the > human gate (1.Q) are ✅ Done (PR #22, 2026-06-15)**; and **node retry (1.S) is ✅ Done (PR #24, 2026-06-15)** -> (ADR-0040 Part A; the user-triggered retry-from-node Part B is deferred to Phase-2). The **pre-egress budget -> governor (1.AC)** is the next workstream. +> (ADR-0040 Part A; the user-triggered retry-from-node Part B is deferred to Phase-2); and the **pre-egress budget +> governor (1.AC) + the `AgentSession` (1.V) entry point are ✅ Done (PR #26, 2026-06-16)** — 1.AC closed **1.m4**. +> The next workstream is **1.U** (the end-to-end Node harness, the M2 milestone), with Lane C continuing at 1.W/1.X. Carry-over hardening is tracked in [deferred-tasks.md](deferred-tasks.md) — pick items up as Phase 1 first touches each file. diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index a0667227..47429e34 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -2,7 +2,7 @@ > Status: Living -> Last updated: 2026-06-15 +> Last updated: 2026-06-16 - **Related**: [current.md](current.md), [README.md](README.md), [phases/phase-0-foundations.md](phases/phase-0-foundations.md) @@ -183,15 +183,19 @@ Severity is the review's verified rating. Check an item off in the PR that resol the host's run-scoped `outputStore` (handle in the marker); applied in `registry.dispatch` (returns `truncated`) under the one cancellation-precedence ladder. Documented in [tool-registry.md §result-bounding-and-spill-to-file](../reference/shared-core/tool-registry.md#result-bounding-and-spill-to-file). -- [ ] **Cumulative cost is not restored on cross-process resume (cost-event persistence) — 1.AC/1.R.** `cost:updated` - is **streamed** (`#nodeEmit` → bus), not persisted via `#emitDurable`, so `reconstructCheckpointState`'s - `cost:updated` fold (checkpoint.ts) never sees it: a resumed run's `cumulativeCostMicrocents` (and the - governor) restart near 0, under-reporting `run:completed.totalCostMicrocents` and under-blocking the budget - after resume. Partially mitigated (2026-06-15): the fold now restores the cumulative from the **durable** - `budget:paused.spentMicrocents`, so a run paused at a **budget** gate resumes with the right spend; a budgeted - run paused at a **plain human** gate (or crashed mid-run) still loses its cost. The general fix is making cost - durable — persist `cost:updated`, or carry `cumulativeCostMicrocents` on the durable `node:completed` (like - `tokensUsed`) — a contract/durability-model change. *(medium · packages/core/src/engine/engine.ts `#nodeEmit` + checkpoint.ts; 1.AC/1.R)* +- [x] **Cumulative cost is not restored on cross-process resume (cost-event persistence) — 1.AC/1.R.** **Done + (maintainer-approved, the node:completed-carry variant).** `cost:updated` is streamed (`#nodeEmit` → bus), + not persisted, so the `reconstructCheckpointState` fold never saw it — a resumed run's + `cumulativeCostMicrocents` (and the governor) restarted near 0. **Fix:** the durable `node:completed` now + carries an optional `cumulativeCostMicrocents` (run-event.ts) — a snapshot of the run-wide running total at + the node boundary, populated by the engine (`#completeNode`) and folded on resume with a monotonic `Math.max` + that reconciles with the existing `budget:paused.spentMicrocents` restore (checkpoint.ts). So a run paused at + **any** gate (plain human OR budget) now resumes with the right spend; a gate-less crashed-mid-run is + reconciled to `run:failed` (not resumed), so its cost-loss is moot. Chosen over persisting `cost:updated` + (which would add hot-path durable writes + a delivery-ordering change): zero new events, one additive + forward-compatible field, folds at boundaries the store already persists — no ADR needed. Pinned by a + checkpoint.test.ts unit test (plain-human-gate restore) + the 1.U flagship harness (post-resume + `run:completed.totalCostMicrocents` reflects the pre-gate cost). *(packages/core/src/engine/engine.ts `#completeNode` + checkpoint.ts; packages/shared/src/run-event.ts; 1.AC/1.R)* - [ ] **Pre-egress token-estimate accuracy — watch item (1.AC).** The ADR-0028 governor blocks on `worstCaseNextEstimate(maxTokens)` from `[defaults].max_tokens_estimate`. Record the open question: does the estimate need provider-accurate token counting (from the seam's model meta / @@ -249,9 +253,14 @@ Severity is the review's verified rating. Check an item off in the PR that resol `tool_call` part has no field for it, so Gemini 3 function-calling continuations cannot replay it (and can themselves 400). Needs a continuation-metadata carrier on the canonical `tool_call`/`reasoning` parts plus adapter capture/replay. *(high · packages/llm/src/adapters/gemini.ts:193-198, packages/shared/src/content.ts:419-441; ADR-0030 follow-up)* -- [ ] **DeepSeek surviving-reasoning replay** — the same per-provider contract applies; confirm whether the - OpenAI-compatible adapter normalizes/replays `reasoning_content` on a same-provider continuation, or currently - drops it. *(medium · packages/llm/src/adapters/openai.ts; ADR-0030 follow-up)* +- [x] **DeepSeek surviving-reasoning replay** — **Confirmed correct + locked (engine-hardening pass).** The + OpenAI-compatible adapter CAPTURES `reasoning_content` inbound (`mapContent` → a `reasoning` part) but + intentionally **drops it on egress** (`toOpenAiMessages` lowers only text + tool_call parts; openai.ts:256-260, + "reasoning is ephemeral and never replayed, ADR-0030"). For DeepSeek this is the CORRECT direction: + `reasoning_content` is output-only — the API 400s if it is echoed back in an input message, and + deepseek-reasoner does not need prior reasoning to continue. So no seam-shape carrier is needed (unlike the + Anthropic-redacted / Gemini-thoughtSignature items above). Pinned by an openai.test.ts lock-test (a prior-turn + reasoning part never reaches the request body). *(packages/llm/src/adapters/openai.ts; ADR-0030/0039)* - [ ] **`output_schema` deep JSON-Schema conformance** — 1.O validates an `agent` node's `output_schema` node-side but **parse-as-JSON only** (the seam's `responseFormat` is a request hint; a schema-violating-but-valid JSON output, e.g. `{"wrong":true}` for a `{ n: number }` schema, currently @@ -264,12 +273,14 @@ Severity is the review's verified rating. Check an item off in the PR that resol `agent:token.model` uses `activeModel` (updated from the *succeeding* attempt record, which fires after the stream), so a *cross-model pre-content failover* attributes that turn's tokens to the prior model. A precise fix needs a `FallbackChain` `onAttemptStart`/attributed-stream hook (a seam change). *(low · packages/core/src/engine/agent-turn.ts; packages/llm/src/fallback-chain.ts)* -- [ ] **Per-attempt pre-egress budget gate (1.AC)** — 1.O leaves a coarse always-pass hook at the tool-loop - turn boundary; the precise per-egress budget check (a `FallbackChain` makes several attempts per turn) is a - chain pre-attempt hook 1.AC adds. *(medium · ADR-0028; ADR-0038; 1.AC)* +- [x] **Per-attempt pre-egress budget gate (1.AC)** — closed by 1.AC (PR #26). The precise per-egress budget + check now rides the `FallbackChain` **pre-attempt** hook, so every attempt — including a failover to a pricier + model — is capped; the loop-top `awaitPreEgress` in `runAgentTurn` adds the zero-egress-on-cancel guard + + primary-model early check (the intentional double gate). *(closed · ADR-0028; ADR-0038; 1.AC, PR #26)* - [ ] **Multi-tool result ordering in the turn core** — `dispatchToolCalls` appends tool-result messages in - dispatch-completion order; for v1.0 (single tool call per `tool_use` stop) this is moot, but a parallel-tool - provider should order by the accumulator's `toolOrder` before 1.V reuses the core. *(low · packages/core/src/engine/agent-turn.ts; 1.V)* + dispatch-completion order; for v1.0 (single tool call per `tool_use` stop) this is moot — and 1.V now reuses + the core on that single-tool path. A parallel-tool provider should order by the accumulator's `toolOrder`; + re-home to whatever future parallel-tool work first enables it. *(low · packages/core/src/engine/agent-turn.ts; future parallel-tool)* - [x] **Secret-into-`run.outputs` runtime taint (ADR-0029(c) follow-up)** — an `agent` node cannot launder a secret into `run.outputs` (it emits LLM text only), so this is **not** 1.O's to own; it belongs to the `transform` / sandbox node (1.P / 1.AB) that can return a secret-derived value. 1.O's only obligation is to @@ -300,10 +311,13 @@ Severity is the review's verified rating. Check an item off in the PR that resol **✅ Added (hardening pass):** `agent-runner.e2e.test.ts` "runs two agent nodes concurrently against the shared executor" — `max_parallel: 2`, two agent vertices on one executor instance, asserting a gap-free global sequence and that each node's `agent:token` events carry their own `nodeId` (no cross-node bleed). -- [ ] **Combined tool-loop DoS bound (turns × corrections)** — `maxToolTurns` (16) and `maxToolCorrections` - (3) are independent budgets; their *product* bounds worst-case egress and the interleaving (a turn mixing - correctable + genuine tool rounds) is untested. Document the combined bound and add an interleaving test. - *(low · packages/core/src/engine/agent-turn.ts)* +- [x] **Combined tool-loop DoS bound (turns × corrections)** — **Done (engine-hardening pass).** The + "product" framing was imprecise: the bounds are NOT multiplicative. `maxToolTurns` is the worst-case + **egress ceiling** (≤ `maxToolTurns + 1` provider calls); `maxToolCorrections` is a **monotonic sub-budget** + that can only end the turn EARLY with `tool_failed` (a genuine round never resets it). Documented on + `AgentTurnLimits` (agent-turn.ts) and pinned by an interleaving test (correctable / genuine / correctable / + correctable → `tool_failed` at turn 4, far under `maxToolTurns`), asserting corrections accumulate across the + interleaved genuine round and egress stays bounded. *(low · packages/core/src/engine/agent-turn.ts)* - [ ] **Multimodal tool-result through the adjacent-message + redaction paths** — all 1.O coverage exercises text/JSON tool args + content; confirm image/media tool-result blocks survive the Anthropic adjacent-role merge (no dropped blocks / no double-merge with `stripReasoningParts`) and the redaction path. *(low · packages/llm/src/adapters/anthropic.ts; 1.AF)* @@ -397,13 +411,15 @@ Severity is the review's verified rating. Check an item off in the PR that resol ## AgentSession (1.V) follow-ups -> **2026-06-15 1.V implementation (ADR-0024) + two pre-merge review passes.** The in-memory `AgentSession` -> entry point landed — multi-turn `start`/`sendMessage`/`cancel` over the **shared turn core** (`runAgentTurn`), -> the hard turn cap → `turn_limit`, session-wide cost, emission via an injected `SessionEventSink`. The -> deferrals below were decided while building it; each has a clear later home, recorded so it isn't lost. -> (The bus wiring + `SessionHandle` is the scheduled **1.W** workstream, persistence + the durable -> `SessionMessage` schema is **1.X**, resume **1.Y**, export **1.Z** — those are workstreams, tracked in -> [phase-1-engine-and-llm.md](phases/phase-1-engine-and-llm.md), not deferred items.) +> **2026-06-16 — 1.V `AgentSession` (ADR-0024) + 1.AC budget governor (ADR-0028) merged in PR #26** (after two +> pre-merge review passes + a Sonnet multi-dimensional review). The in-memory `AgentSession` entry point landed — +> multi-turn `start`/`sendMessage`/`cancel` over the **shared turn core** (`runAgentTurn`), the hard turn cap → +> `turn_limit`, session-wide cost, emission via an injected `SessionEventSink`. The deferrals below were decided +> while building it; each has a clear later home, recorded so it isn't lost. The still-open follow-ons are **1.W** +> (wire the `SessionEventSink` onto the `RunEventBus` + per-session `sequenceNumber`/`SessionHandle`), **1.X** +> (session persistence + the durable `SessionMessage` schema), resume **1.Y**, export **1.Z**, and the deferred +> cost-event persistence (below) — those are workstreams, tracked in +> [phase-1-engine-and-llm.md](phases/phase-1-engine-and-llm.md), not deferred items. - [ ] **Faithful cross-turn transcript (tool + reasoning history) → 1.X/1.Z.** 1.V appends only the final assistant **text** across turns: the turn core keeps the within-turn `tool_use`/`tool_result` pairs internal @@ -502,14 +518,14 @@ Severity is the review's verified rating. Check an item off in the PR that resol accurate from the repo root. Make the glob cwd-tolerant (or document root-only) and add the testing.md **≥90% line+branch** threshold for the engine packages (`packages/core`, `packages/llm`) — per-area, since surfaces are smoke-only. *(minor · vitest.config.ts)* -- [ ] **Coverage floor fires only on a repo-root run + is not a CI gate** — two residues of the - item above (PR #10 review, verified empirically): (1) the per-glob threshold key - (`packages/llm/src/**/*.ts`) is root-relative while a package-scoped `--coverage` run keys the - coverage map cwd-relative (`src/…`), so the floor silently does not fire there — and no single - glob can fix it without wrongly binding shared/db package runs to the engine floor (documented - at the thresholds block). (2) `pnpm coverage` is not yet a required CI step. Resolve both at - once when wiring coverage into CI: a root-run `pnpm coverage` CI step makes the root-relative - glob authoritative and the package-scoped gap moot. *(minor · vitest.config.ts, ci.yml)* +- [x] **Coverage floor fires only on a repo-root run + is not a CI gate** — **Done (engine-hardening + pass, advisory).** Added a repo-ROOT `pnpm coverage` CI job (ci.yml) — a root run is exactly what makes the + root-relative per-glob thresholds (`packages/core|llm/src/**`) authoritative, so the package-scoped cwd gap + (residue 1) is moot. The job is **advisory** (a separate, non-required job like `peer-dep-gate`) so it + surfaces a regression without blocking merge while the core-package **branch** margin is thin (90.29% vs the + 90% floor); promote it to a required check once that margin is confirmed stable under CI's Node 22. The + cwd-sensitivity itself stays documented at the thresholds block (a single glob cannot fix it without wrongly + binding shared/db runs). *(minor · ci.yml; vitest.config.ts)* - [x] **Column-level schema fidelity** — `client.test.ts` proves only that table *names* exist. Add a `PRAGMA table_info()` assertion per table (name/type/notnull/dflt/pk) against an expected fixture, or snapshot `0000_*.sql` byte-for-byte. *(minor · packages/db/src/client.test.ts)* @@ -532,14 +548,15 @@ Severity is the review's verified rating. Check an item off in the PR that resol rejects, pinning the record boundary. *(nit · run.test.ts, run-event.test.ts)* - [x] **Round-trip fixture verbatim** — the workflow no-drift fixture paraphrases multi-line prompts; transcribe them verbatim from the spec or soften the "verbatim" claim. *(nit · workflow.test.ts)* -- [ ] **Conformance: tool-loop + cache-hit recorded scenarios (1.F follow-up)** — the shared - conformance suite covers text / single tool-call / usage / stop / error, but not a **multi-turn - tool loop** (call → result → continuation on the same provider, the path every agent node - exercises) or a **prompt-cache-hit** response (cached-token usage fields folding into the one - canonical `Usage`). Add both as recorded scenarios, and grow a small provider-quirk fixture bank - (reasoning-field variants, tool-call adjacency rules) as quirks are met in the adapters — the - suite is where quirk knowledge belongs, not adapter comments. *(packages/llm conformance; next - adapters-touching PR wave)* +- [x] **Conformance: tool-loop + cache-hit recorded scenarios (1.F follow-up)** — **Done + (engine-hardening pass).** Both landed as recorded scenarios across all four provider suites: (1) a + **multi-turn tool loop** — a new `replayFetchSequence` (+ a `replayFor` single-vs-sequence router; the + Gemini transport indexes per call) drives two generate() calls against one adapter, so turn 2 exercises the + adapter lowering a `tool_result` message back onto the provider's wire (the call → result → continuation + path every agent node runs); and (2) a **prompt-cache-hit** assertion — `ConformanceExpectations.textGenerate` + gained an optional `cacheReadTokens`, asserted in the textGenerate test (DeepSeek's fixture already records + `prompt_cache_hit_tokens: 4` → net input 8, cacheRead 4 folds into the one canonical `Usage`). The + provider-quirk fixture bank can still grow opportunistically as new quirks are met. *(packages/llm conformance)* ## Tooling / CI diff --git a/docs/roadmap/phases/phase-1-engine-and-llm.md b/docs/roadmap/phases/phase-1-engine-and-llm.md index 3af530e1..06f8cae6 100644 --- a/docs/roadmap/phases/phase-1-engine-and-llm.md +++ b/docs/roadmap/phases/phase-1-engine-and-llm.md @@ -27,9 +27,13 @@ > `human_in_the_loop` gate (suspend/resume + the one-shot `setTimer` timeout port: `approve` auto-resolves, > `reject` fails with `run_timeout`). **Node retry (1.S) is ✅ Done (PR #24, 2026-06-15)** — the above-chain > whole-node retry budget ([ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md), Part A; the -> user-triggered retry-from-node Part B is deferred to Phase-2). The lane now continues at the last 1.m4 -> workstream — **1.AC** budget governor — toward **M2**, and Lane C (1.V–1.AA) opens. *(Session persistence, -> 1.X/1.Z, must exclude the reasoning signature — non-persisting.)* +> user-triggered retry-from-node Part B is deferred to Phase-2). **The pre-egress budget governor (1.AC) and +> the `AgentSession` agent-first entry point (1.V) then landed together — ✅ Done (PR #26, 2026-06-16).** 1.AC +> was the last **1.m4** workstream, so **1.m4 is complete** (the full engine stack — node handlers, gate, +> checkpoint/resume, retry, tools, sandbox, budget governor); 1.V opens **Lane C** (the agent-first sub-spine, +> 1.m5). The critical path now reaches **1.U — the end-to-end Node harness (the M2 milestone)**, now unblocked, +> with Lane C continuing at session events (1.W) ‖ persistence (1.X). *(Session persistence, 1.X/1.Z, must +> exclude the reasoning signature — non-persisting.)* > > **Multimodal I/O decided (2026-06-08).** First-class image/audio/video I/O (input **and** output) is a > second pre-freeze seam amendment in the ADR-0030 mould — [ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md) @@ -815,8 +819,8 @@ The engine-side registry that dispatches built-in tools the `AgentRunner` invoke correctly, and emits sanitized tool events; an unlisted `run_command` is refused; `git_commit` is blocked without a gate approval; and an **upstream agent/LLM output wired via `input_mapping` into a tool / `http_request` arg is treated as UNTRUSTED data** — schema-validated against -the tool's declared arg and, **for an outbound-URL tool**, routed through the **same** exact-FQDN allow-list -+ SSRF range-block regardless of provenance (ADR-0029(d) enumerates `baseURL` / `http_request` / MCP), so a +the tool's declared arg and, **for an outbound-URL tool**, routed through the **same** exact-FQDN allow-list + +SSRF range-block regardless of provenance (ADR-0029(d) enumerates `baseURL` / `http_request` / MCP), so a derived URL cannot bypass the egress guard. *(A `web_search`-style query is untrusted-data schema-validated per ADR-0029(c) but transits a query, not an `allowedDomains` FQDN allowlist.)* @@ -838,12 +842,32 @@ The proof that the engine works before any surface exists. then fallback with the run still completing; resume from a mid-run checkpoint reproduces the same final output — **M2 achieved**. +> **Harness shape (decided 2026-06-16, implementing 1.U).** The harness is a **scenario suite** with a +> reusable driver (not a single test) — the seed the Phase-2 CLI regression harness (2.K) grows from, and +> the suite 1.AB's `condition`/`transform` scenario and the determinism ban plug into. Its **happy-path** +> member is the literal 3-node `input → agent → output` (a clean run: live token streaming + a tool call + +> per-attempt cost + a gap-free `sequenceNumber` stream that validates against the canonical +> [`RunEventSchema`](../../reference/contracts/sse-event-schema.md)). Its **flagship** member inserts a +> `human_in_the_loop` gate — `input → agent → human_gate → output` — as the **durable mid-run checkpoint**, +> because the Phase-1 engine resumes **only** from a durable suspend point (a human/budget gate): a gate-less +> interrupted run is reconciled to `run:failed`, never resumed +> ([ADR-0036](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md); the derived +> `Checkpointer`, ADR-0003). The flagship drives, **in one run** across a process boundary: the agent's +> forced provider error → **node retry** ([ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md) +> Part A) → **failover** to the second chain entry (1.K), streaming + a tool call + per-attempt cost; then the +> gate pause (the persisted checkpoint); then a **fresh engine** resumes via `resumeFromCheckpoint`, runs +> `output` to `run:completed`, and reproduces the same final output with `sequenceNumber` continuing gap-free. +> All LLM cost is incurred **pre-gate**, so the plain-human-gate cost-restore deferral +> ([deferred-tasks.md](../deferred-tasks.md)) is off this path. **No new engine code or ADR** — the harness +> composes 1.K/1.N/1.O/1.P/1.Q/1.R/1.S/1.T/1.AB behind the `@relavium/llm` seam and uses only already-exported +> `@relavium/core` symbols. + ### Agent-first sub-spine (1.V–1.AA) — additive, parallel to the M2 critical path These build the `AgentSession` entry point ([ADR-0024](../../decisions/0024-agent-first-entry-point-agentsession.md)). They run **parallel** to 1.L–1.U and do **not** feed the 1.U workflow harness — each is proven by its own harness (1.AA). The `WorkflowEngine` is unchanged; `AgentSession` is an additional entry point on the same substrate. -- **1.V — `AgentSession` entry point.** Wrap `AgentRunner` in a multi-turn session (session context, one bound agent + its fallback chain). This workstream also settles the session's **hard turn/round cap** knob — deliberately distinct from `[chat].max_messages`, which is a history-**trim** threshold ([config-spec.md](../../reference/contracts/config-spec.md)) that continues the session. A session that reaches the hard cap ends **loudly**: `session:turn_completed` carries `error.code: 'turn_limit'` ([sse-event-schema.md](../../reference/contracts/sse-event-schema.md#error-code-taxonomy)) — never a silent stop — and the behavior is pinned by a dedicated regression test (a refactor of the turn loop must not be able to silently drop the cap signal). Context compaction (when it lands, later phases) is **append-only by principle**: the persisted transcript is never rewritten or trimmed in place; `agent:context_compacted` is the reserved signal ([sse-event-schema.md](../../reference/contracts/sse-event-schema.md#workflow-governance-and-reserved-events)). *Acceptance:* a session runs a multi-turn conversation with a tool round-trip through the same `AgentRunner` path a workflow agent node uses; a session driven to its hard turn cap emits the `turn_limit`-coded event, regression-pinned. - - *1.V scope notes (decided 2026-06-15, [agent-session-spec.md](../../reference/contracts/agent-session-spec.md)):* the driver reuses the correlation-agnostic turn core (`runAgentTurn`) directly via an **injected `SessionEventSink`** (so the bus wiring is **1.W** — see below); the hard cap is an **engine-API knob** (finite default **50**), with the `[chat]` surface-default mapping deferred to the surface phases (it is **not** a Phase-1 `[chat]` field). **Deferred within 1.V, to be picked up later:** the cross-turn transcript is **text-only** — the turn core keeps the within-turn `tool_use`/`tool_result` pairs internal (returns only the final non-`tool_use` content), so the transcript carries no orphaned `tool_use` and stays protocol-valid; the assistant reply is appended as text, **dropping reasoning** (a `signature` is a within-turn same-provider replay token, ADR-0030/0039, that must not span turns). Faithful cross-turn tool/reasoning history is **revisited when 1.X persistence / 1.Z export needs it** (it needs the turn core to expose the intermediate messages, currently owned by the concurrent 1.AC edits in `agent-turn.ts`); **per-session tool narrowing** (ADR-0029 narrow-only — 1.V grants `agent.tools` as-is, no per-session narrow); and **session `output_schema`** (chat is free-form text — structured output stays a workflow concern). +- **1.V — `AgentSession` entry point — ✅ Done (PR #26, 2026-06-16).** Wrap `AgentRunner` in a multi-turn session (session context, one bound agent + its fallback chain). This workstream also settles the session's **hard turn/round cap** knob — deliberately distinct from `[chat].max_messages`, which is a history-**trim** threshold ([config-spec.md](../../reference/contracts/config-spec.md)) that continues the session. A session that reaches the hard cap ends **loudly**: `session:turn_completed` carries `error.code: 'turn_limit'` ([sse-event-schema.md](../../reference/contracts/sse-event-schema.md#error-code-taxonomy)) — never a silent stop — and the behavior is pinned by a dedicated regression test (a refactor of the turn loop must not be able to silently drop the cap signal). Context compaction (when it lands, later phases) is **append-only by principle**: the persisted transcript is never rewritten or trimmed in place; `agent:context_compacted` is the reserved signal ([sse-event-schema.md](../../reference/contracts/sse-event-schema.md#workflow-governance-and-reserved-events)). *Acceptance:* a session runs a multi-turn conversation with a tool round-trip through the same `AgentRunner` path a workflow agent node uses; a session driven to its hard turn cap emits the `turn_limit`-coded event, regression-pinned. + - *1.V scope notes (decided 2026-06-15, [agent-session-spec.md](../../reference/contracts/agent-session-spec.md)):* the driver reuses the correlation-agnostic turn core (`runAgentTurn`) directly via an **injected `SessionEventSink`** (so the bus wiring is **1.W** — see below); the hard cap is an **engine-API knob** (finite default **50**), with the `[chat]` surface-default mapping deferred to the surface phases (it is **not** a Phase-1 `[chat]` field). **Deferred within 1.V, to be picked up later:** the cross-turn transcript is **text-only** — the turn core keeps the within-turn `tool_use`/`tool_result` pairs internal (returns only the final non-`tool_use` content), so the transcript carries no orphaned `tool_use` and stays protocol-valid; the assistant reply is appended as text, **dropping reasoning** (a `signature` is a within-turn same-provider replay token, ADR-0030/0039, that must not span turns). Faithful cross-turn tool/reasoning history is **revisited when 1.X persistence / 1.Z export needs it** (it needs the turn core to expose the intermediate messages in `agent-turn.ts` — 1.V and 1.AC landed together in PR #26, so that core is now settled); **per-session tool narrowing** (ADR-0029 narrow-only — 1.V grants `agent.tools` as-is, no per-session narrow); and **session `output_schema`** (chat is free-form text — structured output stays a workflow concern). - **1.W — `session:*` event namespace.** Emit session lifecycle events on the shared `RunEventBus` with the same `sequenceNumber` gap/resync logic ([sse-event-schema.md](../../reference/contracts/sse-event-schema.md)). *Acceptance:* session events are disjoint from `run:*` and gap-detected identically. - *Also owns (from the 1.V injected-sink split, recorded 2026-06-15):* the `SessionEventSink → RunEventBus` adapter; a **`SessionHandle`** (the async-iterable session stream + `cancel`, mirroring `createRunHandle`); and **reconciling the bus session-event gate** — `RunEventBus` validates against `RunEventSchema` and its `RunEventDraft` type both **exclude** the five `session:*` variants (they live only in the separate `SessionEventSchema`), so the bus needs an injected/combined Run+Session schema **and** the matching draft-type widening before it can carry the session lifecycle events 1.V emits. - **1.X — Session persistence.** `agent_sessions` + `session_messages` via `@relavium/db` into `history.db` ([database-schema.md](../../reference/desktop/database-schema.md)). **Authors the durable `SessionMessageSchema`** in `@relavium/shared` (deferred from 1.V, which runs on the in-flight `LlmMessage` form): `{ id, sessionId, sequenceNumber, role, content: DurableContentPart[], modelId?, timestamp }` — the persisted transcript type these tables store. *Acceptance:* a session round-trips to the DB and resumes. **Note:** adding these two tables requires a regenerated Drizzle migration snapshot (the schema-migration drift CI gate). **ADR-0030 ephemerality:** a `reasoning` part's `signature`/`redacted` continuity token must **not** be persisted to `session_messages` — strip it (keep reasoning *text* if a transcript needs it, drop the opaque signature). *Acceptance also asserts:* a round-tripped session row carries no reasoning `signature`. @@ -857,17 +881,17 @@ Per [ADR-0027](../../decisions/0027-expression-sandbox.md): a deterministic, res **Acceptance:** `condition`/`transform` evaluate in the sandbox; a non-deterministic or resource-exhausting expression is rejected/terminated with a typed, secret-free error; a dedicated `condition`/`transform` scenario in the harness suite — alongside 1.AB's own unit tests — asserts sandbox behavior (the 3-node 1.U happy-path does not itself exercise it). -### 1.AC — Resource governor (pre-egress budget) — folds into 1.O +### 1.AC — Resource governor (pre-egress budget) — folds into 1.O · ✅ **Done (PR #26, 2026-06-16)** -Per [ADR-0028](../../decisions/0028-workflow-resource-governance.md): the **pre-egress** budget check, a run `timeout_ms`, and a parallel concurrency cap, with `pause_for_approval` reusing the human-gate seam and emitting `budget:warning` / `budget:paused` / `run:timeout`. The cost formula and `on_exceed` semantics are owned by ADR-0028; this workstream wires them into 1.O. The estimator itself is a **pure function** (model meta + declared estimate in → budget verdict out, no I/O, no ambient state) so it is unit-testable in isolation and reusable wherever a context/token budget is computed (e.g. session context assembly), and its accuracy is a recorded watch item ([deferred-tasks.md](../deferred-tasks.md)). +Per [ADR-0028](../../decisions/0028-workflow-resource-governance.md): the **pre-egress** budget check, a run `timeout_ms`, and a parallel concurrency cap, with `pause_for_approval` reusing the human-gate seam and emitting `budget:warning` / `budget:paused` / `run:timeout`. The cost formula and `on_exceed` semantics are owned by ADR-0028; this workstream wires them into 1.O. The estimator itself is a **pure function** (model meta + declared estimate in → budget verdict out, no I/O, no ambient state) so it is unit-testable in isolation and reusable wherever a context/token budget is computed (e.g. session context assembly), and its accuracy is a recorded watch item ([deferred-tasks.md](../deferred-tasks.md)). The H3 `pause_for_approval` continues the deferred call on approve (a one-shot pre-egress bypass threaded through node retries), and the per-attempt enforcement rides the `FallbackChain` so a failover to a pricier model is still capped. -**Acceptance:** a run that would exceed its budget fails or pauses **before** the next LLM call; the concurrency cap bounds a wide fan-out. +**Acceptance:** ✅ Met (PR #26). A run that would exceed its budget fails or pauses **before** the next LLM call; the concurrency cap bounds a wide fan-out. ### Multimodal I/O sub-spine (1.AD–1.AH) — seam amendment now, behavior additive First-class multimodal I/O (image / audio / video, **input AND output**, incl. a workflow that by rule -**generates** a media file) per [ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md) -+ [ADR-0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md), designed in +**generates** a media file) per [ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md) + +[ADR-0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md), designed in [multimodal-io-design-2026-06-07.md](../../analysis/multimodal-io-design-2026-06-07.md). The **shape** (1.AD) is a second pre-freeze seam amendment in the ADR-0030 mould — it landed **before the exhaustive consumers** (1.K `FallbackChain`, 1.O `AgentRunner`) so adding the `ContentPart`/`StreamChunk` media @@ -959,7 +983,7 @@ the latter being the critical-path milestone for the whole product. | **M1 ✅** | **LLM seam proven: 3 adapters pass the conformance suite (fixtures on PR — live-nightly lane reserved/pending keys; no vendor type across the seam)** *(achieved 2026-06-07, PR #9)* | 1.G, 1.H, 1.I, **1.J** | | 1.m2 ✅ | Policy layers complete: fallback runner + cost tracker (**1.B PR #7, 1.K PR #13**) | 1.B, 1.K | | 1.m3 ✅ | Shared-schema reconciliation + interpolation engine, parse → DAG → run loop emits the canonical event stream (**all components landed — 1.N closed it, PR #17, 2026-06-13**) | **1.L.0**, 1.L, **1.L2**, 1.M, 1.N | -| 1.m4 | Agent + non-agent node handlers, gate, checkpoint/resume, retry, tools, **expression sandbox** + pre-egress budget | 1.O, 1.P, 1.Q, 1.R, 1.S, 1.T, **1.AB**, **1.AC** | +| 1.m4 ✅ | Agent + non-agent node handlers, gate, checkpoint/resume, retry, tools, **expression sandbox** + pre-egress budget (**all components landed — 1.AC closed it, PR #26, 2026-06-16**) | 1.O, 1.P, 1.Q, 1.R, 1.S, 1.T, **1.AB**, **1.AC** | | **M2** | **Engine end-to-end from a Node harness (stream + checkpoint + retry + fallback) — CRITICAL-PATH MILESTONE** | **1.U** | | 1.m5 | Agent-first sub-spine: `AgentSession` + session events + persistence + checkpoint/resume + export, proven by its own harness (**additive, parallel — does NOT gate M2**) | 1.V, 1.W, 1.X, 1.Y, 1.Z, 1.AA | | 1.m6 | Multimodal I/O: seam amendment (**1.AD ✅ Done, PR #11 — landed before 1.K/1.O so the union members are non-breaking**), then media input/engine/output behavior (**additive — does NOT gate M2**) + surfaces threaded into Phases 2–6 ([ADR-0031](../../decisions/0031-llm-seam-shape-amendment-multimodal-io.md)/[0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md)) | **1.AD ✅**, 1.AE, 1.AF, 1.AG, 1.AH | @@ -1113,9 +1137,9 @@ flowchart LR | 1.P | B | 1.O, 1.AB | 1.Q, 1.U | ✅ — **Done (PR #20)** | | 1.S | B | 1.O, 1.R | 1.U | ✅ — **Done (PR #24)** | | 1.Q | B | 1.P, 1.R | 1.AC, 1.U | ✅ — **Done (PR #22)** | -| 1.AC | B | 1.O, 1.Q | 1.U | ✅ folds into 1.O | -| 1.U | B | 1.P, 1.S, 1.Q, 1.R, 1.T, 1.AC | **M2** | ✅ | -| 1.V | C | 1.O | 1.W, 1.X, 1.Z | ◇ | +| 1.AC | B | 1.O, 1.Q | 1.U | ✅ folds into 1.O — **Done (PR #26)** | +| 1.U | B | 1.P, 1.S, 1.Q, 1.R, 1.T, 1.AC | **M2** | ✅ — **next (all deps landed)** | +| 1.V | C | 1.O | 1.W, 1.X, 1.Z | ◇ — **Done (PR #26)** | | 1.W | C | 1.V, 1.N, 1.L.0 | 1.AA | ◇ | | 1.X | C | 1.V, `@relavium/db` (new migration) | 1.Y, 1.AA | ◇ | | 1.Y | C | 1.X, 1.R | 1.AA | ◇ | @@ -1205,10 +1229,12 @@ flowchart LR All must be true to start Phase 2 (CLI): -1. A `relavium`-equivalent invocation from the Node harness (1.U) runs a 3-node - sequential workflow end-to-end: live token streaming, in-process emission of the - canonical run events with monotonic `sequenceNumber`, SQLite-shaped checkpointing, - and resume from a checkpoint. +1. A `relavium`-equivalent invocation from the Node harness (1.U) runs a sequential + workflow end-to-end (the **happy-path** member is a 3-node `input → agent → output`): + live token streaming, in-process emission of the canonical run events with monotonic + `sequenceNumber`, SQLite-shaped checkpointing, and resume from a checkpoint (the resume + rides the suite's gated **flagship** scenario, which adds a `human_in_the_loop` gate — + the engine resumes only from a durable gate checkpoint; see §1.U *Harness shape*). 2. Forcing a provider error triggers node retry and then a fallback to the next provider in the chain, with the run completing and **per-attempt** cost recorded correctly. diff --git a/packages/core/src/engine/agent-turn.test.ts b/packages/core/src/engine/agent-turn.test.ts index 3a4eafd7..ed19ac2c 100644 --- a/packages/core/src/engine/agent-turn.test.ts +++ b/packages/core/src/engine/agent-turn.test.ts @@ -159,6 +159,13 @@ describe('runAgentTurn — streaming + cost', () => { }); describe('runAgentTurn — tool loop', () => { + // A tool-use turn (a tool_call to `echo`, then a tool_use stop) — shared by the tool-loop scenarios. + const toolUseTurn = (id: string): StreamChunk[] => [ + { type: 'tool_call_start', id, name: 'echo' }, + { type: 'tool_call_end', id }, + STOP('tool_use'), + ]; + it('performs a tool round-trip then completes', async () => { const provider = scriptedProvider('anthropic', [ // turn 1: a tool call @@ -246,6 +253,53 @@ describe('runAgentTurn — tool loop', () => { expect(failResult).toBeDefined(); }); + it('combined budget: corrections accumulate across an interleaved genuine round and bound egress (tool_failed before turn_limit)', async () => { + // Pins the COMBINED tool-loop DoS bound: maxToolCorrections is a MONOTONIC sub-budget — a genuine + // (non-correctable) round between correctable ones neither resets nor counts toward it. With + // maxToolCorrections 2, the rounds correctable / genuine / correctable / correctable trip `tool_failed` + // on the 3rd correctable (at turn 4), well under maxToolTurns 16. Proves the two bounds are NOT + // multiplicative: the correction sub-budget ends the turn early; egress stays ≤ the turn count. + let dispatched = 0; + const registry = stubRegistry((call) => { + dispatched += 1; + if (dispatched === 2) { + // the one genuine round, interleaved between correctable ones + const result: ToolResultPart = { type: 'tool_result', toolCallId: call.id, result: 'OK' }; + return { + output: 'OK', + toolResult: markUntrusted(result), + truncated: false, + events: { + call: { toolId: call.name, toolInput: {} }, + result: { toolId: call.name, success: true, outputSummary: 'OK' }, + }, + }; + } + throw new UnknownToolError('echo', ['echo']); // calls 1, 3, 4 are model-correctable + }); + // A 5th turn is scripted but must never be reached (the budget trips on the 4th). + const provider = scriptedProvider('anthropic', [ + toolUseTurn('c1'), + toolUseTurn('c2'), + toolUseTurn('c3'), + toolUseTurn('c4'), + toolUseTurn('c5'), + ]); + const params = baseParams(provider, { + registry, + limits: { maxToolTurns: 16, maxToolCorrections: 2 }, + }); + await expect(runAgentTurn(params)).rejects.toMatchObject({ + code: 'tool_failed', + retryable: false, + }); + expect(dispatched).toBe(4); // exactly 4 tool turns — the correction budget ended it far under maxToolTurns + // the interleaved genuine round actually ran (one successful tool_result between the corrections) + expect( + eventsOf(params).filter((e) => e.type === 'agent:tool_result' && e.success), + ).toHaveLength(1); + }); + it('maps a tool denial to a fatal tool_denied failure (no feedback loop)', async () => { const registry = stubRegistry(() => { throw new ToolPolicyError('echo', 'not_granted', 'tool not granted'); @@ -264,12 +318,6 @@ describe('runAgentTurn — tool loop', () => { }); }); - const toolUseTurn = (id: string): StreamChunk[] => [ - { type: 'tool_call_start', id, name: 'echo' }, - { type: 'tool_call_end', id }, - STOP('tool_use'), - ]; - it('maps ToolCancelledError to cancelled (cancel wins over a tool failure)', async () => { const registry = stubRegistry(() => { throw new ToolCancelledError('echo'); diff --git a/packages/core/src/engine/agent-turn.ts b/packages/core/src/engine/agent-turn.ts index 599ae113..8066a8f4 100644 --- a/packages/core/src/engine/agent-turn.ts +++ b/packages/core/src/engine/agent-turn.ts @@ -47,7 +47,18 @@ import { unwrapUntrusted } from '../tools/untrusted.js'; import { BudgetExceededError, BudgetPauseError } from './budget-governor.js'; import type { NodeStreamEvent } from './node-executor.js'; -/** Loop bounds for one agent turn. The authored hard cap + the `turn_limit` surfacing is the 1.V knob. */ +/** + * Loop bounds for one agent turn. The authored hard cap + the `turn_limit` surfacing is the 1.V knob. + * + * The two bounds are **not multiplicative** — `maxToolTurns` is the worst-case **egress ceiling** (the + * tool loop engages a provider at most `maxToolTurns + 1` times before the guard fails the turn with + * `turn_limit`), while `maxToolCorrections` is a **monotonic sub-budget** *within* that loop: a + * model-correctable tool error (`unknown_tool` / `invalid_args`) increments it and, once exceeded, ends + * the turn EARLY with `tool_failed`. A genuine (non-correctable) tool round never resets it, so + * corrections accumulate across interleaved genuine rounds. Net worst-case egress is `maxToolTurns + 1` + * provider calls regardless of `maxToolCorrections` — the correction budget can only *shorten* a turn, + * never extend its egress (so the DoS bound is the turn budget alone, not the product of the two). + */ export interface AgentTurnLimits { /** Max tool-loop continuations before the run-default DoS guard fails the turn (`turn_limit`). */ readonly maxToolTurns: number; diff --git a/packages/core/src/engine/checkpoint.test.ts b/packages/core/src/engine/checkpoint.test.ts index 658e571a..5c9aa193 100644 --- a/packages/core/src/engine/checkpoint.test.ts +++ b/packages/core/src/engine/checkpoint.test.ts @@ -183,6 +183,9 @@ describe('reconstructCheckpointState', () => { }); it('restores running token + cost tallies so a resumed run keeps cumulative totals', () => { + // Exercises the fold's `cost:updated` arm directly (a defensive branch of the pure function — in a real + // durable log cost:updated is streamed, NOT persisted; the production resume path rides node:completed, + // covered by the tests below). Kept to pin the token tally + the cost:updated running-total fold. const state = reconstructCheckpointState([ started, { @@ -227,6 +230,97 @@ describe('reconstructCheckpointState', () => { expect(state?.cumulativeCostMicrocents).toBe(1600); // the last running total, not a re-sum }); + it('restores the cumulative cost from a durable node:completed at a PLAIN human-gate checkpoint (cost-event persistence)', () => { + // The previously-lost path: a budgeted/costed run paused at a plain HUMAN gate (not a budget gate) had + // no durable cost source (cost:updated is streamed, not persisted) and resumed near 0. The running total + // now rides node:completed.cumulativeCostMicrocents — a REAL durable log (no cost:updated rows) restores it. + const state = reconstructCheckpointState([ + started, + { + type: 'node:completed', + ...base(1), + nodeId: 'agent', + output: 'answer', + tokensUsed: { input: 30, output: 13 }, + durationMs: 1, + cumulativeCostMicrocents: 1600, // the durable snapshot — NO cost:updated in this (real-shaped) log + }, + { + type: 'human_gate:paused', + ...base(2), + nodeId: 'gate', + gateId: 'g1', + gateType: 'approval', + message: 'ok?', + }, + { type: 'run:paused', ...base(3), pendingGateCount: 1, gateIds: ['g1'] }, + ]); + expect(state?.runStatus).toBe('paused'); + expect(state?.cumulativeCostMicrocents).toBe(1600); // survives the plain-human-gate resume (was ~0 before) + }); + + it('reconciles two durable cost sources — a later budget:paused.spentMicrocents above a node:completed snapshot', () => { + // A node completes (running total 800), then the next node's pre-egress trips a budget gate at a higher + // running total (900). Both are durable cost sources; the fold must end at the higher value. + const state = reconstructCheckpointState([ + started, + { + type: 'node:completed', + ...base(1), + nodeId: 'a', + output: 'A', + tokensUsed: { input: 0, output: 0 }, + durationMs: 1, + cumulativeCostMicrocents: 800, + }, + { + type: 'budget:paused', + ...base(2), + nodeId: 'b', + gateId: 'g1', + spentMicrocents: 900, + limitMicrocents: 1000, + }, + { + type: 'human_gate:paused', + ...base(3), + nodeId: 'b', + gateId: 'g1', + gateType: 'approval', + message: 'over budget', + }, + { type: 'run:paused', ...base(4), pendingGateCount: 1, gateIds: ['g1'] }, + ]); + expect(state?.cumulativeCostMicrocents).toBe(900); // the later, higher budget-pause spend wins + }); + + it('never undercounts: a lower node:completed snapshot after a higher budget:paused keeps the higher (Math.max, order-independent)', () => { + // The fold's monotonic guard: were a node:completed to carry a LOWER running total than a prior + // budget:paused (the order-independence case), `Math.max` must keep the higher value — a bare assignment + // would wrongly drop it. Pins that the cost restore can never go backwards. + const state = reconstructCheckpointState([ + started, + { + type: 'budget:paused', + ...base(1), + nodeId: 'a', + gateId: 'g1', + spentMicrocents: 900, + limitMicrocents: 1000, + }, + { + type: 'node:completed', + ...base(2), + nodeId: 'a', + output: 'A', + tokensUsed: { input: 0, output: 0 }, + durationMs: 1, + cumulativeCostMicrocents: 800, // lower than the prior budget:paused — must NOT lower the cumulative + }, + ]); + expect(state?.cumulativeCostMicrocents).toBe(900); // Math.max keeps the higher prior value + }); + it('folds node:retrying as non-state-bearing — a retry-then-recover ends `completed` (1.S)', () => { const state = reconstructCheckpointState([ started, diff --git a/packages/core/src/engine/checkpoint.ts b/packages/core/src/engine/checkpoint.ts index 91cbfe68..7a61d2ed 100644 --- a/packages/core/src/engine/checkpoint.ts +++ b/packages/core/src/engine/checkpoint.ts @@ -64,7 +64,10 @@ export interface CheckpointState { /** Running token totals (summed from `node:completed`), restored so a resumed run's `run:completed` totals stay correct. */ readonly totalInputTokens: number; readonly totalOutputTokens: number; - /** The last `cost:updated.cumulativeCostMicrocents` (a running total), restored so post-resume cost stays cumulative. */ + /** The run-wide cumulative cost (integer micro-cents), restored on resume from the durable + * `node:completed.cumulativeCostMicrocents` snapshot and/or `budget:paused.spentMicrocents` — the higher + * wins (`Math.max`, order-independent). (`cost:updated` is also folded when present, but it is streamed, + * not persisted, so it never appears in a real durable log.) Keeps post-resume cost cumulative. */ readonly cumulativeCostMicrocents: number; } @@ -125,6 +128,16 @@ function applyNodeEvent(acc: ReconAccumulator, event: RunEvent): void { }); acc.totalInputTokens += event.tokensUsed.input; acc.totalOutputTokens += event.tokensUsed.output; + // Restore the run-wide cumulative cost from the durable node boundary (cost:updated is streamed, not + // persisted, so it is otherwise lost on a plain-human-gate / crash resume). `Math.max` keeps it + // monotonic and order-independent — it reconciles with the `budget:paused.spentMicrocents` restore + // (applyGateEvent) regardless of which durable cost source has the higher sequence number. + if (event.cumulativeCostMicrocents !== undefined) { + acc.cumulativeCostMicrocents = Math.max( + acc.cumulativeCostMicrocents, + event.cumulativeCostMicrocents, + ); + } break; case 'node:failed': acc.nodeStates.set(event.nodeId, { @@ -159,10 +172,11 @@ function applyGateEvent(acc: ReconAccumulator, event: RunEvent): void { isBudgetGate: acc.pendingGates.get(event.gateId)?.isBudgetGate === true || event.type === 'budget:paused', }); - // `cost:updated` is streamed (not persisted), so the running cost is otherwise unrecoverable on resume; - // but `budget:paused.spentMicrocents` IS the durable cumulative-at-pause. Restore it so a resumed budgeted - // run keeps its spend and the re-seeded governor blocks correctly (H2). (A budgeted run that paused at a - // *plain human* gate still loses its cost on resume — cost-event persistence is the deferred general fix.) + // `cost:updated` is streamed (not persisted), so the cost cannot be recovered from it; but + // `budget:paused.spentMicrocents` IS the durable cumulative-at-pause. Restore it so a resumed budgeted run + // keeps its spend and the re-seeded governor blocks correctly (H2). A plain-human-gate / crash resume now + // recovers the same total from the durable `node:completed.cumulativeCostMicrocents` (applyNodeEvent above) — + // the two durable sources reconcile via that `Math.max` fold (cost-event persistence is no longer deferred). if (event.type === 'budget:paused') { acc.cumulativeCostMicrocents = event.spentMicrocents; } diff --git a/packages/core/src/engine/engine.ts b/packages/core/src/engine/engine.ts index a330218a..b211a235 100644 --- a/packages/core/src/engine/engine.ts +++ b/packages/core/src/engine/engine.ts @@ -987,6 +987,10 @@ class RunExecution { output: outcome.output, tokensUsed: tokens, durationMs: Math.max(0, this.#elapsedMs() - startedAtMs), + // Snapshot the run-wide cost running total onto the durable boundary so cross-process resume can + // restore it (1.R) — cost:updated is streamed, not persisted. By here #cumulativeCostMicrocents + // already includes this node's cost (its cost:updated fired during execution, before this boundary). + cumulativeCostMicrocents: this.#cumulativeCostMicrocents, // A condition's branch selection — persisted so resume can restore `selectedTargets` (1.R). ...(outcome.kind === 'branch' ? { selected: [...outcome.selected] } : {}), // Which attempt produced the output, when a node-retry recovered (1.S) — absent ⇒ attempt 1. diff --git a/packages/core/src/engine/m2-e2e-harness.e2e.test.ts b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts new file mode 100644 index 00000000..89280b7d --- /dev/null +++ b/packages/core/src/engine/m2-e2e-harness.e2e.test.ts @@ -0,0 +1,435 @@ +/** + * 1.U — the End-to-end Node harness (the **M2** critical-path milestone). The proof that the engine works + * end-to-end before any surface exists: it composes 1.K (FallbackChain) / 1.N (run loop + RunEventBus) / + * 1.O (AgentRunner) / 1.P (node handlers) / 1.Q (human gate) / 1.R (checkpoint/resume) / 1.S (node retry) / + * 1.T (ToolRegistry) / 1.AB (ExpressionSandbox) behind the `@relavium/llm` seam, using only already-exported `@relavium/core` symbols + * and the in-memory `ExecutionHost` reference — zero platform imports, no live network/keys, deterministic. + * + * This is a **scenario suite** (the seed the Phase-2 CLI regression harness, 2.K, grows from — see + * docs/roadmap/phases/phase-1-engine-and-llm.md §1.U *Harness shape*). Its members: + * • **happy-path** — the literal 3-node `input → agent → output` (a clean run, with a tool call): live + * token streaming, per-attempt cost, a gap-free `sequenceNumber` stream that validates against the + * canonical {@link RunEventSchema}. + * • **flagship** — `input → agent → human_gate → output`: in ONE run across a process boundary, the agent's + * forced provider error → **node retry** (ADR-0040) → **failover** to the second chain entry (1.K), with + * per-attempt cost; then a pause at the gate (the durable mid-run checkpoint persisted to the + * SQLite-shaped store); then a **fresh engine** resumes via `resumeFromCheckpoint` and runs `output` to + * `run:completed`, reproducing the same final output with `sequenceNumber` continuing gap-free. The + * `human_in_the_loop` gate is the durable suspend point because the Phase-1 engine resumes ONLY from a + * gate/budget pause — a gate-less interrupted run is reconciled to `run:failed` (ADR-0036). All LLM cost is + * incurred pre-gate and is RESTORED across the resume (run:completed.totalCostMicrocents) — the durable + * node:completed.cumulativeCostMicrocents carries it, closing the cost-event-persistence gap. + * • **determinism** — the same scenario produces an identical event signature + final output on a re-run + * (the no-wall-clock / no-RNG ban the risk table binds to this harness). + * + * The local stub helpers mirror `agent-runner.e2e.test.ts` (the project's e2e convention keeps them inline, + * not on the curated public surface). + */ + +import type { CapabilityFlags, LlmProvider, ProviderId, StreamChunk } from '@relavium/llm'; +import { RunEventSchema, type RunEvent } from '@relavium/shared'; +import { beforeAll, describe, expect, it } from 'vitest'; + +import { createExpressionSandbox, type ExpressionSandbox } from '../expression/sandbox.js'; +import { parseWorkflow } from '../parser.js'; +import type { ToolDef as CoreToolDef, ToolRegistry, ToolResultPart } from '../tools/types.js'; +import { markUntrusted } from '../tools/untrusted.js'; +import { WorkflowEngine } from './engine.js'; +import { createInMemoryHost, InMemoryRunStore } from './execution-host.js'; +import { createStandardNodeExecutor } from './node-handlers/dispatcher.js'; +import type { RunHandle } from './run-handle.js'; + +// --- LLM-provider stubs (mirror agent-runner.e2e.test.ts) ------------------------------------------- + +const CAPS: CapabilityFlags = { + tools: true, + streaming: true, + parallelToolCalls: true, + vision: false, + promptCache: false, + reasoning: true, + media: { + input: { image: false, audio: false, video: false, document: false }, + outputCombinations: [], + }, +}; + +async function* streamOf(chunks: readonly StreamChunk[]): AsyncGenerator { + await Promise.resolve(); + for (const c of chunks) yield c; +} + +/** A provider whose `stream` replays the SAME chunk list every call (e.g. an always-failing primary). */ +function provider(chunks: StreamChunk[], id: ProviderId = 'anthropic'): LlmProvider { + return { + id, + supports: CAPS, + generate: () => { + throw new Error('generate not used in the harness'); + }, + stream: () => streamOf(chunks), + }; +} + +/** A provider that replays a DIFFERENT chunk list per call (call N → scripts[N]) — drives tool/retry turns. */ +function scriptedProvider(scripts: StreamChunk[][], id: ProviderId = 'anthropic'): LlmProvider { + let call = 0; + return { + id, + supports: CAPS, + generate: () => { + throw new Error('generate not used in the harness'); + }, + stream: () => { + // Fail fast on an UNSCRIPTED call — an unintended extra LLM invocation is a harness bug, not a + // silent empty turn (which would mask, e.g., a retry/failover that re-dispatched more than expected). + const chunks = scripts[call]; + call += 1; + if (chunks === undefined) { + throw new Error( + `scriptedProvider: unexpected stream call #${call} (only ${scripts.length} scripted)`, + ); + } + return streamOf(chunks); + }, + }; +} + +const usage = { inputTokens: 10, outputTokens: 5 }; +const STOP = (reason: 'stop' | 'tool_use' = 'stop'): StreamChunk => ({ + type: 'stop', + stopReason: reason, + usage, +}); +const textTurn = (text: string): StreamChunk[] => [{ type: 'text_delta', text }, STOP('stop')]; +const toolUseTurn = (id: string): StreamChunk[] => [ + { type: 'tool_call_start', id, name: 'echo' }, + { type: 'tool_call_end', id }, + STOP('tool_use'), +]; +const retryableError = (providerId: ProviderId): StreamChunk => ({ + type: 'error', + error: { kind: 'overloaded', retryable: true, provider: providerId, message: 'busy' }, +}); + +// --- Tool stubs: a sanitized echo registry + its LLM-visible def (mirror agent-runner.e2e.test.ts) ---- + +const echoRegistry: ToolRegistry = { + has: () => true, + list: () => ['echo'], + dispatch: (call) => { + const result: ToolResultPart = { type: 'tool_result', toolCallId: call.id, result: 'TOOL-OK' }; + return Promise.resolve({ + output: 'TOOL-OK', + toolResult: markUntrusted(result), + truncated: false, + events: { + call: { toolId: call.name, toolInput: {} }, + result: { toolId: call.name, success: true, outputSummary: 'TOOL-OK' }, + }, + }); + }, +}; + +const echoToolDef: CoreToolDef = { + id: 'echo', + source: 'builtin', + description: 'echo', + parseArgs: (raw) => raw, + llmVisibleParams: { type: 'object' }, + policy: { fsScoped: false, spawnsProcess: false, requiresGateApproval: false }, + dispatch: () => Promise.reject(new Error('echoToolDef dispatch is not used directly')), +}; + +// --- Canonical workflows -------------------------------------------------------------------------- + +/** Happy path — the literal 3-node sequential workflow, with a tool call (§1.U tasks bullet a). */ +const HAPPY_PATH = parseWorkflow( + `schema_version: '1.0' +workflow: + id: m2-harness-happy + inputs: + - { name: topic, type: string } + agents: + - id: writer + model: claude-opus-4-8 + provider: anthropic + system_prompt: You summarize. + tools: [echo] + nodes: + - { id: in, type: input } + - { id: work, type: agent, agent_ref: writer, prompt_template: 'Summarize: {{inputs.topic}}' } + - { id: out, type: output } + edges: + - { from: in, to: work } + - { from: work, to: out } +`, +); + +/** Flagship — adds a human gate as the durable mid-run checkpoint; the agent fails over with a retry budget. */ +const FLAGSHIP = parseWorkflow( + `schema_version: '1.0' +workflow: + id: m2-harness-flagship + inputs: + - { name: topic, type: string } + agents: + - id: writer + model: claude-opus-4-8 + provider: anthropic + system_prompt: You summarize. + retry: { max: 2, backoff: linear, backoff_ms: 10 } + fallback_chain: + - { model: claude-sonnet-4-6, provider: openai, max_attempts: 1 } + nodes: + - { id: in, type: input } + - { id: work, type: agent, agent_ref: writer, prompt_template: 'Summarize: {{inputs.topic}}' } + - { id: g, type: human_gate, gate_type: approval } + - { id: out, type: output } + edges: + - { from: in, to: work } + - { from: work, to: g } + - { from: g, to: out } +`, +); + +const INPUTS = { topic: 'the report' } as const; + +// --- The reusable driver --------------------------------------------------------------------------- + +type Host = ReturnType; + +function buildEngine( + host: Host, + resolveProvider: (id: ProviderId) => LlmProvider | undefined, +): WorkflowEngine { + return new WorkflowEngine({ + host, + executor: createStandardNodeExecutor({ + sandbox, + agent: { + resolveProvider, + registry: echoRegistry, + tools: [echoToolDef], + keyFor: () => 'k', + sleep: () => Promise.resolve(), + now: () => 1, + }, + }), + }); +} + +interface DriveResult { + readonly events: RunEvent[]; + readonly gateId: string | undefined; + readonly lastSeq: number; +} + +/** + * Drive a run handle to its terminal — or, with `breakOnPause`, to the first `run:paused` (the "process" + * dies parked at the gate). On every `node:retrying` it arms-then-fires the backoff timer: the timer is + * armed in `#dispatch`'s continuation just AFTER the event is delivered, so the consumer must spin the + * microtask queue until `armedCount() > 0` before firing (the manual timer never fires on a wall clock). + */ +async function drive( + handle: RunHandle, + host: Host, + opts: { breakOnPause?: boolean } = {}, +): Promise { + const events: RunEvent[] = []; + let gateId: string | undefined; + let lastSeq = -1; + for await (const event of handle.events) { + events.push(event); + lastSeq = Math.max(lastSeq, event.sequenceNumber); + if (event.type === 'node:retrying') { + let waited = 0; + while (host.armedCount() === 0) { + if ((waited += 1) > 1000) { + throw new Error('backoff timer was never armed after node:retrying'); + } + await Promise.resolve(); + } + host.fireTimers(); + } + if (opts.breakOnPause === true && event.type === 'run:paused') { + gateId = event.gateIds[0]; + break; + } + } + return { events, gateId, lastSeq }; +} + +/** Assert every event validates against the canonical RunEventSchema (§1.U "matching the canonical schema"). */ +function assertCanonicalSchema(events: readonly RunEvent[]): void { + for (const event of events) { + const parsed = RunEventSchema.safeParse(event); + if (!parsed.success) { + throw new Error(`event ${event.type}#${String(event.sequenceNumber)} is not canonical`); + } + } +} + +/** Assert sequenceNumbers are exactly 0..n-1 — the bus's gap-free, exactly-once guarantee. */ +function assertGapFreeSeq(events: readonly RunEvent[]): void { + const seqs = events.map((e) => e.sequenceNumber).sort((a, b) => a - b); + seqs.forEach((seq, index) => expect(seq).toBe(index)); +} + +const tokensOf = (events: readonly RunEvent[]): string[] => + events.flatMap((e) => (e.type === 'agent:token' ? [e.token] : [])); +const costsOf = (events: readonly RunEvent[]): Extract[] => + events.filter((e): e is Extract => e.type === 'cost:updated'); +const nodeOutput = (events: readonly RunEvent[], nodeId: string): unknown => + events.find( + (e): e is Extract => + e.type === 'node:completed' && e.nodeId === nodeId, + )?.output; + +let sandbox: ExpressionSandbox; + +beforeAll(async () => { + sandbox = await createExpressionSandbox(); +}); + +describe('M2 — end-to-end Node harness (1.U)', () => { + it('happy path: a 3-node input→agent(+tool)→output run streams, records per-attempt cost, gap-free + canonical', async () => { + const host = createInMemoryHost(); + // The primary streams a tool-use turn (echo) then the answer — a real tool round-trip, no fallback. + const engine = buildEngine(host, () => + scriptedProvider([toolUseTurn('c1'), textTurn('a summary')]), + ); + const { events } = await drive(engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }), host); + + expect(events.at(-1)?.type).toBe('run:completed'); + expect(tokensOf(events)).toEqual(['a summary']); // live token streaming over the RunEventBus + expect(events.some((e) => e.type === 'agent:tool_call' && e.toolId === 'echo')).toBe(true); + expect(events.some((e) => e.type === 'agent:tool_result' && e.success)).toBe(true); + expect(nodeOutput(events, 'out')).toBe('a summary'); // the agent's answer flows through to output + + // Per-attempt cost: the tool-use turn AND the answer turn each emit one cost:updated, the cumulative + // rolls up. Pinned to the EXACT count — a `>= 1` would pass even if the tool-turn cost went missing. + const costs = costsOf(events); + expect(costs.length).toBe(2); // tool-use turn + answer turn → one cost:updated each + let running = 0; + for (const c of costs) { + expect(c.model).toBe('claude-opus-4-8'); + expect(c.costMicrocents).toBeGreaterThan(0); + running += c.costMicrocents; + expect(c.cumulativeCostMicrocents).toBe(running); + } + + assertGapFreeSeq(events); + assertCanonicalSchema(events); + }); + + it('flagship: one run — retry then failover, pause at the gate, cross-process resume reproduces the final output', async () => { + // The primary (anthropic) ALWAYS errors retryably pre-content; the fallback (openai) fails the first + // dispatch then succeeds — so dispatch 1 exhausts the chain (→ node retry), dispatch 2 fails over to the + // fallback and completes. "forcing a provider error triggers retry then fallback" (§1.U acceptance). + // Instantiate the stubs ONCE so the fallback's per-call counter persists across the two dispatches + // (a fresh instance per resolve would reset it and never recover — the failover would never succeed). + const primary = provider([retryableError('anthropic')], 'anthropic'); + const fallback = scriptedProvider( + [[retryableError('openai')], textTurn('a summary')], + 'openai', + ); + const resolveProvider = (id: ProviderId): LlmProvider => + id === 'anthropic' ? primary : fallback; + + // --- "Process" #1: run until the gate, persisting node-boundary + gate events to the shared store. --- + const store = new InMemoryRunStore(); + const host1 = createInMemoryHost({ store }); + const engine1 = buildEngine(host1, resolveProvider); + const handle1 = engine1.start({ workflow: FLAGSHIP, inputs: INPUTS }); + const { + events: events1, + gateId, + lastSeq, + } = await drive(handle1, host1, { breakOnPause: true }); + + expect(gateId).toBeDefined(); + // Node retry then failover, all pre-gate; the run parked at the gate (no terminal yet). + expect(events1.filter((e) => e.type === 'node:retrying')).toHaveLength(1); + const retrying = events1.find((e) => e.type === 'node:retrying'); + // Assert the CLASSIFIED code, not `.retryable` — node:retrying is only ever emitted for a retryable + // failure, so asserting retryable===true is tautological; the overloaded chain-exhaustion maps to + // `provider_unavailable` (agent-turn.ts), which a misclassification would fail. + expect(retrying?.type === 'node:retrying' ? retrying.error.code : undefined).toBe( + 'provider_unavailable', + ); + expect(tokensOf(events1)).toEqual(['a summary']); // the fallback streamed the answer + expect(events1.some((e) => e.type === 'human_gate:paused')).toBe(true); + expect(events1.some((e) => e.type === 'run:paused')).toBe(true); + expect(events1.some((e) => e.type === 'run:completed')).toBe(false); + // The expensive agent result is checkpointed (failover output recorded at the node boundary). + expect(nodeOutput(events1, 'work')).toBe('a summary'); + + // Per-attempt cost recorded, attributed to the FALLBACK model — failover cost is accounted (§1.U). + // Exactly ONE cost:updated: only the successful fallback attempt bills (the pre-content error attempts + // carry no usage). `=== 1` catches a double-charge or a billed failed attempt that `>= 1` would miss. + const costs1 = costsOf(events1); + expect(costs1.length).toBe(1); + for (const c of costs1) { + expect(c.model).toBe('claude-sonnet-4-6'); + expect(c.costMicrocents).toBeGreaterThan(0); + } + + // --- "Process" #2: a brand-new engine resumes purely from the persisted store. --- + const host2 = createInMemoryHost({ store }); + const engine2 = buildEngine(host2, resolveProvider); + const handle2 = await engine2.resumeFromCheckpoint({ + runId: handle1.runId, + workflow: FLAGSHIP, + inputs: INPUTS, + gateId: gateId ?? '', + decision: { decision: 'approved', decidedBy: 'tester' }, + }); + const { events: events2 } = await drive(handle2, host2); + + expect(handle2.runId).toBe(handle1.runId); + expect(events2[0]?.type).toBe('human_gate:resumed'); // NOT a re-emitted run:started + expect(tokensOf(events2)).toEqual([]); // the agent was NOT re-run — its output was restored + // The checkpointed `work` (agent) node must NOT be re-dispatched on resume (its output is restored) — + // assert directly, not just via the absence of streamed tokens. + expect(events2.some((e) => e.type === 'node:started' && e.nodeId === 'work')).toBe(false); + expect(events2.some((e) => e.type === 'node:completed' && e.nodeId === 'work')).toBe(false); + expect(events2.some((e) => e.type === 'node:started' && e.nodeId === 'out')).toBe(true); + expect(events2.at(-1)?.type).toBe('run:completed'); // resume reproduces a completed run + expect(nodeOutput(events2, 'out')).toEqual({ decision: 'approved' }); // deterministic final output + + // Cost-event persistence: the pre-gate agent cost is RESTORED across the cross-process resume — the + // durable node:completed.cumulativeCostMicrocents carries it (cost:updated is streamed, not persisted), + // so run:completed.totalCostMicrocents reflects it rather than restarting near 0. + const preGateCost = costs1.at(-1)?.cumulativeCostMicrocents ?? 0; + expect(preGateCost).toBeGreaterThan(0); + const resumedTerminal = events2.find((e) => e.type === 'run:completed'); + expect( + resumedTerminal?.type === 'run:completed' ? resumedTerminal.totalCostMicrocents : -1, + ).toBe(preGateCost); + + // The whole run — across the process boundary — is one gap-free, canonical sequence. + const whole = [...events1, ...events2]; + assertGapFreeSeq(whole); + assertCanonicalSchema(whole); + expect(events2[0]?.sequenceNumber).toBe(lastSeq + 1); // resume continues the counter, no reset/gap + }); + + it('determinism: re-running the happy path yields an identical event signature + final output (no wall-clock/RNG)', async () => { + const runOnce = async (): Promise<{ sig: string; output: unknown }> => { + const host = createInMemoryHost(); + const engine = buildEngine(host, () => + scriptedProvider([toolUseTurn('c1'), textTurn('a summary')]), + ); + const { events } = await drive(engine.start({ workflow: HAPPY_PATH, inputs: INPUTS }), host); + return { + sig: events.map((e) => `${String(e.sequenceNumber)}:${e.type}`).join('|'), + output: nodeOutput(events, 'out'), + }; + }; + const first = await runOnce(); + const second = await runOnce(); + expect(second.sig).toBe(first.sig); + expect(second.output).toEqual(first.output); + }); +}); diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index 03c0471d..781f930f 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -547,6 +547,49 @@ describe('OpenAI-compatible adapter — reasoning + structured output (ADR-0030) expect(parts[1]).toEqual({ type: 'text', text: 'answer' }); }); + it('DROPS a prior-turn reasoning part on egress — reasoning_content is output-only, replay would 400 (ADR-0030/0039)', async () => { + // DeepSeek/Kimi `reasoning_content` is captured INBOUND (mapContent above) but is output-only: the API + // rejects it if echoed back in an input message, and deepseek-reasoner does not require prior reasoning + // to continue. So a same-provider continuation must NOT replay it. This pins the drop so a future change + // cannot start round-tripping reasoning into the request body (which would 400 the whole turn). + let sent: Record = {}; + const adapter = createOpenAiAdapter({ + providerId: 'deepseek', + fetch: (_i, init) => { + sent = parseJsonBody(init); + return Promise.resolve(okResponse()); + }, + maxRetries: 0, + }); + await adapter.generate( + { + model: 'deepseek-reasoner', + messages: [ + { role: 'user', content: [{ type: 'text', text: 'hi' }] }, + // a prior assistant turn the engine replays: the ephemeral reasoning + the visible answer + { + role: 'assistant', + content: [ + { type: 'reasoning', text: 'internal chain of thought' }, + { type: 'text', text: 'the answer' }, + ], + }, + { role: 'user', content: [{ type: 'text', text: 'continue' }] }, + ], + }, + 'k', + ); + const isRecord = (v: unknown): v is Record => + typeof v === 'object' && v !== null; + const messages: readonly unknown[] = Array.isArray(sent['messages']) ? sent['messages'] : []; + const assistant = messages.find( + (m): m is Record => isRecord(m) && m['role'] === 'assistant', + ); + expect(assistant?.['content']).toBe('the answer'); // the visible text survives the replay… + expect(JSON.stringify(sent)).not.toContain('internal chain of thought'); // …the reasoning never does + expect(JSON.stringify(sent)).not.toContain('reasoning_content'); + }); + it('mapUsage surfaces reasoning_tokens as reasoningTokens', () => { expect( mapUsage({ diff --git a/packages/llm/src/conformance/anthropic.conformance.test.ts b/packages/llm/src/conformance/anthropic.conformance.test.ts index 807d9533..15a97045 100644 --- a/packages/llm/src/conformance/anthropic.conformance.test.ts +++ b/packages/llm/src/conformance/anthropic.conformance.test.ts @@ -2,13 +2,14 @@ import { describe, expect, it } from 'vitest'; import { anthropicAdapter, createAnthropicAdapter } from '../adapters/anthropic.js'; import { ANTHROPIC_FIXTURES } from './fixtures/anthropic.js'; -import { replayFetch } from './replay.js'; +import { replayFor } from './replay.js'; import { defineConformanceSuite, type MakeReplayAdapter } from './spec.js'; -// Wire the Anthropic adapter to replay a recorded response — note: no vendor SDK is imported here; +// Wire the Anthropic adapter to replay recorded response(s) — note: no vendor SDK is imported here; // the adapter takes a `fetch` override, so the SDK stays inside src/adapters/* (the seam fence). +// `replayFor` serves one body (one-shot scenarios) or a sequence (the multi-turn tool loop). const makeReplayAdapter: MakeReplayAdapter = (recorded) => - createAnthropicAdapter({ fetch: replayFetch(recorded), maxRetries: 0 }); + createAnthropicAdapter({ fetch: replayFor(recorded), maxRetries: 0 }); defineConformanceSuite('anthropic', makeReplayAdapter, ANTHROPIC_FIXTURES); diff --git a/packages/llm/src/conformance/deepseek.conformance.test.ts b/packages/llm/src/conformance/deepseek.conformance.test.ts index 9a34250a..8f67da5c 100644 --- a/packages/llm/src/conformance/deepseek.conformance.test.ts +++ b/packages/llm/src/conformance/deepseek.conformance.test.ts @@ -2,13 +2,14 @@ import { describe, expect, it } from 'vitest'; import { createOpenAiAdapter, deepseekAdapter } from '../adapters/openai.js'; import { DEEPSEEK_FIXTURES } from './fixtures/deepseek.js'; -import { replayFetch } from './replay.js'; +import { replayFor } from './replay.js'; import { defineConformanceSuite, type MakeReplayAdapter } from './spec.js'; // DeepSeek is served by the SAME OpenAI-compatible adapter pointed at api.deepseek.com — same fold, -// distinct provider id + cache field. The fetch override keeps the SDK inside src/adapters/*. +// distinct provider id + cache field. The fetch override keeps the SDK inside src/adapters/*. `replayFor` +// serves one body (one-shot scenarios) or a sequence (the multi-turn tool loop). const makeReplayAdapter: MakeReplayAdapter = (recorded) => - createOpenAiAdapter({ providerId: 'deepseek', fetch: replayFetch(recorded), maxRetries: 0 }); + createOpenAiAdapter({ providerId: 'deepseek', fetch: replayFor(recorded), maxRetries: 0 }); defineConformanceSuite('deepseek', makeReplayAdapter, DEEPSEEK_FIXTURES); diff --git a/packages/llm/src/conformance/fixtures/anthropic.ts b/packages/llm/src/conformance/fixtures/anthropic.ts index 488f83ce..5df15751 100644 --- a/packages/llm/src/conformance/fixtures/anthropic.ts +++ b/packages/llm/src/conformance/fixtures/anthropic.ts @@ -251,6 +251,11 @@ export const ANTHROPIC_FIXTURES: ConformanceFixtures = { streamError: { status: 200, contentType: 'text/event-stream', body: streamError }, reasoningStream: { status: 200, contentType: 'text/event-stream', body: reasoningStream }, structuredOutput: { status: 200, body: structuredOutput }, + toolLoop: { + turn1: { status: 200, body: toolMessage }, + turn2: { status: 200, body: textMessage }, + expected: { toolName: 'get_weather', finalText: 'Hello, world!' }, + }, expected: { textGenerate: { stopReason: 'stop', text: 'Hello, world!', inputTokens: 12, outputTokens: 7 }, toolGenerate: { toolName: 'get_weather', stopReason: 'tool_use' }, diff --git a/packages/llm/src/conformance/fixtures/deepseek.ts b/packages/llm/src/conformance/fixtures/deepseek.ts index b46d75b1..db21a1e7 100644 --- a/packages/llm/src/conformance/fixtures/deepseek.ts +++ b/packages/llm/src/conformance/fixtures/deepseek.ts @@ -173,9 +173,20 @@ export const DEEPSEEK_FIXTURES: ConformanceFixtures = { streamError: { status: 503, body: streamError }, reasoningStream: { status: 200, contentType: 'text/event-stream', body: reasoningStream }, structuredOutput: { status: 200, body: structuredOutput }, + toolLoop: { + turn1: { status: 200, body: toolMessage }, + turn2: { status: 200, body: textMessage }, + expected: { toolName: 'get_weather', finalText: 'Hello, world!' }, + }, expected: { // 4 of 12 prompt tokens cached → net input 8, cacheRead 4. - textGenerate: { stopReason: 'stop', text: 'Hello, world!', inputTokens: 8, outputTokens: 7 }, + textGenerate: { + stopReason: 'stop', + text: 'Hello, world!', + inputTokens: 8, + outputTokens: 7, + cacheReadTokens: 4, + }, toolGenerate: { toolName: 'get_weather', stopReason: 'tool_use' }, textStream: { stopReason: 'stop', inputTokens: 8, outputTokens: 7 }, toolStream: { toolName: 'get_weather', stopReason: 'tool_use' }, diff --git a/packages/llm/src/conformance/fixtures/gemini.ts b/packages/llm/src/conformance/fixtures/gemini.ts index b4e3b08c..97992d5c 100644 --- a/packages/llm/src/conformance/fixtures/gemini.ts +++ b/packages/llm/src/conformance/fixtures/gemini.ts @@ -94,6 +94,11 @@ export const GEMINI_FIXTURES: ConformanceFixtures = { streamError: { status: 503, body: overloadedError }, reasoningStream: { status: 200, body: reasoningStream }, structuredOutput: { status: 200, body: structuredOutput }, + toolLoop: { + turn1: { status: 200, body: toolResponse }, + turn2: { status: 200, body: textResponse }, + expected: { toolName: 'get_weather', finalText: 'Hello, world!' }, + }, expected: { textGenerate: { stopReason: 'stop', text: 'Hello, world!', inputTokens: 12, outputTokens: 7 }, toolGenerate: { toolName: 'get_weather', stopReason: 'tool_use' }, diff --git a/packages/llm/src/conformance/fixtures/openai.ts b/packages/llm/src/conformance/fixtures/openai.ts index 64421698..0ff4c21f 100644 --- a/packages/llm/src/conformance/fixtures/openai.ts +++ b/packages/llm/src/conformance/fixtures/openai.ts @@ -158,6 +158,11 @@ export const OPENAI_FIXTURES: ConformanceFixtures = { rateLimit: { status: 429, body: rateLimitError }, streamError: { status: 503, body: streamError }, structuredOutput: { status: 200, body: structuredOutput }, + toolLoop: { + turn1: { status: 200, body: toolMessage }, + turn2: { status: 200, body: textMessage }, + expected: { toolName: 'get_weather', finalText: 'Hello, world!' }, + }, expected: { textGenerate: { stopReason: 'stop', text: 'Hello, world!', inputTokens: 12, outputTokens: 7 }, toolGenerate: { toolName: 'get_weather', stopReason: 'tool_use' }, diff --git a/packages/llm/src/conformance/gemini.conformance.test.ts b/packages/llm/src/conformance/gemini.conformance.test.ts index 0ad17d2b..83e937c1 100644 --- a/packages/llm/src/conformance/gemini.conformance.test.ts +++ b/packages/llm/src/conformance/gemini.conformance.test.ts @@ -7,6 +7,7 @@ import { type GeminiTransport, } from '../adapters/gemini.js'; import { GEMINI_FIXTURES } from './fixtures/gemini.js'; +import type { RecordedResponse } from './replay.js'; import { defineConformanceSuite, type MakeReplayAdapter } from './spec.js'; async function* toAsyncIterable(items: readonly GeminiResponse[]): AsyncIterable { @@ -28,20 +29,33 @@ const isGeminiResponseArray = (value: unknown): value is GeminiResponse[] => // SDK-output JSON (single response or an array of streamed responses) is parsed and served through a // fake GeminiTransport — no vendor SDK is imported here. const makeReplayAdapter: MakeReplayAdapter = (recorded) => { - const failure = recorded.status >= 400; - const rejection = (): Promise => - Promise.reject(Object.assign(new Error('replayed gemini error'), { status: recorded.status })); + // One-shot scenarios pass a single RecordedResponse; the multi-turn tool loop passes a sequence served + // by call index (turn 1 → recordings[0], turn 2 → recordings[1]). + const recordings: readonly RecordedResponse[] = 'status' in recorded ? [recorded] : recorded; + let call = 0; + const nextRecording = (): RecordedResponse => { + const next = recordings[call]; + call += 1; + if (next === undefined) { + throw new Error(`gemini replay: no recorded response for call #${String(call)}`); + } + return next; + }; + const rejection = (status: number): Promise => + Promise.reject(Object.assign(new Error('replayed gemini error'), { status })); const transport: GeminiTransport = { generate: () => { - if (failure) return rejection(); - const parsed: unknown = JSON.parse(recorded.body); + const current = nextRecording(); + if (current.status >= 400) return rejection(current.status); + const parsed: unknown = JSON.parse(current.body); return isGeminiResponse(parsed) ? Promise.resolve(parsed) : Promise.reject(new Error('replay fixture is not a GeminiResponse object')); }, stream: () => { - if (failure) return rejection(); - const parsed: unknown = JSON.parse(recorded.body); + const current = nextRecording(); + if (current.status >= 400) return rejection(current.status); + const parsed: unknown = JSON.parse(current.body); return isGeminiResponseArray(parsed) ? Promise.resolve(toAsyncIterable(parsed)) : Promise.reject(new Error('replay fixture is not a GeminiResponse[] array')); diff --git a/packages/llm/src/conformance/openai.conformance.test.ts b/packages/llm/src/conformance/openai.conformance.test.ts index 8ca39a39..8d901dfa 100644 --- a/packages/llm/src/conformance/openai.conformance.test.ts +++ b/packages/llm/src/conformance/openai.conformance.test.ts @@ -2,13 +2,14 @@ import { describe, expect, it } from 'vitest'; import { createOpenAiAdapter, openaiAdapter } from '../adapters/openai.js'; import { OPENAI_FIXTURES } from './fixtures/openai.js'; -import { replayFetch } from './replay.js'; +import { replayFor } from './replay.js'; import { defineConformanceSuite, type MakeReplayAdapter } from './spec.js'; -// Wire the OpenAI adapter to replay a recorded response — no vendor SDK is imported here; the adapter -// takes a `fetch` override, so the SDK stays inside src/adapters/* (the seam fence). +// Wire the OpenAI adapter to replay recorded response(s) — no vendor SDK is imported here; the adapter +// takes a `fetch` override, so the SDK stays inside src/adapters/* (the seam fence). `replayFor` serves +// one body (one-shot scenarios) or a sequence (the multi-turn tool loop). const makeReplayAdapter: MakeReplayAdapter = (recorded) => - createOpenAiAdapter({ providerId: 'openai', fetch: replayFetch(recorded), maxRetries: 0 }); + createOpenAiAdapter({ providerId: 'openai', fetch: replayFor(recorded), maxRetries: 0 }); defineConformanceSuite('openai', makeReplayAdapter, OPENAI_FIXTURES); diff --git a/packages/llm/src/conformance/replay.test.ts b/packages/llm/src/conformance/replay.test.ts index 5a5b67fa..264b3782 100644 --- a/packages/llm/src/conformance/replay.test.ts +++ b/packages/llm/src/conformance/replay.test.ts @@ -1,6 +1,12 @@ import { describe, expect, it } from 'vitest'; -import { looksLikeSecret, recordFetch, replayFetch } from './replay.js'; +import { + looksLikeSecret, + recordFetch, + replayFetch, + replayFetchSequence, + replayFor, +} from './replay.js'; describe('replayFetch', () => { it('serves the recorded response', async () => { @@ -21,6 +27,58 @@ describe('replayFetch', () => { }); }); +describe('replayFetchSequence', () => { + it('serves recorded responses in order, one per call (the multi-turn tool loop)', async () => { + const fetch = replayFetchSequence([ + { status: 200, body: '{"turn":1}' }, + { status: 200, body: '{"turn":2}' }, + ]); + expect(await (await fetch('https://x', { method: 'POST', body: '{}' })).text()).toBe( + '{"turn":1}', + ); + expect(await (await fetch('https://x', { method: 'POST', body: '{}' })).text()).toBe( + '{"turn":2}', + ); + }); + + it('rejects an over-fetch beyond the recorded sequence (a fixture bug fails loud)', async () => { + const fetch = replayFetchSequence([{ status: 200, body: '{}' }]); + await fetch('https://x', { method: 'POST', body: '{}' }); + await expect(fetch('https://x', { method: 'POST', body: '{}' })).rejects.toThrow( + /no recorded response for call #2/, + ); + }); + + it('rejects a request whose body is not valid JSON (parity with replayFetch)', async () => { + const fetch = replayFetchSequence([{ status: 200, body: '{}' }]); + await expect(fetch('https://x', { method: 'POST', body: 'nope' })).rejects.toThrow( + /not valid JSON/, + ); + }); +}); + +describe('replayFor', () => { + it('routes a single RecordedResponse to replayFetch (repeats every call)', async () => { + const fetch = replayFor({ status: 200, body: '{"single":true}' }); + expect(await (await fetch('https://x', { method: 'POST', body: '{}' })).text()).toBe( + '{"single":true}', + ); + // a single response repeats — a second call serves the same body (not an over-fetch error) + expect(await (await fetch('https://x', { method: 'POST', body: '{}' })).text()).toBe( + '{"single":true}', + ); + }); + + it('routes an array to replayFetchSequence (one per call)', async () => { + const fetch = replayFor([ + { status: 200, body: '{"n":1}' }, + { status: 200, body: '{"n":2}' }, + ]); + expect(await (await fetch('https://x', { method: 'POST', body: '{}' })).text()).toBe('{"n":1}'); + expect(await (await fetch('https://x', { method: 'POST', body: '{}' })).text()).toBe('{"n":2}'); + }); +}); + describe('recordFetch', () => { it('captures a clean response as a RecordedResponse', async () => { const real = (): Promise => diff --git a/packages/llm/src/conformance/replay.ts b/packages/llm/src/conformance/replay.ts index 80f6958f..1414e482 100644 --- a/packages/llm/src/conformance/replay.ts +++ b/packages/llm/src/conformance/replay.ts @@ -54,6 +54,50 @@ export function replayFetch(recorded: RecordedResponse): FetchLike { }; } +/** + * A `fetch` that replays a SEQUENCE of recorded responses — the Nth call serves `recordings[N]`. Drives a + * multi-turn scenario (a tool-call turn → a continuation carrying the tool result) offline + deterministically. + * Fails loud if called more times than there are recordings (a scenario that over-fetches is a fixture bug), + * and validates each request body is JSON (like {@link replayFetch}). + */ +export function replayFetchSequence(recordings: readonly RecordedResponse[]): FetchLike { + let call = 0; + return (_input, init) => { + if (typeof init?.body === 'string' && init.body.length > 0) { + try { + JSON.parse(init.body); + } catch { + return Promise.reject(new Error('replayFetchSequence: the request body is not valid JSON')); + } + } + const recorded = recordings[call]; + call += 1; + if (recorded === undefined) { + return Promise.reject( + new Error( + `replayFetchSequence: no recorded response for call #${String(call)} (only ${String(recordings.length)} recorded)`, + ), + ); + } + return Promise.resolve( + new Response(recorded.body, { + status: recorded.status, + headers: { 'content-type': recorded.contentType ?? 'application/json' }, + }), + ); + }; +} + +/** + * Pick the right replay `fetch` for a conformance scenario: a single {@link RecordedResponse} (the one-shot + * scenarios) replays the same body each call; an array replays it as a sequence (multi-turn). The `'status' + * in` check narrows the union cleanly without an unsafe cast (a `RecordedResponse` has `status`; an array + * does not). + */ +export function replayFor(recorded: RecordedResponse | readonly RecordedResponse[]): FetchLike { + return 'status' in recorded ? replayFetch(recorded) : replayFetchSequence(recorded); +} + /** * Wrap a real `fetch` to capture each response as a `RecordedResponse` — the live-mode recorder. It * **refuses to record** a body that looks like it contains a secret, so a captured fixture can never diff --git a/packages/llm/src/conformance/spec.ts b/packages/llm/src/conformance/spec.ts index 81e36017..03f821fd 100644 --- a/packages/llm/src/conformance/spec.ts +++ b/packages/llm/src/conformance/spec.ts @@ -24,6 +24,9 @@ export interface ConformanceExpectations { text: string; inputTokens: number; outputTokens: number; + /** The cached prompt tokens that folded into the canonical `Usage` (prompt-cache hit) — providers + * whose textGenerate fixture records a cache hit assert this; omit for a no-cache fixture. */ + cacheReadTokens?: number; }; readonly toolGenerate: { toolName: string; stopReason: StopReason }; readonly textStream: { stopReason: StopReason; inputTokens: number; outputTokens: number }; @@ -55,12 +58,31 @@ export interface ConformanceFixtures { readonly reasoningStream?: RecordedResponse; /** A non-streaming reply produced under `responseFormat: json` (ADR-0030) — omit if unsupported. */ readonly structuredOutput?: RecordedResponse; + /** + * A multi-turn tool loop (the path every agent node exercises): `turn1` is a tool-call reply; `turn2` is + * the continuation the provider returns AFTER the caller appends the tool result. The conformance test + * drives two generate() calls against one replay-sequence adapter, so `turn2` exercises the adapter + * lowering a `tool_result` message back onto the provider's wire format. Omit if not yet recorded. + */ + readonly toolLoop?: { + readonly turn1: RecordedResponse; + readonly turn2: RecordedResponse; + readonly expected: { readonly toolName: string; readonly finalText: string }; + }; /** The canonical values the above should normalize to. */ readonly expected: ConformanceExpectations; } -/** Build an adapter wired to replay a single recorded response (provider-specific). */ -export type MakeReplayAdapter = (recorded: RecordedResponse) => LlmProvider; +/** + * Build an adapter wired to replay recorded response(s) (provider-specific). A single {@link RecordedResponse} + * serves the one-shot scenarios; an array serves a multi-turn scenario (the Nth provider round-trip gets + * the Nth recording) — e.g. the tool-loop scenario's call → continuation. The provider factory normalizes + * both (a `fetch`-based adapter uses `replayFetch` / `replayFetchSequence`; the Gemini transport indexes the + * array per call). + */ +export type MakeReplayAdapter = ( + recorded: RecordedResponse | readonly RecordedResponse[], +) => LlmProvider; const KEY = 'conformance-test-key'; @@ -124,6 +146,11 @@ export function defineConformanceSuite( expect(result.usage.outputTokens).toBe(expected.textGenerate.outputTokens); expect(result.stopReason).toBe(expected.textGenerate.stopReason); expect(result.raw).toBeDefined(); + // A prompt-cache hit must fold into the ONE canonical Usage (cacheReadTokens), not be lost or + // double-counted into inputTokens — asserted for providers whose fixture records a cache hit. + if (expected.textGenerate.cacheReadTokens !== undefined) { + expect(result.usage.cacheReadTokens).toBe(expected.textGenerate.cacheReadTokens); + } }); it('generate: a tool call normalizes to a tool_call part with the expected name + id', async () => { @@ -257,5 +284,48 @@ export function defineConformanceSuite( } }, ); + + it.skipIf(fixtures.toolLoop === undefined)( + 'tool loop: a tool_call then a continuation carrying the tool_result yields final text (call→result→continuation)', + async () => { + const loop = fixtures.toolLoop; + if (loop === undefined) { + return; // narrow for skipIf + } + // One adapter, a replay SEQUENCE: turn 1 → the tool-call reply, turn 2 → the continuation. + const adapter = makeReplayAdapter([loop.turn1, loop.turn2]); + const r1 = await adapter.generate(TOOL_REQUEST, KEY); + const call = r1.content.find((part) => part.type === 'tool_call'); + expect(call?.type).toBe('tool_call'); + if (call?.type !== 'tool_call') { + return; // narrow + } + expect(call.name).toBe(loop.expected.toolName); + // Turn 2: append the assistant tool_call + the tool RESULT, then continue. SCOPE: this asserts the + // end-to-end call→result→continuation FLOW — the adapter accepts a tool_result message and produces a + // continuation without throwing or dropping the turn. The provider-SPECIFIC tool_result WIRE shape + // (Anthropic tool_result block, OpenAI {role:'tool'}, Gemini functionResponse) is asserted by the + // per-adapter unit tests (anthropic/openai/gemini .test.ts); the replay serves turn2 by call index, + // so this shared suite does not (and should not) re-assert each provider's request wire here. + const r2 = await adapter.generate( + { + ...TOOL_REQUEST, + messages: [ + ...TOOL_REQUEST.messages, + { role: 'assistant', content: [call] }, + { + role: 'tool', + content: [{ type: 'tool_result', toolCallId: call.id, result: 'sunny, 18C' }], + }, + ], + }, + KEY, + ); + expect(LlmResultSchema.safeParse(r2).success).toBe(true); + const text = r2.content.map((part) => (part.type === 'text' ? part.text : '')).join(''); + expect(text).toBe(loop.expected.finalText); + expect(r2.content.every((part) => part.type !== 'tool_call')).toBe(true); // a text continuation, not another call + }, + ); }); } diff --git a/packages/shared/src/run-event.ts b/packages/shared/src/run-event.ts index e7ca2185..b356d034 100644 --- a/packages/shared/src/run-event.ts +++ b/packages/shared/src/run-event.ts @@ -208,6 +208,11 @@ export const NodeCompletedEventSchema = z.object({ output: z.unknown(), tokensUsed: TokensUsedSchema, durationMs: nonNegativeInt, + // The run-wide cost running total AT this node boundary (integer micro-cents) — the SAME counter + // cost:updated carries, snapshotted onto the durable node:completed so checkpoint/resume (1.R) restores a + // run's cumulative cost across a process boundary (cost:updated itself is streamed, not persisted). Optional + // for backward-compat with logs persisted before this field existed; the engine always populates it. + cumulativeCostMicrocents: nonNegativeInt.optional(), // 1-based NODE-RETRY dispatch attempt (1.S, ADR-0040) — the same counter as node:started/node:failed. // Absent ⇒ attempt 1. DISTINCT from cost:updated/agent:* attemptNumber (the within-chain FallbackChain // index, which resets per node re-dispatch); the two counters do NOT join — see cost:updated above. diff --git a/vitest.config.ts b/vitest.config.ts index 65c1f567..2c305078 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -8,8 +8,8 @@ import { defineConfig } from 'vitest/config'; * Coverage uses the V8 provider with branch reporting. The Phase-1 **>= 90% line + branch** * engine floor (docs/standards/testing.md#coverage-expectations) is the threshold the built engine * package(s) must meet under `pnpm coverage` (run from the repo ROOT — the threshold glob below is - * root-relative). NOTE: `pnpm coverage` is NOT yet a required CI gate (CI runs lint/typecheck/test); - * wiring coverage into CI is tracked in deferred-tasks. Surfaces stay smoke-only. + * root-relative). NOTE: `pnpm coverage` runs as an ADVISORY (non-required) `coverage` job in ci.yml + * (promote it to a required check once the core-branch margin is confirmed stable). Surfaces stay smoke-only. */ export default defineConfig({ test: { @@ -37,7 +37,8 @@ export default defineConfig({ // (`pnpm coverage`). A package-scoped run (`cd packages/llm && vitest --coverage`) keys the // coverage map cwd-relative (`src/…`), so NO single glob can both stay package-targeted at // the root and still match there — a cwd-tolerant `src/**` would wrongly bind shared/db - // package runs to the engine floor. Tracked in deferred-tasks.md (coverage-in-CI item). + // package runs to the engine floor. The advisory `coverage` job (ci.yml) runs at the repo root, + // which is exactly where this per-glob threshold is authoritative. thresholds: { 'packages/llm/src/**/*.ts': { lines: 90, branches: 90 }, 'packages/core/src/**/*.ts': { lines: 90, branches: 90 }, // engine floor — core landed at 1.L