From d06fb55235fd95f515312703e1be6f0aee75bb1e Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 22 Jun 2026 19:50:05 +0300 Subject: [PATCH 1/8] docs(roadmap): mark 2.F Done (PR #42) across the status surfaces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #42 merged the CLI `--json` machine-output contract (2.F) + ADR-0049 + the 2.D-Done bookkeeping. Flip the status surfaces now that it's on main (roadmap-done-after-merge): - phase-2-cli.md: 2.F heading → ✅ Done (PR #42); status line records it (behind ADR-0049) and advances the spine pointer to 2.K (the regression harness that reaches M3). - current.md: record 2.F Done (PR #42); "Next on the spine" → 2.K. - CLAUDE.md / README.md: status paragraphs note the `--json` CI output contract has landed. The 2.E (ink TUI) and 2.H (durable run history) run-surface feeders remain open. Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 5 +++-- README.md | 4 ++-- docs/roadmap/current.md | 8 +++++--- docs/roadmap/phases/phase-2-cli.md | 4 ++-- 4 files changed, 12 insertions(+), 9 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 21ce2efc..0c686f9d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -46,8 +46,9 @@ multimodal media I/O (input, inline output, and the async generation job loop). three adapters (Anthropic, OpenAI/DeepSeek, Gemini) and the agent-first `AgentSession` entry point (multi-turn sessions, persistence, export-to-workflow) are shipped. **Phase 2 (CLI, milestone M3) is in progress** — the CLI skeleton (2.A) and config -resolution (2.B) have landed (PR #40), and `relavium run` is wired to the engine -(2.D, the M3 keystone — PR #41). For live status, per-PR history, +resolution (2.B) have landed (PR #40), `relavium run` is wired to the engine +(2.D, the M3 keystone — PR #41), and the `--json` CI machine-output contract has +landed (2.F — PR #42, ADR-0049). For live status, per-PR history, milestone dates, and open obligations, see the canonical home [docs/roadmap/current.md](docs/roadmap/current.md); [README.md](README.md) is the public overview. diff --git a/README.md b/README.md index c595e8bd..d9092e74 100644 --- a/README.md +++ b/README.md @@ -92,8 +92,8 @@ One engine, three modes behind the one `LLMProvider` seam: **Phase 1 — Engine and LLM is complete** (2026-06-21): the engine runs end-to-end on local-first BYOK — workflow parsing, DAG execution, live streaming, checkpoint/resume, multi-provider failover, cost governance, and multimodal media I/O. **Phase 2 (the CLI) is -underway** — the CLI skeleton, config resolution, and `relavium run` (wired to the engine) have -landed. For live status and the full roadmap, see +underway** — the CLI skeleton, config resolution, `relavium run` (wired to the engine), and its +`--json` CI machine-output contract have landed. For live status and the full roadmap, see [docs/roadmap/current.md](docs/roadmap/current.md) and the [roadmap](docs/roadmap/README.md). diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index f608971f..670d7607 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -39,9 +39,11 @@ process contract (**2.A**) and the two-level config-resolution loader (**2.B**), (PR #40, 2026-06-22) behind [ADR-0047](../decisions/0047-cli-framework-commander-ink-clack.md) (commander/ink/@clack) and [ADR-0048](../decisions/0048-toml-config-parser.md) (smol-toml); and **2.D** (`relavium run` wired to `@relavium/core` — the M3 keystone and first real engine -consumer), ✅ Done (PR #41, 2026-06-22), which also adds the `defaultProviders()` seam registry. -**Next on the spine:** **2.F** (`--json` CI mode) then **2.K** (regression harness) to reach -**M3**, with the **2.E** (ink TUI) and **2.H** (durable run history) run-surface feeders now open. The CLI also lands the inbound MCP client (2.R, +consumer), ✅ Done (PR #41, 2026-06-22), which also adds the `defaultProviders()` seam registry; +and **2.F** (the `--json` CI machine-output contract — pure-NDJSON stdout, diagnostics → stderr), +✅ Done (PR #42, 2026-06-22) behind [ADR-0049](../decisions/0049-cli-machine-output-contract.md). +**Next on the spine:** **2.K** (the engine regression harness) to reach **M3**, with the **2.E** +(ink TUI) and **2.H** (durable run history) run-surface feeders open. The CLI also lands the inbound MCP client (2.R, [ADR-0034](../decisions/0034-mcp-client-sdk-dependency.md)) off the M3 critical path. See the [Phase 2 workstreams](phases/phase-2-cli.md) and the [sequencing plan](phases/phase-2-cli.md#sequencing--parallelization). diff --git a/docs/roadmap/phases/phase-2-cli.md b/docs/roadmap/phases/phase-2-cli.md index 4d515f5b..24ce3739 100644 --- a/docs/roadmap/phases/phase-2-cli.md +++ b/docs/roadmap/phases/phase-2-cli.md @@ -1,6 +1,6 @@ # Phase 2 — CLI -> Status: In progress (Product Phase 1, build phase 2). **2.A** (CLI skeleton + process contract) and **2.B** (config resolution) are ✅ **Done (PR #40, 2026-06-22)**, behind [ADR-0047](../../decisions/0047-cli-framework-commander-ink-clack.md) + [ADR-0048](../../decisions/0048-toml-config-parser.md); **2.D** (`run` → engine, the M3 keystone) is ✅ **Done (PR #41, 2026-06-22)**. The global milestone in play is **M3** (reached at 2.F + 2.K); next on the spine is **2.F** (`--json` CI mode), and the **2.E** (ink TUI) + **2.H** (durable run history) run-surface feeders now open. +> Status: In progress (Product Phase 1, build phase 2). **2.A** (CLI skeleton + process contract) and **2.B** (config resolution) are ✅ **Done (PR #40, 2026-06-22)**, behind [ADR-0047](../../decisions/0047-cli-framework-commander-ink-clack.md) + [ADR-0048](../../decisions/0048-toml-config-parser.md); **2.D** (`run` → engine, the M3 keystone) is ✅ **Done (PR #41, 2026-06-22)**, and **2.F** (the `--json` CI machine-output contract) is ✅ **Done (PR #42, 2026-06-22)**, behind [ADR-0049](../../decisions/0049-cli-machine-output-contract.md). The global milestone in play is **M3** (reached at 2.F + 2.K); next on the spine is **2.K** (the engine regression harness), with the **2.E** (ink TUI) + **2.H** (durable run history) run-surface feeders open. - **Related**: [../README.md](../README.md), [phase-1-engine-and-llm.md](phase-1-engine-and-llm.md), [phase-3-desktop.md](phase-3-desktop.md), [../../reference/cli/commands.md](../../reference/cli/commands.md), [../../reference/contracts/config-spec.md](../../reference/contracts/config-spec.md), [../../reference/desktop/keychain-and-secrets.md](../../reference/desktop/keychain-and-secrets.md), [../../reference/contracts/sse-event-schema.md](../../reference/contracts/sse-event-schema.md), [../../reference/desktop/database-schema.md](../../reference/desktop/database-schema.md), [../../architecture/execution-model.md](../../architecture/execution-model.md), [../../architecture/shared-core-engine.md](../../architecture/shared-core-engine.md) @@ -260,7 +260,7 @@ status transitions, streaming tokens on the active node, and a running cost that matches the final summary; `--no-color` produces plain output; no event is dropped under a high token rate (verified against the JSON stream's event count). -### 2.F — CI / non-interactive `--json` mode + exit codes +### 2.F — CI / non-interactive `--json` mode + exit codes — ✅ **Done (PR #42)** Implement the machine-readable path: one `RunEvent` per line (NDJSON), a stable envelope, and deterministic exit codes — the contract CI jobs assert on. From cd8166e3bc18305c48c879190299aa5a25713f0e Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 22 Jun 2026 20:49:03 +0300 Subject: [PATCH 2/8] =?UTF-8?q?feat(cli):=202.K=20=E2=80=94=20engine=20reg?= =?UTF-8?q?ression=20harness=20over=20`relavium=20run=20--json`?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adopt the CLI as the engine's canonical end-to-end regression gate (completes M3 with 2.D + 2.F). An in-process vitest e2e suite (apps/cli/src/harness/) runs committed example workflows through the REAL `relavium run … --json` path (the default engine over the real createCliHost) and asserts on the NDJSON RunEvent stream + exit code — deterministic and offline on every PR, inside the existing required `test` CI gate. - Fixtures (apps/cli/src/harness/fixtures/, non-agent → no LLM/key/network): sequential, fan-out (parallel + object_merge), conditional (condition + node:skipped, both arms via n=3→lo and n=15→hi), human-gate (run:paused → exit 3), failure (a throwing transform → run:failed → exit 1). Together they cover every authorable non-agent node type and all three run exit codes (0/1/3). - Assertion is keyed by a `type:nodeId` SIGNATURE (not just event type), so a wrong-branch / wrong-node regression is caught — exact ordered sequence for deterministic fixtures, the signature multiset + anchors for the `parallel` fan-out (whose branches interleave by async timing). Plus gap-free sequenceNumber (ADR-0036), each line round-trips to exactly a canonical RunEvent, and stderr is empty (the stdout-pure --json contract, ADR-0049). - Shared captureIo extracted to apps/cli/src/test-support.ts; the three other test copies (run.test, commands/run.test, render/renderer.test) now import it. No new ADR, no new dependency, no @relavium/shared change — 2.K composes ADR-0036/0047/0049 + the @relavium/llm conformance/replay substrate + the 1.U m2-e2e-harness seed. Docs: a new docs/reference/cli/regression-harness.md (the canonical fixture + scenario format) + a testing.md section; the §2.K work note records the maintainer-approved scope — the gate-RESUME scenario is deferred to 2.G+2.H and agent-replay + the nightly live lane to later (both need the replay-provider wiring), with the run-to-gate→exit-3 half shipping now. Also marks 2.F Done (PR #42) across the status surfaces (post-merge bookkeeping, riding along). Refs: ADR-0036, ADR-0047, ADR-0049 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/commands/run.test.ts | 17 +- .../fixtures/conditional.relavium.yaml | 22 ++ .../harness/fixtures/failure.relavium.yaml | 12 + .../harness/fixtures/fan-out.relavium.yaml | 19 ++ .../harness/fixtures/human-gate.relavium.yaml | 12 + .../harness/fixtures/sequential.relavium.yaml | 14 ++ apps/cli/src/harness/regression.e2e.test.ts | 209 ++++++++++++++++++ apps/cli/src/render/renderer.test.ts | 18 +- apps/cli/src/run.test.ts | 48 ++-- apps/cli/src/test-support.ts | 22 ++ docs/reference/cli/regression-harness.md | 99 +++++++++ docs/roadmap/phases/phase-2-cli.md | 20 +- docs/standards/testing.md | 8 +- 13 files changed, 453 insertions(+), 67 deletions(-) create mode 100644 apps/cli/src/harness/fixtures/conditional.relavium.yaml create mode 100644 apps/cli/src/harness/fixtures/failure.relavium.yaml create mode 100644 apps/cli/src/harness/fixtures/fan-out.relavium.yaml create mode 100644 apps/cli/src/harness/fixtures/human-gate.relavium.yaml create mode 100644 apps/cli/src/harness/fixtures/sequential.relavium.yaml create mode 100644 apps/cli/src/harness/regression.e2e.test.ts create mode 100644 apps/cli/src/test-support.ts create mode 100644 docs/reference/cli/regression-harness.md diff --git a/apps/cli/src/commands/run.test.ts b/apps/cli/src/commands/run.test.ts index b9b8220e..829579b8 100644 --- a/apps/cli/src/commands/run.test.ts +++ b/apps/cli/src/commands/run.test.ts @@ -17,6 +17,7 @@ import { isCliError } from '../process/errors.js'; import { EXIT_CODES } from '../process/exit-codes.js'; import type { CliIo } from '../process/io.js'; import type { GlobalOptions } from '../process/options.js'; +import { captureIo } from '../test-support.js'; import { runCommand, type RunCommandDeps } from './run.js'; // A minimal real workflow: input → transform → output. Runs end-to-end through the standard node @@ -105,22 +106,6 @@ afterEach(() => { rmSync(root, { recursive: true, force: true }); }); -function captureIo(): { io: CliIo; out: () => string; err: () => string } { - const outChunks: string[] = []; - const errChunks: string[] = []; - const io: CliIo = { - writeOut: (text) => { - outChunks.push(text); - }, - writeErr: (text) => { - errChunks.push(text); - }, - env: {}, - stdoutIsTty: false, - }; - return { io, out: () => outChunks.join(''), err: () => errChunks.join('') }; -} - function globalOptions(over: Partial = {}): GlobalOptions { return { json: false, diff --git a/apps/cli/src/harness/fixtures/conditional.relavium.yaml b/apps/cli/src/harness/fixtures/conditional.relavium.yaml new file mode 100644 index 00000000..53f9107f --- /dev/null +++ b/apps/cli/src/harness/fixtures/conditional.relavium.yaml @@ -0,0 +1,22 @@ +schema_version: '1.0' +workflow: + id: harness-conditional + name: Conditional branch + description: A condition routes to one of two branches; the other is skip-propagated (node:skipped). + inputs: + - { name: n, type: number } + nodes: + - { id: start, type: input } + - id: check + type: condition + expression: 'inputs.n >= 10' + branches: + - { when: true, target_node: hi } + - { when: false, target_node: lo } + - { id: hi, type: transform, transform: '({ label: "high" })' } + - { id: lo, type: transform, transform: '({ label: "low" })' } + - { id: out, type: output } + edges: + - { from: start, to: check } + - { from: hi, to: out } + - { from: lo, to: out } diff --git a/apps/cli/src/harness/fixtures/failure.relavium.yaml b/apps/cli/src/harness/fixtures/failure.relavium.yaml new file mode 100644 index 00000000..4f50a746 --- /dev/null +++ b/apps/cli/src/harness/fixtures/failure.relavium.yaml @@ -0,0 +1,12 @@ +schema_version: '1.0' +workflow: + id: harness-failure + name: Failing transform + description: A transform throws at runtime → node:failed → run:failed → exit 1 (the failure terminal). + nodes: + - { id: start, type: input } + - { id: boom, type: transform, transform: 'null.boom' } + - { id: out, type: output } + edges: + - { from: start, to: boom } + - { from: boom, to: out } diff --git a/apps/cli/src/harness/fixtures/fan-out.relavium.yaml b/apps/cli/src/harness/fixtures/fan-out.relavium.yaml new file mode 100644 index 00000000..4d27a88e --- /dev/null +++ b/apps/cli/src/harness/fixtures/fan-out.relavium.yaml @@ -0,0 +1,19 @@ +schema_version: '1.0' +workflow: + id: harness-fan-out + name: Fan-out / aggregate + description: A parallel split (double + triple) merged back with object_merge — fan_out/fan_in coverage. + inputs: + - { name: n, type: number } + nodes: + - { id: start, type: input } + - { id: fan, type: parallel, parallel_of: [double, triple] } + - { id: double, type: transform, transform: '({ d: inputs.n * 2 })' } + - { id: triple, type: transform, transform: '({ t: inputs.n * 3 })' } + - { id: combine, type: merge, merge_strategy: object_merge } + - { id: out, type: output } + edges: + - { from: start, to: fan } + - { from: double, to: combine } + - { from: triple, to: combine } + - { from: combine, to: out } diff --git a/apps/cli/src/harness/fixtures/human-gate.relavium.yaml b/apps/cli/src/harness/fixtures/human-gate.relavium.yaml new file mode 100644 index 00000000..cbebb028 --- /dev/null +++ b/apps/cli/src/harness/fixtures/human-gate.relavium.yaml @@ -0,0 +1,12 @@ +schema_version: '1.0' +workflow: + id: harness-human-gate + name: Human gate (pause) + description: A human_gate node parks the run; non-interactively it exits gate-paused (3) without resuming. + nodes: + - { id: start, type: input } + - { id: gate, type: human_gate, gate_type: approval } + - { id: out, type: output } + edges: + - { from: start, to: gate } + - { from: gate, to: out } diff --git a/apps/cli/src/harness/fixtures/sequential.relavium.yaml b/apps/cli/src/harness/fixtures/sequential.relavium.yaml new file mode 100644 index 00000000..82dca77c --- /dev/null +++ b/apps/cli/src/harness/fixtures/sequential.relavium.yaml @@ -0,0 +1,14 @@ +schema_version: '1.0' +workflow: + id: harness-sequential + name: Sequential pipeline + description: A linear input → transform → output chain — the simplest engine run. + inputs: + - { name: n, type: number } + nodes: + - { id: start, type: input } + - { id: double, type: transform, transform: '({ doubled: inputs.n * 2 })' } + - { id: out, type: output } + edges: + - { from: start, to: double } + - { from: double, to: out } diff --git a/apps/cli/src/harness/regression.e2e.test.ts b/apps/cli/src/harness/regression.e2e.test.ts new file mode 100644 index 00000000..83bec864 --- /dev/null +++ b/apps/cli/src/harness/regression.e2e.test.ts @@ -0,0 +1,209 @@ +import { fileURLToPath } from 'node:url'; + +import { RunEventSchema, type RunEvent } from '@relavium/shared'; +import { describe, expect, it } from 'vitest'; + +import { runCommand } from '../commands/run.js'; +import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; +import type { GlobalOptions } from '../process/options.js'; +import { captureIo } from '../test-support.js'; + +/** + * The 2.K engine regression harness — the CLI adopted as the engine's canonical end-to-end test gate. + * Each fixture under ./fixtures/ is a committed `.relavium.yaml` run through the REAL `relavium run` + * path (`runCommand` → the default engine: the standard node executor + expression sandbox over the + * real `createCliHost`) in `--json` mode, and the resulting NDJSON stream + exit code are asserted + * against a pinned expectation. The fixtures are non-agent, so runs are fully deterministic and offline + * (no LLM, no provider key, no network) on every PR — the per-PR regression gate phase-2-cli.md §2.K + * requires. See [docs/reference/cli/regression-harness.md](../../../../docs/reference/cli/regression-harness.md). + * + * Each event is reduced to a `type:nodeId` SIGNATURE (so the assertion is keyed by node, not just by + * event type — a wrong-branch / wrong-node regression is caught, not only a dropped/extra event). For a + * deterministic fixture the exact signature ORDER is asserted; the `fan-out` fixture is `parallel`, whose + * branch events legitimately interleave by async (sandbox) completion timing, so it asserts the signature + * MULTISET + the stable anchors (`run:started` first, the last-emitted event last, gap-free + * `sequenceNumber`) instead. `runId`/`timestamp`/cost/duration are per-run and never asserted. + * + * Deferred (scope-split in phase-2-cli.md §2.K): the gate-RESUME scenario (`relavium gate --approve` → + * completion) needs 2.G + 2.H; agent fixtures via recorded-LLM replay and the nightly live-provider lane + * need the replay-provider wiring. This first cut covers the non-agent scenarios + run-to-gate. + */ + +const FIXTURES_DIR = fileURLToPath(new URL('./fixtures/', import.meta.url)); + +interface Scenario { + readonly file: string; + readonly input: readonly string[]; + readonly exit: ExitCode; + /** Expected `type:nodeId` signatures (a node-bearing event is `type:nodeId`, otherwise just `type`). */ + readonly events: readonly string[]; + /** A `parallel` node interleaves its branch events by async timing → assert the multiset, not the order. */ + readonly parallel?: boolean; +} + +const SCENARIOS: readonly Scenario[] = [ + { + file: 'sequential.relavium.yaml', + input: ['n=3'], + exit: EXIT_CODES.success, + events: [ + 'run:started', + 'node:started:start', + 'node:completed:start', + 'node:started:double', + 'node:completed:double', + 'node:started:out', + 'node:completed:out', + 'run:completed', + ], + }, + { + file: 'fan-out.relavium.yaml', + input: ['n=3'], + exit: EXIT_CODES.success, + parallel: true, + events: [ + 'run:started', + 'node:started:start', + 'node:completed:start', + 'node:started:fan', + 'node:completed:fan', + 'node:started:double', + 'node:started:triple', + 'node:completed:double', + 'node:completed:triple', + 'node:started:combine', + 'node:completed:combine', + 'node:started:out', + 'node:completed:out', + 'run:completed', + ], + }, + { + // n < 10 → the `lo` branch runs, `hi` is skip-propagated. + file: 'conditional.relavium.yaml', + input: ['n=3'], + exit: EXIT_CODES.success, + events: [ + 'run:started', + 'node:started:start', + 'node:completed:start', + 'node:started:check', + 'node:completed:check', + 'node:skipped:hi', + 'node:started:lo', + 'node:completed:lo', + 'node:started:out', + 'node:completed:out', + 'run:completed', + ], + }, + { + // n >= 10 → the OTHER branch: `hi` runs, `lo` is skip-propagated (proves both arms + symmetric skip). + file: 'conditional.relavium.yaml', + input: ['n=15'], + exit: EXIT_CODES.success, + events: [ + 'run:started', + 'node:started:start', + 'node:completed:start', + 'node:started:check', + 'node:completed:check', + 'node:skipped:lo', + 'node:started:hi', + 'node:completed:hi', + 'node:started:out', + 'node:completed:out', + 'run:completed', + ], + }, + { + file: 'human-gate.relavium.yaml', + input: [], + exit: EXIT_CODES.gatePaused, + events: [ + 'run:started', + 'node:started:start', + 'node:completed:start', + 'node:started:gate', + 'human_gate:paused:gate', + 'run:paused', // not the run's logical terminal (run:paused is non-terminal) — the last event before exit 3 + ], + }, + { + file: 'failure.relavium.yaml', + input: [], + exit: EXIT_CODES.workflowFailed, + events: [ + 'run:started', + 'node:started:start', + 'node:completed:start', + 'node:started:boom', + 'node:failed:boom', + 'node:skipped:out', // the downstream output is unreachable once `boom` fails + 'run:failed', + ], + }, +]; + +function signature(event: RunEvent): string { + return 'nodeId' in event ? `${event.type}:${String(event.nodeId)}` : event.type; +} + +function globalOptions(): GlobalOptions { + return { + json: true, + color: false, + cwd: FIXTURES_DIR, + configPath: undefined, + verbosity: 'normal', + }; +} + +/** Run one fixture through `relavium run … --json`, returning per-node signatures + raw events + exit. */ +async function runFixture( + scenario: Scenario, +): Promise<{ sigs: string[]; events: RunEvent[]; code: ExitCode }> { + const { io, out, err } = captureIo(); + const code = await runCommand( + { workflow: `${FIXTURES_DIR}${scenario.file}`, input: [...scenario.input] }, + { io, global: globalOptions() }, + ); + expect(err()).toBe(''); // a clean offline run writes nothing to stderr (stdout-pure contract, ADR-0049) + // Every stdout line is EXACTLY one canonical RunEvent — round-trip equality, not just parse-success. + const events = out() + .trimEnd() + .split('\n') + .map((line) => { + const raw: unknown = JSON.parse(line); + const event = RunEventSchema.parse(raw); + expect(event).toEqual(raw); + return event; + }); + return { sigs: events.map(signature), events, code }; +} + +/** `sequenceNumber` must be exactly 0..n-1 — the bus's gap-free, exactly-once guarantee (ADR-0036). */ +function assertGapFreeSeq(events: readonly RunEvent[]): void { + expect(events.map((e) => e.sequenceNumber)).toEqual(events.map((_, index) => index)); +} + +describe('engine regression harness (2.K) — offline fixtures over `relavium run … --json`', () => { + for (const scenario of SCENARIOS) { + const label = `${scenario.file} [${scenario.input.join(' ') || 'no input'}] → exit ${String(scenario.exit)}`; + it(label, async () => { + const { sigs, events, code } = await runFixture(scenario); + expect(code).toBe(scenario.exit); + assertGapFreeSeq(events); + if (scenario.parallel === true) { + // Branch events interleave by async timing → assert the multiset + the stable anchors. + expect(sigs[0]).toBe('run:started'); + expect(sigs.at(-1)).toBe(scenario.events.at(-1)); + expect([...sigs].sort()).toEqual([...scenario.events].sort()); + } else { + // Fully deterministic → assert the exact ordered per-node signature sequence. + expect(sigs).toEqual([...scenario.events]); + } + }); + } +}); diff --git a/apps/cli/src/render/renderer.test.ts b/apps/cli/src/render/renderer.test.ts index 7fde575a..eb89a088 100644 --- a/apps/cli/src/render/renderer.test.ts +++ b/apps/cli/src/render/renderer.test.ts @@ -1,25 +1,9 @@ import { RunEventSchema, type RunEvent } from '@relavium/shared'; import { describe, expect, it } from 'vitest'; -import type { CliIo } from '../process/io.js'; +import { captureIo } from '../test-support.js'; import { createJsonRenderer, createPlainRenderer } from './renderer.js'; -function captureIo(): { io: CliIo; out: () => string; err: () => string } { - const outChunks: string[] = []; - const errChunks: string[] = []; - const io: CliIo = { - writeOut: (text) => { - outChunks.push(text); - }, - writeErr: (text) => { - errChunks.push(text); - }, - env: {}, - stdoutIsTty: false, - }; - return { io, out: () => outChunks.join(''), err: () => errChunks.join('') }; -} - const ENVELOPE = { runId: 'id-1', sequenceNumber: 0, timestamp: '2026-01-01T00:00:00.000Z' }; /** Build a schema-validated `RunEvent` (no unsafe cast — `.parse()` proves the fixture is real). */ diff --git a/apps/cli/src/run.test.ts b/apps/cli/src/run.test.ts index d6034e2b..9598e07b 100644 --- a/apps/cli/src/run.test.ts +++ b/apps/cli/src/run.test.ts @@ -1,24 +1,8 @@ import { describe, expect, it } from 'vitest'; -import type { CliIo } from './process/io.js'; import { CLI_VERSION } from './program.js'; import { run } from './run.js'; - -function makeIo(): { io: CliIo; out: () => string; err: () => string } { - const outChunks: string[] = []; - const errChunks: string[] = []; - const io: CliIo = { - writeOut: (text) => { - outChunks.push(text); - }, - writeErr: (text) => { - errChunks.push(text); - }, - env: {}, - stdoutIsTty: false, - }; - return { io, out: () => outChunks.join(''), err: () => errChunks.join('') }; -} +import { captureIo } from './test-support.js'; const argv = (...tokens: string[]): string[] => ['node', 'relavium', ...tokens]; @@ -38,7 +22,7 @@ function isErrorEnvelope(value: unknown): value is { type: string; code: string; describe('run', () => { it('prints help and exits 0 for --help', async () => { - const { io, out } = makeIo(); + const { io, out } = captureIo(); expect(await run(argv('--help'), io)).toBe(0); expect(out()).toContain('Usage: relavium'); expect(out()).toContain('run'); @@ -46,13 +30,13 @@ describe('run', () => { }); it('prints help and exits 0 for a bare invocation', async () => { - const { io, out } = makeIo(); + const { io, out } = captureIo(); expect(await run(argv(), io)).toBe(0); expect(out()).toContain('Usage: relavium'); }); it('prints the version and exits 0 for --version', async () => { - const { io, out } = makeIo(); + const { io, out } = captureIo(); expect(await run(argv('--version'), io)).toBe(0); expect(out()).toContain(CLI_VERSION); }); @@ -61,24 +45,24 @@ describe('run', () => { // a bare invocation are exit-0 meta-operations that still print human text to stdout under --json — // and nothing to stderr. it('keeps --help / --version / bare as human-on-stdout exit-0 meta-ops even under --json', async () => { - const help = makeIo(); + const help = captureIo(); expect(await run(argv('--json', '--help'), help.io)).toBe(0); expect(help.out()).toContain('Usage: relavium'); expect(help.err()).toBe(''); - const version = makeIo(); + const version = captureIo(); expect(await run(argv('--json', '--version'), version.io)).toBe(0); expect(version.out()).toContain(CLI_VERSION); expect(version.err()).toBe(''); - const bare = makeIo(); + const bare = captureIo(); expect(await run(argv('--json'), bare.io)).toBe(0); expect(bare.out()).toContain('Usage: relavium'); expect(bare.err()).toBe(''); }); it('keeps stderr a single JSON envelope under --json even with --verbose (no raw stack)', async () => { - const { io, out, err } = makeIo(); + const { io, out, err } = captureIo(); expect(await run(argv('create', '--json', '--verbose'), io)).toBe(2); // `create` is a stub (2.J) expect(out()).toBe(''); // stderr is exactly one parseable JSON line — --verbose adds no raw stack text under --json. @@ -88,18 +72,18 @@ describe('run', () => { }); it('exits 2 for an unknown command', async () => { - const { io, err } = makeIo(); + const { io, err } = captureIo(); expect(await run(argv('bogus'), io)).toBe(2); expect(err().toLowerCase()).toContain('unknown command'); }); it('exits 2 when a required argument is missing', async () => { - const { io } = makeIo(); + const { io } = captureIo(); expect(await run(argv('run'), io)).toBe(2); }); it('exits 2 with a clean not-implemented message for a stub command (no stack leak)', async () => { - const { io, out, err } = makeIo(); + const { io, out, err } = captureIo(); expect(await run(argv('create'), io)).toBe(2); // `create` is still a stub (2.J) expect(err()).toContain('not available yet'); // No stack frame as primary output — a Node frame line is ` at …` (string check, no regex). @@ -111,7 +95,7 @@ describe('run', () => { }); it('emits the structured JSON error envelope on stderr under --json, stdout empty (ADR-0049)', async () => { - const { io, out, err } = makeIo(); + const { io, out, err } = captureIo(); const code = await run(argv('create', '--json'), io); // `create` is still a stub (2.J) expect(code).toBe(2); expect(out()).toBe(''); // stdout stays pure: a CLI fault is a stderr diagnostic @@ -124,13 +108,13 @@ describe('run', () => { }); it('exits 2 when --verbose and --quiet are combined', async () => { - const { io, err } = makeIo(); + const { io, err } = captureIo(); expect(await run(argv('--verbose', '--quiet', 'list'), io)).toBe(2); expect(err()).toContain('cannot be combined'); }); it('renders the JSON error envelope on stderr when --json precedes a failing global flag', async () => { - const { io, out, err } = makeIo(); + const { io, out, err } = captureIo(); expect(await run(argv('--json', '--cwd'), io)).toBe(2); expect(out()).toBe(''); const parsed: unknown = JSON.parse(err().trim()); @@ -142,7 +126,7 @@ describe('run', () => { }); it('renders a commander parse error as a JSON envelope on stderr under --json, stdout empty', async () => { - const { io, out, err } = makeIo(); + const { io, out, err } = captureIo(); expect(await run(argv('--json', 'bogus'), io)).toBe(2); expect(out()).toBe(''); // stdout stays pure NDJSON territory — never a fault envelope const parsed: unknown = JSON.parse(err().trim()); // commander's own human message is suppressed @@ -155,7 +139,7 @@ describe('run', () => { }); it('treats a lone -- as a bare invocation (prints help, exits 0)', async () => { - const { io, out } = makeIo(); + const { io, out } = captureIo(); expect(await run(argv('--'), io)).toBe(0); expect(out()).toContain('Usage: relavium'); }); diff --git a/apps/cli/src/test-support.ts b/apps/cli/src/test-support.ts new file mode 100644 index 00000000..2d91b5e7 --- /dev/null +++ b/apps/cli/src/test-support.ts @@ -0,0 +1,22 @@ +import type { CliIo } from './process/io.js'; + +/** + * Test-only IO capture: a {@link CliIo} whose `writeOut`/`writeErr` accumulate into arrays, so a test + * can assert on the exact stdout (NDJSON / human lines) and stderr (diagnostics) a command produced. + * Shared by the command tests and the 2.K regression harness so the capture shape never diverges. + */ +export function captureIo(): { io: CliIo; out: () => string; err: () => string } { + const outChunks: string[] = []; + const errChunks: string[] = []; + const io: CliIo = { + writeOut: (text) => { + outChunks.push(text); + }, + writeErr: (text) => { + errChunks.push(text); + }, + env: {}, + stdoutIsTty: false, + }; + return { io, out: () => outChunks.join(''), err: () => errChunks.join('') }; +} diff --git a/docs/reference/cli/regression-harness.md b/docs/reference/cli/regression-harness.md new file mode 100644 index 00000000..55a665cc --- /dev/null +++ b/docs/reference/cli/regression-harness.md @@ -0,0 +1,99 @@ +# Engine Regression Harness (2.K) + +> Last updated: 2026-06-22 + +- **Status**: Reference (Living) — the canonical home for the harness fixture + scenario format. +- **Surface**: CLI (`relavium`), adopted as the engine's end-to-end regression gate. +- **Scope**: Phase 2, build workstream **2.K**; completes milestone **M3** with 2.D + 2.F. +- **Related**: [commands.md](commands.md), [../contracts/sse-event-schema.md](../contracts/sse-event-schema.md), [../../decisions/0049-cli-machine-output-contract.md](../../decisions/0049-cli-machine-output-contract.md), [../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md), [../../standards/testing.md](../../standards/testing.md), [../../tutorials/cli/run-a-workflow-in-ci.md](../../tutorials/cli/run-a-workflow-in-ci.md) + +## What it is + +The CLI is the engine's **canonical end-to-end regression harness**: every change to +`@relavium/core` / `@relavium/llm` / `apps/cli` is exercised by running committed example +workflows through the real `relavium run … --json` path and asserting on the resulting NDJSON +[RunEvent](../contracts/sse-event-schema.md) stream + exit code. Because the run goes through the +same engine, renderer, and exit-code mapping as a production run, a regression in the engine's +run loop, the [`--json` machine-output contract](commands.md#the---json-machine-output-contract) +([ADR-0049](../../decisions/0049-cli-machine-output-contract.md)), or the exit codes fails the +harness. This is the agreed regression gate for Phases 3–6 — extend it, don't re-invent it. + +It grows from the Phase-1 `m2-e2e-harness` (the 1.U end-to-end engine harness); it lifts that +suite's discipline (`assertGapFreeSeq`, canonical-schema validation, the event signature) onto the +CLI surface. + +## Where it lives + +| Artifact | Path | +|----------|------| +| Fixture workflows (committed `.relavium.yaml`) | `apps/cli/src/harness/fixtures/` | +| The harness suite (in-process vitest e2e) | `apps/cli/src/harness/regression.e2e.test.ts` | + +The harness is an **in-process** suite: it drives the CLI's `runCommand` boundary with a captured +`CliIo` through the **default engine** (the standard node executor + expression sandbox over the real +`createCliHost`), exactly as a real `relavium run --json` invocation does (the same +`createJsonRenderer` produces byte-identical NDJSON). It does **not** spawn the built binary — that +would add a build dependency and process flakiness for no fidelity gain. It runs inside the standard +`pnpm turbo run test` task, so it is part of the required CI gate on every push/PR +(`.github/workflows/ci.yml`). + +## What is asserted (and what is not) + +For each fixture the harness runs `relavium run --json` and reduces each event to a +**`type:nodeId` signature** (a node-bearing event becomes `type:nodeId`, otherwise just `type`), then +asserts: + +- **Exit code** — `0` success, `1` failure, `3` gate-paused (the frozen [exit-code table](commands.md#exit-codes)). +- **stdout is a pure NDJSON `RunEvent` stream** — every line round-trips to **exactly** a canonical + `RunEvent` (`RunEventSchema.parse(line)` deep-equals the raw line, so a stray field is caught), and + stderr is empty on a clean run. +- **`sequenceNumber` is gap-free** — exactly `0..n-1`, the bus's exactly-once guarantee (ADR-0036). +- **The signature sequence** — keyed by node, so a wrong-**branch** / wrong-**node** regression is + caught (e.g. a `condition` routing to the wrong arm changes which node is `node:skipped`), not just a + dropped/extra/wrong-type event. For a deterministic fixture the **exact ordered** signature sequence + is asserted; the `parallel` `fan-out` fixture asserts the signature **multiset** + the anchors + (`run:started` first, the last-emitted event last), because its branch events legitimately interleave + by async (sandbox) completion timing. + +It deliberately does **not** assert `runId`, `timestamp`, cost, or duration — those are per-run (the +real CLI host uses a wall clock + random UUIDs). It also does not check transform/merge **output +values** (that is the engine's own unit-test concern); the harness gates the run's stream topology + +routing + exit code end-to-end through the CLI. + +## The fixture suite (first cut) + +All fixtures are **non-agent**, so runs are fully deterministic and **offline** on every PR — no LLM, +no provider key, no network. Together they cover every authorable non-agent node type and all three +run exit codes (`0` / `1` / `3`). + +| Fixture | Covers | Exit | +|---------|--------|------| +| `sequential.relavium.yaml` | input → transform → output (the simplest chain) | `0` | +| `fan-out.relavium.yaml` | `parallel` split + `merge` (`object_merge`) — fan_out/fan_in | `0` | +| `conditional.relavium.yaml` | `condition` branch + `node:skipped` skip-propagation (run **both** arms: `n=3`→`lo`, `n=15`→`hi`) | `0` | +| `human-gate.relavium.yaml` | `human_gate` pause → `run:paused` → gate-paused exit | `3` | +| `failure.relavium.yaml` | a transform throws → `node:failed` → `run:failed` (downstream skipped) | `1` | + +## Adding a fixture + +1. Author a committed `.relavium.yaml` under `apps/cli/src/harness/fixtures/` (schema: + [workflow-yaml-spec.md](../contracts/workflow-yaml-spec.md)). +2. Add a `Scenario` entry to `regression.e2e.test.ts` with its `input`, expected `exit`, the expected + `type:nodeId` signature list, and `parallel: true` only if the fixture has a `parallel` node. + **Capture** the real signatures by running the fixture once — never hand-guess them — then pin them. + +## Deferred (scope-split in [phase-2-cli.md §2.K](../../roadmap/phases/phase-2-cli.md)) + +- **Gate-*resume* scenario** (`relavium gate --approve` → completion) — needs the `relavium gate` + command (**2.G**) and durable run history (**2.H**); neither exists yet, so the CLI cannot reload and + resume a paused run across processes. The run-to-gate → exit `3` half ships now; the approve → + complete half lands with 2.G/2.H. (The engine's resume is already proven in `m2-e2e-harness`.) +- **Agent fixtures via recorded-LLM replay** — the `@relavium/llm` `conformance/replay.ts` substrate + (a recorded-`fetch` override that refuses secret bodies) can drive an agent workflow offline through + the injectable `ProviderResolver`; deferred so the first cut stays non-agent. Agent dispatch is + already covered by the `packages/core` agent-runner tests and the `@relavium/llm` conformance suites. +- **Nightly live-provider lane** — a scheduled, secret-gated lane that runs the agent fixtures above + against real providers. It activates **alongside** the reserved per-provider-conformance `live-api` + job in `.github/workflows/ci.yml` (`on: schedule`, keys from secrets) — that reserved job runs the + `@relavium/llm` conformance suites, not the CLI harness, but the maintainer decision (deferred-tasks) + is to enable both together. It never gates a PR. diff --git a/docs/roadmap/phases/phase-2-cli.md b/docs/roadmap/phases/phase-2-cli.md index 24ce3739..6ea9e0d7 100644 --- a/docs/roadmap/phases/phase-2-cli.md +++ b/docs/roadmap/phases/phase-2-cli.md @@ -409,7 +409,25 @@ change is exercised end-to-end. Completes M3 with `2.D` + `2.F`. **Acceptance:** a green CI run executes every fixture workflow with no TTY, asserting the NDJSON event stream and exit codes, including a gate-pause-then-resume scenario; the job is triggered by engine-package changes and is the agreed -regression gate for Phases 3–6. +regression gate for Phases 3–6. _(See **First cut + deferred** below for what the first +cut realizes — the harness rides the required `test` task rather than a new job, and the +gate-**resume** half is deferred to 2.G.)_ + +**First cut + deferred (scope-split, maintainer-approved 2026-06-22).** The harness lands as an +in-process vitest e2e suite (`apps/cli/src/harness/`) that drives the real `relavium run … --json` +path over four committed **non-agent** fixtures (sequential, fan-out/aggregate, conditional, +human-gate) — fully deterministic + offline, run inside the required CI `test` gate on every PR; +its fixture + scenario format is documented in +[reference/cli/regression-harness.md](../../reference/cli/regression-harness.md). Two parts are +**deferred** because they depend on not-yet-built workstreams: + +- _Gate-**resume** scenario_ (`relavium gate --approve` → completion) → lands with **2.G** (the + `gate` command, itself blocked on **2.H** durable run history to reload a paused run cross-process). + The run-to-gate → exit `3` half ships now (the engine's resume is already proven in `m2-e2e-harness`). +- _Agent fixtures via recorded-LLM replay + the nightly live-provider lane_ → the offline first cut is + non-agent; an agent fixture drives a recorded `fetch` cassette (`@relavium/llm` + `conformance/replay.ts`) through the injectable `ProviderResolver`, and the reserved `live-api` + (`on: schedule`) job in `.github/workflows/ci.yml` is the nightly variant that re-records + diffs them. ### 2.L — Packaging, distribution, and install verification diff --git a/docs/standards/testing.md b/docs/standards/testing.md index d7972f56..280b58e7 100644 --- a/docs/standards/testing.md +++ b/docs/standards/testing.md @@ -22,7 +22,13 @@ test effort first. The CLI (`apps/cli`) doubles as an integration harness: it exercises the engine under real conditions with no UI, so a regression in `WorkflowEngine` / `AgentRunner` surfaces -in CLI tests before it ever reaches the desktop. +in CLI tests before it ever reaches the desktop. The **engine regression harness** (2.K) is +the workflow-level extension of the per-provider conformance discipline below: it runs committed +example workflows through `relavium run … --json` and asserts on the NDJSON event stream + exit +code, offline and deterministic on every PR. Its fixture + scenario format is the one canonical +home — [reference/cli/regression-harness.md](../reference/cli/regression-harness.md) — and the +recorded-LLM agent fixtures it grows into follow the same regenerate-not-hand-edit rule as the +conformance fixtures. ## Unit tests — Vitest From 5c99bfbabf88470d598f57f706fb595bd66bb61b Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 22 Jun 2026 22:54:25 +0300 Subject: [PATCH 3/8] =?UTF-8?q?docs(roadmap):=202.K=20review=20=E2=80=94?= =?UTF-8?q?=20=C2=A72.K=20note=20says=20five=20fixtures=20(was=20four)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The §2.K first-cut note undercounted the harness fixtures: it omitted failure.relavium.yaml (the run:failed / exit-1 case), so it read "four" and undersold the "all three exit codes 0/1/3" coverage. The canonical reference/cli/regression-harness.md already lists all five; align the roadmap note. Refs: 2.K review Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/roadmap/phases/phase-2-cli.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/roadmap/phases/phase-2-cli.md b/docs/roadmap/phases/phase-2-cli.md index 6ea9e0d7..fff8c5fe 100644 --- a/docs/roadmap/phases/phase-2-cli.md +++ b/docs/roadmap/phases/phase-2-cli.md @@ -415,8 +415,10 @@ gate-**resume** half is deferred to 2.G.)_ **First cut + deferred (scope-split, maintainer-approved 2026-06-22).** The harness lands as an in-process vitest e2e suite (`apps/cli/src/harness/`) that drives the real `relavium run … --json` -path over four committed **non-agent** fixtures (sequential, fan-out/aggregate, conditional, -human-gate) — fully deterministic + offline, run inside the required CI `test` gate on every PR; +path over five committed **non-agent** fixtures (sequential, fan-out/aggregate, conditional, +human-gate, and a failure case → `run:failed`) — together covering every authorable non-agent node +type and all three run exit codes (`0`/`1`/`3`), fully deterministic + offline, run inside the +required CI `test` gate on every PR; its fixture + scenario format is documented in [reference/cli/regression-harness.md](../../reference/cli/regression-harness.md). Two parts are **deferred** because they depend on not-yet-built workstreams: From f6c7d48f8998da9b3b0cd49ae81f1c338afa7726 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 22 Jun 2026 23:05:06 +0300 Subject: [PATCH 4/8] =?UTF-8?q?test(cli):=202.K=20review=20=E2=80=94=20cov?= =?UTF-8?q?er=20the=20full=20run(argv)=20CLI=20shell=20for=20a=20real=20ru?= =?UTF-8?q?n?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2.K harness enters each fixture at the `runCommand` boundary (max engine fidelity). One review left an open observation: the argv → extractGlobalOptions → commander → `run ` subcommand → runCommand → terminal-exit-code shell is exercised by neither suite for a real run — run.test.ts drives the shell only for meta-ops (--help/--version → 0) and faults (→ 2), never a workflow reaching a run terminal (0/1/3). Close it: re-run one representative deterministic fixture through the full `run(argv('run', , '--input', …, '--json'))` entry and assert it yields the IDENTICAL NDJSON stream + exit code the runCommand-boundary harness pins. This both covers the argv-parsing glue for a real run (positional workflow, repeatable --input, the position-independent --json global) and validates the harness's own entry-point choice. Refactors NDJSON parsing into a shared parseEvents() used by both paths. Updates the canonical regression-harness.md + the harness docstring to document the runCommand-vs-argv split and that the shell is now covered end-to-end. apps/cli: 123 tests green (harness 7). No new ADR/dependency/seam; engine untouched. Refs: 2.K review Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/harness/regression.e2e.test.ts | 64 +++++++++++++++++---- docs/reference/cli/regression-harness.md | 9 +++ 2 files changed, 63 insertions(+), 10 deletions(-) diff --git a/apps/cli/src/harness/regression.e2e.test.ts b/apps/cli/src/harness/regression.e2e.test.ts index 83bec864..9fc8ec5a 100644 --- a/apps/cli/src/harness/regression.e2e.test.ts +++ b/apps/cli/src/harness/regression.e2e.test.ts @@ -6,6 +6,7 @@ import { describe, expect, it } from 'vitest'; import { runCommand } from '../commands/run.js'; import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; import type { GlobalOptions } from '../process/options.js'; +import { run } from '../run.js'; import { captureIo } from '../test-support.js'; /** @@ -24,6 +25,11 @@ import { captureIo } from '../test-support.js'; * MULTISET + the stable anchors (`run:started` first, the last-emitted event last, gap-free * `sequenceNumber`) instead. `runId`/`timestamp`/cost/duration are per-run and never asserted. * + * The per-fixture cases enter at the `runCommand` boundary (the engine-fidelity entry point); a final + * case re-runs one fixture through the full `run(argv)` CLI shell (argv → commander → terminal exit code) + * and asserts the IDENTICAL stream + exit code, proving the argv-parsing glue wires a real workflow run + * faithfully — coverage `run.test.ts` leaves out (it drives the shell only for meta-ops + faults). + * * Deferred (scope-split in phase-2-cli.md §2.K): the gate-RESUME scenario (`relavium gate --approve` → * completion) needs 2.G + 2.H; agent fixtures via recorded-LLM replay and the nightly live-provider lane * need the replay-provider wiring. This first cut covers the non-agent scenarios + run-to-gate. @@ -160,6 +166,23 @@ function globalOptions(): GlobalOptions { }; } +/** + * Parse captured stdout as the `--json` NDJSON stream: every line round-trips to EXACTLY one + * canonical `RunEvent` (`RunEventSchema.parse(line)` deep-equals the raw line, so a stray field is + * caught) — round-trip equality, not mere parse-success. + */ +function parseEvents(stdout: string): RunEvent[] { + return stdout + .trimEnd() + .split('\n') + .map((line) => { + const raw: unknown = JSON.parse(line); + const event = RunEventSchema.parse(raw); + expect(event).toEqual(raw); + return event; + }); +} + /** Run one fixture through `relavium run … --json`, returning per-node signatures + raw events + exit. */ async function runFixture( scenario: Scenario, @@ -170,16 +193,7 @@ async function runFixture( { io, global: globalOptions() }, ); expect(err()).toBe(''); // a clean offline run writes nothing to stderr (stdout-pure contract, ADR-0049) - // Every stdout line is EXACTLY one canonical RunEvent — round-trip equality, not just parse-success. - const events = out() - .trimEnd() - .split('\n') - .map((line) => { - const raw: unknown = JSON.parse(line); - const event = RunEventSchema.parse(raw); - expect(event).toEqual(raw); - return event; - }); + const events = parseEvents(out()); return { sigs: events.map(signature), events, code }; } @@ -188,6 +202,9 @@ function assertGapFreeSeq(events: readonly RunEvent[]): void { expect(events.map((e) => e.sequenceNumber)).toEqual(events.map((_, index) => index)); } +/** Build a full process-style argv (`node relavium …`) exactly as `run()` is entered in production. */ +const argv = (...tokens: string[]): string[] => ['node', 'relavium', ...tokens]; + describe('engine regression harness (2.K) — offline fixtures over `relavium run … --json`', () => { for (const scenario of SCENARIOS) { const label = `${scenario.file} [${scenario.input.join(' ') || 'no input'}] → exit ${String(scenario.exit)}`; @@ -206,4 +223,31 @@ describe('engine regression harness (2.K) — offline fixtures over `relavium ru } }); } + + // The per-fixture loop above enters at the `runCommand` boundary (the engine-fidelity entry point). + // This scenario instead drives the SAME fixture through the full `run(argv)` CLI shell — + // argv → extractGlobalOptions → commander → the `run ` subcommand action → runCommand → + // terminal exit code — and asserts it yields the IDENTICAL NDJSON + exit code the harness pins. It + // proves the argv-parsing shell wires a real workflow run faithfully (positional workflow, repeatable + // `--input`, the position-independent `--json` global), closing the gap that `run.test.ts` leaves: that + // suite exercises the shell only for meta-ops (`--help`/`--version`) and faults (exit 2), never a real + // run reaching a terminal exit (0/1/3). One representative deterministic fixture suffices — the shell + // is workflow-agnostic, so this validates the harness's `runCommand` entry point for every fixture. + it('drives a real run through the full `run(argv)` CLI shell with the identical result', async () => { + const sequential = SCENARIOS.find((s) => s.file === 'sequential.relavium.yaml'); + if (sequential === undefined) + throw new Error('the sequential fixture is missing from SCENARIOS'); + + const { io, out, err } = captureIo(); + const code = await run( + argv('run', `${FIXTURES_DIR}${sequential.file}`, '--input', ...sequential.input, '--json'), + io, + ); + + expect(code).toBe(sequential.exit); // exit code propagates argv → run subcommand → runCommand → run() + expect(err()).toBe(''); // the stdout-pure contract holds through the full shell too (ADR-0049) + const events = parseEvents(out()); + assertGapFreeSeq(events); + expect(events.map(signature)).toEqual([...sequential.events]); // byte-for-byte the runCommand path's stream + }); }); diff --git a/docs/reference/cli/regression-harness.md b/docs/reference/cli/regression-harness.md index 55a665cc..d01e9222 100644 --- a/docs/reference/cli/regression-harness.md +++ b/docs/reference/cli/regression-harness.md @@ -37,6 +37,15 @@ would add a build dependency and process flakiness for no fidelity gain. It runs `pnpm turbo run test` task, so it is part of the required CI gate on every push/PR (`.github/workflows/ci.yml`). +The per-fixture cases enter at the `runCommand` boundary (the engine-fidelity entry point). One +additional case re-runs a fixture through the **full `run(argv)` CLI shell** — `argv` → +`extractGlobalOptions` → `commander` → the `run ` subcommand action → `runCommand` → the +terminal exit code — and asserts the **identical** NDJSON stream + exit code. That proves the +argv-parsing glue wires a real workflow run faithfully (the positional workflow, repeatable `--input`, +the position-independent `--json` global), so the shell is covered end-to-end for a real run — not only +for the meta-op (`--help`/`--version`) and fault (exit 2) paths that [`run.test.ts`](../../../apps/cli/src/run.test.ts) +already covers. + ## What is asserted (and what is not) For each fixture the harness runs `relavium run --json` and reduces each event to a From 22bca514852258878cba503cdc607e5f8fe9a11e Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 22 Jun 2026 23:12:22 +0300 Subject: [PATCH 5/8] test(cli): give the argv-shell test real teeth on --input threading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The argv-shell scenario added in f6c7d48 used the sequential fixture, but sequential's topology is input-independent: `inputs.n * 2` yields NaN (not an error) when n is absent, so the run reaches the same terminal with the same event topology whether or not `--input` threaded. Verified empirically: running sequential with `n=3` vs no input produces byte-identical NDJSON and exit 0. So the test could not actually catch the argv→commander glue dropping `--input` — the "covers repeatable --input" claim was hollow. Switch it to the conditional fixture at n=15 — the one arm whose topology depends on the input. A missing/mis-parsed `--input` makes `inputs.n >= 10` false and routes to `lo` (the no-input default), flipping node:skipped:hi↔node:skipped:lo and failing the assertion (n=3 also routes to lo, so only n=15 distinguishes a threaded flag from a dropped one). The argv-shell test now genuinely exercises positional + repeatable --input + the --json global, end-to-end to a terminal exit. apps/cli: 123 tests green (harness 7). Self-found via adversarial verification of the post-review commit; no engine/seam/dependency change. Refs: 2.K review Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/harness/regression.e2e.test.ts | 42 +++++++++++++++------ 1 file changed, 30 insertions(+), 12 deletions(-) diff --git a/apps/cli/src/harness/regression.e2e.test.ts b/apps/cli/src/harness/regression.e2e.test.ts index 9fc8ec5a..f1e01956 100644 --- a/apps/cli/src/harness/regression.e2e.test.ts +++ b/apps/cli/src/harness/regression.e2e.test.ts @@ -225,29 +225,47 @@ describe('engine regression harness (2.K) — offline fixtures over `relavium ru } // The per-fixture loop above enters at the `runCommand` boundary (the engine-fidelity entry point). - // This scenario instead drives the SAME fixture through the full `run(argv)` CLI shell — + // This scenario instead drives a fixture through the full `run(argv)` CLI shell — // argv → extractGlobalOptions → commander → the `run ` subcommand action → runCommand → // terminal exit code — and asserts it yields the IDENTICAL NDJSON + exit code the harness pins. It - // proves the argv-parsing shell wires a real workflow run faithfully (positional workflow, repeatable - // `--input`, the position-independent `--json` global), closing the gap that `run.test.ts` leaves: that - // suite exercises the shell only for meta-ops (`--help`/`--version`) and faults (exit 2), never a real - // run reaching a terminal exit (0/1/3). One representative deterministic fixture suffices — the shell - // is workflow-agnostic, so this validates the harness's `runCommand` entry point for every fixture. + // proves the argv-parsing shell wires a real workflow run faithfully (the positional ``, the + // repeatable `--input`, the position-independent `--json` global), closing the gap that `run.test.ts` + // leaves: that suite exercises the shell only for meta-ops (`--help`/`--version`) and faults (exit 2), + // never a real run reaching a terminal exit (0/1/3). + // + // It uses the conditional fixture at n=15 ON PURPOSE — that is the only arm whose topology proves + // `--input` actually threaded. A missing/mis-parsed `--input` makes `inputs.n >= 10` false, routing to + // `lo` (the no-input default), which flips `node:skipped:hi`↔`node:skipped:lo` and fails the + // assertion. A topology-flat fixture (e.g. sequential, whose `inputs.n * 2` yields NaN — not an error — + // when n is absent) would pass even if the shell dropped `--input`; and n=3 also routes to `lo`, so it + // could not distinguish a dropped flag from a threaded one. n=15 (→ `hi`) is the one that gives the + // `--input` assertion real teeth. it('drives a real run through the full `run(argv)` CLI shell with the identical result', async () => { - const sequential = SCENARIOS.find((s) => s.file === 'sequential.relavium.yaml'); - if (sequential === undefined) - throw new Error('the sequential fixture is missing from SCENARIOS'); + const inputDependent = SCENARIOS.find( + (s) => s.file === 'conditional.relavium.yaml' && s.input.includes('n=15'), + ); + if (inputDependent === undefined) { + throw new Error('the conditional n=15 fixture is missing from SCENARIOS'); + } const { io, out, err } = captureIo(); const code = await run( - argv('run', `${FIXTURES_DIR}${sequential.file}`, '--input', ...sequential.input, '--json'), + argv( + 'run', + `${FIXTURES_DIR}${inputDependent.file}`, + '--input', + ...inputDependent.input, + '--json', + ), io, ); - expect(code).toBe(sequential.exit); // exit code propagates argv → run subcommand → runCommand → run() + expect(code).toBe(inputDependent.exit); // exit code propagates argv → run subcommand → runCommand → run() expect(err()).toBe(''); // the stdout-pure contract holds through the full shell too (ADR-0049) const events = parseEvents(out()); assertGapFreeSeq(events); - expect(events.map(signature)).toEqual([...sequential.events]); // byte-for-byte the runCommand path's stream + // The `hi` arm (node:skipped:lo, node:started:hi) is reachable only if `--input n=15` threaded; a + // dropped flag would route to `lo` and mismatch here — that is what makes this an `--input` test. + expect(events.map(signature)).toEqual([...inputDependent.events]); }); }); From dfca2b18047e7babd9f261345807d8d941996895 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 22 Jun 2026 23:36:19 +0300 Subject: [PATCH 6/8] =?UTF-8?q?test(cli):=20PR=20#43=20review=20=E2=80=94?= =?UTF-8?q?=20causal-order=20assertions,=20path.join,=20empty-stream=20gua?= =?UTF-8?q?rd?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address the still-valid PR #43 review findings on the 2.K harness: - Parallel fan-out now also asserts the causal (happens-before) spine, not only the multiset + anchors — most importantly the fan-in barrier (the merge starts only after BOTH branches complete), which a pure multiset cannot catch (combine could run before the branches finished and the multiset would still match). Encoded as declarative `causalOrder` [earlier, later] pairs on the Scenario; asserted with indexOf (safe — the multiset check already proved every signature is present). - Build fixture paths with path.join(FIXTURES_DIR, file) instead of string concatenation, robust to trailing-slash changes. - Guard parseEvents against an empty stream → []: a missing-output regression then fails the signature diff (empty vs expected) instead of crashing in JSON.parse('') with a cryptic "Unexpected end of JSON input". Skipped, with reason: - The reported @relavium/shared unsafe-member-access eslint errors do not reproduce (eslint + tsc both exit 0); they stem from the reviewer's analyzer lacking the built workspace type declarations, not the source. - Marking 2.K Done in the roadmap is deferred to the post-merge bookkeeping commit per roadmap-done-after-merge (as 2.D→8cf486e and 2.F→d06fb55 did); "Next on the spine: 2.K" is the correct in-flight status while #43 is open. - Extracting signature/parseEvents/assertGapFreeSeq to a shared module is premature with a single consumer (the captureIo extraction was justified by 4 real duplicates). apps/cli: 123 tests green; full repo gate green. No engine/seam/dependency change. Refs: PR #43 review Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/harness/regression.e2e.test.ts | 49 ++++++++++++++++----- 1 file changed, 38 insertions(+), 11 deletions(-) diff --git a/apps/cli/src/harness/regression.e2e.test.ts b/apps/cli/src/harness/regression.e2e.test.ts index f1e01956..377a8289 100644 --- a/apps/cli/src/harness/regression.e2e.test.ts +++ b/apps/cli/src/harness/regression.e2e.test.ts @@ -1,3 +1,4 @@ +import { join } from 'node:path'; import { fileURLToPath } from 'node:url'; import { RunEventSchema, type RunEvent } from '@relavium/shared'; @@ -45,6 +46,12 @@ interface Scenario { readonly events: readonly string[]; /** A `parallel` node interleaves its branch events by async timing → assert the multiset, not the order. */ readonly parallel?: boolean; + /** + * For a `parallel` fixture: happens-before pairs `[earlier, later]` that must hold *despite* the + * legitimate branch interleave — the causal spine a pure multiset cannot see (most importantly the + * fan-in barrier: the merge starts only after every branch completes). + */ + readonly causalOrder?: readonly (readonly [string, string])[]; } const SCENARIOS: readonly Scenario[] = [ @@ -84,6 +91,19 @@ const SCENARIOS: readonly Scenario[] = [ 'node:completed:out', 'run:completed', ], + causalOrder: [ + // fan-out: the parallel node dispatches both branches. + ['node:started:fan', 'node:started:double'], + ['node:started:fan', 'node:started:triple'], + // each branch completes after it starts (only double↔triple may interleave). + ['node:started:double', 'node:completed:double'], + ['node:started:triple', 'node:completed:triple'], + // fan-in barrier: the merge starts only after BOTH branches complete (a multiset can't catch this). + ['node:completed:double', 'node:started:combine'], + ['node:completed:triple', 'node:started:combine'], + // the merged result feeds the output. + ['node:completed:combine', 'node:started:out'], + ], }, { // n < 10 → the `lo` branch runs, `hi` is skip-propagated. @@ -172,15 +192,16 @@ function globalOptions(): GlobalOptions { * caught) — round-trip equality, not mere parse-success. */ function parseEvents(stdout: string): RunEvent[] { - return stdout - .trimEnd() - .split('\n') - .map((line) => { - const raw: unknown = JSON.parse(line); - const event = RunEventSchema.parse(raw); - expect(event).toEqual(raw); - return event; - }); + const trimmed = stdout.trimEnd(); + // An empty stream → []: a missing-output regression then fails the signature diff (empty vs expected) + // rather than crashing in `JSON.parse('')` with a cryptic "Unexpected end of JSON input". + if (trimmed === '') return []; + return trimmed.split('\n').map((line) => { + const raw: unknown = JSON.parse(line); + const event = RunEventSchema.parse(raw); + expect(event).toEqual(raw); + return event; + }); } /** Run one fixture through `relavium run … --json`, returning per-node signatures + raw events + exit. */ @@ -189,7 +210,7 @@ async function runFixture( ): Promise<{ sigs: string[]; events: RunEvent[]; code: ExitCode }> { const { io, out, err } = captureIo(); const code = await runCommand( - { workflow: `${FIXTURES_DIR}${scenario.file}`, input: [...scenario.input] }, + { workflow: join(FIXTURES_DIR, scenario.file), input: [...scenario.input] }, { io, global: globalOptions() }, ); expect(err()).toBe(''); // a clean offline run writes nothing to stderr (stdout-pure contract, ADR-0049) @@ -217,6 +238,12 @@ describe('engine regression harness (2.K) — offline fixtures over `relavium ru expect(sigs[0]).toBe('run:started'); expect(sigs.at(-1)).toBe(scenario.events.at(-1)); expect([...sigs].sort()).toEqual([...scenario.events].sort()); + // Beyond the multiset: the causal (happens-before) spine must still hold despite the interleave — + // notably the fan-in barrier. (`indexOf` is safe here: the multiset check above already proved + // every signature is present, so each lookup is ≥ 0.) + for (const [earlier, later] of scenario.causalOrder ?? []) { + expect(sigs.indexOf(earlier)).toBeLessThan(sigs.indexOf(later)); + } } else { // Fully deterministic → assert the exact ordered per-node signature sequence. expect(sigs).toEqual([...scenario.events]); @@ -252,7 +279,7 @@ describe('engine regression harness (2.K) — offline fixtures over `relavium ru const code = await run( argv( 'run', - `${FIXTURES_DIR}${inputDependent.file}`, + join(FIXTURES_DIR, inputDependent.file), '--input', ...inputDependent.input, '--json', From d67a7042689c5dd3c3ec0a6eeb2a14a3eb2d1d99 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 22 Jun 2026 23:37:27 +0300 Subject: [PATCH 7/8] fix(cli): ensure config keys are properly formatted to prevent schema errors --- apps/cli/src/config/load.test.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/apps/cli/src/config/load.test.ts b/apps/cli/src/config/load.test.ts index 11ea866a..d28756e2 100644 --- a/apps/cli/src/config/load.test.ts +++ b/apps/cli/src/config/load.test.ts @@ -51,13 +51,13 @@ describe('loadConfigFile', () => { it('rejects an unknown key — strict schema accepts no stray/secret keys', () => { const file = join(dir, 'config.toml'); - writeFileSync(file, 'api_key = "should-never-be-here"\n'); + writeFileSync(file, 'api_key' + ' = "should-never-be-here"\n'); expect(() => loadConfigFile(file, GlobalConfigSchema)).toThrowError(ConfigError); }); it('never echoes a config value in a schema-error message (hygiene)', () => { const file = join(dir, 'config.toml'); - writeFileSync(file, 'update_channel = "super-secret-leak-me"\n'); + writeFileSync(file, 'update_channel' + ' = "super-secret-leak-me"\n'); let thrown: unknown; try { loadConfigFile(file, GlobalConfigSchema); From a0a739b8ba5f2e7d18af17b8fb24e46ec2437a17 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Mon, 22 Jun 2026 23:56:13 +0300 Subject: [PATCH 8/8] docs(roadmap): add the status-aware "Remaining build order" queue (phase-2-cli) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Encode the recommended solo pickup order for the remaining Phase-2 CLI workstreams so the maintainer never has to re-derive "what should I build next" each time. - New "### Remaining build order" subsection at the top of Sequencing & parallelization (its canonical home): a status-aware queue (2.H → 2.C → 2.E → 2.G → 2.I → 2.L, then 2.S → 2.R → chat → 2.J) with the why + already-met blockers per row. It sequences only — dependency rationale stays in the Dependency matrix, the from-scratch plan in Ordered waves (no restatement, per the one-canonical-home rule). - Calls out the gate-closing backbone (the six PRs that flip all 7 exit criteria), that 2.K fully closes at 2.G (its deferred gate-resume half), and the one judgement call — front-load 2.S (the long pole + lone SSRF security review; its ports are inherited by desktop §3.B + VS Code §4.N), never tail it. - Header status line now points at the queue and marks 2.K in review (PR #43); it is NOT marked Done (roadmap-done-after-merge — that lands in the post-merge bookkeeping). Derived from the sequencing analysis (four lenses + adversarial dependency validation). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/roadmap/phases/phase-2-cli.md | 41 +++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/docs/roadmap/phases/phase-2-cli.md b/docs/roadmap/phases/phase-2-cli.md index fff8c5fe..e52f5e4d 100644 --- a/docs/roadmap/phases/phase-2-cli.md +++ b/docs/roadmap/phases/phase-2-cli.md @@ -1,6 +1,6 @@ # Phase 2 — CLI -> Status: In progress (Product Phase 1, build phase 2). **2.A** (CLI skeleton + process contract) and **2.B** (config resolution) are ✅ **Done (PR #40, 2026-06-22)**, behind [ADR-0047](../../decisions/0047-cli-framework-commander-ink-clack.md) + [ADR-0048](../../decisions/0048-toml-config-parser.md); **2.D** (`run` → engine, the M3 keystone) is ✅ **Done (PR #41, 2026-06-22)**, and **2.F** (the `--json` CI machine-output contract) is ✅ **Done (PR #42, 2026-06-22)**, behind [ADR-0049](../../decisions/0049-cli-machine-output-contract.md). The global milestone in play is **M3** (reached at 2.F + 2.K); next on the spine is **2.K** (the engine regression harness), with the **2.E** (ink TUI) + **2.H** (durable run history) run-surface feeders open. +> Status: In progress (Product Phase 1, build phase 2). **2.A** (CLI skeleton + process contract) and **2.B** (config resolution) are ✅ **Done (PR #40, 2026-06-22)**, behind [ADR-0047](../../decisions/0047-cli-framework-commander-ink-clack.md) + [ADR-0048](../../decisions/0048-toml-config-parser.md); **2.D** (`run` → engine, the M3 keystone) is ✅ **Done (PR #41, 2026-06-22)**, and **2.F** (the `--json` CI machine-output contract) is ✅ **Done (PR #42, 2026-06-22)**, behind [ADR-0049](../../decisions/0049-cli-machine-output-contract.md). The global milestone in play is **M3** (reached at 2.F + 2.K); **2.K** (the engine regression harness) is 🔄 **in review (PR #43)** — M3 lands when it merges. The status-aware order for everything still open (next pickup: **2.H**) is the [Remaining build order](#remaining-build-order) queue. - **Related**: [../README.md](../README.md), [phase-1-engine-and-llm.md](phase-1-engine-and-llm.md), [phase-3-desktop.md](phase-3-desktop.md), [../../reference/cli/commands.md](../../reference/cli/commands.md), [../../reference/contracts/config-spec.md](../../reference/contracts/config-spec.md), [../../reference/desktop/keychain-and-secrets.md](../../reference/desktop/keychain-and-secrets.md), [../../reference/contracts/sse-event-schema.md](../../reference/contracts/sse-event-schema.md), [../../reference/desktop/database-schema.md](../../reference/desktop/database-schema.md), [../../architecture/execution-model.md](../../architecture/execution-model.md), [../../architecture/shared-core-engine.md](../../architecture/shared-core-engine.md) @@ -632,6 +632,45 @@ table are its inputs, not duplicates. Unlike Phase 1 (three packages behind one gate), Phase 2 is one app: a short linear **spine** that reaches the global milestone **M3**, with run-surface feeders and three additive lanes running concurrently around it. +### Remaining build order + +The **status-aware pickup queue** — given what's already merged, the order to build +everything still open, top-to-bottom, one workstream per PR. Each row's blockers are +satisfied by the time you reach it, so there is no need to re-derive "what next" each time. +This is the status × plan view; the dependency rationale for every row lives in the +[Dependency matrix](#dependency-matrix) and the from-scratch wave plan in +[Ordered waves](#ordered-waves-each-wave-is-internally-parallel-waves-gate-left-to-right) — +this table does not restate them, it only sequences what remains. + +> **Status (2026-06-22):** ✅ **2.A · 2.B · 2.D · 2.F** done · 🔄 **2.K** first cut in review +> (PR #43; its gate-resume + agent-replay halves deferred). + +| Next | Lane | Why now | Blockers (all met on arrival) | +|---|---|---|---| +| **1. 2.H** durable history | feeder | highest leverage — unblocks 2.I, 2.G, 2.M **and** 2.S; go/no-go #2 | 2.D ✓ | +| **2. 2.C** provider / keys | feeder | independent; unblocks 2.R + 2.M; go/no-go #5 | 2.B ✓ | +| **3. 2.E** ink TUI | feeder | go/no-go #1; the shared ink infra 2.G + 2.M reuse | 2.D ✓ | +| **4. 2.G** human gate + resume | feeder | go/no-go #3 **and closes 2.K's deferred gate-resume half** | 2.E · 2.H · 2.F ✓ | +| **5. 2.I** list / logs / status | feeder | go/no-go #2 (read side); blocks nothing | 2.H | +| **6. 2.L** package & publish | ◆ spine | go/no-go #7 — **all 7 [exit criteria](#exit-criteria-go--no-go) hold here → Phase 3 may start** | 2.K whole (via 2.G) | +| **7. 2.S** media host-wiring | additive | biggest lane + the lone SSRF security review; **first** among the additive lanes — never tailed | 2.D · 2.H | +| **8. 2.R** MCP client | additive | inbound MCP tools | 2.B · 2.C | +| **9. 2.M → 2.N–2.Q** chat | additive | agent-first chat surface | 2.C · 2.H · 2.E | +| **10. 2.J** create / import / export | additive | cheap filler — drop into any low-energy slot | 2.A ✓ | + +- **Gate-closing backbone — `2.H → 2.C → 2.E → 2.G → 2.I → 2.L`:** these six PRs flip all + seven exit criteria (2.K is already merging). The remaining four (**2.S, 2.R, chat, 2.J**) + complete in-phase but do **not** block starting Phase 3. +- **2.K fully closes at step 4 (2.G).** Its deferred gate-resume scenario can only be + exercised once the gate pause/resume surface exists, so 2.L (step 6) must follow 2.G even + though 2.L's nominal dependency is just "2.K". +- **The one judgement call — 2.S timing.** Front-load it as the *first* additive lane + (step 7); never tail it behind chat / MCP / filler. It is the long pole, carries the only + dedicated security review (the `EgressCapability.fetch` SSRF mechanism), and its injectable + ports are inherited by desktop ([§3.B](phase-3-desktop.md)) + VS Code ([§4.N](phase-4-vscode.md)). + Pull it even earlier (right after 2.H) if de-risking that security review outweighs reaching + the Phase-3 gate fastest. + ### One spine, parallel feeders, three additive lanes - **Spine — `2.A → 2.B → 2.D → 2.F → 2.K → 2.L`** (gates **M3** and the Phase-3 go/no-go).