From c4ecd51d3ac9b16cd72d0de58bef34670525485b Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 18:29:15 +0300 Subject: [PATCH 01/50] docs(roadmap): cite PR #67 on the 2.5.H status (merged to main via #68) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 2.5.H landed on main (PR #68, development→main). Add the workstream PR reference (#67 — the 2.5.H feature PR to development, matching how 2.5.A–G cite their feature PR; #68 is the dev→main release mechanism, like #66) to the 2.5.H status surfaces: phase-2.5-cli-consolidation.md (top status + the 2.5.H block header + Status line), current.md, and CLAUDE.md. Date-only; no other change. format:check + lint:fence-check clean. Refs: phase-2.5-cli-consolidation.md (2.5.H) Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 2 +- docs/roadmap/current.md | 2 +- docs/roadmap/phases/phase-2.5-cli-consolidation.md | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index cf9ac86a..9f992bb2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -114,7 +114,7 @@ config-write, [ADR-0064](docs/decisions/0064-live-model-catalog.md) live catalog [ADR-0065](docs/decisions/0065-provider-economics-and-extensibility.md) provider economics), across 12 reviewed steps. The additive lane **2.5.H** (reasoning render + live-turn feedback + an actionable error taxonomy — behind **EA6**, a dual-envelope `agent:reasoning` stream event that *amends* [ADR-0036](docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md); -no new top-level ADR) is ✅ **Done (2026-07-07)**, reaching milestone **M2.5-3** with 2.5.E; the remaining additive +no new top-level ADR) is ✅ **Done (PR #67, 2026-07-07)**, reaching milestone **M2.5-3** with 2.5.E; the remaining additive lanes 2.5.I / J run in parallel. For live status, per-PR history, milestone dates, and open obligations, see the canonical home [docs/roadmap/current.md](docs/roadmap/current.md); [README.md](README.md) is the public overview. diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index a9019454..30b152c7 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -193,7 +193,7 @@ cost) — behind three new ADRs ([ADR-0063](../decisions/0063-cli-config-write-c [ADR-0064](../decisions/0064-live-model-catalog.md) live catalog · [ADR-0065](../decisions/0065-provider-economics-and-extensibility.md) provider economics), across 12 reviewed steps. The additive lane **2.5.H** (reasoning render + live-turn feedback + an actionable error taxonomy) is ✅ -**Done (2026-07-07)** — behind **EA6** (a new dual-envelope `agent:reasoning` stream event that *amends* +**Done (PR #67, 2026-07-07)** — behind **EA6** (a new dual-envelope `agent:reasoning` stream event that *amends* [ADR-0036](../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md); no new top-level ADR): a host-emit of the reasoning the `@relavium/llm` seam already carries (ADR-0030), a collapsible "thinking" panel (`/thinking` / `Ctrl+T`), the `Thinking…/Working… {elapsed}s · Esc to stop` timer, a visible `…` elision marker diff --git a/docs/roadmap/phases/phase-2.5-cli-consolidation.md b/docs/roadmap/phases/phase-2.5-cli-consolidation.md index e551bf9f..e6113e3e 100644 --- a/docs/roadmap/phases/phase-2.5-cli-consolidation.md +++ b/docs/roadmap/phases/phase-2.5-cli-consolidation.md @@ -9,7 +9,7 @@ > (PR #64, 2026-07-03)** behind [ADR-0061](../../decisions/0061-cli-input-layer-file-injection-and-shell-escape.md). > **2.5.F** (`/clear` + the `session:compacting` "Summarizing…" moment + the context-fullness footer, completing > the ADR-0062 compaction story) ✅ **Done (PR #65, merged 2026-07-05)**. The additive lane **2.5.H** (reasoning -> render + live-turn feedback + actionable errors, EA6 amending ADR-0036) ✅ **Done (2026-07-07)** — **milestone +> render + live-turn feedback + actionable errors, EA6 amending ADR-0036) ✅ **Done (PR #67, 2026-07-07)** — **milestone > M2.5-3 reached** (with 2.5.E). **Next: 2.5.G** (onboarding wizard + Home `/models`). Remaining additive lanes: > 2.5.I / J. @@ -451,9 +451,9 @@ auto-refresh + partial-failure · **S6 🔒** config-write primitive + global `[ **S11** `provider list --verify` + `[chat].max_turns` reconcile · **S12** docs + adding-a-provider runbook. **Six** are security-flagged: S2, S5, S6, S8, S9, S10. -### 2.5.H — Reasoning rendering and live-turn feedback — ✅ **Done (2026-07-07)** +### 2.5.H — Reasoning rendering and live-turn feedback — ✅ **Done (PR #67, 2026-07-07)** -> **Status:** ✅ **Done (2026-07-07)** — EA6 amends [ADR-0036](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md) +> **Status:** ✅ **Done (PR #67, 2026-07-07)** — EA6 amends [ADR-0036](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md) > (no new top-level ADR; the additive event in the shared union). Delivered across four reviewed steps, each with an > **opus + Sonnet** review round (~15 findings fixed incl. 3 HIGH: a run-path silent-drop of the new event, a > frozen Home live-timer, a persisted-scrollback elision loss; plus a one-shot `agent run` hint-leak). **Step 1** From b4d96195c5ad54864ba373e49ca35dec1b306770 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 19:38:39 +0300 Subject: [PATCH 02/50] =?UTF-8?q?docs(roadmap):=202.5-close=20Step=201=20?= =?UTF-8?q?=E2=80=94=20reconcile=202.5.G=20Done=20+=20phase-2.5=20internal?= =?UTF-8?q?=20fixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Mark 2.5.G done (PR #66) across the phase-2.5 doc, current.md, and CLAUDE.md; milestone M2.5-2 (Home + entry + onboarding) reached. - Fix phase-2.5 internal inconsistencies: EA1-EA7 -> EA1-EA8, the broken #engine-amendments-appendix-ea1ea7 anchor -> -ea1ea8, and add EA8 -> ADR-0061 to the amendment->ADR mapping. - deferred-tasks: check off [chat].max_turns (RESOLVED in 2.5.G S11, PR #66); reword the CLI ToolHost entry to reflect the fs/process (2.5.A) + egress/os (2.5.E) arms are wired behind the approval floor -- only the desktop surface remains. Refs: docs/roadmap/phase-2.5-close-plan.md Step 1 Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 11 ++++--- docs/roadmap/current.md | 11 +++---- docs/roadmap/deferred-tasks.md | 28 ++++++++--------- .../phases/phase-2.5-cli-consolidation.md | 30 ++++++++++++------- 4 files changed, 46 insertions(+), 34 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 9f992bb2..87781915 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -108,11 +108,14 @@ under a new `sessionId` (TTY-interactive only, rejected under `--json`/plain per compaction-moment UX polishes (a `session:compacting` "Summarizing…" event amending ADR-0036, and the footer context-fullness indicator via a pure `@relavium/llm` `contextWindowForModel` helper), completing the ADR-0062 compaction story alongside the earlier-landed model-summarised `/compact` + deterministic `/trim` + automatic -compaction. **2.5.G is now underway** — its scope expanded to **Option A** (a **live** model catalog + a complete -model-pricing story that governs cost) behind three new ADRs ([ADR-0063](docs/decisions/0063-cli-config-write-contract.md) +compaction. **2.5.G** (onboarding wizard + Home `/models` + the live model catalog) is ✅ **Done (PR #66, +2026-07-06)** — its scope expanded to **Option A** (a **live** model catalog + a complete model-pricing story that +governs cost) behind three ADRs ([ADR-0063](docs/decisions/0063-cli-config-write-contract.md) config-write, [ADR-0064](docs/decisions/0064-live-model-catalog.md) live catalog, -[ADR-0065](docs/decisions/0065-provider-economics-and-extensibility.md) provider economics), across 12 reviewed -steps. The additive lane **2.5.H** (reasoning render + live-turn feedback + an actionable error taxonomy — behind +[ADR-0065](docs/decisions/0065-provider-economics-and-extensibility.md) provider economics); all 12 steps landed, +plus the model-UX follow-up ([ADR-0059](docs/decisions/0059-cli-mid-session-model-reseat.md) mid-session reseat + +[ADR-0066](docs/decisions/0066-normalized-reasoning-effort-control.md) reasoning-effort). With it **milestone +M2.5-2** is reached. The additive lane **2.5.H** (reasoning render + live-turn feedback + an actionable error taxonomy — behind **EA6**, a dual-envelope `agent:reasoning` stream event that *amends* [ADR-0036](docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md); no new top-level ADR) is ✅ **Done (PR #67, 2026-07-07)**, reaching milestone **M2.5-3** with 2.5.E; the remaining additive lanes 2.5.I / J run in parallel. diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 30b152c7..7f330bf2 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -186,13 +186,14 @@ all recorded in the ADR-0061 "Refined at implementation" append. **2.5.F** (the **Done (PR #65, merged 2026-07-05)** behind [ADR-0062](../decisions/0062-context-compaction-and-cli-history-commands.md): `/clear` (a host-level fresh-session swap across `chat`/`chat-resume`/Home, TTY-interactive only), the `session:compacting` "Summarizing…" moment event (amending ADR-0036), and the footer context-fullness indicator — -completing compaction alongside the earlier `/compact` + `/trim` + automatic compaction. **2.5.G is now -underway**, its scope **expanded to Option A** — a **live** model catalog (per-key provider discovery + a DB -cache + refresh + a static/live merge) and a complete model-pricing story (user-supplied pricing that governs -cost) — behind three new ADRs ([ADR-0063](../decisions/0063-cli-config-write-contract.md) config-write · +completing compaction alongside the earlier `/compact` + `/trim` + automatic compaction. **2.5.G** (onboarding +wizard + Home `/models` + the live model catalog) is ✅ **Done (PR #66, 2026-07-06)** — its scope **expanded to +Option A**: a **live** model catalog (per-key provider discovery + a DB cache + refresh + a static/live merge) +and a complete model-pricing story (user-supplied pricing that governs cost), behind three ADRs +([ADR-0063](../decisions/0063-cli-config-write-contract.md) config-write · [ADR-0064](../decisions/0064-live-model-catalog.md) live catalog · [ADR-0065](../decisions/0065-provider-economics-and-extensibility.md) provider economics), across 12 reviewed -steps. The additive lane **2.5.H** (reasoning render + live-turn feedback + an actionable error taxonomy) is ✅ +steps (all landed); with it **milestone M2.5-2** is reached. The additive lane **2.5.H** (reasoning render + live-turn feedback + an actionable error taxonomy) is ✅ **Done (PR #67, 2026-07-07)** — behind **EA6** (a new dual-envelope `agent:reasoning` stream event that *amends* [ADR-0036](../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md); no new top-level ADR): a host-emit of the reasoning the `@relavium/llm` seam already carries (ADR-0030), a collapsible "thinking" panel diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index a3e2a7ee..f081f06c 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -487,10 +487,10 @@ Severity is the review's verified rating. Check an item off in the PR that resol session cannot yet **narrow** them per-session (it may only ever narrow, never widen). Add a session-level narrow when a surface needs to restrict a session's tools below the agent's grant. *(low · packages/core/src/engine/agent-session.ts; ADR-0029)* -- [ ] **`[chat].max_turns` surface wiring.** The hard turn cap is an **engine-API** knob in 1.V - (`SessionDeps.maxTurns`, finite default 50); mapping the `[chat]` config default onto it is a surface task - (the CLI/desktop read `[chat]` and pass `maxTurns`). It is deliberately **not** a Phase-1 `[chat]` field. - *(low · config-spec.md + surfaces; Phase 2+)* +- [x] **`[chat].max_turns` surface wiring — RESOLVED in 2.5.G S11 (PR #66).** The hard turn cap is an + **engine-API** knob in 1.V (`SessionDeps.maxTurns`, finite default 50); the CLI now maps the `[chat].max_turns` + config default onto `SessionDeps.maxTurns` (`config/resolve.ts` `max_turns`→`maxTurns`, threaded through + `session-host.ts`, enforced by a `session-host.test.ts` pin). *(config-spec.md + surfaces)* - [ ] **Session `output_schema`.** 1.V ignores `agent.output_schema` (a chat session is free-form text); structured output stays a workflow concern. If a session ever needs it, lower it to `responseFormat` + validate node-side (as the AgentRunner does for an `agent` node). *(low · packages/core/src/engine/agent-session.ts)* @@ -589,16 +589,16 @@ Severity is the review's verified rating. Check an item off in the PR that resol > status* note in [../reference/cli/commands.md](../reference/cli/commands.md), so they are **not** > duplicated here. The one item below is an unscheduled security follow-up with no numbered workstream yet. -- [ ] **CLI `ToolHost` is fail-closed — built-in tool host capabilities (filesystem / process / egress) - are not wired.** 2.D builds the engine's tool registry with an empty `ToolHost` (`createToolRegistry({ - tools: BUILTIN_TOOLS, host: {} })`), so every built-in tool that needs a host capability is cleanly - *unavailable* rather than backed by an insecure stub. Wiring these capabilities is deliberately deferred - to a dedicated, **security-reviewed** workstream: the egress half is the existing host-side SSRF item - above (DNS-resolve + connect-by-validated-IP + per-hop redirect re-validation in `EgressCapability.fetch`), - and the filesystem/process halves need their own scope/permission model ([security-review.md](../standards/security-review.md)). - Until then, a workflow whose agent calls a capability-backed built-in tool surfaces a clean "tool - unavailable" failure, never a half-implemented or unsafe execution. *(medium · apps/cli/src/engine/build-engine.ts; - security-review.md; egress → the SSRF item above)* +- [ ] **CLI `ToolHost` — the built-in host capabilities are wired for the CLI; the DESKTOP surface remains.** + 2.D originally built the tool registry with an empty `ToolHost` (`createToolRegistry({ tools: BUILTIN_TOOLS, + host: {} })`). For the CLI this is now **closed**: 2.5.A wired the `fs` + `process` arms and 2.5.E wired the + `egress` + `os` arms via `assembleToolEnv`, all behind the fail-closed approval floor (see the **RESOLVED** + block below, [ADR-0055](../decisions/0055-cli-host-capability-seam-tool-environment-factory.md) + + [ADR-0057](../decisions/0057-cli-chat-modes-and-per-tool-approval.md)). What remains: the **desktop/Tauri** + surface still binds an empty host (Phase 3), and its host-side `EgressCapability.fetch` SSRF hardening tracks + with the SSRF item above. Until the desktop host is wired, a desktop workflow calling a capability-backed + built-in tool surfaces a clean "tool unavailable" failure, never a half-implemented or unsafe execution. + *(medium · desktop tool host; Phase 3; security-review.md; egress → the SSRF item above)* - [ ] **`relavium run` maps any `run:paused` to exit 3 (gate-paused); revisit when media host-wiring lands.** `run.ts` returns `EXIT_CODES.gatePaused` (3) for any `run:paused`, which is correct in 2.D because a human gate is the **only** `run:paused` source (no `mediaStore`/media-job host is wired, so a media-only diff --git a/docs/roadmap/phases/phase-2.5-cli-consolidation.md b/docs/roadmap/phases/phase-2.5-cli-consolidation.md index e6113e3e..fce27dd0 100644 --- a/docs/roadmap/phases/phase-2.5-cli-consolidation.md +++ b/docs/roadmap/phases/phase-2.5-cli-consolidation.md @@ -10,8 +10,9 @@ > **2.5.F** (`/clear` + the `session:compacting` "Summarizing…" moment + the context-fullness footer, completing > the ADR-0062 compaction story) ✅ **Done (PR #65, merged 2026-07-05)**. The additive lane **2.5.H** (reasoning > render + live-turn feedback + actionable errors, EA6 amending ADR-0036) ✅ **Done (PR #67, 2026-07-07)** — **milestone -> M2.5-3 reached** (with 2.5.E). **Next: 2.5.G** (onboarding wizard + Home `/models`). Remaining additive lanes: -> 2.5.I / J. +> M2.5-3 reached** (with 2.5.E). **2.5.G** (onboarding wizard + Home `/models` + the live model catalog) ✅ +> **Done (PR #66, 2026-07-06)** behind [ADR-0063](../../decisions/0063-cli-config-write-contract.md)/[ADR-0064](../../decisions/0064-live-model-catalog.md)/[ADR-0065](../../decisions/0065-provider-economics-and-extensibility.md) +> (+ the ADR-0059/0066 model-UX follow-up) — **milestone M2.5-2 reached**. Remaining: the consolidation lanes 2.5.I / J. - **Related**: [../README.md](../README.md), [phase-2-cli.md](phase-2-cli.md), [phase-2.6-conversational-authoring.md](phase-2.6-conversational-authoring.md), [phase-3-desktop.md](phase-3-desktop.md), [../../reference/cli/commands.md](../../reference/cli/commands.md), [../../reference/cli/chat-session.md](../../reference/cli/chat-session.md), [../../reference/cli/regression-harness.md](../../reference/cli/regression-harness.md), [../../decisions/README.md](../../decisions/README.md) (ADR-0054–0057) @@ -405,16 +406,23 @@ overflows the context window; the summary is inspectable and the moment is a des > footer guard, the Home double-clear MCP-leak guard, best-effort auto-compaction, and the shared adapter > `CONTEXT_SEAM_DEFAULTS`.** -### 2.5.G — Onboarding wizard, `/models`, and the live model catalog +### 2.5.G — Onboarding wizard, `/models`, and the live model catalog — ✅ **Done (PR #66, 2026-07-06)** + +> **Status:** ✅ **Done (PR #66, 2026-07-06)** — all **12** dependency-ordered steps (S1–S12) landed behind +> [ADR-0063](../../decisions/0063-cli-config-write-contract.md)/[ADR-0064](../../decisions/0064-live-model-catalog.md)/[ADR-0065](../../decisions/0065-provider-economics-and-extensibility.md), +> each with an Opus + Sonnet review round (six with an added security round: S2/S5/S6/S8/S9/S10); the post-2.5.G +> model-UX follow-up ([ADR-0059](../../decisions/0059-cli-mid-session-model-reseat.md) mid-session reseat + +> [ADR-0066](../../decisions/0066-normalized-reasoning-effort-control.md) reasoning-effort) landed in the same PR. +> With 2.5.G done, **milestone M2.5-2** (Home + entry + onboarding) is reached. > **Scope expanded to Option A (2026-07-05).** The maintainer chose to build a **live** model catalog > (per-key provider discovery + a DB cache + refresh + a static/live merge) and a **complete model-pricing -> story** (user-supplied pricing that actually governs cost) now, rather than the static-registry catalog -> originally scoped. It lands behind three ADRs — [ADR-0063](../../decisions/0063-cli-config-write-contract.md) +> story** (user-supplied pricing that actually governs cost), rather than the static-registry catalog +> originally scoped. It landed behind three ADRs — [ADR-0063](../../decisions/0063-cli-config-write-contract.md) > (config-write contract), [ADR-0064](../../decisions/0064-live-model-catalog.md) (live catalog), and > [ADR-0065](../../decisions/0065-provider-economics-and-extensibility.md) (provider economics & extensibility) -> — across **12 dependency-ordered, individually-reviewed steps** (six security-flagged). It pulls the live -> `/v1/models` fetch forward from Phase 3 (the § "Explicitly out of scope" note above is reconciled to match). +> — across **12 dependency-ordered, individually-reviewed steps** (six security-flagged). It pulled the live +> `/v1/models` fetch forward from Phase 3 (the § "Explicitly out of scope" note above was reconciled to match). **Tasks:** a `@clack` first-run wizard from the key-less Home (provider → **hidden** stdin key → keychain, with a write-failure fallback + env-key import) reusing the two ink↔clack custody patterns; the **`listModels?` seam @@ -542,7 +550,7 @@ state; `NO_COLOR` is honoured. | In-phase | Completed by | Outcome | |----------|--------------|---------| | M2.5-1 Secure base ✅ **(PR #60, 2026-06-28)** | 2.5.A | Root-cause closed (capability gap + merge asymmetry); host seam reviewed | -| M2.5-2 Home + entry + onboarding | 2.5.B + 2.5.C + 2.5.D + 2.5.F + 2.5.G | First-class entry + ergonomics + onboarding | +| M2.5-2 Home + entry + onboarding ✅ **(PR #66, 2026-07-06)** | 2.5.B + 2.5.C + 2.5.D + 2.5.F + 2.5.G | First-class entry + ergonomics + onboarding | | M2.5-3 Modes + observability ✅ **(2026-07-07)** | 2.5.E + 2.5.H | Safe reseat-less mode system + per-tool approval + reasoning render + actionable errors | | M2.5-4 Consolidation | 2.5.I + 2.5.J | Harness + concurrency + docs-debt | @@ -557,9 +565,9 @@ parallel at any point. - **Phase 2** complete (chat REPL, history, providers, MCP, the 2.J authoring core that 2.5.A's tool host and Phase 2.6 build on). -- The pure engine amendments EA1–EA7 are filed back against `@relavium/core` / `@relavium/shared` as - Phase-1 amendments, each behind the ADR mapped in the [Engine amendments appendix](#engine-amendments-appendix-ea1ea7) - (EA1/EA2 → ADR-0055; EA3/EA4/EA5/EA7 → ADR-0057; EA6 amends ADR-0036); the engine architecture and the +- The pure engine amendments EA1–EA8 are filed back against `@relavium/core` / `@relavium/shared` as + Phase-1 amendments, each behind the ADR mapped in the [Engine amendments appendix](#engine-amendments-appendix-ea1ea8) + (EA1/EA2 → ADR-0055; EA3/EA4/EA5/EA7 → ADR-0057; EA6 amends ADR-0036; EA8 → ADR-0061); the engine architecture and the platform boundary are unchanged. ## Exit criteria (go / no-go → Phase 2.6) From 6e2e24a74e481e52b765070a3b7bdeefa4c1557c Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 19:50:20 +0300 Subject: [PATCH 03/50] =?UTF-8?q?docs(roadmap):=202.5-close=20Step=201=20O?= =?UTF-8?q?pus-review=20fixes=20=E2=80=94=20ADR-0059=20status=20+=20reseat?= =?UTF-8?q?=20reconciliation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review of b4d9619 surfaced leftover inconsistencies from 2.5.G pulling ADR-0059 forward: - ADR index: 0059 was still "Proposed | 2026-06-28"; its body + this reconciliation treat it as a shipped 2.5.G deliverable. Set to Accepted | 2026-07-06. - phase-2.5 "Explicitly out of scope": drop "mid-session model reseat" from the Phase-2.6 bullet (it shipped in 2.5.G per ADR-0059, PR #66); keep session {{ctx.*}} interpolation (ADR-0060, still Proposed). - phase-2.6 2.6.C: add a status note (reseat shipped early in 2.5.G) and fix the now-false "When ADR-0059 flips to Accepted" conditional (it is Accepted; the ADR-0024 amendment note is already recorded). - deferred-tasks: tighten the desktop ToolHost phrasing ("no tool host wired yet" rather than "binds an empty host" — apps/desktop has no host at all yet). Refs: docs/roadmap/phase-2.5-close-plan.md Step 1 Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/decisions/README.md | 2 +- docs/roadmap/deferred-tasks.md | 2 +- docs/roadmap/phases/phase-2.5-cli-consolidation.md | 8 +++++--- .../phases/phase-2.6-conversational-authoring.md | 12 ++++++++---- 4 files changed, 15 insertions(+), 9 deletions(-) diff --git a/docs/decisions/README.md b/docs/decisions/README.md index f7b8b60f..68f299fa 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -102,7 +102,7 @@ flowchart TD | 0056 | [In-app slash command system driven by a single command manifest](0056-cli-in-app-slash-command-system-and-manifest.md) | Accepted | 2026-06-29 | | 0057 | [Reseat-less chat modes and per-tool approval (with mid-turn abort)](0057-cli-chat-modes-and-per-tool-approval.md) | Accepted | 2026-06-28 | | 0058 | [`@relavium/authoring` package and the conversational-authoring pre-flight contract](0058-relavium-authoring-package-and-conversational-authoring.md) | Proposed | 2026-06-28 | -| 0059 | [Mid-session model switching via host-side reseat (refines ADR-0024)](0059-cli-mid-session-model-reseat.md) | Proposed | 2026-06-28 | +| 0059 | [Mid-session model switching via host-side reseat (refines ADR-0024)](0059-cli-mid-session-model-reseat.md) | Accepted | 2026-07-06 | | 0060 | [Session `{{ctx.*}}` prompt interpolation](0060-session-ctx-prompt-interpolation.md) | Proposed | 2026-06-28 | | 0061 | [CLI chat input-layer file-injection (`@`-mention) and shell-escape (`!`-shell) security model](0061-cli-input-layer-file-injection-and-shell-escape.md) | Accepted | 2026-07-03 | | 0062 | [Context compaction — append-only conversation summarization and the CLI history commands (`/clear` · `/trim` · `/compact`)](0062-context-compaction-and-cli-history-commands.md) | Accepted | 2026-07-04 | diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index f081f06c..6a046523 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -595,7 +595,7 @@ Severity is the review's verified rating. Check an item off in the PR that resol `egress` + `os` arms via `assembleToolEnv`, all behind the fail-closed approval floor (see the **RESOLVED** block below, [ADR-0055](../decisions/0055-cli-host-capability-seam-tool-environment-factory.md) + [ADR-0057](../decisions/0057-cli-chat-modes-and-per-tool-approval.md)). What remains: the **desktop/Tauri** - surface still binds an empty host (Phase 3), and its host-side `EgressCapability.fetch` SSRF hardening tracks + surface has no tool host wired yet (Phase 3), and its host-side `EgressCapability.fetch` SSRF hardening tracks with the SSRF item above. Until the desktop host is wired, a desktop workflow calling a capability-backed built-in tool surfaces a clean "tool unavailable" failure, never a half-implemented or unsafe execution. *(medium · desktop tool host; Phase 3; security-review.md; egress → the SSRF item above)* diff --git a/docs/roadmap/phases/phase-2.5-cli-consolidation.md b/docs/roadmap/phases/phase-2.5-cli-consolidation.md index fce27dd0..602f05a6 100644 --- a/docs/roadmap/phases/phase-2.5-cli-consolidation.md +++ b/docs/roadmap/phases/phase-2.5-cli-consolidation.md @@ -62,9 +62,11 @@ Along the way, close the bounded engine amendments and docs-debt that Phase 2 de ### Explicitly out of scope (→ Phase 2.6 or Phase 3) -- Conversational (model-generated) authoring, the `@relavium/authoring` package promotion, mid-session - **model** reseat, session `{{ctx.*}}` interpolation, and competitor-parity polish — all - [phase-2.6-conversational-authoring.md](phase-2.6-conversational-authoring.md). +- Conversational (model-generated) authoring, the `@relavium/authoring` package promotion, session + `{{ctx.*}}` interpolation, and competitor-parity polish — all + [phase-2.6-conversational-authoring.md](phase-2.6-conversational-authoring.md). (**Mid-session model + reseat** was also listed here for Phase 2.6; its core **shipped early in 2.5.G**'s model-UX follow-up per + [ADR-0059](../../decisions/0059-cli-mid-session-model-reseat.md) — the `/models` mid-chat reseat, PR #66.) - `read_media` input (D12), full-fidelity reseat tool-context, in-app scrollback/pager, a **type-ahead message queue while a turn runs** (the in-flight key-swallow is handled for approval input in 2.5.E, but queuing the *next* message is deferred), and a multi-pane dashboard — diff --git a/docs/roadmap/phases/phase-2.6-conversational-authoring.md b/docs/roadmap/phases/phase-2.6-conversational-authoring.md index 3e8fec1e..abd44c62 100644 --- a/docs/roadmap/phases/phase-2.6-conversational-authoring.md +++ b/docs/roadmap/phases/phase-2.6-conversational-authoring.md @@ -118,6 +118,10 @@ shared with 2.6.A.** ### 2.6.C — Mid-session model reseat (`/models` mid-chat) +> **Note (2026-07-07):** ADR-0059 was pulled forward and the `/models` mid-chat reseat **shipped early in +> 2.5.G**'s model-UX follow-up (PR #66, merged 2026-07-06). This workstream is retained for any residual +> Phase-2.6 polish and as the cross-reference home. + A session binds one model for its lifetime ([ADR-0024](../../decisions/0024-agent-first-entry-point-agentsession.md); the fallback plan is memoized). Switching the **model** mid-chat is therefore a host-side **reseat**: reconstruct the transcript (`reconstructSessionState`) and start a new `AgentSession.resume` bound to the @@ -132,10 +136,10 @@ breakdown. **Acceptance:** `/models` mid-chat continues the conversation on a new model from the next turn; cost/turn carry; per-message model attribution persists; the context-loss notice is shown; "carries full context" is -never claimed. When [ADR-0059](../../decisions/0059-cli-mid-session-model-reseat.md) flips to Accepted, its -refinement of [ADR-0024](../../decisions/0024-agent-first-entry-point-agentsession.md) is recorded **in -place** on ADR-0024 with a dated `> Amended …` note + a Related forward-link (documentation-style §7), since -it refines without reversing. **Required ADR: mid-session reseat (model-only) — refines ADR-0024.** +never claimed. [ADR-0059](../../decisions/0059-cli-mid-session-model-reseat.md) is **Accepted (2026-07-06)** and +its refinement of [ADR-0024](../../decisions/0024-agent-first-entry-point-agentsession.md) is recorded **in +place** on ADR-0024 (the dated `> Amended 2026-07-06 …` note + the Related forward-link, documentation-style §7), +since it refines without reversing. **Required ADR: mid-session reseat (model-only) — refines ADR-0024 (done).** ### 2.6.D — Session `{{ctx.*}}` prompt interpolation From 0d0574ebaff5c86c2cdc22cc4e2eeb13c7f9105f Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 20:04:34 +0300 Subject: [PATCH 04/50] =?UTF-8?q?docs(roadmap):=202.5-close=20Step=201=20S?= =?UTF-8?q?onnet-review=20fixes=20=E2=80=94=20PR=20#66=20merge=20date=20+?= =?UTF-8?q?=20phase-2.6=20reseat=20framing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet review verified PR #66 merged 2026-07-07 (gh api merged_at 2026-07-07T10:10:40Z), not 2026-07-06 (the -06 is the ADR-0059/0066 Accepted date). The wrong date also self-contradicted existing "merged 2026-07-07" text for the same PR. - Correct every "PR #66, 2026-07-06" merge citation to 2026-07-07 across the phase-2.5 doc (top status, 2.5.G header + Status block, M2.5-2 row), current.md, CLAUDE.md, and the phase-2.6 2.6.C note. ADR-date "2026-07-06" strings (ADR-0059 Accepted, the ADR-0024 amendment) are left untouched — they are correct. - phase-2.6: add a single Status-block note so the higher-level Goal / Outcomes / In-scope / Milestones / Exit-criteria mentions of "mid-session model switching" are reconciled to "shipped early in 2.5.G" in one place (they framed a shipped feature as future work). - CLAUDE.md: align terminology to "post-2.5.G model-UX follow-up" (matching the phase-2.5 Status block + current.md). Refs: docs/roadmap/phase-2.5-close-plan.md Step 1 Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 4 ++-- docs/roadmap/current.md | 2 +- docs/roadmap/phases/phase-2.5-cli-consolidation.md | 8 ++++---- docs/roadmap/phases/phase-2.6-conversational-authoring.md | 6 +++++- 4 files changed, 12 insertions(+), 8 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 87781915..47588ed1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,11 +109,11 @@ compaction-moment UX polishes (a `session:compacting` "Summarizing…" event ame context-fullness indicator via a pure `@relavium/llm` `contextWindowForModel` helper), completing the ADR-0062 compaction story alongside the earlier-landed model-summarised `/compact` + deterministic `/trim` + automatic compaction. **2.5.G** (onboarding wizard + Home `/models` + the live model catalog) is ✅ **Done (PR #66, -2026-07-06)** — its scope expanded to **Option A** (a **live** model catalog + a complete model-pricing story that +2026-07-07)** — its scope expanded to **Option A** (a **live** model catalog + a complete model-pricing story that governs cost) behind three ADRs ([ADR-0063](docs/decisions/0063-cli-config-write-contract.md) config-write, [ADR-0064](docs/decisions/0064-live-model-catalog.md) live catalog, [ADR-0065](docs/decisions/0065-provider-economics-and-extensibility.md) provider economics); all 12 steps landed, -plus the model-UX follow-up ([ADR-0059](docs/decisions/0059-cli-mid-session-model-reseat.md) mid-session reseat + +plus the post-2.5.G model-UX follow-up ([ADR-0059](docs/decisions/0059-cli-mid-session-model-reseat.md) mid-session reseat + [ADR-0066](docs/decisions/0066-normalized-reasoning-effort-control.md) reasoning-effort). With it **milestone M2.5-2** is reached. The additive lane **2.5.H** (reasoning render + live-turn feedback + an actionable error taxonomy — behind **EA6**, a dual-envelope `agent:reasoning` stream event that *amends* [ADR-0036](docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md); diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 7f330bf2..95f668ec 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -187,7 +187,7 @@ all recorded in the ADR-0061 "Refined at implementation" append. **2.5.F** (the `/clear` (a host-level fresh-session swap across `chat`/`chat-resume`/Home, TTY-interactive only), the `session:compacting` "Summarizing…" moment event (amending ADR-0036), and the footer context-fullness indicator — completing compaction alongside the earlier `/compact` + `/trim` + automatic compaction. **2.5.G** (onboarding -wizard + Home `/models` + the live model catalog) is ✅ **Done (PR #66, 2026-07-06)** — its scope **expanded to +wizard + Home `/models` + the live model catalog) is ✅ **Done (PR #66, 2026-07-07)** — its scope **expanded to Option A**: a **live** model catalog (per-key provider discovery + a DB cache + refresh + a static/live merge) and a complete model-pricing story (user-supplied pricing that governs cost), behind three ADRs ([ADR-0063](../decisions/0063-cli-config-write-contract.md) config-write · diff --git a/docs/roadmap/phases/phase-2.5-cli-consolidation.md b/docs/roadmap/phases/phase-2.5-cli-consolidation.md index 602f05a6..a689fda0 100644 --- a/docs/roadmap/phases/phase-2.5-cli-consolidation.md +++ b/docs/roadmap/phases/phase-2.5-cli-consolidation.md @@ -11,7 +11,7 @@ > the ADR-0062 compaction story) ✅ **Done (PR #65, merged 2026-07-05)**. The additive lane **2.5.H** (reasoning > render + live-turn feedback + actionable errors, EA6 amending ADR-0036) ✅ **Done (PR #67, 2026-07-07)** — **milestone > M2.5-3 reached** (with 2.5.E). **2.5.G** (onboarding wizard + Home `/models` + the live model catalog) ✅ -> **Done (PR #66, 2026-07-06)** behind [ADR-0063](../../decisions/0063-cli-config-write-contract.md)/[ADR-0064](../../decisions/0064-live-model-catalog.md)/[ADR-0065](../../decisions/0065-provider-economics-and-extensibility.md) +> **Done (PR #66, 2026-07-07)** behind [ADR-0063](../../decisions/0063-cli-config-write-contract.md)/[ADR-0064](../../decisions/0064-live-model-catalog.md)/[ADR-0065](../../decisions/0065-provider-economics-and-extensibility.md) > (+ the ADR-0059/0066 model-UX follow-up) — **milestone M2.5-2 reached**. Remaining: the consolidation lanes 2.5.I / J. - **Related**: [../README.md](../README.md), [phase-2-cli.md](phase-2-cli.md), [phase-2.6-conversational-authoring.md](phase-2.6-conversational-authoring.md), [phase-3-desktop.md](phase-3-desktop.md), [../../reference/cli/commands.md](../../reference/cli/commands.md), [../../reference/cli/chat-session.md](../../reference/cli/chat-session.md), [../../reference/cli/regression-harness.md](../../reference/cli/regression-harness.md), [../../decisions/README.md](../../decisions/README.md) (ADR-0054–0057) @@ -408,9 +408,9 @@ overflows the context window; the summary is inspectable and the moment is a des > footer guard, the Home double-clear MCP-leak guard, best-effort auto-compaction, and the shared adapter > `CONTEXT_SEAM_DEFAULTS`.** -### 2.5.G — Onboarding wizard, `/models`, and the live model catalog — ✅ **Done (PR #66, 2026-07-06)** +### 2.5.G — Onboarding wizard, `/models`, and the live model catalog — ✅ **Done (PR #66, 2026-07-07)** -> **Status:** ✅ **Done (PR #66, 2026-07-06)** — all **12** dependency-ordered steps (S1–S12) landed behind +> **Status:** ✅ **Done (PR #66, 2026-07-07)** — all **12** dependency-ordered steps (S1–S12) landed behind > [ADR-0063](../../decisions/0063-cli-config-write-contract.md)/[ADR-0064](../../decisions/0064-live-model-catalog.md)/[ADR-0065](../../decisions/0065-provider-economics-and-extensibility.md), > each with an Opus + Sonnet review round (six with an added security round: S2/S5/S6/S8/S9/S10); the post-2.5.G > model-UX follow-up ([ADR-0059](../../decisions/0059-cli-mid-session-model-reseat.md) mid-session reseat + @@ -552,7 +552,7 @@ state; `NO_COLOR` is honoured. | In-phase | Completed by | Outcome | |----------|--------------|---------| | M2.5-1 Secure base ✅ **(PR #60, 2026-06-28)** | 2.5.A | Root-cause closed (capability gap + merge asymmetry); host seam reviewed | -| M2.5-2 Home + entry + onboarding ✅ **(PR #66, 2026-07-06)** | 2.5.B + 2.5.C + 2.5.D + 2.5.F + 2.5.G | First-class entry + ergonomics + onboarding | +| M2.5-2 Home + entry + onboarding ✅ **(PR #66, 2026-07-07)** | 2.5.B + 2.5.C + 2.5.D + 2.5.F + 2.5.G | First-class entry + ergonomics + onboarding | | M2.5-3 Modes + observability ✅ **(2026-07-07)** | 2.5.E + 2.5.H | Safe reseat-less mode system + per-tool approval + reasoning render + actionable errors | | M2.5-4 Consolidation | 2.5.I + 2.5.J | Harness + concurrency + docs-debt | diff --git a/docs/roadmap/phases/phase-2.6-conversational-authoring.md b/docs/roadmap/phases/phase-2.6-conversational-authoring.md index abd44c62..2f229ea7 100644 --- a/docs/roadmap/phases/phase-2.6-conversational-authoring.md +++ b/docs/roadmap/phases/phase-2.6-conversational-authoring.md @@ -3,6 +3,10 @@ > Status: Planned. Depends on the Phase 2.5 spine (the wired tool-environment and the per-tool > approval / mode system). Spine: 2.6.A (`@relavium/authoring` package) → 2.6.B (conversational > authoring agent). Additive: 2.6.C / D / E. +> +> **Note (2026-07-07):** **2.6.C**'s mid-session `/models` model **reseat shipped early in 2.5.G** (ADR-0059, +> PR #66, merged 2026-07-07); the "mid-session model switching" mentions in the Goal / Outcomes / In-scope / +> Milestones / Exit-criteria below are therefore already realized — see §2.6.C. This phase is otherwise Planned. - **Related**: [../README.md](../README.md), [phase-2.5-cli-consolidation.md](phase-2.5-cli-consolidation.md), [phase-2-cli.md](phase-2-cli.md), [phase-3-desktop.md](phase-3-desktop.md), [phase-4-vscode.md](phase-4-vscode.md), [../../reference/contracts/workflow-yaml-spec.md](../../reference/contracts/workflow-yaml-spec.md), [../../reference/contracts/agent-yaml-spec.md](../../reference/contracts/agent-yaml-spec.md), [../../reference/shared-core/node-types.md](../../reference/shared-core/node-types.md), [../../decisions/README.md](../../decisions/README.md) (ADR-0058–0060) @@ -119,7 +123,7 @@ shared with 2.6.A.** ### 2.6.C — Mid-session model reseat (`/models` mid-chat) > **Note (2026-07-07):** ADR-0059 was pulled forward and the `/models` mid-chat reseat **shipped early in -> 2.5.G**'s model-UX follow-up (PR #66, merged 2026-07-06). This workstream is retained for any residual +> 2.5.G**'s model-UX follow-up (PR #66, merged 2026-07-07). This workstream is retained for any residual > Phase-2.6 polish and as the cross-reference home. A session binds one model for its lifetime ([ADR-0024](../../decisions/0024-agent-first-entry-point-agentsession.md); From 377a272c0acb9ee7d744af4c13573ab8ae17fed4 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 20:08:50 +0300 Subject: [PATCH 05/50] =?UTF-8?q?docs:=202.5-close=20Step=202=20=E2=80=94?= =?UTF-8?q?=20correct=20surface-blind=20"encrypted=20history"=20wording=20?= =?UTF-8?q?+=20README=20Phase-2.5=20prose?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 2.5.J Task 1. Per ADR-0050 the CLI history.db is UNENCRYPTED at rest (guarded by 0700/0600 owner-only permissions + the OS keychain; no credentials stored); only the desktop surface uses a SQLCipher-encrypted store. - uvp.md / vision.md: drop the false "encrypted local history" claim; state the real protection (owner-only file perms + keychain). - tutorials/cli/start-a-chat-session.md: "encrypted history.db" -> unencrypted + 0700/0600 per ADR-0050 (linked). - architecture/agent-sessions.md: the two surface-blind "encrypted ... SQLCipher" statements now carry the CLI-unencrypted vs desktop-SQLCipher split (same defect class the Task-1 scope missed; the phase-2.5 2.5.J task is about surface-blindness). - README.md: add a Phase-2.5 (CLI Consolidation) prose sentence to the Status para (per close-plan D5 — expand prose, do NOT mint global M3.5/M3.6 IDs). Left untouched (accurate): the CI-secret "encrypted CI secret" (GitHub Actions), and the desktop/VS Code docs whose surface genuinely uses SQLCipher. Refs: docs/roadmap/phase-2.5-close-plan.md Step 2 Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 6 +++++- docs/architecture/agent-sessions.md | 9 ++++++--- docs/tutorials/cli/start-a-chat-session.md | 4 +++- docs/uvp.md | 3 ++- docs/vision.md | 3 ++- 5 files changed, 18 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 4c258ab8..e04f69ec 100644 --- a/README.md +++ b/README.md @@ -132,7 +132,11 @@ surface); and the **inbound MCP client** — agents consume external MCP servers proven by a real-spawn end-to-end test; and the **YAML-authoring lifecycle** — `relavium create` (a wizard scaffolding an agent or a minimal single-agent workflow), `import`, and a share-safe `export` (re-serialized from the validated AST, no provider key by construction). With every in-phase workstream merged, the CLI is -cut as **v0.1.1** (the public npm publish is the pending final maintainer step). For live status and the full roadmap, see +cut as **v0.1.1** (the public npm publish is the pending final maintainer step). **Phase 2.5 (CLI +Consolidation) is well underway** — the conversational Home, the slash-command system, reseat-less chat +modes with per-tool approval, context compaction, a live model catalog with provider economics, and +reasoning render have all landed; only the regression-harness/concurrency and documentation-consolidation +lanes remain. For live status and the full roadmap, see [docs/roadmap/current.md](docs/roadmap/current.md) and the [roadmap](docs/roadmap/README.md). diff --git a/docs/architecture/agent-sessions.md b/docs/architecture/agent-sessions.md index 684be58c..28953420 100644 --- a/docs/architecture/agent-sessions.md +++ b/docs/architecture/agent-sessions.md @@ -89,8 +89,10 @@ the `SessionContext`, and the `SessionMessage` shape are canonical in ## Checkpoint and persist to `history.db` A session is **auto-persisted and resumable** — there is no separate "save" step and no -`sessions.db`. Sessions and their transcripts live in the existing encrypted local -`history.db` (SQLCipher) in two new tables: +`sessions.db`. Sessions and their transcripts live in the existing local `history.db` in two new tables +(its at-rest posture is surface-specific per [ADR-0050](../decisions/0050-cli-history-db-at-rest-posture.md): +the **CLI** store is unencrypted, guarded by `0700`/`0600` owner-only permissions + the OS keychain; only the +**desktop** surface uses a SQLCipher-encrypted store): - **`agent_sessions`** — one row per session: the bound `agentRef`/model, the `SessionContext`, and lifecycle timestamps. @@ -124,7 +126,8 @@ The same tool-policy hardening applies as for workflows ([ADR-0029](../decisions/0029-tool-policy-hardening.md)): a session may only **narrow** the agent's tools (never escalate), a `secret`-typed value is **never** interpolated into a prompt or tool text, and `http_request`/MCP egress obeys the same SSRF policy. A user's -own conversational content is the user's data — persisted in the encrypted `history.db`, +own conversational content is the user's data — persisted in `history.db` (unencrypted at rest on the CLI +per [ADR-0050](../decisions/0050-cli-history-db-at-rest-posture.md); SQLCipher on desktop), not treated as a managed secret; that boundary is stated in [../standards/security-review.md](../standards/security-review.md). diff --git a/docs/tutorials/cli/start-a-chat-session.md b/docs/tutorials/cli/start-a-chat-session.md index 565b9727..75cf3822 100644 --- a/docs/tutorials/cli/start-a-chat-session.md +++ b/docs/tutorials/cli/start-a-chat-session.md @@ -111,7 +111,9 @@ chat` opened an [agent session](../../reference/contracts/agent-session-spec.md) same `@relavium/core` engine `relavium run` uses, each turn drove the agent loop through the shared `ToolRegistry` and the `@relavium/llm` seam, every tool call ran under the same filesystem scope and command allowlist a workflow would enforce, and the whole transcript -was checkpointed to the encrypted `history.db` — so it is resumable and, when you are +was checkpointed to `history.db` (unencrypted at rest, guarded by `0700`/`0600` owner-only file +permissions per [ADR-0050](../../decisions/0050-cli-history-db-at-rest-posture.md); no credentials are +stored there — API keys live in the OS keychain) — so it is resumable and, when you are ready, exportable. Harden once, both entry points inherit (see [shared-core-engine.md](../../architecture/shared-core-engine.md)). diff --git a/docs/uvp.md b/docs/uvp.md index 1db83360..bd19a2ef 100644 --- a/docs/uvp.md +++ b/docs/uvp.md @@ -19,7 +19,8 @@ (usage, quota, governance), not a fourth identical-engine runtime. 2. **No chat-driven assistant lets you keep — and ship — the conversation.** Relavium agent sessions are **persistent, resumable, and exportable**: - auto-saved to encrypted local history, resumable on any surface, and one-click + auto-saved to durable local history (owner-only file permissions; keys stay in + the OS keychain, never at rest), resumable on any surface, and one-click exportable to a reviewable `.relavium.yaml` scaffold. Competitors' sessions evaporate; Relavium's become committable, re-runnable workflows. See [decisions/0024-agent-first-entry-point-agentsession.md](decisions/0024-agent-first-entry-point-agentsession.md) diff --git a/docs/vision.md b/docs/vision.md index 74031d25..730313de 100644 --- a/docs/vision.md +++ b/docs/vision.md @@ -148,7 +148,8 @@ The full positioning and competitor matrix lives in [uvp.md](uvp.md). workflow. See [decisions/0026-session-export-to-workflow.md](decisions/0026-session-export-to-workflow.md). - **Persistent, resumable agent sessions** — every conversation is auto-saved to - the encrypted local history and resumable on any surface; no run is ever + durable local history (owner-only `0700`/`0600` file permissions; API keys live in + the OS keychain, never at rest) and resumable on any surface; no run is ever ephemeral. *(The `AgentSession` engine lands in build Phase 1; the first user-facing surface is CLI `relavium chat` in build phase 2 — all within Product Phase 1.)* See From d3267f8970f873a90bc4f5a604967895db5f4f4b Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 20:18:24 +0300 Subject: [PATCH 06/50] =?UTF-8?q?docs:=202.5-close=20Step=202=20Opus-revie?= =?UTF-8?q?w=20fixes=20=E2=80=94=20finish=20the=20surface-blind=20SQLCiphe?= =?UTF-8?q?r=20sweep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review confirmed the commit was correct/complete for its named scope and surfaced same-class pre-existing leftovers. Completing the sweep so no CLI / cross-surface doc still over-claims CLI-side encryption: - architecture/local-first-and-security.md: the "Run history" security-table row now splits CLI-unencrypted (0600/0700 + keychain) vs desktop-SQLCipher. - tech-stack.md: the Database cell + the local-first-storage bullet now state the CLI store is unencrypted at rest (ADR-0050), keeping the desktop-SQLCipher fact. - decisions/0029: append-only dated amendment note (mirrors ADR-0024) clarifying rule (c)'s "encrypted in history.db" is surface-specific; body unchanged. - architecture/agent-sessions.md: tighten the keychain phrasing (keys live there; it does not "guard" history.db). - deferred-tasks.md: track the VS Code docs' "SQLCipher history.db = same store as CLI" contradiction as a Phase-4 forward-design follow-up (blocked on the cross-host physical-store decision; genuinely out of scope here). Refs: docs/roadmap/phase-2.5-close-plan.md Step 2 Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/architecture/agent-sessions.md | 4 ++-- docs/architecture/local-first-and-security.md | 2 +- docs/decisions/0029-tool-policy-hardening.md | 2 ++ docs/roadmap/deferred-tasks.md | 9 +++++++++ docs/tech-stack.md | 7 ++++--- 5 files changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/architecture/agent-sessions.md b/docs/architecture/agent-sessions.md index 28953420..f591e480 100644 --- a/docs/architecture/agent-sessions.md +++ b/docs/architecture/agent-sessions.md @@ -91,8 +91,8 @@ the `SessionContext`, and the `SessionMessage` shape are canonical in A session is **auto-persisted and resumable** — there is no separate "save" step and no `sessions.db`. Sessions and their transcripts live in the existing local `history.db` in two new tables (its at-rest posture is surface-specific per [ADR-0050](../decisions/0050-cli-history-db-at-rest-posture.md): -the **CLI** store is unencrypted, guarded by `0700`/`0600` owner-only permissions + the OS keychain; only the -**desktop** surface uses a SQLCipher-encrypted store): +the **CLI** store is unencrypted, guarded by `0700`/`0600` owner-only permissions (no credentials at rest — +API keys live in the OS keychain); only the **desktop** surface uses a SQLCipher-encrypted store): - **`agent_sessions`** — one row per session: the bound `agentRef`/model, the `SessionContext`, and lifecycle timestamps. diff --git a/docs/architecture/local-first-and-security.md b/docs/architecture/local-first-and-security.md index 6c541902..691abaab 100644 --- a/docs/architecture/local-first-and-security.md +++ b/docs/architecture/local-first-and-security.md @@ -70,7 +70,7 @@ API calls: |------|----------------|-------| | Workflow definitions | `.relavium/*.relavium.yaml` on disk | git-committable; see [workflow-yaml-spec.md](../reference/contracts/workflow-yaml-spec.md) | | Agent definitions | `*.agent.yaml` on disk | git-committable; see [agent-yaml-spec.md](../reference/contracts/agent-yaml-spec.md) | -| Run history, events, outputs | local SQLite (SQLCipher) | DDL in [database-schema.md](../reference/desktop/database-schema.md) | +| Run history, events, outputs | local SQLite — **CLI: unencrypted** (`0600`/`0700` owner-only perms + keychain; no credentials at rest); **desktop: SQLCipher** | DDL + at-rest posture in [database-schema.md](../reference/desktop/database-schema.md); [ADR-0050](../decisions/0050-cli-history-db-at-rest-posture.md) | | Cost records | local SQLite | per-node and per-run | | API keys | OS keychain | never on disk in plaintext, never in the DB | | Global config | `~/.relavium/` | global preferences, MCP registrations | diff --git a/docs/decisions/0029-tool-policy-hardening.md b/docs/decisions/0029-tool-policy-hardening.md index 4c796d30..f1cee83a 100644 --- a/docs/decisions/0029-tool-policy-hardening.md +++ b/docs/decisions/0029-tool-policy-hardening.md @@ -6,6 +6,8 @@ > **Amended 2026-06-18 by [ADR-0044](0044-media-access-governance-read-media-save-to-cost.md).** A refinement, not a reversal: ADR-0044 adds the `read_media` scope-set authz and an **additive** `ToolPolicyDenyReason` member `media_scope_denied` (alongside this ADR's existing six reasons). This ADR's tool-policy decisions are unchanged. +> **Amended 2026-07-07 (append-only, no reversal — surface-specific at-rest posture).** Rule (c)'s parenthetical below says user conversational content is "encrypted in `history.db`". That phrasing predates [ADR-0050](0050-cli-history-db-at-rest-posture.md) and is **surface-specific**: on the **CLI** surface `history.db` is **not** encrypted at rest — it is guarded by `0700`/`0600` owner-only permissions with API keys in the OS keychain only (no credentials at rest); only the **desktop** surface uses a SQLCipher-encrypted store. Read the phrase surface-specifically (mirrors the [ADR-0024](0024-agent-first-entry-point-agentsession.md) amendment note). This ADR's tool-policy decisions are unchanged. + ## Context Four tool-policy ambiguities in the current specs are individually small and collectively a real diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index 6a046523..0515586e 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -599,6 +599,15 @@ Severity is the review's verified rating. Check an item off in the PR that resol with the SSRF item above. Until the desktop host is wired, a desktop workflow calling a capability-backed built-in tool surfaces a clean "tool unavailable" failure, never a half-implemented or unsafe execution. *(medium · desktop tool host; Phase 3; security-review.md; egress → the SSRF item above)* +- [ ] **VS Code docs claim a SQLCipher `history.db` "same store as CLI" — contradicts ADR-0050.** The + Phase-4 VS Code docs ([reference/vscode/extension-api.md](../reference/vscode/extension-api.md) L46/67, + [phases/phase-4-vscode.md](phases/phase-4-vscode.md) L376/378) still carry the pre-ADR-0050 assumption that + the extension host opens the **SQLCipher** `history.db` that is **"the same store as CLI/desktop"**. Per + [ADR-0050](../decisions/0050-cli-history-db-at-rest-posture.md) the CLI store is `better-sqlite3` + **unencrypted** and a SQLCipher file cannot be the same physical file; there is no cross-surface shared + session/run store until a Phase-3/4 ADR reconciles it. Reword the VS Code at-rest posture once the + cross-host physical-store decision lands. *(low · Phase 4 forward-design docs; blocked on the cross-host + store decision; surfaced during the 2.5.J encrypted-wording sweep)* - [ ] **`relavium run` maps any `run:paused` to exit 3 (gate-paused); revisit when media host-wiring lands.** `run.ts` returns `EXIT_CODES.gatePaused` (3) for any `run:paused`, which is correct in 2.D because a human gate is the **only** `run:paused` source (no `mediaStore`/media-job host is wired, so a media-only diff --git a/docs/tech-stack.md b/docs/tech-stack.md index 77b7c92b..ce1bfeef 100644 --- a/docs/tech-stack.md +++ b/docs/tech-stack.md @@ -24,7 +24,7 @@ adversarially reviewed by a 10-agent workflow before being locked. | Orchestration engine | **Pure TypeScript** (`packages/core`) | No LangGraph-Python; the same concepts are implementable in TS. Engine is framework-agnostic and runs identically on every surface. | | YAML parser (engine) | **`yaml`** (eemeli/yaml — pure-JS, confined to `packages/core`) | Decodes an authored `.relavium.yaml` string to a plain JS object for the strict `WorkflowSchema`; pure-JS (no native binding) preserves the zero-platform-imports invariant (Tauri WebView / Bun), and strict Zod does all schema enforcement. The engine's first runtime dependency, landed with the 1.L parser and pinned in the `catalog:`. See [ADR-0035](decisions/0035-yaml-parser-dependency.md). | | Expression sandbox (engine) | **QuickJS compiled to WebAssembly** — via `quickjs-emscripten-core` + a single-file **sync** variant (candidate `@jitl/quickjs-singlefile-mjs-release-sync`), instantiated via the standard `WebAssembly` global from embedded bytes; **never** the default `getQuickJS()` loader (it imports `node:fs`). Confirmed + version-pinned in the `catalog:` by the 1.AB perf spike. | Deterministic, resource-capped evaluation of `condition` / `transform` / `merge_fn` JS expressions that runs in **every** host the engine runs in, including the Tauri WebView. `isolated-vm` (native binding) and Node `vm` (`node:vm`) are rejected for breaking the zero-platform-imports invariant; `new Function` / `eval` are rejected on **sandbox-safety** grounds — they evaluate with full host access (no isolation, no determinism, no resource caps), so they cannot provide the sandbox's guarantees regardless of portability. The engine's second runtime dependency (after the `yaml` loader). See [ADR-0027](decisions/0027-expression-sandbox.md) (and its 2026-06-12 contract addendum) and the canonical contract in [expression-sandbox-spec.md](reference/shared-core/expression-sandbox-spec.md). | -| Database (local, Phase 1) | **SQLite + Drizzle ORM** (SQLCipher) | Tauri plugin available; encrypted at rest with SQLCipher. Node-side consumers (CLI, `@relavium/db` tests) use the **`better-sqlite3`** driver — see [ADR-0021](decisions/0021-node-sqlite-driver-better-sqlite3.md). | +| Database (local, Phase 1) | **SQLite + Drizzle ORM** | Tauri plugin available; the **desktop** store is encrypted at rest with SQLCipher. Node-side consumers (CLI, `@relavium/db` tests) use the **`better-sqlite3`** driver ([ADR-0021](decisions/0021-node-sqlite-driver-better-sqlite3.md)); the **CLI** store is **unencrypted at rest** — guarded by `0600`/`0700` owner-only permissions + the OS keychain, no credentials stored ([ADR-0050](decisions/0050-cli-history-db-at-rest-posture.md)). | | Database (cloud, Phase 2) | **PostgreSQL 16 + Redis 7 + BullMQ** | *Phase 2 only.* Same Drizzle schema, different driver. | | API key storage | **OS keychain**, one `KeychainStore` interface with a per-surface accessor — desktop `tauri-plugin-keychain` (Rust), **CLI `@napi-rs/keyring`** (Node; *not* the archived `keytar`, see [ADR-0019](decisions/0019-cli-node-keychain-library.md)), VS Code `vscode.SecretStorage` | macOS Keychain / Windows Credential Manager / libsecret. Never plaintext, never sent to the frontend. See [ADR-0006](decisions/0006-os-keychain-for-api-keys.md). | | CLI | **TypeScript + commander.js + ink** (`@clack/prompts` setup wizards; bundled to a single ESM `bin` with `tsup`) | Same language as the engine; React for the TUI. **Distribution** ([ADR-0051](decisions/0051-cli-distribution-thin-bundle-private-engine.md)): an *engine-inlined* bundle — the proprietary `@relavium/*` is inlined, every third-party dep (incl. prebuilt native addons) is externalized + declared; published to npm as `relavium` via a tag-triggered, cross-OS-smoke-gated `Release CLI` workflow. | @@ -54,8 +54,9 @@ adversarially reviewed by a 10-agent workflow before being locked. imports, so it runs identically inside the Tauri WebView, the VS Code extension host, the Node.js CLI, and (Phase 2) a Bun API server. This is the decision that makes cross-surface consistency real rather than aspirational. -- **Local-first storage.** SQLite + SQLCipher locally; the same Drizzle schema - ports to PostgreSQL for the Phase 2 cloud layer. See +- **Local-first storage.** SQLite locally (SQLCipher-encrypted on desktop; + unencrypted with owner-only permissions on the CLI, [ADR-0050](decisions/0050-cli-history-db-at-rest-posture.md)); + the same Drizzle schema ports to PostgreSQL for the Phase 2 cloud layer. See [reference/desktop/database-schema.md](reference/desktop/database-schema.md) and the SQLite-vs-Postgres porting notes there. - **Keys never touch the frontend.** API keys live only in the OS keychain, From d6544ebe4476dfb5cf500ce84630335656f1ad4c Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 20:25:26 +0300 Subject: [PATCH 07/50] =?UTF-8?q?docs:=202.5-close=20Step=202=20Sonnet-rev?= =?UTF-8?q?iew=20fixes=20=E2=80=94=20surface-neutral=20UVP/vision=20+=20ru?= =?UTF-8?q?ns.db=20unencrypted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet review confirmed the commits accurate/complete and surfaced two refinements: - uvp.md / vision.md: the added "owner-only file permissions" parenthetical is a CLI-specific control, but the sentence is about sessions "resumable on any surface" — attributing it across surfaces understates desktop's SQLCipher. Make it surface-neutral ("kept on your machine; API keys ... in the OS keychain, never at rest"), matching the plan's "drop 'encrypted', do not overclaim". - reference/desktop/tauri-plugins.md: the tauri-plugin-sql row grouped runs.db with history.db under SQLCipher, but runs.db is intentionally unencrypted + git-committed (metadata only) on every surface. Scope encryption to history.db. Refs: docs/roadmap/phase-2.5-close-plan.md Step 2 Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/reference/desktop/tauri-plugins.md | 2 +- docs/uvp.md | 2 +- docs/vision.md | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/desktop/tauri-plugins.md b/docs/reference/desktop/tauri-plugins.md index 8301e646..0289c0e2 100644 --- a/docs/reference/desktop/tauri-plugins.md +++ b/docs/reference/desktop/tauri-plugins.md @@ -30,7 +30,7 @@ flowchart LR |--------|---------|---------| | `tauri-plugin-fs` | Scoped filesystem read/write. Path access is validated against a configured scope before any syscall. | Loading/saving `.relavium.yaml` & `.agent.yaml`; the `read_file` / `write_file` / `list_directory` built-in tools | | `tauri-plugin-shell` | Spawning OS child processes from an explicit allowlist. | The `run_command` / `git_status` / `git_commit` built-in tools; stdio MCP server processes | -| `tauri-plugin-sql` | SQLite access, with the **SQLCipher** feature for encryption at rest. | The local run-history database (`history.db`, `runs.db`) — see [database-schema.md](database-schema.md) | +| `tauri-plugin-sql` | SQLite access, with the **SQLCipher** feature for encryption at rest. | The desktop `history.db` (SQLCipher-encrypted at rest); the per-project `runs.db` is intentionally **unencrypted** (git-committed metadata) — see [database-schema.md](database-schema.md) | | `tauri-plugin-keychain` | OS-native secret storage (macOS Keychain / Windows Credential Manager / Linux libsecret). | API-key and DB-passphrase storage — see [keychain-and-secrets.md](keychain-and-secrets.md) | | `tauri-plugin-tray` | System tray icon, menu, and badge. | Active-run monitor, awaiting-gate badge, "New Run" quick menu | | `tauri-plugin-global-shortcut` | OS-level global hotkeys. | Command-palette hotkey (`Cmd/Ctrl+Shift+A`) and "run on selection" (`Cmd/Ctrl+Shift+R`) | diff --git a/docs/uvp.md b/docs/uvp.md index bd19a2ef..b4112600 100644 --- a/docs/uvp.md +++ b/docs/uvp.md @@ -19,7 +19,7 @@ (usage, quota, governance), not a fourth identical-engine runtime. 2. **No chat-driven assistant lets you keep — and ship — the conversation.** Relavium agent sessions are **persistent, resumable, and exportable**: - auto-saved to durable local history (owner-only file permissions; keys stay in + auto-saved to durable local history (kept on your machine; API keys stay in the OS keychain, never at rest), resumable on any surface, and one-click exportable to a reviewable `.relavium.yaml` scaffold. Competitors' sessions evaporate; Relavium's become committable, re-runnable workflows. See diff --git a/docs/vision.md b/docs/vision.md index 730313de..da3fd274 100644 --- a/docs/vision.md +++ b/docs/vision.md @@ -148,8 +148,8 @@ The full positioning and competitor matrix lives in [uvp.md](uvp.md). workflow. See [decisions/0026-session-export-to-workflow.md](decisions/0026-session-export-to-workflow.md). - **Persistent, resumable agent sessions** — every conversation is auto-saved to - durable local history (owner-only `0700`/`0600` file permissions; API keys live in - the OS keychain, never at rest) and resumable on any surface; no run is ever + durable local history (kept on your machine; API keys live in the OS keychain, + never at rest) and resumable on any surface; no run is ever ephemeral. *(The `AgentSession` engine lands in build Phase 1; the first user-facing surface is CLI `relavium chat` in build phase 2 — all within Product Phase 1.)* See From d12b99739d588f12929431e0baaaa09616eae1ec Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 20:31:46 +0300 Subject: [PATCH 08/50] =?UTF-8?q?fix(db):=202.5-close=20Step=203=20?= =?UTF-8?q?=E2=80=94=20loadFull=20reads=20its=20snapshot=20in=20one=20read?= =?UTF-8?q?=20transaction=20(2.5.I=20S1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit session-store.loadFull issued two independent SELECTs (session row, then messages). A concurrent writer -- a second `relavium` process, or a `run` sharing history.db -- committing an append + a session-total update BETWEEN them yields a torn read (a session whose totals do not match the returned transcript). Wrap the two reads in one deferred `db.transaction` so both come from a single WAL snapshot (mirrors the house transaction pattern; a read-only body COMMITs a no-op). Tests (packages/db/src/session-store.test.ts): a two-connection, temp-file snapshot suite -- WAL isolation needs a shared file, since two :memory: handles are separate DBs. One test reproduces loadFull's read pair inside the reader's transaction with a real committed write interleaved between the reads and asserts the snapshot is consistent (the write is invisible; a fresh read after sees it); a control shows the same interleave WITHOUT the transaction tears (stale session + post-write transcript). The write-side BEGIN IMMEDIATE + SQLITE_BUSY retry this pairs with lands in Step 4 (ADR-0064 §5 amendment). Refs: docs/roadmap/phase-2.5-close-plan.md Step 3 Co-Authored-By: Claude Opus 4.8 (1M context) --- packages/db/src/session-store.test.ts | 85 +++++++++++++++++++++++++++ packages/db/src/session-store.ts | 15 +++-- 2 files changed, 96 insertions(+), 4 deletions(-) diff --git a/packages/db/src/session-store.test.ts b/packages/db/src/session-store.test.ts index d75677cb..02067575 100644 --- a/packages/db/src/session-store.test.ts +++ b/packages/db/src/session-store.test.ts @@ -1,3 +1,7 @@ +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + import { AgentSchema, type AgentSessionRecord, type SessionMessage } from '@relavium/shared'; import { eq } from 'drizzle-orm'; import { afterEach, beforeEach, describe, expect, it } from 'vitest'; @@ -461,3 +465,84 @@ describe('agent_sessions CHECK constraints (raw insert bypassing the mapper)', ( }).toThrow(); }); }); + +/** + * 2.5.I — `loadFull` reads its session row and its transcript inside ONE deferred read transaction, so the + * pair is a single consistent snapshot even while another connection (a second `relavium` process, or a + * `run` sharing this `history.db`) commits between the two SELECTs. WAL snapshot isolation needs a shared + * FILE — two `:memory:` connections are separate databases — so these use a temp-file DB with two + * connections and interleave a real committed write mid-read. + */ +describe('SessionStore — loadFull snapshot isolation (2.5.I)', () => { + let dir: string; + let writer: DbClient; + let reader: DbClient; + let writerStore: SessionStore; + let readerStore: SessionStore; + + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'relavium-session-snap-')); + const path = join(dir, 'history.db'); + writer = createClient(path); + runMigrations(writer.db); + reader = createClient(path); // a second connection on the SAME file (WAL shared readers) + writerStore = createSessionStore(writer.db); + readerStore = createSessionStore(reader.db); + }); + + afterEach(() => { + writer.sqlite.close(); + reader.sqlite.close(); + rmSync(dir, { recursive: true, force: true }); + }); + + it("loadFull's read transaction hides a concurrent writer's mid-read commit (no torn read)", () => { + // Session v1 with two messages; totalOutputTokens reflects them. + writerStore.createSession(makeSession({ totalOutputTokens: 2 })); + writerStore.appendMessage(makeMessage(0)); + writerStore.appendMessage(makeMessage(1, { role: 'assistant' })); + + // Reproduce loadFull's structure (read session, then read messages) inside the reader's OWN read + // transaction — exactly what loadFull wraps — and commit an append + a session-total bump on the OTHER + // connection BETWEEN the two reads. The deferred transaction pinned its snapshot at read 1, so read 2 + // must not observe the interleaved write. + const snapshot = reader.db.transaction(() => { + const session = readerStore.loadSession('sess-1'); // read 1 — pins the WAL snapshot + writerStore.appendMessage(makeMessage(2, { role: 'user' })); // concurrent, auto-committed + writerStore.updateSession( + makeSession({ totalOutputTokens: 3, updatedAt: '2026-06-17T09:00:00.000Z' }), + ); + const messages = readerStore.loadMessages('sess-1'); // read 2 — same snapshot + return { session, messages }; + }); + + // Consistent: session v1 (totalOutputTokens 2) paired with exactly its two messages — the mid-read + // append (seq 2) and the total bump are invisible to the pinned snapshot. + expect(snapshot.session?.totalOutputTokens).toBe(2); + expect(snapshot.messages.map((m) => m.sequenceNumber)).toEqual([0, 1]); + + // The write really committed: a fresh loadFull after the transaction sees the bump + the new message. + const after = readerStore.loadFull('sess-1'); + expect(after?.session.totalOutputTokens).toBe(3); + expect(after?.messages.map((m) => m.sequenceNumber)).toEqual([0, 1, 2]); + }); + + it('WITHOUT a read transaction the same interleave tears (why loadFull wraps its reads)', () => { + writerStore.createSession(makeSession({ totalOutputTokens: 2 })); + writerStore.appendMessage(makeMessage(0)); + writerStore.appendMessage(makeMessage(1, { role: 'assistant' })); + + // Two INDEPENDENT reads (no surrounding transaction) with a committed write between them. + const session = readerStore.loadSession('sess-1'); // read 1 — no snapshot held + writerStore.appendMessage(makeMessage(2, { role: 'user' })); + writerStore.updateSession( + makeSession({ totalOutputTokens: 3, updatedAt: '2026-06-17T09:00:00.000Z' }), + ); + const messages = readerStore.loadMessages('sess-1'); // read 2 — sees the post-write transcript + + // Torn: the stale session (totalOutputTokens 2) is paired with the 3-message post-write transcript — + // the mismatch loadFull's transaction prevents. + expect(session?.totalOutputTokens).toBe(2); + expect(messages.map((m) => m.sequenceNumber)).toEqual([0, 1, 2]); + }); +}); diff --git a/packages/db/src/session-store.ts b/packages/db/src/session-store.ts index 3036337c..45a11491 100644 --- a/packages/db/src/session-store.ts +++ b/packages/db/src/session-store.ts @@ -255,9 +255,16 @@ export function createSessionStore(db: Db): SessionStore { db.insert(sessionMessages).values(toSessionMessageRow(message, meta)).run(); }, loadMessages, - loadFull: (sessionId) => { - const session = loadSession(sessionId); - return session === undefined ? undefined : { session, messages: loadMessages(sessionId) }; - }, + loadFull: (sessionId) => + // One read transaction so the session row and its transcript come from a SINGLE consistent snapshot. + // Without it the two SELECTs are independent reads: a concurrent writer — another `relavium` process, or + // a `run` sharing this `history.db` — committing an append + a session-total update BETWEEN them yields a + // torn read (a session whose totals do not match the returned messages). In WAL mode the deferred + // transaction pins one snapshot for both reads (2.5.I). A read-only body COMMITs a no-op. The write-side + // `BEGIN IMMEDIATE` + `SQLITE_BUSY` retry that this pairs with is ADR-0064 §5 (landed alongside). + db.transaction(() => { + const session = loadSession(sessionId); + return session === undefined ? undefined : { session, messages: loadMessages(sessionId) }; + }), }; } From 2d14989e7f6fbb8c4a7a42238399b0442a700adf Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 20:42:18 +0300 Subject: [PATCH 09/50] =?UTF-8?q?fix(db):=202.5-close=20Step=203=20Opus-re?= =?UTF-8?q?view=20fixes=20=E2=80=94=20bind=20the=20regression=20to=20loadF?= =?UTF-8?q?ull=20+=20comment/teardown?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review confirmed the fix correct on every concurrency dimension and surfaced that the regression test guarded SQLite/drizzle snapshot behavior but NOT loadFull's wrapper (removing the wrapper kept both tests green). - Add a coupling test that spies the reader connection's `transaction` and asserts loadFull opens exactly one (single-read methods open zero). Verified it FAILS when the wrapper is reverted, so it now genuinely guards the fix. - session-store.ts: correct the forward-reference comment ("(landed alongside)" was false — the write-side BEGIN IMMEDIATE + retry lands in Step 4; drop the premature §5 pin). - Failure-isolate the temp-file teardown (try/finally) + vi.restoreAllMocks. - deferred-tasks: track the analogous run-resume reconstruction torn-read exposure (lower impact — event-sourced fold self-heals) surfaced in review. Refs: docs/roadmap/phase-2.5-close-plan.md Step 3 Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/roadmap/deferred-tasks.md | 7 ++++++ packages/db/src/session-store.test.ts | 33 +++++++++++++++++++++++---- packages/db/src/session-store.ts | 3 ++- 3 files changed, 38 insertions(+), 5 deletions(-) diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index 0515586e..4214d9bb 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -608,6 +608,13 @@ Severity is the review's verified rating. Check an item off in the PR that resol session/run store until a Phase-3/4 ADR reconciles it. Reword the VS Code at-rest posture once the cross-host physical-store decision lands. *(low · Phase 4 forward-design docs; blocked on the cross-host store decision; surfaced during the 2.5.J encrypted-wording sweep)* +- [ ] **Run-resume reconstruction reads (`loadRun` + `loadRunEvents` + `loadStepExecutions`) are separate + reads at the caller level — the same torn-read class 2.5.I S1 fixed for session `loadFull`.** A concurrent + writer committing between them could yield a run row + event/step reads from different snapshots. Lower + impact than the session case: run history is **event-sourced** and the checkpoint fold tolerates partial + state, so a torn read self-heals on the next fold. If tightened, wrap the caller-level reconstruction in one + read transaction (as `loadFull` now does). *(low · packages/db run-history-store consumers + the resume + caller; surfaced during 2.5.I S1 review)* - [ ] **`relavium run` maps any `run:paused` to exit 3 (gate-paused); revisit when media host-wiring lands.** `run.ts` returns `EXIT_CODES.gatePaused` (3) for any `run:paused`, which is correct in 2.D because a human gate is the **only** `run:paused` source (no `mediaStore`/media-job host is wired, so a media-only diff --git a/packages/db/src/session-store.test.ts b/packages/db/src/session-store.test.ts index 02067575..42a50529 100644 --- a/packages/db/src/session-store.test.ts +++ b/packages/db/src/session-store.test.ts @@ -4,7 +4,7 @@ import { join } from 'node:path'; import { AgentSchema, type AgentSessionRecord, type SessionMessage } from '@relavium/shared'; import { eq } from 'drizzle-orm'; -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { createClient, runMigrations, type DbClient } from './client.js'; import { agentSessions, llmProviders, modelCatalog, sessionMessages } from './schema.js'; @@ -491,9 +491,34 @@ describe('SessionStore — loadFull snapshot isolation (2.5.I)', () => { }); afterEach(() => { - writer.sqlite.close(); - reader.sqlite.close(); - rmSync(dir, { recursive: true, force: true }); + vi.restoreAllMocks(); + // Best-effort teardown: close both connections even if one throws, then remove the temp dir. + try { + writer.sqlite.close(); + } finally { + reader.sqlite.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('loadFull routes its two reads through ONE db.transaction (guards the wrapper is not removed)', () => { + // Bind the regression to the real method: spy the reader connection's `transaction` (in place, so the + // store's captured handle sees it) and assert loadFull opens exactly one — deleting the wrapper from + // loadFull drops this to zero. The default spy calls through, so the reads still run and return. + writerStore.createSession(makeSession()); + writerStore.appendMessage(makeMessage(0)); + const txnSpy = vi.spyOn(reader.db, 'transaction'); + + const full = readerStore.loadFull('sess-1'); + expect(full?.messages.map((m) => m.sequenceNumber)).toEqual([0]); + expect(txnSpy).toHaveBeenCalledTimes(1); + + // Sanity: the single-read methods do NOT open a transaction, so the assertion above is meaningful + // (it is loadFull's wrapper being counted, not an incidental transaction). + txnSpy.mockClear(); + readerStore.loadSession('sess-1'); + readerStore.loadMessages('sess-1'); + expect(txnSpy).not.toHaveBeenCalled(); }); it("loadFull's read transaction hides a concurrent writer's mid-read commit (no torn read)", () => { diff --git a/packages/db/src/session-store.ts b/packages/db/src/session-store.ts index 45a11491..69f73eef 100644 --- a/packages/db/src/session-store.ts +++ b/packages/db/src/session-store.ts @@ -261,7 +261,8 @@ export function createSessionStore(db: Db): SessionStore { // a `run` sharing this `history.db` — committing an append + a session-total update BETWEEN them yields a // torn read (a session whose totals do not match the returned messages). In WAL mode the deferred // transaction pins one snapshot for both reads (2.5.I). A read-only body COMMITs a no-op. The write-side - // `BEGIN IMMEDIATE` + `SQLITE_BUSY` retry that this pairs with is ADR-0064 §5 (landed alongside). + // `BEGIN IMMEDIATE` + `SQLITE_BUSY` retry this pairs with lands in Step 4 (ADR-0064 amendment note — + // DB write-path concurrency). db.transaction(() => { const session = loadSession(sessionId); return session === undefined ? undefined : { session, messages: loadMessages(sessionId) }; From 6eec63ce971d07c9409802485081ccf5250d82d0 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 20:50:15 +0300 Subject: [PATCH 10/50] =?UTF-8?q?test(db):=202.5-close=20Step=203=20Sonnet?= =?UTF-8?q?-review=20fix=20=E2=80=94=20fully=20failure-isolate=20the=20tem?= =?UTF-8?q?p-file=20teardown?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet review mutation-verified the fix + coupling test (both correct) and found the teardown still asymmetric: if reader.close() threw, rmSync never ran and the temp dir leaked. Nest the finally so the dir is swept regardless of which close fails. Refs: docs/roadmap/phase-2.5-close-plan.md Step 3 Co-Authored-By: Claude Opus 4.8 (1M context) --- packages/db/src/session-store.test.ts | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/packages/db/src/session-store.test.ts b/packages/db/src/session-store.test.ts index 42a50529..e7c209cb 100644 --- a/packages/db/src/session-store.test.ts +++ b/packages/db/src/session-store.test.ts @@ -492,12 +492,16 @@ describe('SessionStore — loadFull snapshot isolation (2.5.I)', () => { afterEach(() => { vi.restoreAllMocks(); - // Best-effort teardown: close both connections even if one throws, then remove the temp dir. + // Best-effort teardown: close both connections and remove the temp dir even if a close throws + // (nested finally so the dir is always swept regardless of which close fails). try { writer.sqlite.close(); } finally { - reader.sqlite.close(); - rmSync(dir, { recursive: true, force: true }); + try { + reader.sqlite.close(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } } }); From a60c3621becf0b55ed3c5823959a7f9173429fec Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 21:06:20 +0300 Subject: [PATCH 11/50] =?UTF-8?q?fix(db):=202.5-close=20Step=204=20?= =?UTF-8?q?=E2=80=94=20BEGIN=20IMMEDIATE=20+=20fail-loud=20SQLITE=5FBUSY?= =?UTF-8?q?=20retry=20on=20write=20paths=20(2.5.I=20S2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `history.db` is a single shared file two concurrent `relavium` processes may write (a `run` persisting events while a `chat` refreshes the model catalog — ADR-0064 §5). Harden the write path: - New packages/db/src/retry.ts (`withBusyRetry`): a bounded, FAIL-LOUD SQLITE_BUSY/ SQLITE_LOCKED retry with DETERMINISTIC linear backoff (no jitter, never Math.random, per ADR-0040) via a synchronous Atomics.wait sleep. Rethrows the original error on an exhausted budget or any non-lock fault — never silently drops a write (ADR-0050). - Switch the multi-statement write transactions to BEGIN IMMEDIATE (`{ behavior: 'immediate' }`) so a DEFERRED read→write lock-upgrade race (SQLITE_BUSY_SNAPSHOT, which busy_timeout does NOT cover) cannot occur: persistEvent (run history), replaceProviderModels (the model-catalog bulk live-upsert), and the provider `upsert` read-then-write (previously an unguarded read-then-write). Each is wrapped in withBusyRetry. Single-statement writes rely on SQLite's built-in busy handler. Docs: - ADR-0064: the approved append-only amendment note (DB write-path concurrency), recording the two repo-wide conventions (every write txn IMMEDIATE; every write routes through the retry) as the concrete realization of §5's concurrent-process requirement. No new ADR. - database-schema.md: a new "Concurrency & transaction behavior" section — the one canonical home retry.ts and the store doc-comments cite. Tests (203 pass): retry.test.ts unit cases (deterministic backoff, fail-loud, SQLITE_LOCKED, maxAttempts) + a REAL SQLITE_BUSY contention suite (a held write lock released during the injected backoff; and a never-released lock that fails loud). Plus an IMMEDIATE-coupling test in each of the three store suites (spies db.transaction, asserts `{ behavior: 'immediate' }`) — a revert to DEFERRED fails them. Refs: docs/roadmap/phase-2.5-close-plan.md Step 4; ADR-0064 §5 amendment note Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/decisions/0064-live-model-catalog.md | 25 +++ docs/reference/desktop/database-schema.md | 13 ++ packages/db/src/model-catalog-store.test.ts | 15 +- packages/db/src/model-catalog-store.ts | 193 ++++++++++---------- packages/db/src/provider-store.test.ts | 15 +- packages/db/src/provider-store.ts | 96 +++++----- packages/db/src/retry.test.ts | 186 +++++++++++++++++++ packages/db/src/retry.ts | 82 +++++++++ packages/db/src/run-history-store.test.ts | 14 +- packages/db/src/run-history-store.ts | 14 +- 10 files changed, 511 insertions(+), 142 deletions(-) create mode 100644 packages/db/src/retry.test.ts create mode 100644 packages/db/src/retry.ts diff --git a/docs/decisions/0064-live-model-catalog.md b/docs/decisions/0064-live-model-catalog.md index ff3f24ad..c9a04f75 100644 --- a/docs/decisions/0064-live-model-catalog.md +++ b/docs/decisions/0064-live-model-catalog.md @@ -8,6 +8,31 @@ > **Clarified 2026-07-06 (2.5.G key-awareness — append-only, body unchanged):** §6 availability also requires a **resolvable key**. The merge gains an optional `keyedProviders` input (the providers with a keychain/env key); a model whose provider is **not** keyed is `available: false` with `unavailableReason: 'no-key'`, regardless of live/static presence — because with no key the model is genuinely uncallable (a chat started on it would only fail `provider_auth`), so the `/models` picker dims it (naming the remedy) and makes it **non-selectable**. This **refines, not reverses**, §6's "never everything unavailable" static-presence safe default: that default applies only to a **keyed** provider with no live data (never dimming a whole provider the user can actually use). `keyedProviders` is optional — **absent ⇒ availability is not key-gated**: the `available` BOOLEAN is unchanged from the pre-clarification behavior (the only new output is the additive-optional `unavailableReason`), so only a key-resolving surface (the CLI Home) opts in. The pre-existing "not available on your key" dim (a keyed provider whose live list omits a static model) is now labeled `unavailableReason: 'not-on-key'`. +> **Clarified 2026-07-07 (2.5.I — DB write-path concurrency — append-only, body unchanged):** §5's Negative +> bullet already requires the background refresh to "tolerate two concurrent `relavium` processes racing the DB +> write (WAL + `busy_timeout` already exist)". 2.5.I gives that requirement its concrete, **repo-wide** +> realization across every `history.db` writer, establishing two conventions the next store author follows. +> **(1)** Every write transaction opens with **`BEGIN IMMEDIATE`** (not drizzle's `DEFERRED` default), taking +> the write lock up front to close the read→write lock-upgrade race — applied to `persistEvent` (run history), +> the `replaceProviderModels` bulk live-upsert, and the provider `upsert` read-then-write. **(2)** Every such +> write routes through a bounded, **fail-loud** `SQLITE_BUSY`/`SQLITE_LOCKED` retry helper +> ([retry.ts](../../packages/db/src/retry.ts)) with **deterministic** backoff — **no jitter**, never +> `Math.random`, following the no-jitter/deterministic-replay convention of +> [ADR-0040](0040-node-retry-budget-above-the-chain.md) — that **surfaces** the error (never silently drops a +> write) once the bounded attempt budget is exhausted, preserving +> [ADR-0050](0050-cli-history-db-at-rest-posture.md)'s durability-first `persistEvent` posture. Single-statement +> writes rely on SQLite's built-in busy handler (`busy_timeout`). Symmetrically, `sessionStore.loadFull` reads +> its session row and its messages inside one **read transaction**, for a torn-read-free snapshot. This +> **extends** §5's already-accepted concurrent-process clause: it reverses nothing, adds **no dependency** +> (`retry.ts` is an in-house helper), and changes **no** at-rest/credential posture (ADR-0050 / +> [ADR-0006](0006-os-keychain-for-api-keys.md) / +> [ADR-0036](0036-run-loop-substrate-event-bus-and-execution-host.md) untouched — concurrency is a +> data-integrity/liveness concern, not the credential boundary). The mechanism's one canonical home is the +> "Concurrency & transaction behavior" section of +> [database-schema.md](../reference/desktop/database-schema.md); the `0600`/`0700` guard is a documented Windows +> no-op (ADR-0050), so the 2.5.I test lane gates POSIX-permission assertions off Windows, while the +> `BEGIN IMMEDIATE` + retry mechanism behaves identically cross-OS. + ## Context The model catalog is **static in-code**: `MODEL_PRICING` ([pricing.ts](../../packages/llm/src/pricing.ts)) diff --git a/docs/reference/desktop/database-schema.md b/docs/reference/desktop/database-schema.md index 4496f46c..5ef526ca 100644 --- a/docs/reference/desktop/database-schema.md +++ b/docs/reference/desktop/database-schema.md @@ -485,6 +485,19 @@ CREATE INDEX idx_media_references_handle ON media_references (handle); > Postgres `DISTINCT ON (workflow_id)` for "latest run per workflow" is **not** supported in SQLite. Use `ROW_NUMBER() OVER (PARTITION BY workflow_id ORDER BY created_at DESC)` instead — it works identically in both engines, easing the Phase-2 port. +## Concurrency & transaction behavior + +`history.db` is a **single shared file** two concurrent `relavium` processes may write at once — e.g. a `run` persisting events while a `chat` refreshes the live model catalog ([ADR-0064](../../decisions/0064-live-model-catalog.md) §5). The Node/CLI path (`better-sqlite3`, [ADR-0021](../../decisions/0021-node-sqlite-driver-better-sqlite3.md)) hardens this at the connection and the transaction level; this is the one canonical home for the policy that `packages/db/src/retry.ts` and the store doc-comments cite. + +- **Connection PRAGMAs** ([`client.ts`](../../../packages/db/src/client.ts)): `journal_mode = WAL` (readers never block the single writer, and vice-versa), `busy_timeout = 5000` (SQLite's built-in busy handler waits up to 5 s for a contended lock before returning `SQLITE_BUSY`), `synchronous = NORMAL` (the recommended durability/throughput trade-off under WAL), and `foreign_keys = ON`. +- **Write transactions use `BEGIN IMMEDIATE`**, never drizzle's `DEFERRED` default. A DEFERRED transaction that reads before it writes takes a read lock first and must *upgrade* to a write lock on the first write — if another connection committed in between, that upgrade fails immediately with `SQLITE_BUSY` (`SQLITE_BUSY_SNAPSHOT`), which `busy_timeout` does **not** cover. `BEGIN IMMEDIATE` takes the write lock up front, so the upgrade race cannot occur. Applied to every multi-statement writer: `persistEvent` (run history), `replaceProviderModels` (the model-catalog bulk live-upsert), and the provider `upsert` read-then-write. +- **The bounded retry** (`packages/db/src/retry.ts`, `withBusyRetry`) wraps those write transactions and retries only `SQLITE_BUSY`/`SQLITE_LOCKED` up to a bounded attempt budget (default 5) with a **deterministic** linear backoff — **no jitter**, never `Math.random`, per the no-jitter/deterministic-replay convention of [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md). It is **fail-loud**: on an exhausted budget (or any non-lock fault) it rethrows the original error and never silently drops a write, preserving [ADR-0050](../../decisions/0050-cli-history-db-at-rest-posture.md)'s durability-first `persistEvent` posture. A retried transaction rolls back with no partial write and re-runs the whole (idempotent) body. +- **Single-statement writes** (e.g. `appendMessage`, `setKeychainRef`) go straight for the write lock and rely on SQLite's built-in busy handler (`busy_timeout`); they need no explicit transaction. +- **Reads that must be consistent across statements** use a read transaction: `sessionStore.loadFull` reads the session row and its transcript inside one deferred transaction so the pair is a single WAL snapshot, never a torn read (a session whose totals disagree with its returned messages). +- **Cross-platform:** `BEGIN IMMEDIATE` + the retry behave identically on every OS. The `0600`/`0700` at-rest guard below is a documented Windows no-op, so the concurrency test lane gates POSIX-permission assertions off Windows only. + +This realizes the concurrent-process write requirement recorded in the [ADR-0064](../../decisions/0064-live-model-catalog.md) §5 amendment note (2.5.I). + ## Encryption at rest At-rest encryption of `history.db` is **per-surface**: diff --git a/packages/db/src/model-catalog-store.test.ts b/packages/db/src/model-catalog-store.test.ts index 9633d679..b23318ad 100644 --- a/packages/db/src/model-catalog-store.test.ts +++ b/packages/db/src/model-catalog-store.test.ts @@ -1,5 +1,5 @@ import { eq } from 'drizzle-orm'; -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { createClient, runMigrations, type DbClient } from './client.js'; import { @@ -39,9 +39,22 @@ describe('createModelCatalogStore (2.S — media routing + load-check reader)', }); afterEach(() => { + vi.restoreAllMocks(); client.sqlite.close(); }); + it('replaceProviderModels opens an IMMEDIATE write transaction (2.5.I — the ADR-0064 §5 concurrent-refresh path)', () => { + const txnSpy = vi.spyOn(client.db, 'transaction'); + store.replaceProviderModels( + providerId, + [{ modelId: 'bare-model', displayName: 'Bare' }], + TS_MS, + ); + // The bulk live-upsert reads existing rows then writes — BEGIN IMMEDIATE is what lets two concurrent + // `relavium` refreshes race the DB write safely (ADR-0064 §5); a DEFERRED begin drops this config arg. + expect(txnSpy).toHaveBeenCalledWith(expect.any(Function), { behavior: 'immediate' }); + }); + it('upserts a generative-surface row and reads it back (record shape + parsed capabilities)', () => { const rec = store.upsert({ providerId, diff --git a/packages/db/src/model-catalog-store.ts b/packages/db/src/model-catalog-store.ts index 83e3ee0b..ff437e75 100644 --- a/packages/db/src/model-catalog-store.ts +++ b/packages/db/src/model-catalog-store.ts @@ -7,6 +7,7 @@ import { import { and, asc, eq, isNull, notInArray, sql } from 'drizzle-orm'; import type { Db } from './client.js'; +import { withBusyRetry } from './retry.js'; import { modelCatalog, type ModelCatalogRow, type NewModelCatalogRow } from './schema.js'; /** @@ -471,99 +472,107 @@ export function createModelCatalogStore(db: Db, deps: ModelCatalogStoreDeps): Mo // The transaction RETURNS the tallies so they are observed from WITHIN the serialized write — a concurrent // same-provider refresh can never miscount them (an external before/after `listByProvider` diff would read a // stale `before` and could double-count). drizzle's better-sqlite3 `transaction()` returns the callback value. - db.transaction(() => { - // Only LIVE rows are tallied: `added` on a true INSERT, `updated` on an existing-live-row UPDATE (a non-live - // `static`/`user` row hits the provenance `continue` below and is counted in NEITHER), `deactivated` from the - // soft-deactivate UPDATE's `.changes` (its WHERE is already `source='live'`-scoped) — so the counts carry the - // same LIVE-only intent the write enforces, with no separate source filter needed. - let added = 0; - let updated = 0; - for (const input of rows) { - const displayName = input.displayName.trim() === '' ? input.modelId : input.displayName; - // `0` is the NOT-NULL "unknown" sentinel (ADR-0064 §3) — an absent live limit stores as 0. - const contextWindowTokens = input.contextWindowTokens ?? 0; - const maxOutputTokens = input.maxOutputTokens ?? 0; - // Find the existing (provider, model) row (deletedAt IS NULL — the partial-unique scope), whether it - // is active or soft-deactivated. Reuse its id so FK targets stay stable. - const existing = db - .select() - .from(modelCatalog) - .where( - and( - eq(modelCatalog.providerId, providerId), - eq(modelCatalog.modelId, input.modelId), - isNull(modelCatalog.deletedAt), - ), - ) - .get(); - if (existing !== undefined && existing.source !== 'live') { - // A `source='user'` (user pricing, ADR-0065 §1) or `source='static'` (a media-routing seed — - // media_surface/capabilities/rates) row already represents this model. A live refresh must NEVER - // clobber it (that would drop user pricing or regress media routing), so it is left UNTOUCHED and, - // being non-`live`, is also never deactivated below — the model stays represented by its own row. - // It is counted in neither `added` nor `updated` (provenance-protected — never part of the live delta). - continue; - } - if (existing === undefined) { - const row: NewModelCatalogRow = { - id: deps.uuid(), - providerId, - modelId: input.modelId, - displayName, - contextWindowTokens, - maxOutputTokens, - source: 'live', - lastRefreshedAt: now, - isActive: true, - createdAt: now, - updatedAt: now, - }; - db.insert(modelCatalog).values(row).run(); - added += 1; - } else { - // Reactivate + refresh the existing live row in place (id/created_at/FK refs preserved); only the - // discovery columns + provenance/freshness are written — pricing/media columns are left as-is. - db.update(modelCatalog) - .set({ - displayName, - contextWindowTokens, - maxOutputTokens, - source: 'live', - lastRefreshedAt: now, - isActive: true, - updatedAt: now, - }) - .where(eq(modelCatalog.id, existing.id)) + // `BEGIN IMMEDIATE` — this reads existing rows then writes, so a DEFERRED begin would hit the read→write + // lock-upgrade race — plus `withBusyRetry` for residual cross-process contention (ADR-0064 amendment note). + withBusyRetry(() => + db.transaction( + () => { + // Only LIVE rows are tallied: `added` on a true INSERT, `updated` on an existing-live-row UPDATE (a non-live + // `static`/`user` row hits the provenance `continue` below and is counted in NEITHER), `deactivated` from the + // soft-deactivate UPDATE's `.changes` (its WHERE is already `source='live'`-scoped) — so the counts carry the + // same LIVE-only intent the write enforces, with no separate source filter needed. + let added = 0; + let updated = 0; + for (const input of rows) { + const displayName = + input.displayName.trim() === '' ? input.modelId : input.displayName; + // `0` is the NOT-NULL "unknown" sentinel (ADR-0064 §3) — an absent live limit stores as 0. + const contextWindowTokens = input.contextWindowTokens ?? 0; + const maxOutputTokens = input.maxOutputTokens ?? 0; + // Find the existing (provider, model) row (deletedAt IS NULL — the partial-unique scope), whether it + // is active or soft-deactivated. Reuse its id so FK targets stay stable. + const existing = db + .select() + .from(modelCatalog) + .where( + and( + eq(modelCatalog.providerId, providerId), + eq(modelCatalog.modelId, input.modelId), + isNull(modelCatalog.deletedAt), + ), + ) + .get(); + if (existing !== undefined && existing.source !== 'live') { + // A `source='user'` (user pricing, ADR-0065 §1) or `source='static'` (a media-routing seed — + // media_surface/capabilities/rates) row already represents this model. A live refresh must NEVER + // clobber it (that would drop user pricing or regress media routing), so it is left UNTOUCHED and, + // being non-`live`, is also never deactivated below — the model stays represented by its own row. + // It is counted in neither `added` nor `updated` (provenance-protected — never part of the live delta). + continue; + } + if (existing === undefined) { + const row: NewModelCatalogRow = { + id: deps.uuid(), + providerId, + modelId: input.modelId, + displayName, + contextWindowTokens, + maxOutputTokens, + source: 'live', + lastRefreshedAt: now, + isActive: true, + createdAt: now, + updatedAt: now, + }; + db.insert(modelCatalog).values(row).run(); + added += 1; + } else { + // Reactivate + refresh the existing live row in place (id/created_at/FK refs preserved); only the + // discovery columns + provenance/freshness are written — pricing/media columns are left as-is. + db.update(modelCatalog) + .set({ + displayName, + contextWindowTokens, + maxOutputTokens, + source: 'live', + lastRefreshedAt: now, + isActive: true, + updatedAt: now, + }) + .where(eq(modelCatalog.id, existing.id)) + .run(); + updated += 1; + } + } + // Soft-deactivate the vanished live rows: every currently-active `source='live'` row of THIS provider + // whose model id is absent from the new list. `isActive=false` with `deletedAt` untouched keeps the + // partial-unique slot occupied so a reappearing model reuses the SAME row (reactivated above). NEVER a + // hard-DELETE (FK target from five tables); NEVER touches `source='user'`/`source='static'`. + const incomingModelIds = rows.map((r) => r.modelId); + const deactivateScope = and( + eq(modelCatalog.providerId, providerId), + eq(modelCatalog.isActive, true), + eq(modelCatalog.source, 'live'), + isNull(modelCatalog.deletedAt), + ); + const deactivateResult = db + .update(modelCatalog) + .set({ isActive: false, updatedAt: now }) + // An empty new list deactivates ALL of the provider's live rows (no `notInArray([])` — its semantics + // vary; the guard makes the "everything vanished" case explicit). + .where( + incomingModelIds.length === 0 + ? deactivateScope + : and(deactivateScope, notInArray(modelCatalog.modelId, incomingModelIds)), + ) .run(); - updated += 1; - } - } - // Soft-deactivate the vanished live rows: every currently-active `source='live'` row of THIS provider - // whose model id is absent from the new list. `isActive=false` with `deletedAt` untouched keeps the - // partial-unique slot occupied so a reappearing model reuses the SAME row (reactivated above). NEVER a - // hard-DELETE (FK target from five tables); NEVER touches `source='user'`/`source='static'`. - const incomingModelIds = rows.map((r) => r.modelId); - const deactivateScope = and( - eq(modelCatalog.providerId, providerId), - eq(modelCatalog.isActive, true), - eq(modelCatalog.source, 'live'), - isNull(modelCatalog.deletedAt), - ); - const deactivateResult = db - .update(modelCatalog) - .set({ isActive: false, updatedAt: now }) - // An empty new list deactivates ALL of the provider's live rows (no `notInArray([])` — its semantics - // vary; the guard makes the "everything vanished" case explicit). - .where( - incomingModelIds.length === 0 - ? deactivateScope - : and(deactivateScope, notInArray(modelCatalog.modelId, incomingModelIds)), - ) - .run(); - // better-sqlite3's `RunResult.changes` = the rows the UPDATE matched (each flips isActive true→false, so - // every matched row is genuinely modified) = the number of live rows soft-deactivated this refresh. - return { added, updated, deactivated: deactivateResult.changes }; - }), + // better-sqlite3's `RunResult.changes` = the rows the UPDATE matched (each flips isActive true→false, so + // every matched row is genuinely modified) = the number of live rows soft-deactivated this refresh. + return { added, updated, deactivated: deactivateResult.changes }; + }, + { behavior: 'immediate' }, + ), + ), providerRefreshedAt: (providerId) => { const row = db diff --git a/packages/db/src/provider-store.test.ts b/packages/db/src/provider-store.test.ts index 05811293..772ec91e 100644 --- a/packages/db/src/provider-store.test.ts +++ b/packages/db/src/provider-store.test.ts @@ -1,5 +1,5 @@ import { eq, sql } from 'drizzle-orm'; -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { createClient, runMigrations, type DbClient } from './client.js'; import { llmProviders } from './schema.js'; @@ -23,9 +23,22 @@ describe('createProviderStore', () => { }); afterEach(() => { + vi.restoreAllMocks(); client.sqlite.close(); }); + it('upsert opens an IMMEDIATE write transaction (2.5.I — read-then-write serialized, no lock-upgrade race)', () => { + const txnSpy = vi.spyOn(client.db, 'transaction'); + store.upsert({ + name: 'anthropic', + displayName: 'Anthropic', + baseUrl: 'https://api.anthropic.com', + }); + // The read (`activeRow`) then write (insert/update) run under one BEGIN IMMEDIATE; a DEFERRED begin would + // drop this config arg (and reintroduce the read→write upgrade race two concurrent writers could hit). + expect(txnSpy).toHaveBeenCalledWith(expect.any(Function), { behavior: 'immediate' }); + }); + it('upserts a provider row and reads it back', () => { const rec = store.upsert({ name: 'anthropic', diff --git a/packages/db/src/provider-store.ts b/packages/db/src/provider-store.ts index 197d4614..8f2ffe22 100644 --- a/packages/db/src/provider-store.ts +++ b/packages/db/src/provider-store.ts @@ -2,6 +2,7 @@ import { PROVIDER_KINDS, type ProviderKind } from '@relavium/shared'; import { and, asc, eq, isNull } from 'drizzle-orm'; import type { Db } from './client.js'; +import { withBusyRetry } from './retry.js'; import { llmProviders, type LlmProviderRow, type NewLlmProviderRow } from './schema.js'; import { epochMsToIso } from './time.js'; @@ -137,48 +138,59 @@ export function createProviderStore(db: Db, deps: ProviderStoreDeps): ProviderSt upsert: (input) => { const t = deps.now(); - const existing = activeRow(input.name); - if (existing === undefined) { - const row: NewLlmProviderRow = { - id: deps.uuid(), - name: input.name, - displayName: input.displayName, - baseUrl: input.baseUrl, - defaultHeaders: JSON.stringify(input.defaultHeaders ?? {}), - ...(input.kind === undefined ? {} : { kind: input.kind }), - ...(input.pricingReferenceUrl === undefined - ? {} - : { pricingReferenceUrl: input.pricingReferenceUrl }), - createdAt: t, - updatedAt: t, - }; - db.insert(llmProviders).values(row).run(); - } else { - db.update(llmProviders) - .set({ - displayName: input.displayName, - baseUrl: input.baseUrl, - // `existing.defaultHeaders` is already the stored JSON STRING — keep it verbatim when the caller - // supplies none; stringify only a fresh value. (Re-stringifying the string would double-encode it.) - defaultHeaders: - input.defaultHeaders === undefined - ? existing.defaultHeaders - : JSON.stringify(input.defaultHeaders), - // Preserve an existing kind / pricing URL when the caller omits it (like defaultHeaders). - ...(input.kind === undefined ? {} : { kind: input.kind }), - ...(input.pricingReferenceUrl === undefined - ? {} - : { pricingReferenceUrl: input.pricingReferenceUrl }), - updatedAt: t, - }) - .where(eq(llmProviders.id, existing.id)) - .run(); - } - const row = activeRow(input.name); - if (row === undefined) { - throw new Error(`provider '${input.name}' not found after upsert`); // unreachable — just inserted/updated - } - return fromRow(row); + // Read-then-write upsert under ONE IMMEDIATE transaction: two concurrent `provider add` (or a refresh) + // writers must not both read `existing === undefined` and double-insert, nor race a DEFERRED read→write + // lock upgrade. `BEGIN IMMEDIATE` serializes them; `withBusyRetry` waits out residual cross-process + // contention, fail-loud (ADR-0064 amendment note — DB write-path concurrency). + return withBusyRetry(() => + db.transaction( + () => { + const existing = activeRow(input.name); + if (existing === undefined) { + const row: NewLlmProviderRow = { + id: deps.uuid(), + name: input.name, + displayName: input.displayName, + baseUrl: input.baseUrl, + defaultHeaders: JSON.stringify(input.defaultHeaders ?? {}), + ...(input.kind === undefined ? {} : { kind: input.kind }), + ...(input.pricingReferenceUrl === undefined + ? {} + : { pricingReferenceUrl: input.pricingReferenceUrl }), + createdAt: t, + updatedAt: t, + }; + db.insert(llmProviders).values(row).run(); + } else { + db.update(llmProviders) + .set({ + displayName: input.displayName, + baseUrl: input.baseUrl, + // `existing.defaultHeaders` is already the stored JSON STRING — keep it verbatim when the caller + // supplies none; stringify only a fresh value. (Re-stringifying the string would double-encode it.) + defaultHeaders: + input.defaultHeaders === undefined + ? existing.defaultHeaders + : JSON.stringify(input.defaultHeaders), + // Preserve an existing kind / pricing URL when the caller omits it (like defaultHeaders). + ...(input.kind === undefined ? {} : { kind: input.kind }), + ...(input.pricingReferenceUrl === undefined + ? {} + : { pricingReferenceUrl: input.pricingReferenceUrl }), + updatedAt: t, + }) + .where(eq(llmProviders.id, existing.id)) + .run(); + } + const row = activeRow(input.name); + if (row === undefined) { + throw new Error(`provider '${input.name}' not found after upsert`); // unreachable — just inserted/updated + } + return fromRow(row); + }, + { behavior: 'immediate' }, + ), + ); }, setKeychainRef: (name, ref) => { diff --git a/packages/db/src/retry.test.ts b/packages/db/src/retry.test.ts new file mode 100644 index 00000000..ee864fb0 --- /dev/null +++ b/packages/db/src/retry.test.ts @@ -0,0 +1,186 @@ +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { describe, expect, it, vi } from 'vitest'; + +import { createClient, runMigrations } from './client.js'; +import { withBusyRetry } from './retry.js'; + +/** A `better-sqlite3`-shaped lock error: an `Error` with the string `.code` the driver sets. */ +const lockError = (code: 'SQLITE_BUSY' | 'SQLITE_LOCKED'): Error => + Object.assign(new Error('database is locked'), { code }); + +describe('withBusyRetry — unit (2.5.I)', () => { + it('returns the value on first success (no retry, no sleep)', () => { + const sleep = vi.fn(); + expect(withBusyRetry(() => 42, { sleep })).toBe(42); + expect(sleep).not.toHaveBeenCalled(); + }); + + it('retries SQLITE_BUSY then succeeds, with a deterministic linear backoff (no jitter)', () => { + const sleeps: number[] = []; + let calls = 0; + const result = withBusyRetry( + () => { + calls += 1; + if (calls < 3) throw lockError('SQLITE_BUSY'); + return 'ok'; + }, + { baseDelayMs: 25, sleep: (ms) => sleeps.push(ms) }, + ); + expect(result).toBe('ok'); + expect(calls).toBe(3); + // 25×1 then 25×2 — linear, deterministic, no jitter (ADR-0040 convention). + expect(sleeps).toEqual([25, 50]); + }); + + it('retries SQLITE_LOCKED as well', () => { + let calls = 0; + const result = withBusyRetry( + () => { + calls += 1; + if (calls < 2) throw lockError('SQLITE_LOCKED'); + return 'ok'; + }, + { sleep: () => {} }, + ); + expect(result).toBe('ok'); + expect(calls).toBe(2); + }); + + it('rethrows a NON-lock error immediately, unchanged (no retry, no sleep)', () => { + const sleep = vi.fn(); + const constraint = Object.assign(new Error('constraint failed'), { code: 'SQLITE_CONSTRAINT' }); + let calls = 0; + let caught: unknown; + try { + withBusyRetry( + () => { + calls += 1; + throw constraint; + }, + { sleep }, + ); + } catch (err) { + caught = err; + } + expect(caught).toBe(constraint); // the ORIGINAL error object, unwrapped + expect(calls).toBe(1); + expect(sleep).not.toHaveBeenCalled(); + }); + + it('fails loud after exhausting maxAttempts, rethrowing the ORIGINAL lock error', () => { + const busy = lockError('SQLITE_BUSY'); + let calls = 0; + let caught: unknown; + try { + withBusyRetry( + () => { + calls += 1; + throw busy; + }, + { maxAttempts: 4, sleep: () => {} }, + ); + } catch (err) { + caught = err; + } + expect(caught).toBe(busy); // never swallowed — a dropped write is silent data loss (ADR-0050) + expect(calls).toBe(4); // the full budget was spent + }); + + it('maxAttempts:1 makes a single attempt (retry disabled)', () => { + let calls = 0; + expect(() => + withBusyRetry( + () => { + calls += 1; + throw lockError('SQLITE_BUSY'); + }, + { maxAttempts: 1, sleep: () => {} }, + ), + ).toThrow(); + expect(calls).toBe(1); + }); +}); + +/** + * Real SQLITE_BUSY contention: two connections on one file, one holding the single WAL write lock. The + * injected `sleep` is the interleave hook — releasing the lock during the backoff lets the retry succeed, + * so the test is deterministic without any real waiting or threads. + */ +describe('withBusyRetry — real SQLITE_BUSY contention (2.5.I)', () => { + it('retries a write blocked by a held write lock; succeeds once the lock is released in the backoff', () => { + const dir = mkdtempSync(join(tmpdir(), 'relavium-retry-')); + const holder = createClient(join(dir, 'c.db')); + const writer = createClient(join(dir, 'c.db')); + try { + runMigrations(holder.db); + holder.sqlite.exec('CREATE TABLE contention (x INTEGER)'); // auto-committed → visible to writer + writer.sqlite.pragma('busy_timeout = 0'); // surface BUSY immediately instead of waiting 5s + const write = writer.sqlite.prepare('INSERT INTO contention (x) VALUES (2)'); + + // holder takes the single WAL write lock and keeps it: + holder.sqlite.exec('BEGIN IMMEDIATE'); + holder.sqlite.prepare('INSERT INTO contention (x) VALUES (1)').run(); + + let attempts = 0; + let released = false; + withBusyRetry( + () => { + attempts += 1; + write.run(); // throws SQLITE_BUSY while holder owns the lock + }, + { + baseDelayMs: 1, + sleep: () => { + if (!released) { + holder.sqlite.exec('COMMIT'); // release the lock so the retry can take it + released = true; + } + }, + }, + ); + + expect(attempts).toBe(2); // first attempt BUSY, retry after release succeeds + expect(Number(writer.sqlite.prepare('SELECT count(*) FROM contention').pluck().get())).toBe( + 2, + ); + } finally { + holder.sqlite.close(); + writer.sqlite.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('fails loud when the lock is never released (exhausts the budget, rethrows SQLITE_BUSY)', () => { + const dir = mkdtempSync(join(tmpdir(), 'relavium-retry-')); + const holder = createClient(join(dir, 'c.db')); + const writer = createClient(join(dir, 'c.db')); + try { + runMigrations(holder.db); + holder.sqlite.exec('CREATE TABLE contention (x INTEGER)'); + writer.sqlite.pragma('busy_timeout = 0'); + const write = writer.sqlite.prepare('INSERT INTO contention (x) VALUES (2)'); + holder.sqlite.exec('BEGIN IMMEDIATE'); + holder.sqlite.prepare('INSERT INTO contention (x) VALUES (1)').run(); + + let attempts = 0; + expect(() => + withBusyRetry( + () => { + attempts += 1; + write.run(); + }, + { maxAttempts: 3, sleep: () => {} }, // the lock is never released → every attempt is BUSY + ), + ).toThrow(/SQLITE_BUSY|database is locked/); + expect(attempts).toBe(3); // the whole budget was spent before failing loud + holder.sqlite.exec('COMMIT'); // release for teardown + } finally { + holder.sqlite.close(); + writer.sqlite.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/packages/db/src/retry.ts b/packages/db/src/retry.ts new file mode 100644 index 00000000..568ac5d1 --- /dev/null +++ b/packages/db/src/retry.ts @@ -0,0 +1,82 @@ +/** + * Bounded, **fail-loud** `SQLITE_BUSY` / `SQLITE_LOCKED` retry for the write path (2.5.I). + * + * `history.db` is opened with `busy_timeout = 5000` ([client.ts](./client.ts)), so SQLite's own busy handler + * already waits out most lock contention *inside* a single statement or `BEGIN`. This helper covers the + * residual: a write that still surfaces `SQLITE_BUSY`/`SQLITE_LOCKED` after that wait (heavy multi-process + * contention, or the `SQLITE_BUSY_SNAPSHOT` a stale deferred writer hits — which `BEGIN IMMEDIATE` avoids and + * this retries if it ever escapes). It is the concrete realization of the concurrent-process write requirement + * recorded in the **ADR-0064 amendment note** (DB write-path concurrency): every multi-statement write + * transaction opens `BEGIN IMMEDIATE` **and** routes through this policy. + * + * The backoff is **deterministic — no jitter, never `Math.random`** — following the no-jitter / deterministic + * convention of [ADR-0040 §backoff](../../../docs/decisions/0040-node-retry-budget-above-the-chain.md): a + * linear `baseDelayMs × attempt` pause between attempts (1×, 2×, … up to `maxAttempts − 1`). On an exhausted + * budget — or any non-lock fault — it **rethrows the original error**; it never silently drops a write + * (ADR-0050's durability-first `persistEvent` posture: a swallowed write is silent data loss). + * + * The wrapped `fn` MUST be re-runnable: a lock fault rolls the transaction back with no partial write, and the + * retry re-runs the whole `fn` from scratch. Both engine writers (`persistEvent`'s fold, the model-catalog + * `replaceProviderModels` upsert, the provider `upsert`) are idempotent given the same input + DB state. + */ + +/** Driver error codes we wait out: a lock we can retry. Anything else is a real fault → rethrow. */ +const RETRYABLE_CODES: ReadonlySet = new Set(['SQLITE_BUSY', 'SQLITE_LOCKED']); + +/** Default total attempts (the first try + up to 4 retries). */ +const DEFAULT_MAX_ATTEMPTS = 5; +/** Default linear-backoff base; the nth retry sleeps `base × n` ms. */ +const DEFAULT_BASE_DELAY_MS = 25; + +export interface BusyRetryOptions { + /** Total attempts INCLUDING the first (default {@link DEFAULT_MAX_ATTEMPTS}). Must be ≥ 1. */ + readonly maxAttempts?: number; + /** Linear-backoff base in ms (default {@link DEFAULT_BASE_DELAY_MS}); the nth retry sleeps `base × n`. */ + readonly baseDelayMs?: number; + /** Injectable synchronous sleep — tests pass a no-op/recorder so they never actually block. */ + readonly sleep?: (ms: number) => void; +} + +/** A `better-sqlite3` `SqliteError` carries a string `.code` (e.g. `SQLITE_BUSY`); match structurally. */ +function isRetryableLockError(err: unknown): boolean { + return ( + typeof err === 'object' && + err !== null && + 'code' in err && + typeof err.code === 'string' && + RETRYABLE_CODES.has(err.code) + ); +} + +/** + * Synchronous sleep via `Atomics.wait` on a throwaway `SharedArrayBuffer` — `better-sqlite3` is synchronous, + * so the backoff must be too. This parks the thread for `ms` (nothing ever notifies the location, so it always + * times out) without a busy-loop, and is deterministic. `Atomics.wait` is permitted on Node's main thread. + */ +function sleepSync(ms: number): void { + if (ms <= 0) return; + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); +} + +/** + * Run `fn`, retrying only on `SQLITE_BUSY`/`SQLITE_LOCKED` up to `maxAttempts`, with a deterministic linear + * backoff between attempts. Returns `fn`'s value on success; rethrows the original error on a non-lock fault + * or an exhausted budget (fail-loud). Synchronous — it wraps a synchronous `db.transaction(...)` call. + */ +export function withBusyRetry(fn: () => T, options: BusyRetryOptions = {}): T { + const maxAttempts = options.maxAttempts ?? DEFAULT_MAX_ATTEMPTS; + const baseDelayMs = options.baseDelayMs ?? DEFAULT_BASE_DELAY_MS; + const sleep = options.sleep ?? sleepSync; + for (let attempt = 1; ; attempt += 1) { + try { + return fn(); + } catch (err) { + // Fail loud: the last attempt, or any non-lock fault, rethrows the ORIGINAL error unchanged. + if (attempt >= maxAttempts || !isRetryableLockError(err)) { + throw err; + } + // Deterministic linear backoff (no jitter): let the contending writer commit before we re-take the lock. + sleep(baseDelayMs * attempt); + } + } +} diff --git a/packages/db/src/run-history-store.test.ts b/packages/db/src/run-history-store.test.ts index 25464593..5ac81049 100644 --- a/packages/db/src/run-history-store.test.ts +++ b/packages/db/src/run-history-store.test.ts @@ -1,6 +1,6 @@ import { RunEventSchema, type RunEvent } from '@relavium/shared'; import { eq } from 'drizzle-orm'; -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { createClient, runMigrations, type DbClient } from './client.js'; import { runCosts, runEvents, runs, stepExecutions, workflows } from './schema.js'; @@ -67,9 +67,21 @@ describe('createRunHistoryStore', () => { }); afterEach(() => { + vi.restoreAllMocks(); client.sqlite.close(); }); + it('persistEvent opens an IMMEDIATE write transaction (2.5.I — guards against a DEFERRED regression)', async () => { + const workflowId = await store.resolveWorkflowId('demo'); + const txnSpy = vi.spyOn(client.db, 'transaction'); + await store.persistEvent( + ev('run:started', 0, { workflowId, inputs: { n: 3 }, executionMode: 'local' }), + ); + // BEGIN IMMEDIATE takes the write lock up front (no DEFERRED read→write upgrade race); reverting to the + // default deferred transaction drops the config arg and fails this assertion. + expect(txnSpy).toHaveBeenCalledWith(expect.any(Function), { behavior: 'immediate' }); + }); + /** Resolve the workflow + persist a `run:started` so a `runs` row (FK target) exists. Returns the workflow UUID. */ async function startRun(): Promise { const workflowId = await store.resolveWorkflowId('demo'); diff --git a/packages/db/src/run-history-store.ts b/packages/db/src/run-history-store.ts index bb3913ca..35933197 100644 --- a/packages/db/src/run-history-store.ts +++ b/packages/db/src/run-history-store.ts @@ -7,6 +7,7 @@ import { import { and, asc, desc, eq, getTableColumns, inArray, isNull, notInArray, sql } from 'drizzle-orm'; import type { Db } from './client.js'; +import { withBusyRetry } from './retry.js'; import { runCosts, runEvents, @@ -436,11 +437,14 @@ export function createRunHistoryStore(db: Db, deps: RunHistoryStoreDeps): RunHis throw new Error(`run-history store received a non-run event: ${parsed.type}`); } const ts = isoToEpochMs(parsed.timestamp); - // One transaction per event: the run_events append and its derived rows land atomically, so a crash - // can never leave a derived row without its event (or vice-versa). - db.transaction(() => { - fold(parsed, runId, ts); - }); + // One IMMEDIATE transaction per event: the run_events append and its derived rows land atomically, so a + // crash can never leave a derived row without its event (or vice-versa). `BEGIN IMMEDIATE` takes the + // write lock up front (never a DEFERRED read→write upgrade race), and `withBusyRetry` waits out residual + // cross-process lock contention — fail-loud, so a swallowed write can never be silent data loss + // (ADR-0050; the ADR-0064 amendment note — DB write-path concurrency). + withBusyRetry(() => + db.transaction(() => fold(parsed, runId, ts), { behavior: 'immediate' }), + ); return Promise.resolve(); } catch (error) { return Promise.reject(error instanceof Error ? error : new Error(String(error))); From 7b7d9ff933ea43530a7101484c1972be5bcd0fb2 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 21:20:10 +0300 Subject: [PATCH 12/50] =?UTF-8?q?fix(db):=202.5-close=20Step=204=20Opus-re?= =?UTF-8?q?view=20fixes=20=E2=80=94=204th=20write=20path=20+=20doc=20accur?= =?UTF-8?q?acy=20+=20retry=20hardening?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review confirmed the mechanism correct and surfaced a missed writer + doc overclaims: - HIGH: model-catalog `upsert` (the `relavium models pricing` read-then-write) was a FOURTH unguarded multi-statement writer that can race `models refresh` on the same (provider, model) — a lost update or a UNIQUE crash. Wrap it in the same withBusyRetry + BEGIN IMMEDIATE, and add its IMMEDIATE-coupling test. Update the database-schema.md + ADR-0064 enumerations to list all four writers (the docs had claimed "every multi-statement writer" while covering three). - MEDIUM: the database-schema.md loadFull bullet overclaimed. loadFull's read-txn guarantees SNAPSHOT consistency, not TURN atomicity — the CLI persister writes a turn's messages + totals as separate auto-committed statements, so a snapshot can observe messages ahead of their totals. Reword to what is actually guaranteed + track the non-atomic turn persister as a follow-up (deferred-tasks). - LOW: document the ~25s compounded worst-case block (5 attempts × 5s busy_timeout); floor maxAttempts at 1 (Math.max) so a stray 0/negative can't disable the first try; add a full-default-budget backoff-sequence test [25,50,75,100] pinning base×attempt. - retry.ts: caveat that BEGIN IMMEDIATE applies only to the outermost txn (a nested call demotes to SAVEPOINT); mirror it in database-schema.md. - NIT: failure-isolate the two contention-test teardowns (nested finally). Full db gate green; 207 tests pass. Refs: docs/roadmap/phase-2.5-close-plan.md Step 4 Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/decisions/0064-live-model-catalog.md | 3 +- docs/reference/desktop/database-schema.md | 6 +- docs/roadmap/deferred-tasks.md | 8 + packages/db/src/model-catalog-store.test.ts | 8 + packages/db/src/model-catalog-store.ts | 196 +++++++++++--------- packages/db/src/retry.test.ts | 43 ++++- packages/db/src/retry.ts | 13 +- 7 files changed, 173 insertions(+), 104 deletions(-) diff --git a/docs/decisions/0064-live-model-catalog.md b/docs/decisions/0064-live-model-catalog.md index c9a04f75..3472ae10 100644 --- a/docs/decisions/0064-live-model-catalog.md +++ b/docs/decisions/0064-live-model-catalog.md @@ -14,7 +14,8 @@ > realization across every `history.db` writer, establishing two conventions the next store author follows. > **(1)** Every write transaction opens with **`BEGIN IMMEDIATE`** (not drizzle's `DEFERRED` default), taking > the write lock up front to close the read→write lock-upgrade race — applied to `persistEvent` (run history), -> the `replaceProviderModels` bulk live-upsert, and the provider `upsert` read-then-write. **(2)** Every such +> the model-catalog `replaceProviderModels` bulk live-upsert and its per-model `upsert`, and the provider +> `upsert` read-then-write. **(2)** Every such > write routes through a bounded, **fail-loud** `SQLITE_BUSY`/`SQLITE_LOCKED` retry helper > ([retry.ts](../../packages/db/src/retry.ts)) with **deterministic** backoff — **no jitter**, never > `Math.random`, following the no-jitter/deterministic-replay convention of diff --git a/docs/reference/desktop/database-schema.md b/docs/reference/desktop/database-schema.md index 5ef526ca..06b5f65f 100644 --- a/docs/reference/desktop/database-schema.md +++ b/docs/reference/desktop/database-schema.md @@ -490,10 +490,10 @@ CREATE INDEX idx_media_references_handle ON media_references (handle); `history.db` is a **single shared file** two concurrent `relavium` processes may write at once — e.g. a `run` persisting events while a `chat` refreshes the live model catalog ([ADR-0064](../../decisions/0064-live-model-catalog.md) §5). The Node/CLI path (`better-sqlite3`, [ADR-0021](../../decisions/0021-node-sqlite-driver-better-sqlite3.md)) hardens this at the connection and the transaction level; this is the one canonical home for the policy that `packages/db/src/retry.ts` and the store doc-comments cite. - **Connection PRAGMAs** ([`client.ts`](../../../packages/db/src/client.ts)): `journal_mode = WAL` (readers never block the single writer, and vice-versa), `busy_timeout = 5000` (SQLite's built-in busy handler waits up to 5 s for a contended lock before returning `SQLITE_BUSY`), `synchronous = NORMAL` (the recommended durability/throughput trade-off under WAL), and `foreign_keys = ON`. -- **Write transactions use `BEGIN IMMEDIATE`**, never drizzle's `DEFERRED` default. A DEFERRED transaction that reads before it writes takes a read lock first and must *upgrade* to a write lock on the first write — if another connection committed in between, that upgrade fails immediately with `SQLITE_BUSY` (`SQLITE_BUSY_SNAPSHOT`), which `busy_timeout` does **not** cover. `BEGIN IMMEDIATE` takes the write lock up front, so the upgrade race cannot occur. Applied to every multi-statement writer: `persistEvent` (run history), `replaceProviderModels` (the model-catalog bulk live-upsert), and the provider `upsert` read-then-write. -- **The bounded retry** (`packages/db/src/retry.ts`, `withBusyRetry`) wraps those write transactions and retries only `SQLITE_BUSY`/`SQLITE_LOCKED` up to a bounded attempt budget (default 5) with a **deterministic** linear backoff — **no jitter**, never `Math.random`, per the no-jitter/deterministic-replay convention of [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md). It is **fail-loud**: on an exhausted budget (or any non-lock fault) it rethrows the original error and never silently drops a write, preserving [ADR-0050](../../decisions/0050-cli-history-db-at-rest-posture.md)'s durability-first `persistEvent` posture. A retried transaction rolls back with no partial write and re-runs the whole (idempotent) body. +- **Write transactions use `BEGIN IMMEDIATE`**, never drizzle's `DEFERRED` default. A DEFERRED transaction that reads before it writes takes a read lock first and must *upgrade* to a write lock on the first write — if another connection committed in between, that upgrade fails immediately with `SQLITE_BUSY` (`SQLITE_BUSY_SNAPSHOT`), which `busy_timeout` does **not** cover. `BEGIN IMMEDIATE` takes the write lock up front, so the upgrade race cannot occur. Applied to every multi-statement writer: `persistEvent` (run history), the model-catalog `replaceProviderModels` (bulk live-upsert) and `upsert` (per-model pricing), and the provider `upsert` read-then-write. It applies only to the OUTERMOST `BEGIN` — a store method called inside another transaction is demoted to a `SAVEPOINT` and the IMMEDIATE behavior is ignored, so a future batch-in-one-transaction caller must itself open `BEGIN IMMEDIATE`. +- **The bounded retry** (`packages/db/src/retry.ts`, `withBusyRetry`) wraps those write transactions and retries only `SQLITE_BUSY`/`SQLITE_LOCKED` up to a bounded attempt budget (default 5) with a **deterministic** linear backoff — **no jitter**, never `Math.random`, per the no-jitter/deterministic-replay convention of [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md). It is **fail-loud**: on an exhausted budget (or any non-lock fault) it rethrows the original error and never silently drops a write, preserving [ADR-0050](../../decisions/0050-cli-history-db-at-rest-posture.md)'s durability-first `persistEvent` posture. A retried transaction rolls back with no partial write and re-runs the whole (idempotent) body. Because each attempt's `BEGIN IMMEDIATE` can itself wait up to `busy_timeout` (5 s) for the lock, the compounded worst case under sustained contention is ~25 s (5 attempts × 5 s + the sub-300 ms backoffs) of a synchronous block before the fail-loud rethrow — a deliberate durability-over-latency trade on a path that only stalls under pathological multi-writer contention. - **Single-statement writes** (e.g. `appendMessage`, `setKeychainRef`) go straight for the write lock and rely on SQLite's built-in busy handler (`busy_timeout`); they need no explicit transaction. -- **Reads that must be consistent across statements** use a read transaction: `sessionStore.loadFull` reads the session row and its transcript inside one deferred transaction so the pair is a single WAL snapshot, never a torn read (a session whose totals disagree with its returned messages). +- **Reads that must be consistent across statements** use a read transaction: `sessionStore.loadFull` reads the session row and its transcript inside one deferred transaction so **both reads observe a single consistent DB snapshot** (never a two-`SELECT` straddle across a concurrent commit). Note this guarantees *snapshot* consistency, not *turn* atomicity: the CLI persister writes a turn's messages and its updated session totals as separate auto-committed statements, so a snapshot can still legitimately observe messages ahead of their totals. A "totals always match the returned messages" guarantee would additionally require the host to persist each turn in one transaction (a tracked follow-up). - **Cross-platform:** `BEGIN IMMEDIATE` + the retry behave identically on every OS. The `0600`/`0700` at-rest guard below is a documented Windows no-op, so the concurrency test lane gates POSIX-permission assertions off Windows only. This realizes the concurrent-process write requirement recorded in the [ADR-0064](../../decisions/0064-live-model-catalog.md) §5 amendment note (2.5.I). diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index 4214d9bb..0692cd1a 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -615,6 +615,14 @@ Severity is the review's verified rating. Check an item off in the PR that resol state, so a torn read self-heals on the next fold. If tightened, wrap the caller-level reconstruction in one read transaction (as `loadFull` now does). *(low · packages/db run-history-store consumers + the resume caller; surfaced during 2.5.I S1 review)* +- [ ] **The CLI chat persister writes a turn non-atomically — messages then session totals in separate + auto-committed statements.** `apps/cli/src/chat/persister.ts` appends the user + assistant messages and then + `updateSession`s the running totals as separate writes, so the DB legitimately passes through a state where a + turn's messages are present but its totals are stale. `sessionStore.loadFull`'s read transaction (2.5.I S1) + guarantees *snapshot* consistency (both reads see one DB snapshot) but not *turn* atomicity — a snapshot can + observe messages ahead of their totals. To make "totals always match the returned messages" hold, wrap each + turn's message-appends + `updateSession` in one host-side `db.transaction` (BEGIN IMMEDIATE). Bounded, host-side. + *(low · apps/cli/src/chat/persister.ts; surfaced during 2.5.I S2 review)* - [ ] **`relavium run` maps any `run:paused` to exit 3 (gate-paused); revisit when media host-wiring lands.** `run.ts` returns `EXIT_CODES.gatePaused` (3) for any `run:paused`, which is correct in 2.D because a human gate is the **only** `run:paused` source (no `mediaStore`/media-job host is wired, so a media-only diff --git a/packages/db/src/model-catalog-store.test.ts b/packages/db/src/model-catalog-store.test.ts index b23318ad..2a09da96 100644 --- a/packages/db/src/model-catalog-store.test.ts +++ b/packages/db/src/model-catalog-store.test.ts @@ -55,6 +55,14 @@ describe('createModelCatalogStore (2.S — media routing + load-check reader)', expect(txnSpy).toHaveBeenCalledWith(expect.any(Function), { behavior: 'immediate' }); }); + it('upsert opens an IMMEDIATE write transaction (2.5.I — the models-pricing read-then-write path)', () => { + const txnSpy = vi.spyOn(client.db, 'transaction'); + store.upsert({ providerId, modelId: 'gpt-4o', displayName: 'GPT-4o' }); + // The `models pricing` read-then-write must serialize against a concurrent `models refresh` on the same + // (provider, model); a DEFERRED begin drops this config arg (and reopens the lost-update / UNIQUE race). + expect(txnSpy).toHaveBeenCalledWith(expect.any(Function), { behavior: 'immediate' }); + }); + it('upserts a generative-surface row and reads it back (record shape + parsed capabilities)', () => { const rec = store.upsert({ providerId, diff --git a/packages/db/src/model-catalog-store.ts b/packages/db/src/model-catalog-store.ts index ff437e75..139351db 100644 --- a/packages/db/src/model-catalog-store.ts +++ b/packages/db/src/model-catalog-store.ts @@ -349,97 +349,111 @@ export function createModelCatalogStore(db: Db, deps: ModelCatalogStoreDeps): Mo upsert: (input) => { const t = deps.now(); - const existing = db - .select() - .from(modelCatalog) - .where( - and( - eq(modelCatalog.providerId, input.providerId), - eq(modelCatalog.modelId, input.modelId), - isNull(modelCatalog.deletedAt), - ), - ) - .get(); - const id = existing?.id ?? deps.uuid(); - const shared = { - // Display name + token limits also follow the "never clobber an omitted field on update" invariant (2.5.G - // S10): a pricing-only `models pricing` patch omits them, so they PRESERVE the existing row's values — incl. - // a soft-deactivated live row the command's active-only read cannot see (else a re-price would silently zero - // the discovered name/context). A true INSERT defaults `displayName` → the model id, tokens → the `0` - // "unknown" sentinel; a full-row caller (media fixture / a sync) passes all three, so it is unchanged. - displayName: input.displayName ?? existing?.displayName ?? input.modelId, - contextWindowTokens: input.contextWindowTokens ?? existing?.contextWindowTokens ?? 0, - maxOutputTokens: input.maxOutputTokens ?? existing?.maxOutputTokens ?? 0, - // Media routing / capability columns follow the SAME "never clobber an omitted field on update" invariant - // as the pricing + provenance columns below (2.5.G S10): a partial upsert — e.g. `models pricing` writing a - // `source='user'` row over a model the live refresh discovered — must NOT reset a live/seed row's - // `media_surface` back to `'chat'` (silently disabling generative routing) or blank its capabilities. On a - // true INSERT (`existing` undefined) each still falls to its documented default, so every full-row caller - // (the media fixture, a re-seed) is byte-for-byte unchanged (it always passes these). - mediaSurface: input.mediaSurface ?? existing?.mediaSurface ?? 'chat', - supportsVision: input.supportsVision ?? existing?.supportsVision ?? false, - capabilities: - input.capabilities !== undefined - ? JSON.stringify(input.capabilities) - : (existing?.capabilities ?? JSON.stringify({})), - // The media cost fields are `number | null` (unlike the NOT-NULL text-token prices below): OMITTED - // (`undefined`) preserves the existing row's value (the "never clobber" invariant), an explicit `null` - // CLEARS it, and a number writes it — so a `!== undefined` check (not `??`, which would treat a clearing - // `null` as an omission and preserve the old rate). A true INSERT (`existing` undefined) still defaults to - // `null`. - mediaImageCostMicrocents: - input.mediaImageCostMicrocents !== undefined - ? input.mediaImageCostMicrocents - : (existing?.mediaImageCostMicrocents ?? null), - mediaAudioCostMicrocents: - input.mediaAudioCostMicrocents !== undefined - ? input.mediaAudioCostMicrocents - : (existing?.mediaAudioCostMicrocents ?? null), - mediaVideoCostMicrocents: - input.mediaVideoCostMicrocents !== undefined - ? input.mediaVideoCostMicrocents - : (existing?.mediaVideoCostMicrocents ?? null), - // USER text-token pricing (2.5.G S10) — write the supplied prices, else PRESERVE the existing row's (an - // update that omits them must not zero a hand-entered price), else the NOT-NULL default `0`. - inputCostPerMtokMicrocents: - input.inputCostPerMtokMicrocents ?? existing?.inputCostPerMtokMicrocents ?? 0, - outputCostPerMtokMicrocents: - input.outputCostPerMtokMicrocents ?? existing?.outputCostPerMtokMicrocents ?? 0, - cachedInputCostPerMtokMicrocents: - input.cachedInputCostPerMtokMicrocents ?? existing?.cachedInputCostPerMtokMicrocents ?? 0, - // Provenance + freshness (ADR-0064 §4/§5). On a true INSERT (`existing` undefined) these fall to - // `'static'` / `null`, so every existing media-routing caller (which passes neither) writes a static, - // never-refreshed row unchanged. On an UPDATE they PRESERVE the existing row's `source`/`lastRefreshedAt` - // when the caller omits them — a caller that omits `source` (e.g. a future provider-sync patch) must - // NEVER demote a live-refreshed row back to `'static'` or null its stamp (the "never clobber" invariant, - // symmetric with `replaceProviderModels`). - source: input.source ?? existing?.source ?? 'static', - lastRefreshedAt: input.lastRefreshedAt ?? existing?.lastRefreshedAt ?? null, - // An upsert (re)activates the row: keep `isActive` in lockstep with `activeRow`'s `isActive = true` - // filter so a re-upserted, previously-deactivated row is reachable again and the returned record never - // disagrees with a subsequent `getByModelId` (which filters inactive rows out). - isActive: true, - updatedAt: t, - } satisfies Partial; - if (existing === undefined) { - const row: NewModelCatalogRow = { - id, - providerId: input.providerId, - modelId: input.modelId, - createdAt: t, - ...shared, - }; - db.insert(modelCatalog).values(row).run(); - } else { - db.update(modelCatalog).set(shared).where(eq(modelCatalog.id, id)).run(); - } - // Re-read by the exact id written (not by modelId — that would return the earliest row for a model id - // offered by multiple providers, not necessarily the one just upserted). - const row = rowById(id); - if (row === undefined) { - throw new Error(`model_catalog '${input.modelId}' not found after upsert`); // unreachable — just inserted/updated - } - return fromRow(row); + // Read-then-write upsert under ONE IMMEDIATE transaction (the twin of provider `upsert`): a `models + // pricing` write racing a concurrent `models refresh` (`replaceProviderModels`) on the same + // (provider, model) must not straddle the read→write — BEGIN IMMEDIATE serializes them, avoiding both a + // lost update and a UNIQUE(provider_id, model_id) crash from two concurrent INSERTs; `withBusyRetry` + // waits out residual cross-process contention (ADR-0064 amendment note — DB write-path concurrency). + return withBusyRetry(() => + db.transaction( + () => { + const existing = db + .select() + .from(modelCatalog) + .where( + and( + eq(modelCatalog.providerId, input.providerId), + eq(modelCatalog.modelId, input.modelId), + isNull(modelCatalog.deletedAt), + ), + ) + .get(); + const id = existing?.id ?? deps.uuid(); + const shared = { + // Display name + token limits also follow the "never clobber an omitted field on update" invariant (2.5.G + // S10): a pricing-only `models pricing` patch omits them, so they PRESERVE the existing row's values — incl. + // a soft-deactivated live row the command's active-only read cannot see (else a re-price would silently zero + // the discovered name/context). A true INSERT defaults `displayName` → the model id, tokens → the `0` + // "unknown" sentinel; a full-row caller (media fixture / a sync) passes all three, so it is unchanged. + displayName: input.displayName ?? existing?.displayName ?? input.modelId, + contextWindowTokens: input.contextWindowTokens ?? existing?.contextWindowTokens ?? 0, + maxOutputTokens: input.maxOutputTokens ?? existing?.maxOutputTokens ?? 0, + // Media routing / capability columns follow the SAME "never clobber an omitted field on update" invariant + // as the pricing + provenance columns below (2.5.G S10): a partial upsert — e.g. `models pricing` writing a + // `source='user'` row over a model the live refresh discovered — must NOT reset a live/seed row's + // `media_surface` back to `'chat'` (silently disabling generative routing) or blank its capabilities. On a + // true INSERT (`existing` undefined) each still falls to its documented default, so every full-row caller + // (the media fixture, a re-seed) is byte-for-byte unchanged (it always passes these). + mediaSurface: input.mediaSurface ?? existing?.mediaSurface ?? 'chat', + supportsVision: input.supportsVision ?? existing?.supportsVision ?? false, + capabilities: + input.capabilities !== undefined + ? JSON.stringify(input.capabilities) + : (existing?.capabilities ?? JSON.stringify({})), + // The media cost fields are `number | null` (unlike the NOT-NULL text-token prices below): OMITTED + // (`undefined`) preserves the existing row's value (the "never clobber" invariant), an explicit `null` + // CLEARS it, and a number writes it — so a `!== undefined` check (not `??`, which would treat a clearing + // `null` as an omission and preserve the old rate). A true INSERT (`existing` undefined) still defaults to + // `null`. + mediaImageCostMicrocents: + input.mediaImageCostMicrocents !== undefined + ? input.mediaImageCostMicrocents + : (existing?.mediaImageCostMicrocents ?? null), + mediaAudioCostMicrocents: + input.mediaAudioCostMicrocents !== undefined + ? input.mediaAudioCostMicrocents + : (existing?.mediaAudioCostMicrocents ?? null), + mediaVideoCostMicrocents: + input.mediaVideoCostMicrocents !== undefined + ? input.mediaVideoCostMicrocents + : (existing?.mediaVideoCostMicrocents ?? null), + // USER text-token pricing (2.5.G S10) — write the supplied prices, else PRESERVE the existing row's (an + // update that omits them must not zero a hand-entered price), else the NOT-NULL default `0`. + inputCostPerMtokMicrocents: + input.inputCostPerMtokMicrocents ?? existing?.inputCostPerMtokMicrocents ?? 0, + outputCostPerMtokMicrocents: + input.outputCostPerMtokMicrocents ?? existing?.outputCostPerMtokMicrocents ?? 0, + cachedInputCostPerMtokMicrocents: + input.cachedInputCostPerMtokMicrocents ?? + existing?.cachedInputCostPerMtokMicrocents ?? + 0, + // Provenance + freshness (ADR-0064 §4/§5). On a true INSERT (`existing` undefined) these fall to + // `'static'` / `null`, so every existing media-routing caller (which passes neither) writes a static, + // never-refreshed row unchanged. On an UPDATE they PRESERVE the existing row's `source`/`lastRefreshedAt` + // when the caller omits them — a caller that omits `source` (e.g. a future provider-sync patch) must + // NEVER demote a live-refreshed row back to `'static'` or null its stamp (the "never clobber" invariant, + // symmetric with `replaceProviderModels`). + source: input.source ?? existing?.source ?? 'static', + lastRefreshedAt: input.lastRefreshedAt ?? existing?.lastRefreshedAt ?? null, + // An upsert (re)activates the row: keep `isActive` in lockstep with `activeRow`'s `isActive = true` + // filter so a re-upserted, previously-deactivated row is reachable again and the returned record never + // disagrees with a subsequent `getByModelId` (which filters inactive rows out). + isActive: true, + updatedAt: t, + } satisfies Partial; + if (existing === undefined) { + const row: NewModelCatalogRow = { + id, + providerId: input.providerId, + modelId: input.modelId, + createdAt: t, + ...shared, + }; + db.insert(modelCatalog).values(row).run(); + } else { + db.update(modelCatalog).set(shared).where(eq(modelCatalog.id, id)).run(); + } + // Re-read by the exact id written (not by modelId — that would return the earliest row for a model id + // offered by multiple providers, not necessarily the one just upserted). + const row = rowById(id); + if (row === undefined) { + throw new Error(`model_catalog '${input.modelId}' not found after upsert`); // unreachable — just inserted/updated + } + return fromRow(row); + }, + { behavior: 'immediate' }, + ), + ); }, listByProvider: (providerId) => diff --git a/packages/db/src/retry.test.ts b/packages/db/src/retry.test.ts index ee864fb0..02c90040 100644 --- a/packages/db/src/retry.test.ts +++ b/packages/db/src/retry.test.ts @@ -35,6 +35,25 @@ describe('withBusyRetry — unit (2.5.I)', () => { expect(sleeps).toEqual([25, 50]); }); + it('the full default budget follows the exact linear schedule [25, 50, 75, 100] (locks base×attempt)', () => { + const sleeps: number[] = []; + let caught: unknown; + try { + // Never succeeds → spend the whole default budget (5 attempts ⇒ 4 backoffs). Pins the formula end-to-end + // so a mutation to a constant delay, or base×(attempt−1), would fail here even though the 2-sleep case passes. + withBusyRetry( + () => { + throw lockError('SQLITE_BUSY'); + }, + { baseDelayMs: 25, sleep: (ms) => sleeps.push(ms) }, + ); + } catch (err) { + caught = err; + } + expect(caught).toBeInstanceOf(Error); + expect(sleeps).toEqual([25, 50, 75, 100]); + }); + it('retries SQLITE_LOCKED as well', () => { let calls = 0; const result = withBusyRetry( @@ -147,9 +166,15 @@ describe('withBusyRetry — real SQLITE_BUSY contention (2.5.I)', () => { 2, ); } finally { - holder.sqlite.close(); - writer.sqlite.close(); - rmSync(dir, { recursive: true, force: true }); + try { + holder.sqlite.close(); + } finally { + try { + writer.sqlite.close(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + } } }); @@ -178,9 +203,15 @@ describe('withBusyRetry — real SQLITE_BUSY contention (2.5.I)', () => { expect(attempts).toBe(3); // the whole budget was spent before failing loud holder.sqlite.exec('COMMIT'); // release for teardown } finally { - holder.sqlite.close(); - writer.sqlite.close(); - rmSync(dir, { recursive: true, force: true }); + try { + holder.sqlite.close(); + } finally { + try { + writer.sqlite.close(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + } } }); }); diff --git a/packages/db/src/retry.ts b/packages/db/src/retry.ts index 568ac5d1..675a988f 100644 --- a/packages/db/src/retry.ts +++ b/packages/db/src/retry.ts @@ -16,8 +16,14 @@ * (ADR-0050's durability-first `persistEvent` posture: a swallowed write is silent data loss). * * The wrapped `fn` MUST be re-runnable: a lock fault rolls the transaction back with no partial write, and the - * retry re-runs the whole `fn` from scratch. Both engine writers (`persistEvent`'s fold, the model-catalog - * `replaceProviderModels` upsert, the provider `upsert`) are idempotent given the same input + DB state. + * retry re-runs the whole `fn` from scratch. The wrapped writers (`persistEvent`'s fold, the model-catalog + * `replaceProviderModels` bulk-upsert + `upsert`, the provider `upsert`) are idempotent given the same input + + * DB state. + * + * CAVEAT: `{ behavior: 'immediate' }` only applies to the OUTERMOST `BEGIN`. If a wrapped store method is ever + * called INSIDE another `db.transaction`, better-sqlite3 demotes it to a `SAVEPOINT` and the IMMEDIATE behavior + * is silently ignored. All current call sites invoke these as top-level store methods; a future batch-in-one- + * transaction caller must take the outer `BEGIN IMMEDIATE` itself. */ /** Driver error codes we wait out: a lock we can retry. Anything else is a real fault → rethrow. */ @@ -64,7 +70,8 @@ function sleepSync(ms: number): void { * or an exhausted budget (fail-loud). Synchronous — it wraps a synchronous `db.transaction(...)` call. */ export function withBusyRetry(fn: () => T, options: BusyRetryOptions = {}): T { - const maxAttempts = options.maxAttempts ?? DEFAULT_MAX_ATTEMPTS; + // Floor at 1 so a stray `0`/negative can never disable the first attempt (or spin) — always at least one try. + const maxAttempts = Math.max(1, options.maxAttempts ?? DEFAULT_MAX_ATTEMPTS); const baseDelayMs = options.baseDelayMs ?? DEFAULT_BASE_DELAY_MS; const sleep = options.sleep ?? sleepSync; for (let attempt = 1; ; attempt += 1) { From d1d4b20cc5e094865551477c16154791b3516779 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 21:33:45 +0300 Subject: [PATCH 13/50] =?UTF-8?q?fix(db):=202.5-close=20Step=204=20Sonnet-?= =?UTF-8?q?review=20fixes=20=E2=80=94=20harden=20the=20media-reference=20G?= =?UTF-8?q?C=20writers=20+=20retry=20floor=20test?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet review mutation-tested the mechanism (all correct) and found the docs' "every multi-statement writer" claim overclaimed: media-reference-store's addReference / removeRunReferences / reclaimExpired are multi-statement read-then-write GC writers with no transaction wrapper — and media GC IS host-wired (2.S), so a chat GC sweep racing a run's media production is the exact ADR-0064 §5 scenario. reclaimExpired even documented an (now-false-under-concurrency) "single-connection ⇒ select-then-update is consistent" assumption. - Wrap all three in withBusyRetry + BEGIN IMMEDIATE (recordObject stays a single atomic upsert — no wrap). reclaimExpired now takes one clock read for both the cutoff and the soft-delete stamp (stable across retries). Add a coupling test asserting all three open an IMMEDIATE transaction. - Update the database-schema.md + ADR-0064 enumerations to list the media-reference GC writers, so "every multi-statement writer" is now accurate. - Add a retry test pinning the maxAttempts:0 floor (the 7b7d9ff hardening had no regression test). Full db gate green (207 tests); apps/cli media-gc consumers (19) still pass. Refs: docs/roadmap/phase-2.5-close-plan.md Step 4; ADR-0064 §5 amendment note Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/decisions/0064-live-model-catalog.md | 5 +- docs/reference/desktop/database-schema.md | 2 +- packages/db/src/media-reference-store.test.ts | 21 ++- packages/db/src/media-reference-store.ts | 169 +++++++++++------- packages/db/src/retry.test.ts | 16 ++ 5 files changed, 141 insertions(+), 72 deletions(-) diff --git a/docs/decisions/0064-live-model-catalog.md b/docs/decisions/0064-live-model-catalog.md index 3472ae10..d53e1714 100644 --- a/docs/decisions/0064-live-model-catalog.md +++ b/docs/decisions/0064-live-model-catalog.md @@ -14,8 +14,9 @@ > realization across every `history.db` writer, establishing two conventions the next store author follows. > **(1)** Every write transaction opens with **`BEGIN IMMEDIATE`** (not drizzle's `DEFERRED` default), taking > the write lock up front to close the read→write lock-upgrade race — applied to `persistEvent` (run history), -> the model-catalog `replaceProviderModels` bulk live-upsert and its per-model `upsert`, and the provider -> `upsert` read-then-write. **(2)** Every such +> the model-catalog `replaceProviderModels` bulk live-upsert and its per-model `upsert`, the provider +> `upsert` read-then-write, and the media-reference GC writes (`addReference` / `removeRunReferences` / +> `reclaimExpired`). **(2)** Every such > write routes through a bounded, **fail-loud** `SQLITE_BUSY`/`SQLITE_LOCKED` retry helper > ([retry.ts](../../packages/db/src/retry.ts)) with **deterministic** backoff — **no jitter**, never > `Math.random`, following the no-jitter/deterministic-replay convention of diff --git a/docs/reference/desktop/database-schema.md b/docs/reference/desktop/database-schema.md index 06b5f65f..24a412b8 100644 --- a/docs/reference/desktop/database-schema.md +++ b/docs/reference/desktop/database-schema.md @@ -490,7 +490,7 @@ CREATE INDEX idx_media_references_handle ON media_references (handle); `history.db` is a **single shared file** two concurrent `relavium` processes may write at once — e.g. a `run` persisting events while a `chat` refreshes the live model catalog ([ADR-0064](../../decisions/0064-live-model-catalog.md) §5). The Node/CLI path (`better-sqlite3`, [ADR-0021](../../decisions/0021-node-sqlite-driver-better-sqlite3.md)) hardens this at the connection and the transaction level; this is the one canonical home for the policy that `packages/db/src/retry.ts` and the store doc-comments cite. - **Connection PRAGMAs** ([`client.ts`](../../../packages/db/src/client.ts)): `journal_mode = WAL` (readers never block the single writer, and vice-versa), `busy_timeout = 5000` (SQLite's built-in busy handler waits up to 5 s for a contended lock before returning `SQLITE_BUSY`), `synchronous = NORMAL` (the recommended durability/throughput trade-off under WAL), and `foreign_keys = ON`. -- **Write transactions use `BEGIN IMMEDIATE`**, never drizzle's `DEFERRED` default. A DEFERRED transaction that reads before it writes takes a read lock first and must *upgrade* to a write lock on the first write — if another connection committed in between, that upgrade fails immediately with `SQLITE_BUSY` (`SQLITE_BUSY_SNAPSHOT`), which `busy_timeout` does **not** cover. `BEGIN IMMEDIATE` takes the write lock up front, so the upgrade race cannot occur. Applied to every multi-statement writer: `persistEvent` (run history), the model-catalog `replaceProviderModels` (bulk live-upsert) and `upsert` (per-model pricing), and the provider `upsert` read-then-write. It applies only to the OUTERMOST `BEGIN` — a store method called inside another transaction is demoted to a `SAVEPOINT` and the IMMEDIATE behavior is ignored, so a future batch-in-one-transaction caller must itself open `BEGIN IMMEDIATE`. +- **Write transactions use `BEGIN IMMEDIATE`**, never drizzle's `DEFERRED` default. A DEFERRED transaction that reads before it writes takes a read lock first and must *upgrade* to a write lock on the first write — if another connection committed in between, that upgrade fails immediately with `SQLITE_BUSY` (`SQLITE_BUSY_SNAPSHOT`), which `busy_timeout` does **not** cover. `BEGIN IMMEDIATE` takes the write lock up front, so the upgrade race cannot occur. Applied to every multi-statement writer: `persistEvent` (run history), the model-catalog `replaceProviderModels` (bulk live-upsert) and `upsert` (per-model pricing), the provider `upsert` read-then-write, and the media-reference GC writes (`addReference`, `removeRunReferences`, `reclaimExpired`). It applies only to the OUTERMOST `BEGIN` — a store method called inside another transaction is demoted to a `SAVEPOINT` and the IMMEDIATE behavior is ignored, so a future batch-in-one-transaction caller must itself open `BEGIN IMMEDIATE`. - **The bounded retry** (`packages/db/src/retry.ts`, `withBusyRetry`) wraps those write transactions and retries only `SQLITE_BUSY`/`SQLITE_LOCKED` up to a bounded attempt budget (default 5) with a **deterministic** linear backoff — **no jitter**, never `Math.random`, per the no-jitter/deterministic-replay convention of [ADR-0040](../../decisions/0040-node-retry-budget-above-the-chain.md). It is **fail-loud**: on an exhausted budget (or any non-lock fault) it rethrows the original error and never silently drops a write, preserving [ADR-0050](../../decisions/0050-cli-history-db-at-rest-posture.md)'s durability-first `persistEvent` posture. A retried transaction rolls back with no partial write and re-runs the whole (idempotent) body. Because each attempt's `BEGIN IMMEDIATE` can itself wait up to `busy_timeout` (5 s) for the lock, the compounded worst case under sustained contention is ~25 s (5 attempts × 5 s + the sub-300 ms backoffs) of a synchronous block before the fail-loud rethrow — a deliberate durability-over-latency trade on a path that only stalls under pathological multi-writer contention. - **Single-statement writes** (e.g. `appendMessage`, `setKeychainRef`) go straight for the write lock and rely on SQLite's built-in busy handler (`busy_timeout`); they need no explicit transaction. - **Reads that must be consistent across statements** use a read transaction: `sessionStore.loadFull` reads the session row and its transcript inside one deferred transaction so **both reads observe a single consistent DB snapshot** (never a two-`SELECT` straddle across a concurrent commit). Note this guarantees *snapshot* consistency, not *turn* atomicity: the CLI persister writes a turn's messages and its updated session totals as separate auto-committed statements, so a snapshot can still legitimately observe messages ahead of their totals. A "totals always match the returned messages" guarantee would additionally require the host to persist each turn in one transaction (a tracked follow-up). diff --git a/packages/db/src/media-reference-store.test.ts b/packages/db/src/media-reference-store.test.ts index 549b4e70..b02188cc 100644 --- a/packages/db/src/media-reference-store.test.ts +++ b/packages/db/src/media-reference-store.test.ts @@ -1,5 +1,5 @@ import { eq } from 'drizzle-orm'; -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { createClient, runMigrations, type DbClient } from './client.js'; import { @@ -21,12 +21,29 @@ describe('MediaReferenceStore (1.AF/D12c + D11 — media_objects/media_reference let tick = 1_000; store = createMediaReferenceStore(client.db, () => (tick += 1)); }); - afterEach(() => client.sqlite.close()); + afterEach(() => { + vi.restoreAllMocks(); + client.sqlite.close(); + }); function record(): void { store.recordObject({ handle: HANDLE, mimeType: 'image/png', modality: 'image', byteLength: 5 }); } + it('the three multi-statement GC writers each open an IMMEDIATE transaction (2.5.I write-path convention)', () => { + record(); // recordObject (single-statement upsert) opens no transaction; it seeds the FK target + const txnSpy = vi.spyOn(client.db, 'transaction'); + store.addReference(HANDLE, 'run', 'run-1'); // INSERT + cursor UPDATE + store.removeRunReferences('run-1'); // SELECT + DELETE + cursor UPDATE + store.reclaimExpired(0); // SELECT + SELECT + soft-delete + // All three serialize under BEGIN IMMEDIATE so a concurrent sweep/refresh can't straddle their statements + // (ADR-0064 §5); a DEFERRED begin would drop the config arg on any of them. + expect(txnSpy).toHaveBeenCalledTimes(3); + for (const call of txnSpy.mock.calls) { + expect(call[1]).toEqual({ behavior: 'immediate' }); + } + }); + it('describe returns the durable metadata + only the session/workspace authz scopes', () => { record(); store.addReference(HANDLE, 'run', 'run-1'); // lifetime — never grants read diff --git a/packages/db/src/media-reference-store.ts b/packages/db/src/media-reference-store.ts index 829de17d..57042c20 100644 --- a/packages/db/src/media-reference-store.ts +++ b/packages/db/src/media-reference-store.ts @@ -10,6 +10,7 @@ import type { import { and, eq, inArray, isNull, lte, notInArray } from 'drizzle-orm'; import type { Db } from './client.js'; +import { withBusyRetry } from './retry.js'; import { mediaObjects, mediaReferences } from './schema.js'; /** Max handles per `handle IN (…)` UPDATE — under SQLite's 999-bound-parameter floor (older builds). */ @@ -105,21 +106,35 @@ export function createMediaReferenceStore( addReference(handle: string, scopeKind: MediaScopeKind, scopeId: string): void { const ts = now(); - db.insert(mediaReferences) - .values({ id: randomUUID(), handle, scopeKind, scopeId, createdAt: ts }) - // A scope references a handle at most once (the refcount is the distinct-row count). - .onConflictDoNothing({ - target: [mediaReferences.handle, mediaReferences.scopeKind, mediaReferences.scopeId], - }) - .run(); - // Fresh reference activity ⇒ refresh the GC cursor. The grace window is measured from - // `last_referenced_at` (ADR-0042 §4), so it must track the last reference-SET mutation, not just - // production time — else a handle re-referenced long after it was produced inherits a stale, already- - // expired cursor and is reclaimed on the next sweep instead of getting a fresh grace window. - db.update(mediaObjects) - .set({ lastReferencedAt: ts }) - .where(eq(mediaObjects.handle, handle)) - .run(); + // INSERT (the reference) + UPDATE (the GC cursor) under ONE BEGIN IMMEDIATE so a concurrent GC sweep + // (reclaimExpired) can't observe state between the two statements and reclaim a still-referenced handle — + // the 2.5.I write-path convention (ADR-0064 amendment note). + withBusyRetry(() => + db.transaction( + () => { + db.insert(mediaReferences) + .values({ id: randomUUID(), handle, scopeKind, scopeId, createdAt: ts }) + // A scope references a handle at most once (the refcount is the distinct-row count). + .onConflictDoNothing({ + target: [ + mediaReferences.handle, + mediaReferences.scopeKind, + mediaReferences.scopeId, + ], + }) + .run(); + // Fresh reference activity ⇒ refresh the GC cursor. The grace window is measured from + // `last_referenced_at` (ADR-0042 §4), so it must track the last reference-SET mutation, not just + // production time — else a handle re-referenced long after it was produced inherits a stale, already- + // expired cursor and is reclaimed on the next sweep instead of getting a fresh grace window. + db.update(mediaObjects) + .set({ lastReferencedAt: ts }) + .where(eq(mediaObjects.handle, handle)) + .run(); + }, + { behavior: 'immediate' }, + ), + ); }, describe(handle: string): MediaHandleRecord | undefined { @@ -153,29 +168,38 @@ export function createMediaReferenceStore( removeRunReferences(runId: string): number { const ts = now(); - // Capture the handles this run referenced BEFORE the delete, so the grace window of every handle this - // sweep drops toward zero starts NOW (ADR-0042 §4 — measured from `last_referenced_at`), not from the - // handle's production time. Without this, a long-lived handle losing its last reference is reclaimed - // on the very next sweep with zero effective grace. - const affected = db - .selectDistinct({ handle: mediaReferences.handle }) - .from(mediaReferences) - .where(and(eq(mediaReferences.scopeKind, 'run'), eq(mediaReferences.scopeId, runId))) - .all(); - const result = db - .delete(mediaReferences) - .where(and(eq(mediaReferences.scopeKind, 'run'), eq(mediaReferences.scopeId, runId))) - .run(); - const handles = affected.map((row) => row.handle); - // CHUNK the `handle IN (…)` refresh under SQLite's bound-parameter floor (a wide fan-out run can - // reference many handles); one shared `ts` keeps the batches consistent. - for (let i = 0; i < handles.length; i += SQLITE_INARRAY_CHUNK) { - db.update(mediaObjects) - .set({ lastReferencedAt: ts }) - .where(inArray(mediaObjects.handle, handles.slice(i, i + SQLITE_INARRAY_CHUNK))) - .run(); - } - return result.changes; + // SELECT + DELETE + UPDATE-loop under ONE BEGIN IMMEDIATE (2.5.I write-path convention): the cursor + // refresh must be atomic with the delete so a concurrent GC sweep never reclaims a handle mid-drop. + return withBusyRetry(() => + db.transaction( + () => { + // Capture the handles this run referenced BEFORE the delete, so the grace window of every handle this + // sweep drops toward zero starts NOW (ADR-0042 §4 — measured from `last_referenced_at`), not from the + // handle's production time. Without this, a long-lived handle losing its last reference is reclaimed + // on the very next sweep with zero effective grace. + const affected = db + .selectDistinct({ handle: mediaReferences.handle }) + .from(mediaReferences) + .where(and(eq(mediaReferences.scopeKind, 'run'), eq(mediaReferences.scopeId, runId))) + .all(); + const result = db + .delete(mediaReferences) + .where(and(eq(mediaReferences.scopeKind, 'run'), eq(mediaReferences.scopeId, runId))) + .run(); + const handles = affected.map((row) => row.handle); + // CHUNK the `handle IN (…)` refresh under SQLite's bound-parameter floor (a wide fan-out run can + // reference many handles); one shared `ts` keeps the batches consistent. + for (let i = 0; i < handles.length; i += SQLITE_INARRAY_CHUNK) { + db.update(mediaObjects) + .set({ lastReferencedAt: ts }) + .where(inArray(mediaObjects.handle, handles.slice(i, i + SQLITE_INARRAY_CHUNK))) + .run(); + } + return result.changes; + }, + { behavior: 'immediate' }, + ), + ); }, listObjectHandles(): string[] { @@ -196,35 +220,46 @@ export function createMediaReferenceStore( }, reclaimExpired(graceMs: number): string[] { - const cutoff = now() - graceMs; - const referenced = db.select({ handle: mediaReferences.handle }).from(mediaReferences); - const expired = db - .select({ handle: mediaObjects.handle }) - .from(mediaObjects) - .where( - and( - isNull(mediaObjects.deletedAt), // not already reclaimed - lte(mediaObjects.lastReferencedAt, cutoff), // past the grace window - notInArray(mediaObjects.handle, referenced), // zero references (refcount = row count) - ), - ) - .all(); - const handles = expired.map((row) => row.handle); - if (handles.length > 0) { - // Soft-delete EXACTLY the expired handles found above (not a re-run of the 0-ref filter, which - // would ignore the grace window). better-sqlite3 is single-connection, so select-then-update is - // consistent within this method. CHUNK the `handle IN (…)` list so a large sweep never exceeds - // SQLite's bound-parameter limit (SQLITE_MAX_VARIABLE_NUMBER, 999 on older builds) — a multi-day - // orphan backlog can easily surpass it; one shared `deletedAt` keeps the batches consistent. - const deletedAt = now(); - for (let i = 0; i < handles.length; i += SQLITE_INARRAY_CHUNK) { - db.update(mediaObjects) - .set({ deletedAt }) - .where(inArray(mediaObjects.handle, handles.slice(i, i + SQLITE_INARRAY_CHUNK))) - .run(); - } - } - return handles; + const ts = now(); // one clock read for the cutoff AND the soft-delete stamp (stable across retries) + const cutoff = ts - graceMs; + // SELECT (referenced) + SELECT (expired) + UPDATE-loop under ONE BEGIN IMMEDIATE (2.5.I): the reads and + // the soft-delete are one atomic snapshot, so a reference added concurrently is never reclaimed. (The old + // "single-connection ⇒ select-then-update is consistent" note held only WITHIN one process; two `relavium` + // processes share this file — ADR-0064 §5.) + return withBusyRetry(() => + db.transaction( + () => { + const referenced = db.select({ handle: mediaReferences.handle }).from(mediaReferences); + const expired = db + .select({ handle: mediaObjects.handle }) + .from(mediaObjects) + .where( + and( + isNull(mediaObjects.deletedAt), // not already reclaimed + lte(mediaObjects.lastReferencedAt, cutoff), // past the grace window + notInArray(mediaObjects.handle, referenced), // zero references (refcount = row count) + ), + ) + .all(); + const handles = expired.map((row) => row.handle); + if (handles.length > 0) { + // Soft-delete EXACTLY the expired handles found above (not a re-run of the 0-ref filter, which + // would ignore the grace window). Now serialized under BEGIN IMMEDIATE (2.5.I), so the select-then- + // update snapshot is consistent even across two `relavium` processes. CHUNK the `handle IN (…)` list + // so a large sweep never exceeds SQLite's bound-parameter limit (SQLITE_MAX_VARIABLE_NUMBER, 999 on + // older builds) — a multi-day orphan backlog can easily surpass it; one shared `ts` keeps the batches consistent. + for (let i = 0; i < handles.length; i += SQLITE_INARRAY_CHUNK) { + db.update(mediaObjects) + .set({ deletedAt: ts }) + .where(inArray(mediaObjects.handle, handles.slice(i, i + SQLITE_INARRAY_CHUNK))) + .run(); + } + } + return handles; + }, + { behavior: 'immediate' }, + ), + ); }, }; } diff --git a/packages/db/src/retry.test.ts b/packages/db/src/retry.test.ts index 02c90040..36668e18 100644 --- a/packages/db/src/retry.test.ts +++ b/packages/db/src/retry.test.ts @@ -121,6 +121,22 @@ describe('withBusyRetry — unit (2.5.I)', () => { ).toThrow(); expect(calls).toBe(1); }); + + it('a stray maxAttempts of 0 is floored to one attempt (never disables the first try, never spins)', () => { + const sleep = vi.fn(); + let calls = 0; + expect(() => + withBusyRetry( + () => { + calls += 1; + throw lockError('SQLITE_BUSY'); + }, + { maxAttempts: 0, sleep }, + ), + ).toThrow(); + expect(calls).toBe(1); // Math.max(1, 0) — the fn still ran exactly once + expect(sleep).not.toHaveBeenCalled(); + }); }); /** From 7769c5d1895adbdc5512022788ab57e496917905 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 21:55:54 +0300 Subject: [PATCH 14/50] =?UTF-8?q?test(cli):=202.5-close=20Step=205=20?= =?UTF-8?q?=E2=80=94=20concurrent=20chat+run=20e2e=20(2.5.I=20S3)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two scenarios (close-plan D3) proving a `relavium chat` and a `relavium run` share one history.db safely (ADR-0064 §5): - Coexistence (in-process, two connections): the REAL runCommand persists run events on its own connection while a REAL SessionStore writes a transcript on a second connection, interleaved. Asserts both land intact and the loadFull snapshot is consistent. (One synchronous better-sqlite3 process can't overlap two txns, so this proves coexistence, not lock contention — see the two-process test for that.) - Contention (real two OS processes): two child processes race a burst of provider upserts (the BEGIN IMMEDIATE + withBusyRetry path) against the same file; asserts both exit 0 with empty stderr and all 2×N distinct providers landed (no escaped SQLITE_BUSY, no lost/corrupt row). This is the only form that reproduces real cross-process WAL write-lock contention. The child (concurrent-writer.mjs) imports the BUILT @relavium/db by an argv abs path (it can't use vitest source resolution); the test is visibly it.skipIf-skipped, never silently passed, if the dist is absent. Refs: docs/roadmap/phase-2.5-close-plan.md Step 5 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/harness/concurrency.e2e.test.ts | 208 ++++++++++++++++++ .../harness/fixtures/concurrent-writer.mjs | 40 ++++ 2 files changed, 248 insertions(+) create mode 100644 apps/cli/src/harness/concurrency.e2e.test.ts create mode 100644 apps/cli/src/harness/fixtures/concurrent-writer.mjs diff --git a/apps/cli/src/harness/concurrency.e2e.test.ts b/apps/cli/src/harness/concurrency.e2e.test.ts new file mode 100644 index 00000000..90c46146 --- /dev/null +++ b/apps/cli/src/harness/concurrency.e2e.test.ts @@ -0,0 +1,208 @@ +import { spawn } from 'node:child_process'; +import { existsSync, mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import type { WorkflowDefinition } from '@relavium/core'; +import { + createClient, + createProviderStore, + createRunHistoryStore, + createSessionStore, + runMigrations, +} from '@relavium/db'; +import { RunEventSchema, type AgentSessionRecord, type SessionMessage } from '@relavium/shared'; +import { randomUUID } from 'node:crypto'; +import { describe, expect, it } from 'vitest'; + +import { runCommand } from '../commands/run.js'; +import type { OpenedHistory } from '../history/open.js'; +import type { GlobalOptions } from '../process/options.js'; +import { captureIo } from '../test-support.js'; + +/** + * 2.5.I S3 — concurrency e2e: a `relavium chat` and a `relavium run` share one `history.db` + * ([ADR-0064](../../../../docs/decisions/0064-live-model-catalog.md) §5). Two scenarios, per the close-plan D3: + * + * 1. **Two-connection coexistence** (in-process): the REAL `runCommand` (its own connection) writes run + * events while a REAL `SessionStore` (a second connection — the chat process's handle) writes a transcript, + * interleaved. Proves the run and chat write paths coexist on one file with no interference and consistent + * reads. A single Node process (synchronous better-sqlite3) can't truly overlap two transactions, so this + * proves coexistence, not lock contention — + * 2. **Two-process contention** (real child processes): the genuine cross-process WAL path. Two child + * processes race a burst of provider `upsert`s (the BEGIN IMMEDIATE + `withBusyRetry` write path) against + * the same file; both must land every write with no escaped `SQLITE_BUSY`. This is the only form that + * reproduces the real write-lock contention ADR-0064 §5 names. It needs the built `@relavium/db` (the child + * can't use vitest's source resolution); it is visibly SKIPPED — never silently passed — if the dist is + * absent (a `pnpm turbo run build` produces it; CI builds upstream packages before this test). + */ + +const FIXTURES_DIR = fileURLToPath(new URL('./fixtures/', import.meta.url)); +const DB_DIST = fileURLToPath(new URL('../../../../packages/db/dist/index.js', import.meta.url)); +const CHILD_SCRIPT = fileURLToPath(new URL('./fixtures/concurrent-writer.mjs', import.meta.url)); +const ISO = '2026-07-07T00:00:00.000Z'; + +function globalOptions(): GlobalOptions { + return { + json: true, + color: false, + cwd: FIXTURES_DIR, + configPath: undefined, + verbosity: 'normal', + }; +} + +const session = (id: string, totalOutputTokens = 0): AgentSessionRecord => ({ + id, + agentSlug: 'chatter', + context: { workingDir: '/workspace', fsScopeTier: 'sandboxed' }, + status: 'active', + totalInputTokens: 0, + totalOutputTokens, + totalCostMicrocents: 0, + createdAt: ISO, + updatedAt: ISO, +}); + +const message = (sessionId: string, seq: number): SessionMessage => ({ + id: `${sessionId}-m${seq}`, + sessionId, + sequenceNumber: seq, + role: seq % 2 === 0 ? 'user' : 'assistant', + content: [{ type: 'text', text: `turn ${seq}` }], + timestamp: ISO, +}); + +/** Spawn a child process; resolve with its exit code + captured stderr (empty on a clean run). */ +function runChild(args: readonly string[]): Promise<{ code: number; stderr: string }> { + return new Promise((resolve, reject) => { + const child = spawn(process.execPath, [CHILD_SCRIPT, ...args], { + stdio: ['ignore', 'ignore', 'pipe'], + }); + let stderr = ''; + child.stderr.on('data', (chunk: Buffer) => { + stderr += chunk.toString('utf8'); + }); + child.on('error', reject); + child.on('close', (code) => resolve({ code: code ?? -1, stderr })); + }); +} + +describe('concurrency e2e (2.5.I S3) — a run and a chat share one history.db', () => { + it('coexistence: a real run and a real chat session write the same file concurrently — both land', async () => { + const dir = mkdtempSync(join(tmpdir(), 'relavium-concurrency-')); + const dbPath = join(dir, 'history.db'); + const runClient = createClient(dbPath); + runMigrations(runClient.db); + const chatClient = createClient(dbPath); // a SEPARATE connection — the chat process's handle + const TURNS = 12; + try { + const sessions = createSessionStore(chatClient.db); + const sessionId = 'sess-coexist'; + sessions.createSession(session(sessionId)); + + // The chat side: append a transcript, yielding between turns so the run interleaves at await points. + const chatWrites = (async () => { + for (let seq = 0; seq < TURNS; seq += 1) { + sessions.appendMessage(message(sessionId, seq)); + sessions.updateSession(session(sessionId, seq + 1)); + await Promise.resolve(); + } + })(); + + // The run side: the REAL `runCommand` persisting to its OWN connection on the same file. + const openRunStore = (workflow: WorkflowDefinition): OpenedHistory => ({ + store: createRunHistoryStore(runClient.db, { + uuid: () => randomUUID(), + now: () => Date.now(), + workflow: { + slug: workflow.workflow.id, + name: workflow.workflow.name ?? workflow.workflow.id, + definitionJson: JSON.stringify(workflow), + }, + }), + db: runClient.db, + close: () => {}, + }); + const runIo = captureIo(); + const runPromise = runCommand( + { workflow: join(FIXTURES_DIR, 'sequential.relavium.yaml'), input: [] }, + { io: runIo.io, global: globalOptions(), openRunStore }, + ); + + const [runCode] = await Promise.all([runPromise, chatWrites]); + + // The run completed cleanly and its events persisted (stdout-pure --json contract). + expect(runCode).toBe(0); + expect(runIo.err()).toBe(''); + const runEvents = runIo + .out() + .trimEnd() + .split('\n') + .map((line) => RunEventSchema.parse(JSON.parse(line))); + expect(runEvents[0]?.type).toBe('run:started'); + expect(runEvents.at(-1)?.type).toBe('run:completed'); + + // The chat transcript persisted intact + the loadFull snapshot is consistent with the session totals. + const full = sessions.loadFull(sessionId); + expect(full?.messages.map((m) => m.sequenceNumber)).toEqual( + Array.from({ length: TURNS }, (_, i) => i), + ); + expect(full?.session.totalOutputTokens).toBe(TURNS); + } finally { + try { + runClient.sqlite.close(); + } finally { + try { + chatClient.sqlite.close(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + } + } + }); + + it.skipIf(!existsSync(DB_DIST))( + 'contention: two child processes race provider upserts on one file — all writes land, no SQLITE_BUSY escapes', + async () => { + const dir = mkdtempSync(join(tmpdir(), 'relavium-concurrency-2p-')); + const dbPath = join(dir, 'history.db'); + // The parent migrates ONCE so the children never race the migrator — they only write. + const setup = createClient(dbPath); + runMigrations(setup.db); + setup.sqlite.close(); + const PER_CHILD = 40; + try { + const [a, b] = await Promise.all([ + runChild([DB_DIST, dbPath, 'a', String(PER_CHILD)]), + runChild([DB_DIST, dbPath, 'b', String(PER_CHILD)]), + ]); + // Both children exited cleanly (no escaped SQLITE_BUSY / no thrown write) — a failure prints to stderr. + expect(a.stderr).toBe(''); + expect(a.code).toBe(0); + expect(b.stderr).toBe(''); + expect(b.code).toBe(0); + + // Every write of BOTH children landed: 2 × PER_CHILD distinct providers, none lost or corrupted. + const verify = createClient(dbPath); + try { + const providers = createProviderStore(verify.db, { + uuid: () => randomUUID(), + now: () => 0, + }); + const names = providers.list().map((p) => p.name); + expect(names).toHaveLength(2 * PER_CHILD); + for (let i = 0; i < PER_CHILD; i += 1) { + expect(names).toContain(`a-${i}`); + expect(names).toContain(`b-${i}`); + } + } finally { + verify.sqlite.close(); + } + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }, + ); +}); diff --git a/apps/cli/src/harness/fixtures/concurrent-writer.mjs b/apps/cli/src/harness/fixtures/concurrent-writer.mjs new file mode 100644 index 00000000..d0e80f52 --- /dev/null +++ b/apps/cli/src/harness/fixtures/concurrent-writer.mjs @@ -0,0 +1,40 @@ +// A child process for the 2.5.I two-process contention smoke (concurrency.e2e.test.ts). It opens a SHARED, +// already-migrated `history.db` via the BUILT `@relavium/db` and writes a burst of provider `upsert`s — the +// BEGIN IMMEDIATE + `withBusyRetry` write path (ADR-0064 §5). Two of these racing the same file exercise the +// REAL cross-process WAL write-lock contention that a single process (synchronous better-sqlite3) cannot +// reproduce. It writes distinct provider names, so success = every write of BOTH children landed with no +// escaped `SQLITE_BUSY` and no lost/corrupt row (the parent asserts the union). +// +// argv: [node, thisFile, , , , ] +/* global process, console -- a Node child-process fixture (not TS source); it uses only these Node globals. */ +const [, , distPath, dbPath, prefix, countArg] = process.argv; +const count = Number(countArg); + +let client; +try { + // test-harness mechanism: the child can't use vitest's source resolution, so it imports the BUILT + // @relavium/db by an argv-provided abs path. Not a seam bypass — @relavium/db carries no provider SDK + // (ADR-0011); the fence targets vendor-SDK smuggling in the LLM seam. + // eslint-disable-next-line no-restricted-syntax + const { createClient, createProviderStore } = await import(distPath); + // The parent migrated the file before spawning, so children never race the migrator — they only write. + client = createClient(dbPath); + let uuidN = 0; + const providers = createProviderStore(client.db, { + // Per-child UUID namespace (the `prefix` first char) so two children never collide on a row PK. + uuid: () => `${prefix}0000000-0000-4000-8000-${String(++uuidN).padStart(12, '0')}`, + now: () => 1_700_000_000_000, + }); + for (let i = 0; i < count; i += 1) { + providers.upsert({ + name: `${prefix}-${i}`, + displayName: `provider ${prefix} ${i}`, + baseUrl: 'https://api.example/v1', + }); + } +} catch (err) { + console.error(err instanceof Error ? (err.stack ?? err.message) : String(err)); + process.exitCode = 1; +} finally { + client?.sqlite.close(); +} From bca913d82e3c2ee81780ef2bd2464fc9d3b963bd Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 22:15:38 +0300 Subject: [PATCH 15/50] =?UTF-8?q?test(cli):=202.5-close=20Step=205=20Opus-?= =?UTF-8?q?review=20fixes=20=E2=80=94=20real=20held-lock=20contention=20+?= =?UTF-8?q?=20honest=20framing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review empirically showed (≈100 controlled runs) the two-process test was a reliable cross-process COEXISTENCE smoke but a weak CONTENTION guard: distinct-name upserts + the always-on busy_timeout rarely contend, so it caught an IMMEDIATE-only regression ~2.5% and a retry-only regression 0%. The precise clause-guards already live deterministically in @relavium/db (provider-store.test.ts IMMEDIATE spy; retry.test.ts held-lock). - Give it real teeth deterministically: the parent now HOLDS the WAL write lock (BEGIN IMMEDIATE on a third connection) while the children spawn, so their first upsert genuinely busy-waits cross-process every run; released after a short hold. The pass condition never depends on the timing (best-effort force, not a flake). - Reframe the docstring + title honestly: a cross-process safety/coexistence smoke; point at provider-store.test.ts + retry.test.ts as the precise Step-4 guards. - Fix the stderr flake vector: assert exit 0 + no /SQLITE|Error/ line, not exact-empty stderr (a CI runner's benign Node warning would have broken it). - NIT: group node:crypto with the other node: builtins. 3× stable locally; typecheck + lint clean. Refs: docs/roadmap/phase-2.5-close-plan.md Step 5 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/harness/concurrency.e2e.test.ts | 53 +++++++++++++++----- 1 file changed, 41 insertions(+), 12 deletions(-) diff --git a/apps/cli/src/harness/concurrency.e2e.test.ts b/apps/cli/src/harness/concurrency.e2e.test.ts index 90c46146..e1b0e1c8 100644 --- a/apps/cli/src/harness/concurrency.e2e.test.ts +++ b/apps/cli/src/harness/concurrency.e2e.test.ts @@ -1,4 +1,5 @@ import { spawn } from 'node:child_process'; +import { randomUUID } from 'node:crypto'; import { existsSync, mkdtempSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; @@ -13,7 +14,6 @@ import { runMigrations, } from '@relavium/db'; import { RunEventSchema, type AgentSessionRecord, type SessionMessage } from '@relavium/shared'; -import { randomUUID } from 'node:crypto'; import { describe, expect, it } from 'vitest'; import { runCommand } from '../commands/run.js'; @@ -30,12 +30,16 @@ import { captureIo } from '../test-support.js'; * interleaved. Proves the run and chat write paths coexist on one file with no interference and consistent * reads. A single Node process (synchronous better-sqlite3) can't truly overlap two transactions, so this * proves coexistence, not lock contention — - * 2. **Two-process contention** (real child processes): the genuine cross-process WAL path. Two child - * processes race a burst of provider `upsert`s (the BEGIN IMMEDIATE + `withBusyRetry` write path) against - * the same file; both must land every write with no escaped `SQLITE_BUSY`. This is the only form that - * reproduces the real write-lock contention ADR-0064 §5 names. It needs the built `@relavium/db` (the child - * can't use vitest's source resolution); it is visibly SKIPPED — never silently passed — if the dist is - * absent (a `pnpm turbo run build` produces it; CI builds upstream packages before this test). + * 2. **Two-process cross-process safety** (real child processes): the genuine two-OS-process WAL path a + * single synchronous process cannot reach (separate SQLite instances + real OS file locks). Two children + * race a burst of provider `upsert`s (the BEGIN IMMEDIATE + `withBusyRetry` write path) while the parent + * briefly HOLDS the write lock, forcing a real cross-process busy-wait every run; both children must then + * land every write with no escaped `SQLITE_BUSY`. This is a cross-process *safety/coexistence* smoke — the + * precise clause-guards for the Step-4 fix are the DETERMINISTIC white-box tests in `@relavium/db`: + * `provider-store.test.ts` (spies `db.transaction(..., { behavior: 'immediate' })`) and `retry.test.ts` + * (a held-lock released mid-backoff exercising `withBusyRetry` + the fail-loud budget). It needs the built + * `@relavium/db` (the child can't use vitest's source resolution); it is visibly SKIPPED — never silently + * passed — if the dist is absent (a `pnpm turbo run build` produces it; CI builds upstream packages first). */ const FIXTURES_DIR = fileURLToPath(new URL('./fixtures/', import.meta.url)); @@ -164,7 +168,7 @@ describe('concurrency e2e (2.5.I S3) — a run and a chat share one history.db', }); it.skipIf(!existsSync(DB_DIST))( - 'contention: two child processes race provider upserts on one file — all writes land, no SQLITE_BUSY escapes', + 'cross-process safety: two child processes contend on one file (parent holds the lock) — all writes land', async () => { const dir = mkdtempSync(join(tmpdir(), 'relavium-concurrency-2p-')); const dbPath = join(dir, 'history.db'); @@ -173,16 +177,39 @@ describe('concurrency e2e (2.5.I S3) — a run and a chat share one history.db', runMigrations(setup.db); setup.sqlite.close(); const PER_CHILD = 40; + // A third connection that HOLDS the single WAL write lock while the children start, so their first + // upsert genuinely busy-waits cross-process every run (not luck-of-the-overlap). Released after a short + // hold; both children then acquire the lock in turn and complete. + const holder = createClient(dbPath); + let released = false; + const releaseLock = (): void => { + if (!released) { + released = true; + holder.sqlite.exec('COMMIT'); + } + }; try { - const [a, b] = await Promise.all([ + holder.sqlite.exec('BEGIN IMMEDIATE'); // acquire the write lock; the children block on it + const childrenDone = Promise.all([ runChild([DB_DIST, dbPath, 'a', String(PER_CHILD)]), runChild([DB_DIST, dbPath, 'b', String(PER_CHILD)]), ]); - // Both children exited cleanly (no escaped SQLITE_BUSY / no thrown write) — a failure prints to stderr. - expect(a.stderr).toBe(''); + // Give both children time to spawn + reach their first (blocked) write, then release the lock. If the + // release beats a slow spawn, the run simply proceeds without contention — the pass condition never + // depends on the timing, so this is a best-effort force, not a flake vector. + await new Promise((resolve) => { + setTimeout(resolve, 150); + }); + releaseLock(); + const [a, b] = await childrenDone; + + // Both children exited cleanly. A real failure prints the error to stderr + exits 1; a benign Node + // warning line (deprecation, experimental flag) is tolerated — assert the exit code and the ABSENCE + // of a SQLite/Error line, not exact-empty stderr (which a CI runner's warnings could break). expect(a.code).toBe(0); - expect(b.stderr).toBe(''); + expect(a.stderr).not.toMatch(/SQLITE|Error/i); expect(b.code).toBe(0); + expect(b.stderr).not.toMatch(/SQLITE|Error/i); // Every write of BOTH children landed: 2 × PER_CHILD distinct providers, none lost or corrupted. const verify = createClient(dbPath); @@ -201,6 +228,8 @@ describe('concurrency e2e (2.5.I S3) — a run and a chat share one history.db', verify.sqlite.close(); } } finally { + releaseLock(); // release even if an assertion threw before the release point + holder.sqlite.close(); rmSync(dir, { recursive: true, force: true }); } }, From 9b68a07eff487300445ed37911ec154bd75618af Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 22:31:04 +0300 Subject: [PATCH 16/50] =?UTF-8?q?test(cli):=202.5-close=20Step=205=20Sonne?= =?UTF-8?q?t-review=20fixes=20=E2=80=94=20READY=20handshake=20forces=20rea?= =?UTF-8?q?l=20contention?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet review MEASURED that the 150ms fixed hold was shorter than the child's ~183ms startup (Node bootstrap + the dynamic @relavium/db import), so contention never fired (0/23 trials); "forcing a busy-wait every run" was empirically false. - HIGH: replace the fixed timer with a deterministic READY handshake. The child prints READY immediately before its first write; the parent releases the held write lock only after BOTH children signal + a small margin — so a genuine cross-process busy-wait happens every run, independent of startup speed. Empirically verified: both children now block ~120-140ms on the held lock (was 0ms). A `close`-resolves-`ready` fallback means a dead child can never hang the wait. - MEDIUM: reconcile the docstring to the handshake mechanism (no more "every run" claim that the inline comment contradicted). - LOW: harden releaseLock cleanup — guard on `holder.sqlite.inTransaction` (never COMMIT with no open txn) and nest the finally (release → close → rmSync) so a failure in one step never skips the rest. 3× stable; typecheck + lint + format clean. Refs: docs/roadmap/phase-2.5-close-plan.md Step 5 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/harness/concurrency.e2e.test.ts | 91 ++++++++++++------- .../harness/fixtures/concurrent-writer.mjs | 4 + 2 files changed, 63 insertions(+), 32 deletions(-) diff --git a/apps/cli/src/harness/concurrency.e2e.test.ts b/apps/cli/src/harness/concurrency.e2e.test.ts index e1b0e1c8..05485d0f 100644 --- a/apps/cli/src/harness/concurrency.e2e.test.ts +++ b/apps/cli/src/harness/concurrency.e2e.test.ts @@ -33,8 +33,10 @@ import { captureIo } from '../test-support.js'; * 2. **Two-process cross-process safety** (real child processes): the genuine two-OS-process WAL path a * single synchronous process cannot reach (separate SQLite instances + real OS file locks). Two children * race a burst of provider `upsert`s (the BEGIN IMMEDIATE + `withBusyRetry` write path) while the parent - * briefly HOLDS the write lock, forcing a real cross-process busy-wait every run; both children must then - * land every write with no escaped `SQLITE_BUSY`. This is a cross-process *safety/coexistence* smoke — the + * HOLDS the write lock and releases it only after a **READY handshake** — each child signals just before + * its first write — so a real cross-process busy-wait happens every run (deterministic, not overlap-luck). + * Both children must then land every write with no escaped `SQLITE_BUSY`. This is a cross-process + * *safety/coexistence* smoke — the * precise clause-guards for the Step-4 fix are the DETERMINISTIC white-box tests in `@relavium/db`: * `provider-store.test.ts` (spies `db.transaction(..., { behavior: 'immediate' })`) and `retry.test.ts` * (a held-lock released mid-backoff exercising `withBusyRetry` + the fail-loud budget). It needs the built @@ -78,19 +80,39 @@ const message = (sessionId: string, seq: number): SessionMessage => ({ timestamp: ISO, }); -/** Spawn a child process; resolve with its exit code + captured stderr (empty on a clean run). */ -function runChild(args: readonly string[]): Promise<{ code: number; stderr: string }> { - return new Promise((resolve, reject) => { - const child = spawn(process.execPath, [CHILD_SCRIPT, ...args], { - stdio: ['ignore', 'ignore', 'pipe'], - }); - let stderr = ''; - child.stderr.on('data', (chunk: Buffer) => { - stderr += chunk.toString('utf8'); - }); +interface SpawnedChild { + /** Resolves when the child prints `READY` (about to write) OR exits — so the parent never hangs on a dead child. */ + readonly ready: Promise; + /** Resolves with the child's exit code + captured stderr once it closes. */ + readonly done: Promise<{ code: number; stderr: string }>; +} + +/** Spawn a child; expose a `ready` handshake (printed just before its first write) + its final exit/stderr. */ +function runChild(args: readonly string[]): SpawnedChild { + const child = spawn(process.execPath, [CHILD_SCRIPT, ...args], { + stdio: ['ignore', 'pipe', 'pipe'], + }); + let stdout = ''; + let stderr = ''; + let signalReady: () => void = () => {}; + const ready = new Promise((resolve) => { + signalReady = resolve; + }); + child.stdout.on('data', (chunk: Buffer) => { + stdout += chunk.toString('utf8'); + if (stdout.includes('READY')) signalReady(); + }); + child.stderr.on('data', (chunk: Buffer) => { + stderr += chunk.toString('utf8'); + }); + const done = new Promise<{ code: number; stderr: string }>((resolve, reject) => { child.on('error', reject); - child.on('close', (code) => resolve({ code: code ?? -1, stderr })); + child.on('close', (code) => { + signalReady(); // a child that died before printing READY must not hang the parent's ready-wait + resolve({ code: code ?? -1, stderr }); + }); }); + return { ready, done }; } describe('concurrency e2e (2.5.I S3) — a run and a chat share one history.db', () => { @@ -178,30 +200,28 @@ describe('concurrency e2e (2.5.I S3) — a run and a chat share one history.db', setup.sqlite.close(); const PER_CHILD = 40; // A third connection that HOLDS the single WAL write lock while the children start, so their first - // upsert genuinely busy-waits cross-process every run (not luck-of-the-overlap). Released after a short - // hold; both children then acquire the lock in turn and complete. + // upsert genuinely busy-waits cross-process. `inTransaction` is the truth of whether the lock is held — + // it double-guards release and never COMMITs when no transaction is open. const holder = createClient(dbPath); - let released = false; const releaseLock = (): void => { - if (!released) { - released = true; - holder.sqlite.exec('COMMIT'); - } + if (holder.sqlite.inTransaction) holder.sqlite.exec('COMMIT'); }; try { holder.sqlite.exec('BEGIN IMMEDIATE'); // acquire the write lock; the children block on it - const childrenDone = Promise.all([ - runChild([DB_DIST, dbPath, 'a', String(PER_CHILD)]), - runChild([DB_DIST, dbPath, 'b', String(PER_CHILD)]), - ]); - // Give both children time to spawn + reach their first (blocked) write, then release the lock. If the - // release beats a slow spawn, the run simply proceeds without contention — the pass condition never - // depends on the timing, so this is a best-effort force, not a flake vector. + const childA = runChild([DB_DIST, dbPath, 'a', String(PER_CHILD)]); + const childB = runChild([DB_DIST, dbPath, 'b', String(PER_CHILD)]); + + // Deterministic handshake: both children print READY the instant before their first write, so once + // both signal they are (about to be) busy-waiting on the held lock. A small margin then covers the + // microgap between the signal and the child's BEGIN IMMEDIATE, so releasing here guarantees a real + // cross-process busy-wait every run — independent of Node/import startup speed (which can exceed a + // fixed timer). The `close`-resolves-`ready` fallback means a dead child can never hang this wait. + await Promise.all([childA.ready, childB.ready]); await new Promise((resolve) => { - setTimeout(resolve, 150); + setTimeout(resolve, 100); }); releaseLock(); - const [a, b] = await childrenDone; + const [a, b] = await Promise.all([childA.done, childB.done]); // Both children exited cleanly. A real failure prints the error to stderr + exits 1; a benign Node // warning line (deprecation, experimental flag) is tolerated — assert the exit code and the ABSENCE @@ -228,9 +248,16 @@ describe('concurrency e2e (2.5.I S3) — a run and a chat share one history.db', verify.sqlite.close(); } } finally { - releaseLock(); // release even if an assertion threw before the release point - holder.sqlite.close(); - rmSync(dir, { recursive: true, force: true }); + // Nested so a failure in one cleanup step never skips the rest (release → close → remove temp dir). + try { + releaseLock(); + } finally { + try { + holder.sqlite.close(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + } } }, ); diff --git a/apps/cli/src/harness/fixtures/concurrent-writer.mjs b/apps/cli/src/harness/fixtures/concurrent-writer.mjs index d0e80f52..8783b40f 100644 --- a/apps/cli/src/harness/fixtures/concurrent-writer.mjs +++ b/apps/cli/src/harness/fixtures/concurrent-writer.mjs @@ -25,6 +25,10 @@ try { uuid: () => `${prefix}0000000-0000-4000-8000-${String(++uuidN).padStart(12, '0')}`, now: () => 1_700_000_000_000, }); + // Signal readiness (the import + the connection are up) so the parent releases the held write lock only + // once we are about to write — a deterministic handshake that forces real cross-process contention every + // run, independent of Node/import startup latency (which can exceed a fixed parent-side timer). + process.stdout.write('READY\n'); for (let i = 0; i < count; i += 1) { providers.upsert({ name: `${prefix}-${i}`, From 018036265489cd94257d644c7436d5fc4bec0a28 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 22:49:10 +0300 Subject: [PATCH 17/50] =?UTF-8?q?test(cli):=202.5-close=20Step=206=20?= =?UTF-8?q?=E2=80=94=20Home=E2=86=92chat=E2=86=92resume=E2=86=92export=20c?= =?UTF-8?q?hain=20e2e=20(2.5.I=20S4)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The coverage the 2.K regression harness (offline `run` fixtures only) lacks: a transcript's full journey over a real file-backed history.db. - A fresh cassette-driven session (the Home graduating into chat via buildChatSession) persists a turn through the real createSessionPersister. - A chat-resume on a FRESH connection to the same file (loadFull → buildResumedChatSession) continues it, appending PAST the persisted max sequenceNumber. - exportSession serializes it to a .relavium.yaml. Asserts the roles land ['user','assistant','user','assistant'] gap-free [0,1,2,3] and both user turns survive into the exported scaffold. The interactive ink Home/chat TUI is out of scope (needs a render-test dep — SKIP-listed); this drives the exact session-host + persister + export seam the Home graduates into. Refs: docs/roadmap/phase-2.5-close-plan.md Step 6 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../cli/src/harness/session-chain.e2e.test.ts | 162 ++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 apps/cli/src/harness/session-chain.e2e.test.ts diff --git a/apps/cli/src/harness/session-chain.e2e.test.ts b/apps/cli/src/harness/session-chain.e2e.test.ts new file mode 100644 index 00000000..bc795262 --- /dev/null +++ b/apps/cli/src/harness/session-chain.e2e.test.ts @@ -0,0 +1,162 @@ +import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { + createClient, + createSessionStore, + runMigrations, + type DbClient, + type SessionStore, +} from '@relavium/db'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; + +import type { ResolvedChatConfig } from '../config/resolve.js'; +import { exportSession } from '../chat/export.js'; +import { cassetteResolver, type Cassette } from '../chat/fixture.js'; +import { createSessionPersister } from '../chat/persister.js'; +import { buildChatSession, buildResumedChatSession } from '../chat/session-host.js'; +import { textTurn } from '../chat/test-support.js'; + +/** + * 2.5.I S4 — the Home → chat → resume → export CHAIN over a real file-backed `history.db`, the coverage the + * 2.K regression harness (offline `run` fixtures only) does not carry. It proves a transcript survives the + * whole journey a user takes: a fresh session graduated from the Home (`buildChatSession`, cassette-driven) + * that persists a turn, a `chat-resume` on a FRESH connection to the same file (`loadFull` → + * `buildResumedChatSession`) that continues it, and a `chat-export` (`exportSession`) that serializes it to a + * `.relavium.yaml`. The interactive ink Home/chat surface itself is a TUI (out of scope without a render-test + * dependency); this drives the exact session-host + persister + export seam the Home graduates into. + */ + +const EMPTY_CHAT: ResolvedChatConfig = { + defaultModel: undefined, + fsScope: undefined, + maxTurns: undefined, + maxMessages: undefined, + autoCompact: undefined, + compactThreshold: undefined, + maxCostMicrocents: undefined, + onExceed: undefined, + allowedCommands: undefined, + allowedCommandGlobs: undefined, + reasoningEffort: undefined, +}; + +/** A one-turn cassette: the single `stream()` call replays the given assistant reply (the seam holds — all + * chunks are Relavium `StreamChunk`s). A fresh + a resumed build each get their OWN cassette (own call counter). */ +const oneTurnCassette = (reply: string): Cassette => ({ + schema_version: '1.0', + provider: 'anthropic', + calls: [textTurn(reply)], +}); + +const SESSION_ID = 'sess-chain'; + +describe('session chain e2e (2.5.I S4) — Home→chat→resume→export over a real history.db', () => { + let dir: string; + let dbPath: string; + + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'relavium-session-chain-')); + dbPath = join(dir, 'history.db'); + }); + + afterEach(() => { + rmSync(dir, { recursive: true, force: true }); + }); + + /** Open a fresh connection + a migrated session store on the shared file (each "process" gets its own). */ + function openStore(): { client: DbClient; store: SessionStore } { + const client = createClient(dbPath); + runMigrations(client.db); // idempotent — mirrors migrate-on-open + return { client, store: createSessionStore(client.db) }; + } + + it('a transcript survives fresh-chat → persist → resume → continue → export', async () => { + let tick = Date.parse('2026-07-07T00:00:00.000Z'); + const now = (): number => tick++; + let msgSeq = 0; + const uuidMsg = (): string => `m-${msgSeq++}`; + + // 1. FRESH session (the Home graduating into chat), cassette-driven, on connection #1. + const first = openStore(); + try { + const built = await buildChatSession({ + chat: EMPTY_CHAT, + agentRef: undefined, + cwd: dir, + projectConfigDir: undefined, + now, + uuid: () => SESSION_ID, + providers: cassetteResolver(oneTurnCassette('reply from the fresh session')), + }); + const persister = createSessionPersister({ + store: first.store, + handle: built.handle, + sessionId: built.sessionId, + agent: built.agent, + context: built.context, + now, + uuid: uuidMsg, + }); + persister.start(); + built.session.start(); // a fresh session must be started before its first turn (the resumed one lands idle) + persister.beginUserTurn('first user message'); // record the user text for the in-flight turn (as the REPL does) + await built.session.sendMessage('first user message'); + persister.close(); + } finally { + first.client.sqlite.close(); // the chat process exits + } + + // 2. RESUME on a FRESH connection (a separate `chat-resume` process reopening the file). + const second = openStore(); + try { + const loaded = second.store.loadFull(SESSION_ID); + if (loaded === undefined) { + throw new Error('expected the fresh session to have persisted before resume'); + } + // The fresh turn persisted a user + an assistant message. + expect(loaded.messages.map((m) => m.role)).toEqual(['user', 'assistant']); + + const resumed = await buildResumedChatSession({ + chat: EMPTY_CHAT, + record: loaded.session, + messages: loaded.messages, + now, + providers: cassetteResolver(oneTurnCassette('reply from the resumed session')), + }); + const persister = createSessionPersister({ + store: second.store, + handle: resumed.handle, + sessionId: resumed.sessionId, + agent: resumed.agent, + context: resumed.context, + now, + uuid: uuidMsg, + initialSequenceNumber: resumed.nextSequenceNumber, // continue PAST the persisted max seq + }); + persister.start(); + persister.beginUserTurn('second user message'); + await resumed.session.sendMessage('second user message'); + persister.close(); + + // The full chain persisted, in order, gap-free: two complete turns. + const full = second.store.loadFull(SESSION_ID); + expect(full?.messages.map((m) => m.role)).toEqual(['user', 'assistant', 'user', 'assistant']); + expect(full?.messages.map((m) => m.sequenceNumber)).toEqual([0, 1, 2, 3]); + + // 3. EXPORT → a share-safe `.relavium.yaml`; the transcript's user turns survive into the scaffold. + const result = exportSession({ + store: second.store, + sessionId: SESSION_ID, + cwd: dir, + force: false, + }); + const yaml = readFileSync(result.path, 'utf8'); + expect(yaml).toContain('first user message'); + expect(yaml).toContain('second user message'); + } finally { + second.client.sqlite.close(); + } + }); +}); From edf75088050aeaa35623c479277bedd7db91e79b Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 22:58:35 +0300 Subject: [PATCH 18/50] =?UTF-8?q?test(cli):=202.5-close=20Step=206=20Opus-?= =?UTF-8?q?review=20fixes=20=E2=80=94=20prove=20context-threading=20+=20to?= =?UTF-8?q?tal=20accumulation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review confirmed the chain genuine (real cross-process resume, deterministic, leak-free, both cassettes drive) and flagged assertions the test should make (all values verified to flow correctly, just unasserted): - MEDIUM: the resumed turn now uses a CAPTURING resolver that records its provider request; assert the request carried the reconstructed PRIOR transcript ('first user message' + 'reply from the fresh session') into the model's context — the defining behavior of resume, previously untested at this file-backed seam. - MEDIUM: assert the session totals DOUBLE across resume (adopt-then-accumulate) — a resume that failed to seed the prior totals would reset the row to 1×. - MEDIUM: assert both assistant replies survive into the exported YAML (not just the user turns). - LOW: assert resumed.sessionId === SESSION_ID; narrow `full` before mapping. - NIT: disableMcp: true on the fresh build to state the offline intent explicitly. Refs: docs/roadmap/phase-2.5-close-plan.md Step 6 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../cli/src/harness/session-chain.e2e.test.ts | 71 +++++++++++++++++-- 1 file changed, 65 insertions(+), 6 deletions(-) diff --git a/apps/cli/src/harness/session-chain.e2e.test.ts b/apps/cli/src/harness/session-chain.e2e.test.ts index bc795262..9a1d1be3 100644 --- a/apps/cli/src/harness/session-chain.e2e.test.ts +++ b/apps/cli/src/harness/session-chain.e2e.test.ts @@ -9,6 +9,7 @@ import { type DbClient, type SessionStore, } from '@relavium/db'; +import type { LlmProvider, LlmRequest } from '@relavium/llm'; import { afterEach, beforeEach, describe, expect, it } from 'vitest'; import type { ResolvedChatConfig } from '../config/resolve.js'; @@ -17,6 +18,8 @@ import { cassetteResolver, type Cassette } from '../chat/fixture.js'; import { createSessionPersister } from '../chat/persister.js'; import { buildChatSession, buildResumedChatSession } from '../chat/session-host.js'; import { textTurn } from '../chat/test-support.js'; +import type { ProviderResolver } from '../engine/providers.js'; +import { CHAT_TEXT_CAPABILITY_FLAGS } from '../test-support.js'; /** * 2.5.I S4 — the Home → chat → resume → export CHAIN over a real file-backed `history.db`, the coverage the @@ -50,6 +53,36 @@ const oneTurnCassette = (reply: string): Cassette => ({ calls: [textTurn(reply)], }); +/** + * A resolver that RECORDS each `stream()` request (unlike the cassette, which ignores it) and replays `reply` — + * used for the resumed turn so the test can prove `reconstructSessionState` threaded the PRIOR transcript into + * the next turn's provider request, not just that the DB rows accumulated. + */ +function capturingResolver(reply: string): { resolver: ProviderResolver; requests: LlmRequest[] } { + const requests: LlmRequest[] = []; + const provider: LlmProvider = { + id: 'anthropic', + supports: CHAT_TEXT_CAPABILITY_FLAGS, + generate: () => { + throw new Error('capturingResolver.generate is not used (the session path streams)'); + }, + stream: (req) => { + requests.push(req); + return (async function* () { + await Promise.resolve(); + for (const chunk of textTurn(reply)) yield chunk; + })(); + }, + }; + return { + resolver: { + resolveProvider: (id) => (id === 'anthropic' ? provider : undefined), + keyFor: () => 'test-key', + }, + requests, + }; +} + const SESSION_ID = 'sess-chain'; describe('session chain e2e (2.5.I S4) — Home→chat→resume→export over a real history.db', () => { @@ -89,6 +122,7 @@ describe('session chain e2e (2.5.I S4) — Home→chat→resume→export over a now, uuid: () => SESSION_ID, providers: cassetteResolver(oneTurnCassette('reply from the fresh session')), + disableMcp: true, // fully offline: the default agent declares no mcp_servers, so make the intent explicit }); const persister = createSessionPersister({ store: first.store, @@ -115,16 +149,22 @@ describe('session chain e2e (2.5.I S4) — Home→chat→resume→export over a if (loaded === undefined) { throw new Error('expected the fresh session to have persisted before resume'); } - // The fresh turn persisted a user + an assistant message. + // The fresh turn persisted a user + an assistant message, with usage/cost recorded. expect(loaded.messages.map((m) => m.role)).toEqual(['user', 'assistant']); + const freshInput = loaded.session.totalInputTokens; + const freshOutput = loaded.session.totalOutputTokens; + const freshCost = loaded.session.totalCostMicrocents; + expect(freshInput).toBeGreaterThan(0); // the fresh turn recorded real usage (so the doubling below has teeth) + const capture = capturingResolver('reply from the resumed session'); const resumed = await buildResumedChatSession({ chat: EMPTY_CHAT, record: loaded.session, messages: loaded.messages, now, - providers: cassetteResolver(oneTurnCassette('reply from the resumed session')), + providers: capture.resolver, }); + expect(resumed.sessionId).toBe(SESSION_ID); // resume continues the SAME session, never a fresh one const persister = createSessionPersister({ store: second.store, handle: resumed.handle, @@ -140,12 +180,29 @@ describe('session chain e2e (2.5.I S4) — Home→chat→resume→export over a await resumed.session.sendMessage('second user message'); persister.close(); + // The resumed turn's provider request carried the reconstructed PRIOR transcript into the model's context + // (the defining behavior of resume) — not merely appended DB rows. + const resumedRequest = capture.requests[0]; + expect(resumedRequest).toBeDefined(); + const requestJson = JSON.stringify(resumedRequest?.messages); + expect(requestJson).toContain('first user message'); + expect(requestJson).toContain('reply from the fresh session'); + expect(requestJson).toContain('second user message'); + // The full chain persisted, in order, gap-free: two complete turns. const full = second.store.loadFull(SESSION_ID); - expect(full?.messages.map((m) => m.role)).toEqual(['user', 'assistant', 'user', 'assistant']); - expect(full?.messages.map((m) => m.sequenceNumber)).toEqual([0, 1, 2, 3]); - - // 3. EXPORT → a share-safe `.relavium.yaml`; the transcript's user turns survive into the scaffold. + if (full === undefined) { + throw new Error('expected the resumed session to still exist after the second turn'); + } + expect(full.messages.map((m) => m.role)).toEqual(['user', 'assistant', 'user', 'assistant']); + expect(full.messages.map((m) => m.sequenceNumber)).toEqual([0, 1, 2, 3]); + // Resume ADOPTED the fresh turn's totals and ACCUMULATED an identical turn → exact double (a resume that + // failed to seed the prior totals would reset the row, and this would be 1×, not 2×). + expect(full.session.totalInputTokens).toBe(2 * freshInput); + expect(full.session.totalOutputTokens).toBe(2 * freshOutput); + expect(full.session.totalCostMicrocents).toBe(2 * freshCost); + + // 3. EXPORT → a share-safe `.relavium.yaml`; both turns (user + the replayed assistant reply) survive. const result = exportSession({ store: second.store, sessionId: SESSION_ID, @@ -154,7 +211,9 @@ describe('session chain e2e (2.5.I S4) — Home→chat→resume→export over a }); const yaml = readFileSync(result.path, 'utf8'); expect(yaml).toContain('first user message'); + expect(yaml).toContain('reply from the fresh session'); expect(yaml).toContain('second user message'); + expect(yaml).toContain('reply from the resumed session'); } finally { second.client.sqlite.close(); } From bfa5ee4a32094ef4dea19c5b553599c00e2cec47 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 23:05:50 +0300 Subject: [PATCH 19/50] =?UTF-8?q?test(cli):=202.5-close=20Step=206=20Sonne?= =?UTF-8?q?t-review=20fixes=20=E2=80=94=20exact=20reconstruction=20shape?= =?UTF-8?q?=20+=20full=20usage=20guards?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet review traced the engine cost/reconstruction mechanics (all confirmed correct) and flagged assertions that could false-green: - MEDIUM: the context-threading proof checked only that the turn strings appeared SOMEWHERE in the request — a scrambled/duplicated/mis-tagged reconstruction would pass. Now assert the EXACT role+order+text array ([user, assistant, user] with the precise texts) reconstructSessionState threaded into the request. - MEDIUM: add `> 0` guards for freshOutput + freshCost (only freshInput had one), so the 2× accumulation assertion keeps teeth on all three axes if a fixture/price drifts. - LOW: assert exactly one provider call drove the resumed turn (toHaveLength(1)) before reading requests[0]. Refs: docs/roadmap/phase-2.5-close-plan.md Step 6 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../cli/src/harness/session-chain.e2e.test.ts | 21 ++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/apps/cli/src/harness/session-chain.e2e.test.ts b/apps/cli/src/harness/session-chain.e2e.test.ts index 9a1d1be3..1f37bbec 100644 --- a/apps/cli/src/harness/session-chain.e2e.test.ts +++ b/apps/cli/src/harness/session-chain.e2e.test.ts @@ -154,7 +154,11 @@ describe('session chain e2e (2.5.I S4) — Home→chat→resume→export over a const freshInput = loaded.session.totalInputTokens; const freshOutput = loaded.session.totalOutputTokens; const freshCost = loaded.session.totalCostMicrocents; - expect(freshInput).toBeGreaterThan(0); // the fresh turn recorded real usage (so the doubling below has teeth) + // The fresh turn recorded real usage on every axis, so the doubling below has teeth on all three (a 0 + // would make `2 × 0 === 0` pass trivially and silently lose the accumulation guarantee). + expect(freshInput).toBeGreaterThan(0); + expect(freshOutput).toBeGreaterThan(0); + expect(freshCost).toBeGreaterThan(0); const capture = capturingResolver('reply from the resumed session'); const resumed = await buildResumedChatSession({ @@ -181,13 +185,16 @@ describe('session chain e2e (2.5.I S4) — Home→chat→resume→export over a persister.close(); // The resumed turn's provider request carried the reconstructed PRIOR transcript into the model's context - // (the defining behavior of resume) — not merely appended DB rows. + // (the defining behavior of resume) — not merely appended DB rows. Assert the exact role + order + text of + // the reconstructed history followed by the new user turn, so a scrambled/duplicated/mis-tagged + // reconstruction (which a substring check would miss) fails here. + expect(capture.requests).toHaveLength(1); // exactly one provider call drove the resumed turn const resumedRequest = capture.requests[0]; - expect(resumedRequest).toBeDefined(); - const requestJson = JSON.stringify(resumedRequest?.messages); - expect(requestJson).toContain('first user message'); - expect(requestJson).toContain('reply from the fresh session'); - expect(requestJson).toContain('second user message'); + expect(resumedRequest?.messages).toEqual([ + { role: 'user', content: [{ type: 'text', text: 'first user message' }] }, + { role: 'assistant', content: [{ type: 'text', text: 'reply from the fresh session' }] }, + { role: 'user', content: [{ type: 'text', text: 'second user message' }] }, + ]); // The full chain persisted, in order, gap-free: two complete turns. const full = second.store.loadFull(SESSION_ID); From fcad2bb956ca17c9623a0c4b28309fa8fa9c515c Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 23:12:33 +0300 Subject: [PATCH 20/50] =?UTF-8?q?test(cli):=202.5-close=20Step=207=20?= =?UTF-8?q?=E2=80=94=20query-shape=20perf=20budgets=20(2.5.I=20S5)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Query-shape budgets for the hot read paths (the 2.5.B Home recent-sessions/runs strips + `relavium list`), per close-plan D2 (shape, not flaky wall-clock): EXPLAIN QUERY PLAN asserts each read is INDEX-served with NO filesort (USE TEMP B-TREE) and NO full-table SCAN — the concrete guarantee the store docs claim ("served off idx_… (no filesort)"). - listSessions → idx_agent_sessions_updated - listRuns → idx_runs_created - loadFull's message read → a single indexed range scan (no N+1, no full scan) A dropped/renamed index, or an ORDER BY the index stops covering, flips the plan and fails here. The other §2.5.I perf item — the 80×24 narrow-terminal degrade — already exists and is asserted in render/tui/home-projection.test.ts (homeFitsTerminal / tooSmallMessage at the boundary), so it needs no new coverage (close-plan D4). Refs: docs/roadmap/phase-2.5-close-plan.md Step 7 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/harness/perf-budget.e2e.test.ts | 90 ++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 apps/cli/src/harness/perf-budget.e2e.test.ts diff --git a/apps/cli/src/harness/perf-budget.e2e.test.ts b/apps/cli/src/harness/perf-budget.e2e.test.ts new file mode 100644 index 00000000..ec7e45a6 --- /dev/null +++ b/apps/cli/src/harness/perf-budget.e2e.test.ts @@ -0,0 +1,90 @@ +import { + agentSessions, + createClient, + runMigrations, + runs, + sessionMessages, + type DbClient, +} from '@relavium/db'; +import { asc, desc, eq, isNull, type SQL } from 'drizzle-orm'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; + +/** + * 2.5.I S5 — query-shape perf budgets for the CLI's hot read paths (the 2.5.B Home "recent sessions/runs" + * strips + the `relavium list` reads). Rather than a flaky wall-clock number, this asserts the *shape* of the + * plan SQLite chooses: the read is served off its intended index with **no filesort** (`USE TEMP B-TREE`) and + * **no full table SCAN** — the concrete guarantee the store docs claim ("served off `idx_…` (no filesort)"). + * A dropped/renamed index, or an `ORDER BY` the index no longer covers, flips the plan and fails here. + * + * (The other §2.5.I perf item — the 80×24 narrow-terminal degrade — already exists and is asserted directly + * in `render/tui/home-projection.test.ts` (`homeFitsTerminal` / `tooSmallMessage` at the 80×24 boundary), so + * it needs no new coverage here.) + */ + +describe('query-shape perf budgets (2.5.I S5) — the hot reads stay index-served, no filesort', () => { + let client: DbClient; + + beforeEach(() => { + client = createClient(':memory:'); + runMigrations(client.db); + }); + + afterEach(() => { + client.sqlite.close(); + }); + + /** The `EXPLAIN QUERY PLAN` `detail` lines for a drizzle query (its `?`-placeholder SQL + bound params). */ + function planFor(query: { toSQL: () => { sql: string; params: unknown[] } }): string[] { + const { sql, params } = query.toSQL(); + // The drizzle `params` are valid SQLite bind values (strings/numbers); EXPLAIN ignores their values but the + // `?` placeholders must still be bound. Read the `detail` column defensively at the DB boundary. + const bind = params as ReadonlyArray; + const rows = client.sqlite.prepare(`EXPLAIN QUERY PLAN ${sql}`).all(...bind); + return rows.map((row) => (row as { detail?: string }).detail ?? ''); + } + + /** A read is within budget when it uses an index for the scan AND never sorts in a temp b-tree (filesort). */ + function expectIndexServedNoFilesort(plan: string[], indexName: string): void { + const joined = plan.join('\n'); + expect(joined).toContain('USING INDEX'); // an index serves the scan, not a full-table SCAN + expect(joined).toContain(indexName); // and specifically the intended index + expect(joined).not.toMatch(/USE TEMP B-TREE/); // the ORDER BY is index-served — no filesort + expect(joined).not.toMatch(/SCAN (agent_sessions|runs)(?! USING)/); // no bare full-table scan + } + + it('listSessions (recent-sessions strip / chat-list) is served off idx_agent_sessions_updated', () => { + // Mirrors session-store.listSessions: non-deleted, most-recently-updated first, id tiebreak, top-N. + const query = client.db + .select() + .from(agentSessions) + .where(isNull(agentSessions.deletedAt)) + .orderBy(desc(agentSessions.updatedAt), desc(agentSessions.id)) + .limit(8); + expectIndexServedNoFilesort(planFor(query), 'idx_agent_sessions_updated'); + }); + + it('listRuns (recent-runs strip / relavium list) is served off idx_runs_created', () => { + // Mirrors run-history.listRuns: non-deleted, newest-first, id tiebreak, top-N. + const query = client.db + .select() + .from(runs) + .where(isNull(runs.deletedAt)) + .orderBy(desc(runs.createdAt), desc(runs.id)) + .limit(8); + expectIndexServedNoFilesort(planFor(query), 'idx_runs_created'); + }); + + it('loadFull messages read is a single indexed range scan (no N+1, no full scan)', () => { + // Mirrors session-store.loadMessages: one ordered range read of a session's transcript, keyed by session_id. + const filter: SQL = eq(sessionMessages.sessionId, 'sess-1'); + const query = client.db + .select() + .from(sessionMessages) + .where(filter) + .orderBy(asc(sessionMessages.sequenceNumber)); + const plan = planFor(query).join('\n'); + // A single index-backed lookup by session_id — never a per-message (N+1) fan-out or a full-table scan. + expect(plan).toContain('USING INDEX'); + expect(plan).not.toMatch(/SCAN session_messages(?! USING)/); + }); +}); From c276145a6b2ea80fd351a6c9d20c664499b4a0df Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 23:22:26 +0300 Subject: [PATCH 21/50] =?UTF-8?q?test(cli,db):=202.5-close=20Step=207=20Op?= =?UTF-8?q?us-review=20fixes=20=E2=80=94=20close=20the=20loadMessages=20fi?= =?UTF-8?q?lesort=20gap=20+=20drift=20coupling?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review verified the replicas match the stores exactly and inspected real plans; two MEDIUMs worth closing: - MEDIUM: the loadMessages budget FALSE-PASSED a filesort regression — it omitted the index-name pin + the USE TEMP B-TREE guard, so dropping idx_session_messages_seq (which falls to idx_session_messages_session + a temp b-tree) still passed. Parameterize the helper's table name and run loadMessages through it, pinning idx_session_messages_seq and forbidding the temp b-tree — now all three reads share one enforced contract. - MEDIUM: the test replicates the store queries (the stores don't expose their SQL); strengthen the drift coupling from a bare "Mirrors" comment to explicit bidirectional file:line back-references — added at the session-store listSessions/loadMessages and run-history-store listRuns query sites, so a store ORDER BY edit trips the reviewer. - LOW: drop `.limit` so the plan covers both the bounded Home strip AND the unbounded chat-list/`relavium list` (identical plan); soften the "no N+1" wording to what a single-statement EXPLAIN actually proves ("one indexed range scan, not a per-message read"). - NIT: document the covering-index + SQLite-wording caveats on the helper. Refs: docs/roadmap/phase-2.5-close-plan.md Step 7 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/harness/perf-budget.e2e.test.ts | 42 +++++++++++--------- packages/db/src/run-history-store.ts | 2 + packages/db/src/session-store.ts | 4 ++ 3 files changed, 30 insertions(+), 18 deletions(-) diff --git a/apps/cli/src/harness/perf-budget.e2e.test.ts b/apps/cli/src/harness/perf-budget.e2e.test.ts index ec7e45a6..02eba65e 100644 --- a/apps/cli/src/harness/perf-budget.e2e.test.ts +++ b/apps/cli/src/harness/perf-budget.e2e.test.ts @@ -43,48 +43,54 @@ describe('query-shape perf budgets (2.5.I S5) — the hot reads stay index-serve return rows.map((row) => (row as { detail?: string }).detail ?? ''); } - /** A read is within budget when it uses an index for the scan AND never sorts in a temp b-tree (filesort). */ - function expectIndexServedNoFilesort(plan: string[], indexName: string): void { + /** + * A read is within budget when the ORDER BY is served off its intended index (no filesort) and the scan is + * index-backed (no full-table SCAN). The `indexName` pin is the primary guard (a dropped/renamed index fails + * here); the SCAN regex is belt-and-suspenders and is coupled to modern SQLite's `SCAN ` plan wording. + * (If a future `.select()` narrows to only-indexed columns, SQLite emits `USING COVERING INDEX` — still + * index-served, but the `USING INDEX` substring would need loosening then.) + */ + function expectIndexServedNoFilesort(plan: string[], table: string, indexName: string): void { const joined = plan.join('\n'); expect(joined).toContain('USING INDEX'); // an index serves the scan, not a full-table SCAN - expect(joined).toContain(indexName); // and specifically the intended index + expect(joined).toContain(indexName); // and specifically the intended index — the real teeth of the budget expect(joined).not.toMatch(/USE TEMP B-TREE/); // the ORDER BY is index-served — no filesort - expect(joined).not.toMatch(/SCAN (agent_sessions|runs)(?! USING)/); // no bare full-table scan + expect(joined).not.toMatch(new RegExp(`SCAN ${table}(?! USING)`)); // no bare full-table scan } it('listSessions (recent-sessions strip / chat-list) is served off idx_agent_sessions_updated', () => { - // Mirrors session-store.listSessions: non-deleted, most-recently-updated first, id tiebreak, top-N. + // Mirrors session-store.ts `listSessions` (keep in sync — that query site back-references this budget): + // non-deleted, most-recently-updated first, id tiebreak. No `.limit` — the plan is identical for the bounded + // Home strip and the unbounded `chat-list`, so this one case covers both callers. const query = client.db .select() .from(agentSessions) .where(isNull(agentSessions.deletedAt)) - .orderBy(desc(agentSessions.updatedAt), desc(agentSessions.id)) - .limit(8); - expectIndexServedNoFilesort(planFor(query), 'idx_agent_sessions_updated'); + .orderBy(desc(agentSessions.updatedAt), desc(agentSessions.id)); + expectIndexServedNoFilesort(planFor(query), 'agent_sessions', 'idx_agent_sessions_updated'); }); it('listRuns (recent-runs strip / relavium list) is served off idx_runs_created', () => { - // Mirrors run-history.listRuns: non-deleted, newest-first, id tiebreak, top-N. + // Mirrors run-history-store.ts `listRuns` (the no-status branch; keep in sync — back-referenced there): + // non-deleted, newest-first, id tiebreak. Unbounded — same plan as the bounded Home strip. const query = client.db .select() .from(runs) .where(isNull(runs.deletedAt)) - .orderBy(desc(runs.createdAt), desc(runs.id)) - .limit(8); - expectIndexServedNoFilesort(planFor(query), 'idx_runs_created'); + .orderBy(desc(runs.createdAt), desc(runs.id)); + expectIndexServedNoFilesort(planFor(query), 'runs', 'idx_runs_created'); }); - it('loadFull messages read is a single indexed range scan (no N+1, no full scan)', () => { - // Mirrors session-store.loadMessages: one ordered range read of a session's transcript, keyed by session_id. + it('loadFull messages read is one indexed range scan (not a per-message read), served off idx_session_messages_seq', () => { + // Mirrors session-store.ts `loadMessages` (keep in sync — back-referenced there): one ordered range read of + // a session's transcript, keyed by session_id. Pinning the (session_id, sequence_number) index + forbidding a + // temp b-tree is what stops SQLite silently falling to the (session_id, created_at) index with a filesort. const filter: SQL = eq(sessionMessages.sessionId, 'sess-1'); const query = client.db .select() .from(sessionMessages) .where(filter) .orderBy(asc(sessionMessages.sequenceNumber)); - const plan = planFor(query).join('\n'); - // A single index-backed lookup by session_id — never a per-message (N+1) fan-out or a full-table scan. - expect(plan).toContain('USING INDEX'); - expect(plan).not.toMatch(/SCAN session_messages(?! USING)/); + expectIndexServedNoFilesort(planFor(query), 'session_messages', 'idx_session_messages_seq'); }); }); diff --git a/packages/db/src/run-history-store.ts b/packages/db/src/run-history-store.ts index 35933197..0fa2a967 100644 --- a/packages/db/src/run-history-store.ts +++ b/packages/db/src/run-history-store.ts @@ -502,6 +502,8 @@ export function createRunHistoryReader(db: Db): RunHistoryReader { opts?.status === undefined ? isNull(runs.deletedAt) : and(eq(runs.status, opts.status), isNull(runs.deletedAt)); + // The no-status ORDER BY is index-served (idx_runs_created, no filesort); its plan is pinned by + // apps/cli/src/harness/perf-budget.e2e.test.ts (2.5.I S5) — keep the WHERE/ORDER BY in sync with it. const query = db .select() .from(runs) diff --git a/packages/db/src/session-store.ts b/packages/db/src/session-store.ts index 69f73eef..55d32b2a 100644 --- a/packages/db/src/session-store.ts +++ b/packages/db/src/session-store.ts @@ -211,6 +211,8 @@ export function createSessionStore(db: Db): SessionStore { }; const listSessions = (opts?: { readonly limit?: number }): AgentSessionRecord[] => { + // This ORDER BY is index-served (idx_agent_sessions_updated, no filesort); its plan is pinned by + // apps/cli/src/harness/perf-budget.e2e.test.ts (2.5.I S5) — keep the WHERE/ORDER BY in sync with that budget. const query = db .select() .from(agentSessions) @@ -228,6 +230,8 @@ export function createSessionStore(db: Db): SessionStore { }; const loadMessages = (sessionId: string): SessionMessage[] => + // Ordered range read by session_id — index-served (idx_session_messages_seq, no filesort); its plan is + // pinned by apps/cli/src/harness/perf-budget.e2e.test.ts (2.5.I S5). db .select() .from(sessionMessages) From 3eebb522bd17441ae904b21b283cb66c489419ab Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 23:32:56 +0300 Subject: [PATCH 22/50] =?UTF-8?q?test(cli):=202.5-close=20Step=207=20Sonne?= =?UTF-8?q?t-review=20fixes=20=E2=80=94=20covering-index=20tolerance=20+?= =?UTF-8?q?=20closed=20table=20union?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet mutation-tested every budget (dropped each pinned index, confirmed all three now fail correctly + that the pre-fix loadMessages false-passed the filesort). Two cheap LOW hardenings: - Loosen the index-served check to /USING (COVERING )?INDEX/ so a future .select() narrowing (which flips SQLite to "USING COVERING INDEX") doesn't false-FAIL a still-index-served plan. - Type the helper's `table` as a closed union ('agent_sessions'|'runs'| 'session_messages') so a computed/metachar table name can never reach the RegExp. Refs: docs/roadmap/phase-2.5-close-plan.md Step 7 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/harness/perf-budget.e2e.test.ts | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/apps/cli/src/harness/perf-budget.e2e.test.ts b/apps/cli/src/harness/perf-budget.e2e.test.ts index 02eba65e..5d8caf72 100644 --- a/apps/cli/src/harness/perf-budget.e2e.test.ts +++ b/apps/cli/src/harness/perf-budget.e2e.test.ts @@ -50,9 +50,13 @@ describe('query-shape perf budgets (2.5.I S5) — the hot reads stay index-serve * (If a future `.select()` narrows to only-indexed columns, SQLite emits `USING COVERING INDEX` — still * index-served, but the `USING INDEX` substring would need loosening then.) */ - function expectIndexServedNoFilesort(plan: string[], table: string, indexName: string): void { + function expectIndexServedNoFilesort( + plan: string[], + table: 'agent_sessions' | 'runs' | 'session_messages', // closed union — no computed/metachar table names + indexName: string, + ): void { const joined = plan.join('\n'); - expect(joined).toContain('USING INDEX'); // an index serves the scan, not a full-table SCAN + expect(joined).toMatch(/USING (COVERING )?INDEX/); // an index serves the scan, not a full-table SCAN expect(joined).toContain(indexName); // and specifically the intended index — the real teeth of the budget expect(joined).not.toMatch(/USE TEMP B-TREE/); // the ORDER BY is index-served — no filesort expect(joined).not.toMatch(new RegExp(`SCAN ${table}(?! USING)`)); // no bare full-table scan From 0059f0d23ce3eee57f8dafa77f502d307e232717 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 23:40:36 +0300 Subject: [PATCH 23/50] =?UTF-8?q?ci(cli):=202.5-close=20Step=208=20?= =?UTF-8?q?=E2=80=94=20Windows=20concurrency=20CI=20job=20+=20headless=20r?= =?UTF-8?q?aw-mode=20smoke=20(2.5.I=20S6)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The DB write-path hardening (BEGIN IMMEDIATE + the retry's Atomics.wait sleep + WAL locking), the two-process concurrency e2e, and the no-TTY raw-mode path are the parts most likely to diverge on Windows — but ci.yml ran ubuntu-only (release.yml already smokes the built binary cross-OS, but never the test suite). - Add an advisory `windows-concurrency` job to ci.yml: install + build, then the @relavium/db concurrency suite + the CLI concurrency/perf/session-chain harness on windows-latest, plus a headless raw-mode smoke (--help loads the native addons; a non-TTY `chat` must exit cleanly, never a setRawMode crash). The required check stays the ubuntu `ci` job; POSIX 0600/0700 assertions are not exercised (Windows no-op, ADR-0050). - Fix a real cross-OS bug the Windows job would expose: the two-process e2e's child did `await import()`, which fails on Windows (a bare `C:\…` path is not a valid import specifier). Pass the built @relavium/db as a `file://` URL (cross-OS); keep the path form only for the existence gate. Still green on macOS. Refs: docs/roadmap/phase-2.5-close-plan.md Step 8 Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/ci.yml | 42 ++++++++++++++++++++ apps/cli/src/harness/concurrency.e2e.test.ts | 11 +++-- 2 files changed, 49 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4eb8c046..b9db1822 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -175,6 +175,48 @@ jobs: - name: Engine coverage floor (>=90% line+branch) run: pnpm coverage + # Cross-OS concurrency + headless gate (2.5.I S6). The DB write-path hardening (BEGIN IMMEDIATE + the + # SQLITE_BUSY retry's Atomics.wait sleep + WAL locking), the two-process concurrency e2e (a child spawn + + # a file:// import of the built @relavium/db), and the no-TTY raw-mode path are the parts most likely to + # behave differently on Windows. Runs the @relavium/db concurrency suite + the CLI concurrency/perf harness + # there, plus a headless raw-mode smoke. A SEPARATE, advisory job (the required check stays the ubuntu `ci` + # job) — promote to required in branch protection once confirmed stable. POSIX 0600/0700 perm assertions are + # NOT exercised here (a documented Windows no-op — ADR-0050). + windows-concurrency: + name: windows · db concurrency + headless smoke (advisory) + runs-on: windows-latest + timeout-minutes: 25 + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + persist-credentials: false + - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + with: + node-version-file: .nvmrc + cache: pnpm + - name: Install (frozen lockfile) + run: pnpm install --frozen-lockfile + # The two-process concurrency e2e spawns children that import the BUILT @relavium/db (by file:// URL), + # so the dist must exist before the tests run. + - name: Build + run: pnpm turbo run build + - name: DB concurrency suite (BEGIN IMMEDIATE + SQLITE_BUSY retry + WAL behave identically on Windows) + run: pnpm --filter @relavium/db test + - name: CLI concurrency + query-shape harness (cross-process spawn + EXPLAIN plans on Windows) + run: pnpm --filter relavium exec vitest run concurrency perf-budget session-chain + - name: Headless raw-mode smoke (no TTY ⇒ the CLI must not crash entering raw mode) + shell: bash + run: | + set -euo pipefail + node apps/cli/dist/index.js --help >/dev/null + # `chat` needs an interactive TTY; a piped (non-TTY) stdin must exit cleanly — never a setRawMode crash. + out="$(printf '' | node apps/cli/dist/index.js chat 2>&1 || true)" + if printf '%s' "$out" | grep -qiE 'setRawMode|raw mode'; then + echo "::error::chat crashed entering raw mode without a TTY"; exit 1 + fi + echo "✓ windows headless raw-mode smoke passed" + # --- Reserved Phase-1 lanes (TODO: enable with the first provider adapter) ------------ # The per-provider conformance suite and the nightly live-API lane land WITH the adapters # in Phase 1 (testing.md); only their CI slots are reserved here so the testing standard diff --git a/apps/cli/src/harness/concurrency.e2e.test.ts b/apps/cli/src/harness/concurrency.e2e.test.ts index 05485d0f..0fb21d6c 100644 --- a/apps/cli/src/harness/concurrency.e2e.test.ts +++ b/apps/cli/src/harness/concurrency.e2e.test.ts @@ -45,7 +45,10 @@ import { captureIo } from '../test-support.js'; */ const FIXTURES_DIR = fileURLToPath(new URL('./fixtures/', import.meta.url)); -const DB_DIST = fileURLToPath(new URL('../../../../packages/db/dist/index.js', import.meta.url)); +// The child receives the built db as a file:// URL (its `import()` needs a URL — a bare Windows path like +// `C:\…` is not a valid import specifier); the path form is only for the existence gate. +const DB_DIST_URL = new URL('../../../../packages/db/dist/index.js', import.meta.url); +const DB_DIST_PATH = fileURLToPath(DB_DIST_URL); const CHILD_SCRIPT = fileURLToPath(new URL('./fixtures/concurrent-writer.mjs', import.meta.url)); const ISO = '2026-07-07T00:00:00.000Z'; @@ -189,7 +192,7 @@ describe('concurrency e2e (2.5.I S3) — a run and a chat share one history.db', } }); - it.skipIf(!existsSync(DB_DIST))( + it.skipIf(!existsSync(DB_DIST_PATH))( 'cross-process safety: two child processes contend on one file (parent holds the lock) — all writes land', async () => { const dir = mkdtempSync(join(tmpdir(), 'relavium-concurrency-2p-')); @@ -208,8 +211,8 @@ describe('concurrency e2e (2.5.I S3) — a run and a chat share one history.db', }; try { holder.sqlite.exec('BEGIN IMMEDIATE'); // acquire the write lock; the children block on it - const childA = runChild([DB_DIST, dbPath, 'a', String(PER_CHILD)]); - const childB = runChild([DB_DIST, dbPath, 'b', String(PER_CHILD)]); + const childA = runChild([DB_DIST_URL.href, dbPath, 'a', String(PER_CHILD)]); + const childB = runChild([DB_DIST_URL.href, dbPath, 'b', String(PER_CHILD)]); // Deterministic handshake: both children print READY the instant before their first write, so once // both signal they are (about to be) busy-waiting on the held lock. A small margin then covers the From 60b2cc8a9db3e04c4b0d0a5f724a0b0de5bc81f2 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 7 Jul 2026 23:52:03 +0300 Subject: [PATCH 24/50] =?UTF-8?q?ci(cli,db):=202.5-close=20Step=208=20Opus?= =?UTF-8?q?-review=20fixes=20=E2=80=94=20Windows=20rmSync=20flake=20guard?= =?UTF-8?q?=20+=20honest=20smoke=20framing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review verified the file:// fix (empirically on Node 22), the YAML/pins, and the vitest filter, and flagged: - MEDIUM: the "raw-mode smoke" mislabeled what it does — `chat` selects the PLAIN driver when stdout is a pipe (the driver-selection gate is TTY-gated), so ink/setRawMode is never entered on ANY OS. Reword the job/step comments to what actually runs: the bundle loads, and a piped `chat` picks the plain driver + exits cleanly on EOF (the smoke guards that driver-selection gate, not Windows raw mode). - MEDIUM: `rmSync(dir, { recursive, force })` has no EPERM/EBUSY retry — a real Windows flake when deleting a dir that just held SQLite WAL sidecars. Add maxRetries:5 + retryDelay:100 to all 13 SQLite-teardown rmSync sites the Windows job runs (the concurrency + session-chain harness + the @relavium/db suite). - LOW: add the Turborepo cache restore to the Windows job (parity with ci/coverage; the cold build was tight against the 25-min budget). Soften the --help native-addon claim (the real Windows addon exercise is the DB vitest step). Full db suite (207) + CLI harness (6) green; lint + format clean. Refs: docs/roadmap/phase-2.5-close-plan.md Step 8 Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/ci.yml | 33 ++++++++++++------- apps/cli/src/harness/concurrency.e2e.test.ts | 4 +-- .../cli/src/harness/session-chain.e2e.test.ts | 2 +- packages/db/src/client.test.ts | 2 +- packages/db/src/media-store.test.ts | 8 ++--- packages/db/src/media-write.test.ts | 4 +-- packages/db/src/retry.test.ts | 4 +-- packages/db/src/session-store.test.ts | 2 +- 8 files changed, 35 insertions(+), 24 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b9db1822..f5c1307b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -176,12 +176,14 @@ jobs: run: pnpm coverage # Cross-OS concurrency + headless gate (2.5.I S6). The DB write-path hardening (BEGIN IMMEDIATE + the - # SQLITE_BUSY retry's Atomics.wait sleep + WAL locking), the two-process concurrency e2e (a child spawn + - # a file:// import of the built @relavium/db), and the no-TTY raw-mode path are the parts most likely to - # behave differently on Windows. Runs the @relavium/db concurrency suite + the CLI concurrency/perf harness - # there, plus a headless raw-mode smoke. A SEPARATE, advisory job (the required check stays the ubuntu `ci` - # job) — promote to required in branch protection once confirmed stable. POSIX 0600/0700 perm assertions are - # NOT exercised here (a documented Windows no-op — ADR-0050). + # SQLITE_BUSY retry's Atomics.wait sleep + WAL locking) and the two-process concurrency e2e (a child spawn + + # a file:// import of the built @relavium/db) are the parts most likely to behave differently on Windows — + # ci.yml ran ubuntu-only (release.yml already smokes the built binary cross-OS, but never the test suite). + # Runs the @relavium/db concurrency suite (the real Windows native-addon + WAL exercise) + the CLI + # concurrency/perf harness there, plus a headless no-TTY smoke. A SEPARATE, advisory job (the required check + # stays the ubuntu `ci` job) — promote to required in branch protection once confirmed stable. POSIX 0600/0700 + # perm assertions are NOT exercised here (a documented Windows no-op — ADR-0050); nor is ink's raw-mode code, + # which is TTY-gated (the driver-selection gate picks the plain driver without a TTY — see the smoke below). windows-concurrency: name: windows · db concurrency + headless smoke (advisory) runs-on: windows-latest @@ -197,6 +199,13 @@ jobs: cache: pnpm - name: Install (frozen lockfile) run: pnpm install --frozen-lockfile + - name: Restore Turborepo cache + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: .turbo + key: turbo-${{ runner.os }}-${{ github.sha }} + restore-keys: | + turbo-${{ runner.os }}- # The two-process concurrency e2e spawns children that import the BUILT @relavium/db (by file:// URL), # so the dist must exist before the tests run. - name: Build @@ -205,17 +214,19 @@ jobs: run: pnpm --filter @relavium/db test - name: CLI concurrency + query-shape harness (cross-process spawn + EXPLAIN plans on Windows) run: pnpm --filter relavium exec vitest run concurrency perf-budget session-chain - - name: Headless raw-mode smoke (no TTY ⇒ the CLI must not crash entering raw mode) + - name: Headless no-TTY smoke (the bundle loads; a piped chat picks the plain driver and exits cleanly) shell: bash run: | set -euo pipefail - node apps/cli/dist/index.js --help >/dev/null - # `chat` needs an interactive TTY; a piped (non-TTY) stdin must exit cleanly — never a setRawMode crash. + node apps/cli/dist/index.js --help >/dev/null # the tsup bundle loads + --help renders + # `chat` is TTY-gated: with a piped (non-TTY) stdin the driver-selection gate picks the PLAIN driver + # (ink/setRawMode is never entered), and the plain driver must exit cleanly on EOF — never crash. If a + # regression routed to ink without a TTY it would throw "Raw mode is not supported…", which this catches. out="$(printf '' | node apps/cli/dist/index.js chat 2>&1 || true)" if printf '%s' "$out" | grep -qiE 'setRawMode|raw mode'; then - echo "::error::chat crashed entering raw mode without a TTY"; exit 1 + echo "::error::chat entered raw mode without a TTY (the driver-selection gate regressed)"; exit 1 fi - echo "✓ windows headless raw-mode smoke passed" + echo "✓ windows headless no-TTY smoke passed" # --- Reserved Phase-1 lanes (TODO: enable with the first provider adapter) ------------ # The per-provider conformance suite and the nightly live-API lane land WITH the adapters diff --git a/apps/cli/src/harness/concurrency.e2e.test.ts b/apps/cli/src/harness/concurrency.e2e.test.ts index 0fb21d6c..a38c3a1f 100644 --- a/apps/cli/src/harness/concurrency.e2e.test.ts +++ b/apps/cli/src/harness/concurrency.e2e.test.ts @@ -186,7 +186,7 @@ describe('concurrency e2e (2.5.I S3) — a run and a chat share one history.db', try { chatClient.sqlite.close(); } finally { - rmSync(dir, { recursive: true, force: true }); + rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); } } } @@ -258,7 +258,7 @@ describe('concurrency e2e (2.5.I S3) — a run and a chat share one history.db', try { holder.sqlite.close(); } finally { - rmSync(dir, { recursive: true, force: true }); + rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); } } } diff --git a/apps/cli/src/harness/session-chain.e2e.test.ts b/apps/cli/src/harness/session-chain.e2e.test.ts index 1f37bbec..cda89743 100644 --- a/apps/cli/src/harness/session-chain.e2e.test.ts +++ b/apps/cli/src/harness/session-chain.e2e.test.ts @@ -95,7 +95,7 @@ describe('session chain e2e (2.5.I S4) — Home→chat→resume→export over a }); afterEach(() => { - rmSync(dir, { recursive: true, force: true }); + rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); }); /** Open a fresh connection + a migrated session store on the shared file (each "process" gets its own). */ diff --git a/packages/db/src/client.test.ts b/packages/db/src/client.test.ts index 05d74071..497ecb83 100644 --- a/packages/db/src/client.test.ts +++ b/packages/db/src/client.test.ts @@ -60,7 +60,7 @@ beforeAll(() => { afterAll(() => { client.sqlite.close(); - rmSync(tmpDir, { recursive: true, force: true }); + rmSync(tmpDir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); }); describe('@relavium/db migrations + client', () => { diff --git a/packages/db/src/media-store.test.ts b/packages/db/src/media-store.test.ts index 5bd1ad52..93205b9d 100644 --- a/packages/db/src/media-store.test.ts +++ b/packages/db/src/media-store.test.ts @@ -19,7 +19,7 @@ describe('FilesystemMediaStore (1.AF, ADR-0042 — content-addressed CAS)', () = root = mkdtempSync(join(tmpdir(), 'relavium-media-')); store = new FilesystemMediaStore(root); }); - afterAll(() => rmSync(root, { recursive: true, force: true })); + afterAll(() => rmSync(root, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 })); it('put returns the canonical media://sha256- handle = sha256 of the bytes', async () => { const handle = await store.put(HELLO); @@ -58,7 +58,7 @@ describe('FilesystemMediaStore (1.AF, ADR-0042 — content-addressed CAS)', () = writeFileSync(casPath, new Uint8Array([0xff])); // overwrite the stored bytes in place await expect(tamperStore.get(handle)).rejects.toThrow(/content-address/); } finally { - rmSync(tamperRoot, { recursive: true, force: true }); + rmSync(tamperRoot, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); } }); }); @@ -114,7 +114,7 @@ describe('MediaStore.readRange (1.AF/D13 — byte-delivery Range gate)', () => { /handle/, ); } finally { - rmSync(root, { recursive: true, force: true }); + rmSync(root, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); } }); }); @@ -126,7 +126,7 @@ describe('FilesystemMediaStore — host GC support (2.S/D-GC: delete + listHandl root = mkdtempSync(join(tmpdir(), 'relavium-media-gc-')); store = new FilesystemMediaStore(root); }); - afterEach(() => rmSync(root, { recursive: true, force: true })); + afterEach(() => rmSync(root, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 })); it('listHandles enumerates every stored handle (with mtime); an empty or absent root yields []', async () => { expect(await store.listHandles()).toEqual([]); // the root exists but is empty diff --git a/packages/db/src/media-write.test.ts b/packages/db/src/media-write.test.ts index b2b91a10..f42d14be 100644 --- a/packages/db/src/media-write.test.ts +++ b/packages/db/src/media-write.test.ts @@ -17,8 +17,8 @@ describe('createFilesystemMediaWrite (1.AF/D16, ADR-0044 §2 — save_to write p outside = mkdtempSync(join(tmpdir(), 'relavium-outside-')); }); afterEach(() => { - rmSync(root, { recursive: true, force: true }); - rmSync(outside, { recursive: true, force: true }); + rmSync(root, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); + rmSync(outside, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); }); it('writes the bytes to a nested relative path under the scope root, creating dirs', async () => { diff --git a/packages/db/src/retry.test.ts b/packages/db/src/retry.test.ts index 36668e18..f4c2fd85 100644 --- a/packages/db/src/retry.test.ts +++ b/packages/db/src/retry.test.ts @@ -188,7 +188,7 @@ describe('withBusyRetry — real SQLITE_BUSY contention (2.5.I)', () => { try { writer.sqlite.close(); } finally { - rmSync(dir, { recursive: true, force: true }); + rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); } } } @@ -225,7 +225,7 @@ describe('withBusyRetry — real SQLITE_BUSY contention (2.5.I)', () => { try { writer.sqlite.close(); } finally { - rmSync(dir, { recursive: true, force: true }); + rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); } } } diff --git a/packages/db/src/session-store.test.ts b/packages/db/src/session-store.test.ts index e7c209cb..4fcf46c9 100644 --- a/packages/db/src/session-store.test.ts +++ b/packages/db/src/session-store.test.ts @@ -500,7 +500,7 @@ describe('SessionStore — loadFull snapshot isolation (2.5.I)', () => { try { reader.sqlite.close(); } finally { - rmSync(dir, { recursive: true, force: true }); + rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); } } }); From 6a37f89584742270be2508eb012ab8fb0bd4488b Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 00:01:11 +0300 Subject: [PATCH 25/50] =?UTF-8?q?ci:=202.5-close=20Step=208=20Sonnet-revie?= =?UTF-8?q?w=20fix=20=E2=80=94=20name=20the=20correct=20(stdout)=20TTY=20g?= =?UTF-8?q?ate=20in=20the=20smoke=20comment?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet review verified everything empirically (207 db + 6 harness tests, YAML parse, tag→SHA, live rmSync repro) and found the step-level comment still misattributed the driver-selection gate to stdin. The gate is process.stdout.isTTY (selectChatDriver / io.stdoutIsTty); stdin's TTY-ness plays no role. Reword to name the actual gate (the smoke still exercises it — stdout is captured non-TTY inside the $(...)). Refs: docs/roadmap/phase-2.5-close-plan.md Step 8 Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/ci.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f5c1307b..f6551b79 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -219,9 +219,10 @@ jobs: run: | set -euo pipefail node apps/cli/dist/index.js --help >/dev/null # the tsup bundle loads + --help renders - # `chat` is TTY-gated: with a piped (non-TTY) stdin the driver-selection gate picks the PLAIN driver - # (ink/setRawMode is never entered), and the plain driver must exit cleanly on EOF — never crash. If a - # regression routed to ink without a TTY it would throw "Raw mode is not supported…", which this catches. + # `chat` is TTY-gated on STDOUT (selectChatDriver / io.stdoutIsTty): with stdout captured (non-TTY) + # here, the driver-selection gate picks the PLAIN driver (ink/setRawMode is never entered), and the + # plain driver must exit cleanly on EOF — never crash. A regression routing to ink without a TTY would + # throw "Raw mode is not supported…", which this catches. out="$(printf '' | node apps/cli/dist/index.js chat 2>&1 || true)" if printf '%s' "$out" | grep -qiE 'setRawMode|raw mode'; then echo "::error::chat entered raw mode without a TTY (the driver-selection gate regressed)"; exit 1 From b90cdba13b161b7045dae104a213c3cf0a877525 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 00:03:11 +0300 Subject: [PATCH 26/50] =?UTF-8?q?docs(reference):=202.5-close=20Step=209?= =?UTF-8?q?=20=E2=80=94=20reconcile=20regression-harness.md=20with=20the?= =?UTF-8?q?=202.5.I=20coverage=20(2.5.I=20S7)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The canonical harness reference documented only the 2.K run-fixture stream contract. Reconcile it with the Phase-2.5.I extensions: - Bump the Last-updated + Scope; add ADR-0064/0050 + database-schema.md to Related. - Add the new harness suites to "Where it lives": the concurrency e2e (+ its child fixture), the session-chain e2e, and the query-shape perf budgets. - Add a "§2.5.I extensions" section documenting each suite (what it proves, what it doesn't — e.g. the concurrency e2e is a cross-process safety smoke, the precise BEGIN IMMEDIATE/retry guards live in @relavium/db white-box tests; the 80×24 degrade is in home-projection.test.ts) + the advisory windows-concurrency CI lane. Refs: docs/roadmap/phase-2.5-close-plan.md Step 9 Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/reference/cli/regression-harness.md | 50 ++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/docs/reference/cli/regression-harness.md b/docs/reference/cli/regression-harness.md index 9214ea4d..b8251530 100644 --- a/docs/reference/cli/regression-harness.md +++ b/docs/reference/cli/regression-harness.md @@ -1,11 +1,11 @@ # Engine Regression Harness (2.K) -> Last updated: 2026-06-22 +> Last updated: 2026-07-08 - **Status**: Reference (Living) — the canonical home for the harness fixture + scenario format. - **Surface**: CLI (`relavium`), adopted as the engine's end-to-end regression gate. -- **Scope**: Phase 2, build workstream **2.K**; completes milestone **M3** with 2.D + 2.F. -- **Related**: [commands.md](commands.md), [../contracts/sse-event-schema.md](../contracts/sse-event-schema.md), [../../decisions/0049-cli-machine-output-contract.md](../../decisions/0049-cli-machine-output-contract.md), [../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md), [../../standards/testing.md](../../standards/testing.md), [../../tutorials/cli/run-a-workflow-in-ci.md](../../tutorials/cli/run-a-workflow-in-ci.md) +- **Scope**: Phase 2, build workstream **2.K** (completes milestone **M3** with 2.D + 2.F); **extended by Phase 2.5.I** with concurrency, session-chain, and query-shape coverage + a Windows lane. +- **Related**: [commands.md](commands.md), [../contracts/sse-event-schema.md](../contracts/sse-event-schema.md), [../desktop/database-schema.md](../desktop/database-schema.md), [../../decisions/0049-cli-machine-output-contract.md](../../decisions/0049-cli-machine-output-contract.md), [../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md), [../../decisions/0064-live-model-catalog.md](../../decisions/0064-live-model-catalog.md) (§5 concurrent-process write requirement), [../../decisions/0050-cli-history-db-at-rest-posture.md](../../decisions/0050-cli-history-db-at-rest-posture.md), [../../standards/testing.md](../../standards/testing.md), [../../tutorials/cli/run-a-workflow-in-ci.md](../../tutorials/cli/run-a-workflow-in-ci.md) ## What it is @@ -27,7 +27,10 @@ CLI surface. | Artifact | Path | |----------|------| | Fixture workflows (committed `.relavium.yaml`) | `apps/cli/src/harness/fixtures/` | -| The harness suite (in-process vitest e2e) | `apps/cli/src/harness/regression.e2e.test.ts` | +| The `run`-fixture suite (in-process vitest e2e) | `apps/cli/src/harness/regression.e2e.test.ts` | +| Concurrency e2e — a run + a chat share one `history.db`; a real two-process contention smoke (2.5.I S3) | `apps/cli/src/harness/concurrency.e2e.test.ts` (+ its child `fixtures/concurrent-writer.mjs`) | +| Session-chain e2e — Home→chat→resume→export over a real file-backed db (2.5.I S4) | `apps/cli/src/harness/session-chain.e2e.test.ts` | +| Query-shape perf budgets — `EXPLAIN QUERY PLAN` of the hot reads (2.5.I S5) | `apps/cli/src/harness/perf-budget.e2e.test.ts` | The harness is an **in-process** suite: it drives the CLI's `runCommand` boundary with a captured `CliIo` through the **default engine** (the standard node executor + expression sandbox over the real @@ -91,6 +94,45 @@ run exit codes (`0` / `1` / `3`). `type:nodeId` signature list, and `parallel: true` only if the fixture has a `parallel` node. **Capture** the real signatures by running the fixture once — never hand-guess them — then pin them. +## §2.5.I extensions — concurrency, the session chain, and query-shape budgets + +Phase 2.5.I extended the harness beyond the `run`-fixture stream contract, to the parts a single +`relavium run` cannot exercise: two `relavium` processes sharing one `history.db`, the chat→resume→export +journey, and the read-path query shapes. These are additive `.e2e.test.ts` suites under +`apps/cli/src/harness/`, run by the same `pnpm turbo run test` gate. + +- **Concurrency e2e** (`concurrency.e2e.test.ts`, 2.5.I S3) — proves a `run` and a `chat` share one + `history.db` safely ([ADR-0064](../../decisions/0064-live-model-catalog.md) §5). Two scenarios: an + **in-process two-connection** coexistence case (the real `runCommand` + a real `SessionStore` interleave + on one file — proves coexistence, not lock contention, since a single synchronous `better-sqlite3` process + can't overlap two transactions), and a **real two-process** case where the parent holds the WAL write lock + and releases it only after a **READY handshake** from both children, forcing genuine cross-process + busy-wait every run. The child (`fixtures/concurrent-writer.mjs`) imports the **built** `@relavium/db` by a + `file://` URL — so this case is skipped, visibly, when the dist is absent. The *precise* clause-guards for + the `BEGIN IMMEDIATE` + `withBusyRetry` write path are the deterministic white-box tests in `@relavium/db` + (`provider-store.test.ts` / `retry.test.ts`); this suite is a cross-process safety/coexistence smoke. +- **Session-chain e2e** (`session-chain.e2e.test.ts`, 2.5.I S4) — the Home→chat→resume→export journey a + single `run` fixture cannot cover: a fresh cassette-driven session persists a turn, a `chat-resume` on a + **fresh connection** to the same file continues it (asserting the reconstructed transcript threads into the + resumed turn's provider request + the totals accumulate), and `exportSession` serializes it. The + interactive ink Home/chat TUI is out of scope (no render-test dependency); this drives the session-host + + persister + export seam the Home graduates into. +- **Query-shape perf budgets** (`perf-budget.e2e.test.ts`, 2.5.I S5) — `EXPLAIN QUERY PLAN` asserts the hot + reads (`listSessions` / `listRuns` / `loadFull`'s messages) are **index-served with no filesort** (no + `USE TEMP B-TREE`, no full-table SCAN) — the shape the store docs claim, not a flaky wall-clock. The store + query sites back-reference this budget so a store `ORDER BY` edit trips the reviewer. (The sibling §2.5.I + perf item, the 80×24 narrow-terminal degrade, is asserted in `render/tui/home-projection.test.ts`.) + +### The Windows lane (2.5.I S6) + +An advisory `windows-concurrency` job in [`.github/workflows/ci.yml`](../../../.github/workflows/ci.yml) runs +the `@relavium/db` concurrency suite + the CLI concurrency/perf/session-chain harness on `windows-latest` +(the WAL locking, the retry's `Atomics.wait` sleep, the two-process child spawn + `file://` import, and the +native `better-sqlite3` addon are the parts most likely to diverge), plus a headless no-TTY smoke. It is a +**separate, advisory** job — the required check stays the ubuntu `ci` job. POSIX `0600`/`0700` permission +assertions are a documented Windows no-op ([ADR-0050](../../decisions/0050-cli-history-db-at-rest-posture.md)) +and are not exercised there; `release.yml` separately smokes the built binary across ubuntu/macOS/Windows. + ## Deferred (scope-split in [phase-2-cli.md §2.K](../../roadmap/phases/phase-2-cli.md)) - ~~**Gate-*resume* scenario** (`relavium gate --approve` → completion)~~ — **landed with 2.G**: the From 1b01239a2a7b2e9e28810a9a47c8dfda3a2db0bf Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 00:08:25 +0300 Subject: [PATCH 27/50] =?UTF-8?q?docs(reference):=202.5-close=20Step=209?= =?UTF-8?q?=20Opus-review=20fixes=20=E2=80=94=20scope=20the=20in-process?= =?UTF-8?q?=20claim=20+=20full-suite=20wording?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review verified the reconcile is highly accurate (every suite claim, store back-reference, Windows lane, and link checks out); two wording precisions: - LOW: "The harness is an in-process suite … does not spawn the built binary" now sits above a table listing the concurrency e2e, whose cross-process case DOES spawn child node processes. Scope the subject to "The `run`-fixture harness" and note the §2.5.I concurrency suite is the deliberate exception. - NIT: the Windows lane runs the FULL @relavium/db test suite, not a concurrency-only split — reword. Refs: docs/roadmap/phase-2.5-close-plan.md Step 9 Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/reference/cli/regression-harness.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/reference/cli/regression-harness.md b/docs/reference/cli/regression-harness.md index b8251530..236fefb6 100644 --- a/docs/reference/cli/regression-harness.md +++ b/docs/reference/cli/regression-harness.md @@ -32,11 +32,13 @@ CLI surface. | Session-chain e2e — Home→chat→resume→export over a real file-backed db (2.5.I S4) | `apps/cli/src/harness/session-chain.e2e.test.ts` | | Query-shape perf budgets — `EXPLAIN QUERY PLAN` of the hot reads (2.5.I S5) | `apps/cli/src/harness/perf-budget.e2e.test.ts` | -The harness is an **in-process** suite: it drives the CLI's `runCommand` boundary with a captured -`CliIo` through the **default engine** (the standard node executor + expression sandbox over the real +The **`run`-fixture** harness is an **in-process** suite: it drives the CLI's `runCommand` boundary with a +captured `CliIo` through the **default engine** (the standard node executor + expression sandbox over the real `createCliHost`), exactly as a real `relavium run --json` invocation does (the same `createJsonRenderer` produces byte-identical NDJSON). It does **not** spawn the built binary — that -would add a build dependency and process flakiness for no fidelity gain. It runs inside the standard +would add a build dependency and process flakiness for no fidelity gain. (The additive §2.5.I concurrency +suite below is the one exception — its cross-process case deliberately spawns child `node` processes to +reach the real two-OS-process WAL path a single process cannot.) It runs inside the standard `pnpm turbo run test` task, so it is part of the required CI gate on every push/PR (`.github/workflows/ci.yml`). @@ -126,7 +128,8 @@ journey, and the read-path query shapes. These are additive `.e2e.test.ts` suite ### The Windows lane (2.5.I S6) An advisory `windows-concurrency` job in [`.github/workflows/ci.yml`](../../../.github/workflows/ci.yml) runs -the `@relavium/db` concurrency suite + the CLI concurrency/perf/session-chain harness on `windows-latest` +the full `@relavium/db` test suite (the concurrency/retry white-box tests among it) + the CLI +concurrency/perf/session-chain harness on `windows-latest` (the WAL locking, the retry's `Atomics.wait` sleep, the two-process child spawn + `file://` import, and the native `better-sqlite3` addon are the parts most likely to diverge), plus a headless no-TTY smoke. It is a **separate, advisory** job — the required check stays the ubuntu `ci` job. POSIX `0600`/`0700` permission From 4867366042da3a295a490f02b54f2ba58b4cf6d7 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 00:22:09 +0300 Subject: [PATCH 28/50] =?UTF-8?q?feat(cli):=202.5-close=20Step=2010=20?= =?UTF-8?q?=E2=80=94=20NO=5FCOLOR/FORCE=5FCOLOR=20env=20support=20+=20--co?= =?UTF-8?q?lor=20flag=20(2.5.J=20S3)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Color was resolvable only via --no-color (raw.color !== false); no env input. Add the standard env conventions + the symmetric --color flag (close-plan D6/D7). - resolveColor(raw, env) with the precedence: --color/--no-color flag > NO_COLOR (any non-empty ⇒ off, the no-color.org accessibility contract, wins over FORCE_COLOR) > FORCE_COLOR (not 0/false/empty ⇒ on) > default-on. FORCE_COLOR is force-ON only; disabling is NO_COLOR / --no-color. - Add --color to BOOLEAN_FLAGS (symmetric with --no-color; both present ⇒ last wins). - resolveGlobalOptions(raw, defaultCwd, env = {}); run.ts passes io.env. - Color stays orthogonal to the output MODE (--json/CI/non-TTY suppress ANSI separately in detectOutputMode — unchanged). - Document in program.ts --help + commands.md (the precedence + the mode-orthogonality). - Tests: the full precedence matrix in options.test.ts (NO_COLOR any-value-off incl. '0'; empty falls through; FORCE_COLOR truthy-on / 0-false-empty fall-through; NO_COLOR beats FORCE_COLOR; flag beats both directions) + --color extraction. Refs: docs/roadmap/phase-2.5-close-plan.md Step 10; ADR-0047 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/process/options.test.ts | 43 +++++++++++++++++++++++++++- apps/cli/src/process/options.ts | 34 ++++++++++++++++++++-- apps/cli/src/program.ts | 5 +++- apps/cli/src/run.ts | 2 +- docs/reference/cli/commands.md | 14 +++++++-- 5 files changed, 89 insertions(+), 9 deletions(-) diff --git a/apps/cli/src/process/options.test.ts b/apps/cli/src/process/options.test.ts index 1b45d190..83298eee 100644 --- a/apps/cli/src/process/options.test.ts +++ b/apps/cli/src/process/options.test.ts @@ -29,8 +29,12 @@ describe('extractGlobalOptions', () => { expect(extractGlobalOptions(argv('--cwd=/y', 'list')).raw.cwd).toBe('/y'); }); - it('treats --no-color as color off', () => { + it('treats --no-color as color off and --color as color on', () => { expect(extractGlobalOptions(argv('--no-color')).raw.color).toBe(false); + expect(extractGlobalOptions(argv('--color')).raw.color).toBe(true); + // Both present ⇒ last flag wins (idempotent boolean flags, applied in argv order). + expect(extractGlobalOptions(argv('--no-color', '--color')).raw.color).toBe(true); + expect(extractGlobalOptions(argv('--color', '--no-color')).raw.color).toBe(false); }); it('reports (not throws) invalid_invocation when --cwd / --config has no argument', () => { @@ -92,6 +96,43 @@ describe('resolveGlobalOptions', () => { 'cannot be combined', ); }); + + describe('color precedence (2.5.J): flag > NO_COLOR > FORCE_COLOR > on', () => { + const color = ( + raw: Parameters[0], + env: Record = {}, + ): boolean => resolveGlobalOptions(raw, '/w', env).color; + + it('defaults ON with no flag and no env', () => { + expect(color({})).toBe(true); + }); + + it('an explicit flag wins over any env (both directions)', () => { + expect(color({ color: false }, { FORCE_COLOR: '1' })).toBe(false); // --no-color beats FORCE_COLOR + expect(color({ color: true }, { NO_COLOR: '1' })).toBe(true); // --color beats NO_COLOR + }); + + it('NO_COLOR turns color OFF for ANY non-empty value (no-color.org)', () => { + expect(color({}, { NO_COLOR: '1' })).toBe(false); + expect(color({}, { NO_COLOR: 'anything' })).toBe(false); + expect(color({}, { NO_COLOR: '0' })).toBe(false); // even '0' is non-empty ⇒ off + expect(color({}, { NO_COLOR: '' })).toBe(true); // empty ⇒ not set ⇒ falls through + }); + + it('NO_COLOR beats FORCE_COLOR (opt-out wins over opt-in)', () => { + expect(color({}, { NO_COLOR: '1', FORCE_COLOR: '1' })).toBe(false); + }); + + it('FORCE_COLOR turns color ON only for a truthy value; 0/false/empty fall through to default-on', () => { + expect(color({}, { FORCE_COLOR: '1' })).toBe(true); + expect(color({}, { FORCE_COLOR: 'true' })).toBe(true); + // These do not FORCE on (and there is no NO_COLOR), so the default-on applies — FORCE_COLOR never + // turns color OFF; that is NO_COLOR / --no-color's job. + expect(color({}, { FORCE_COLOR: '0' })).toBe(true); + expect(color({}, { FORCE_COLOR: 'false' })).toBe(true); + expect(color({}, { FORCE_COLOR: '' })).toBe(true); + }); + }); }); describe('assertNoGlobalOptionConflicts', () => { diff --git a/apps/cli/src/process/options.ts b/apps/cli/src/process/options.ts index 04463b6c..c9c77c19 100644 --- a/apps/cli/src/process/options.ts +++ b/apps/cli/src/process/options.ts @@ -19,7 +19,7 @@ export interface GlobalOptions { /** The raw global-flag values harvested from argv (before normalization). */ export interface RawGlobalOptions { json?: boolean; - /** `false` for `--no-color`; otherwise absent (color on). */ + /** `true` for `--color`, `false` for `--no-color`; absent ⇒ resolved from `NO_COLOR`/`FORCE_COLOR`/default. */ color?: boolean; cwd?: string; config?: string; @@ -46,6 +46,9 @@ const BOOLEAN_FLAGS: Readonly void>> = '--no-color': (raw) => { raw.color = false; }, + '--color': (raw) => { + raw.color = true; + }, '--verbose': (raw) => { raw.verbose = true; }, @@ -163,11 +166,36 @@ function resolveVerbosity(raw: RawGlobalOptions): Verbosity { return 'normal'; } -export function resolveGlobalOptions(raw: RawGlobalOptions, defaultCwd: string): GlobalOptions { +/** + * Resolve the effective color setting — ANSI STYLING only, orthogonal to the `--json`/CI/non-TTY output MODE + * (which suppresses ANSI separately in `detectOutputMode`, output-mode.ts). Precedence (2.5.J; the flag pair + * is [ADR-0047](../../../docs/decisions/0047-cli-framework-commander-ink-clack.md)'s global-flag set): + * 1. an explicit `--color` / `--no-color` flag (a per-invocation override); + * 2. `NO_COLOR` — ANY non-empty value ⇒ OFF (the no-color.org accessibility contract; wins over FORCE_COLOR); + * 3. `FORCE_COLOR` — any value other than `0`/`false`/`''` ⇒ ON; + * 4. default ON. + * `NO_COLOR` intentionally beats `FORCE_COLOR`: a user who opts OUT of color must win over a tool/CI opting in. + */ +function resolveColor( + raw: RawGlobalOptions, + env: Readonly>, +): boolean { + if (raw.color !== undefined) return raw.color; // 1. the explicit flag wins + if ((env['NO_COLOR'] ?? '') !== '') return false; // 2. NO_COLOR opts out (accessibility floor) + const force = env['FORCE_COLOR']; // 3. FORCE_COLOR opts in (force-ON only; disabling is NO_COLOR/--no-color) + if (force !== undefined && force !== '' && force !== '0' && force !== 'false') return true; + return true; // 4. default on +} + +export function resolveGlobalOptions( + raw: RawGlobalOptions, + defaultCwd: string, + env: Readonly> = {}, +): GlobalOptions { assertNoGlobalOptionConflicts(raw); return { json: raw.json === true, - color: raw.color !== false, + color: resolveColor(raw, env), cwd: raw.cwd ?? defaultCwd, configPath: raw.config, verbosity: resolveVerbosity(raw), diff --git a/apps/cli/src/program.ts b/apps/cli/src/program.ts index 9a317e57..21216e91 100644 --- a/apps/cli/src/program.ts +++ b/apps/cli/src/program.ts @@ -26,11 +26,14 @@ export const CLI_VERSION = const GLOBAL_OPTIONS_HELP = ` Global options (usable anywhere on the command line): --json emit machine-readable NDJSON output (disables the TUI) + --color force colored output on --no-color disable colored output --cwd run as if started in --config use an explicit config file -v, --verbose print verbose diagnostics to stderr - -q, --quiet suppress non-essential output`; + -q, --quiet suppress non-essential output + +Color precedence: --color/--no-color > NO_COLOR (any value ⇒ off) > FORCE_COLOR (⇒ on) > on.`; /** * Build the `commander` program: the full subcommand surface plus a bare-invocation help diff --git a/apps/cli/src/run.ts b/apps/cli/src/run.ts index f15e548e..bb9d30dd 100644 --- a/apps/cli/src/run.ts +++ b/apps/cli/src/run.ts @@ -40,7 +40,7 @@ export async function run( let global: GlobalOptions; try { - global = resolveGlobalOptions(raw, process.cwd()); // also enforces the --verbose/--quiet rule + global = resolveGlobalOptions(raw, process.cwd(), io.env); // env feeds NO_COLOR/FORCE_COLOR; also enforces --verbose/--quiet } catch (err) { renderError(err, renderCtx, io); return toUserFacing(err).exitCode; diff --git a/docs/reference/cli/commands.md b/docs/reference/cli/commands.md index df06e312..03fb2379 100644 --- a/docs/reference/cli/commands.md +++ b/docs/reference/cli/commands.md @@ -37,10 +37,17 @@ environment disables the interactive TUI but does not by itself switch stdout to ([ADR-0049](../../decisions/0049-cli-machine-output-contract.md)). Exit codes are CI-friendly (see [Exit codes](#exit-codes)). -`--no-color` does **not** change the mode — the interactive TUI stays active and only ANSI color/dim -are suppressed (plain output without a renderer swap). A swap to the Plain renderer happens only on +`--no-color` (and `--color`) do **not** change the mode — the interactive TUI stays active and only ANSI +color/dim are suppressed (plain output without a renderer swap). A swap to the Plain renderer happens only on no-TTY / `CI=true`, and to NDJSON only on `--json`. +**Color precedence.** ANSI styling is orthogonal to the output mode above, resolved with this precedence: +an explicit `--color` / `--no-color` flag > the `NO_COLOR` env var (**any** non-empty value ⇒ off, the +[no-color.org](https://no-color.org) accessibility contract) > the `FORCE_COLOR` env var (any value other than +`0`/`false`/empty ⇒ on) > **on** by default. `NO_COLOR` intentionally beats `FORCE_COLOR` — a user who opts +out of color wins over a tool/CI that opts in. (A `--json`/CI/no-TTY stream carries no ANSI regardless — that +is the separate output-mode selection above.) + ### The `--json` machine-output contract Under `relavium run --json`, the CLI emits a stable machine contract a CI job can pipe and assert @@ -86,7 +93,8 @@ before parsing the subcommand). | Flag | Effect | |------|--------| | `--json` | Emit machine-readable NDJSON output (disables the TUI) — see [Output modes](#output-modes). | -| `--no-color` | Disable colored output. | +| `--color` | Force colored (ANSI) output on. | +| `--no-color` | Disable colored (ANSI) output. | | `--cwd ` | Run as if started in `` (project discovery and relative paths resolve from here). | | `--config ` | Use an explicit global config file instead of `~/.relavium/config.toml` — the project `.relavium/` layers still apply ([config-spec.md](../contracts/config-spec.md)). | | `-v, --verbose` | Print verbose diagnostics to stderr. | From 77eee127ee1df7ecd4f93a97cbb2a709e342953e Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 00:30:30 +0300 Subject: [PATCH 29/50] =?UTF-8?q?feat(cli):=202.5-close=20Step=2010=20Opus?= =?UTF-8?q?-review=20fixes=20=E2=80=94=20give=20FORCE=5FCOLOR=20real=20tee?= =?UTF-8?q?th=20+=20fix=20doc-link=20depth?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review proved the FORCE_COLOR branch was inert: `color` defaults to `true` and the force-ON-only branch could only ALSO return `true`, so it never changed the outcome — yet it was documented + tested as a live feature (dead code shipped as a feature). - Adopt the supports-color convention so FORCE_COLOR has an observable effect: FORCE_COLOR=0/false ⇒ OFF (distinct from the default-on); any other value stays on. NO_COLOR still wins (checked first); an explicit --color/--no-color flag beats both. Update program.ts --help + commands.md to match, and rewrite the tests so FORCE_COLOR=0/false now assert OFF (the teeth) + a flag-beats-FORCE_COLOR=0 case. - LOW: fix the resolveColor JSDoc ADR-0047 link depth (../../../ → ../../../../, matching output-mode.ts / render/select.ts in the same tree). 20 options tests pass; typecheck + lint + format + fence clean. Refs: docs/roadmap/phase-2.5-close-plan.md Step 10; ADR-0047 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/process/options.test.ts | 14 +++++++++----- apps/cli/src/process/options.ts | 15 ++++++++------- apps/cli/src/program.ts | 2 +- docs/reference/cli/commands.md | 8 ++++---- 4 files changed, 22 insertions(+), 17 deletions(-) diff --git a/apps/cli/src/process/options.test.ts b/apps/cli/src/process/options.test.ts index 83298eee..9fe6a617 100644 --- a/apps/cli/src/process/options.test.ts +++ b/apps/cli/src/process/options.test.ts @@ -123,15 +123,19 @@ describe('resolveGlobalOptions', () => { expect(color({}, { NO_COLOR: '1', FORCE_COLOR: '1' })).toBe(false); }); - it('FORCE_COLOR turns color ON only for a truthy value; 0/false/empty fall through to default-on', () => { + it('FORCE_COLOR=0/false turns color OFF (the supports-color convention); other values keep the default on', () => { + // 0/false is the one FORCE_COLOR value with an observable effect — it opts out (distinct from default-on). + expect(color({}, { FORCE_COLOR: '0' })).toBe(false); + expect(color({}, { FORCE_COLOR: 'false' })).toBe(false); + // A truthy or empty value is consistent with the default-on (color already defaults on). expect(color({}, { FORCE_COLOR: '1' })).toBe(true); expect(color({}, { FORCE_COLOR: 'true' })).toBe(true); - // These do not FORCE on (and there is no NO_COLOR), so the default-on applies — FORCE_COLOR never - // turns color OFF; that is NO_COLOR / --no-color's job. - expect(color({}, { FORCE_COLOR: '0' })).toBe(true); - expect(color({}, { FORCE_COLOR: 'false' })).toBe(true); expect(color({}, { FORCE_COLOR: '' })).toBe(true); }); + + it('an explicit flag beats FORCE_COLOR=0 (the per-invocation override wins over env opt-out)', () => { + expect(color({ color: true }, { FORCE_COLOR: '0' })).toBe(true); + }); }); }); diff --git a/apps/cli/src/process/options.ts b/apps/cli/src/process/options.ts index c9c77c19..bd9157f7 100644 --- a/apps/cli/src/process/options.ts +++ b/apps/cli/src/process/options.ts @@ -169,11 +169,12 @@ function resolveVerbosity(raw: RawGlobalOptions): Verbosity { /** * Resolve the effective color setting — ANSI STYLING only, orthogonal to the `--json`/CI/non-TTY output MODE * (which suppresses ANSI separately in `detectOutputMode`, output-mode.ts). Precedence (2.5.J; the flag pair - * is [ADR-0047](../../../docs/decisions/0047-cli-framework-commander-ink-clack.md)'s global-flag set): + * is [ADR-0047](../../../../docs/decisions/0047-cli-framework-commander-ink-clack.md)'s global-flag set): * 1. an explicit `--color` / `--no-color` flag (a per-invocation override); * 2. `NO_COLOR` — ANY non-empty value ⇒ OFF (the no-color.org accessibility contract; wins over FORCE_COLOR); - * 3. `FORCE_COLOR` — any value other than `0`/`false`/`''` ⇒ ON; - * 4. default ON. + * 3. `FORCE_COLOR` — `0`/`false` ⇒ OFF (the `supports-color` convention; the value is checked, unlike NO_COLOR); + * 4. default ON (a truthy `FORCE_COLOR` is consistent with this — it can only opt OUT here, since color already + * defaults on and is consulted only on a TTY where it is on). * `NO_COLOR` intentionally beats `FORCE_COLOR`: a user who opts OUT of color must win over a tool/CI opting in. */ function resolveColor( @@ -181,10 +182,10 @@ function resolveColor( env: Readonly>, ): boolean { if (raw.color !== undefined) return raw.color; // 1. the explicit flag wins - if ((env['NO_COLOR'] ?? '') !== '') return false; // 2. NO_COLOR opts out (accessibility floor) - const force = env['FORCE_COLOR']; // 3. FORCE_COLOR opts in (force-ON only; disabling is NO_COLOR/--no-color) - if (force !== undefined && force !== '' && force !== '0' && force !== 'false') return true; - return true; // 4. default on + if ((env['NO_COLOR'] ?? '') !== '') return false; // 2. NO_COLOR opts out (any non-empty; accessibility floor) + const force = env['FORCE_COLOR']; // 3. FORCE_COLOR=0/false opts out (supports-color convention) + if (force === '0' || force === 'false') return false; + return true; // 4. default on (incl. a truthy FORCE_COLOR) } export function resolveGlobalOptions( diff --git a/apps/cli/src/program.ts b/apps/cli/src/program.ts index 21216e91..4610c771 100644 --- a/apps/cli/src/program.ts +++ b/apps/cli/src/program.ts @@ -33,7 +33,7 @@ Global options (usable anywhere on the command line): -v, --verbose print verbose diagnostics to stderr -q, --quiet suppress non-essential output -Color precedence: --color/--no-color > NO_COLOR (any value ⇒ off) > FORCE_COLOR (⇒ on) > on.`; +Color precedence: --color/--no-color > NO_COLOR (any value ⇒ off) > FORCE_COLOR (0/false ⇒ off) > on.`; /** * Build the `commander` program: the full subcommand surface plus a bare-invocation help diff --git a/docs/reference/cli/commands.md b/docs/reference/cli/commands.md index 03fb2379..e25e986f 100644 --- a/docs/reference/cli/commands.md +++ b/docs/reference/cli/commands.md @@ -43,10 +43,10 @@ no-TTY / `CI=true`, and to NDJSON only on `--json`. **Color precedence.** ANSI styling is orthogonal to the output mode above, resolved with this precedence: an explicit `--color` / `--no-color` flag > the `NO_COLOR` env var (**any** non-empty value ⇒ off, the -[no-color.org](https://no-color.org) accessibility contract) > the `FORCE_COLOR` env var (any value other than -`0`/`false`/empty ⇒ on) > **on** by default. `NO_COLOR` intentionally beats `FORCE_COLOR` — a user who opts -out of color wins over a tool/CI that opts in. (A `--json`/CI/no-TTY stream carries no ANSI regardless — that -is the separate output-mode selection above.) +[no-color.org](https://no-color.org) accessibility contract) > the `FORCE_COLOR` env var (`0`/`false` ⇒ off, +the `supports-color` convention) > **on** by default. `NO_COLOR` intentionally beats `FORCE_COLOR` — a user +who opts out of color wins over a tool/CI that opts in. (A `--json`/CI/no-TTY stream carries no ANSI regardless +— that is the separate output-mode selection above.) ### The `--json` machine-output contract From 5a9d841bf6b2b67e3d69dc4488372e2b2eac87f5 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 00:40:22 +0300 Subject: [PATCH 30/50] =?UTF-8?q?test(cli):=202.5-close=20Step=2010=20Sonn?= =?UTF-8?q?et-review=20fix=20=E2=80=94=20honest=20NO=5FCOLOR/FORCE=5FCOLOR?= =?UTF-8?q?=20framing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet mutation-tested the fix (removing the FORCE_COLOR=0 branch fails a test; observ- ability confirmed end-to-end) and found a residual honesty gap: after FORCE_COLOR became opt-OUT-only, the "NO_COLOR beats FORCE_COLOR (opt-out wins over opt-in)" test + the JSDoc "intentionally beats" framing describe an opt-in FORCE_COLOR no longer has — the check ORDER is now inconsequential (a truthy FORCE_COLOR never force-ONs). - Reword the JSDoc: both env vars are opt-OUT signals; a truthy FORCE_COLOR has no independent effect, so there is no NO_COLOR-vs-FORCE_COLOR conflict to resolve. - Reword the test to what it actually verifies (NO_COLOR turns off even alongside FORCE_COLOR=1 — a truthy FORCE_COLOR never force-ONs over NO_COLOR). - (Also reconciled the untracked close-plan's D6 row to the shipped semantics.) Refs: docs/roadmap/phase-2.5-close-plan.md Step 10 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/process/options.test.ts | 2 +- apps/cli/src/process/options.ts | 9 +++++---- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/apps/cli/src/process/options.test.ts b/apps/cli/src/process/options.test.ts index 9fe6a617..d611877d 100644 --- a/apps/cli/src/process/options.test.ts +++ b/apps/cli/src/process/options.test.ts @@ -119,7 +119,7 @@ describe('resolveGlobalOptions', () => { expect(color({}, { NO_COLOR: '' })).toBe(true); // empty ⇒ not set ⇒ falls through }); - it('NO_COLOR beats FORCE_COLOR (opt-out wins over opt-in)', () => { + it('NO_COLOR turns off even alongside FORCE_COLOR=1 (a truthy FORCE_COLOR never force-ONs over NO_COLOR)', () => { expect(color({}, { NO_COLOR: '1', FORCE_COLOR: '1' })).toBe(false); }); diff --git a/apps/cli/src/process/options.ts b/apps/cli/src/process/options.ts index bd9157f7..8ccd5850 100644 --- a/apps/cli/src/process/options.ts +++ b/apps/cli/src/process/options.ts @@ -171,11 +171,12 @@ function resolveVerbosity(raw: RawGlobalOptions): Verbosity { * (which suppresses ANSI separately in `detectOutputMode`, output-mode.ts). Precedence (2.5.J; the flag pair * is [ADR-0047](../../../../docs/decisions/0047-cli-framework-commander-ink-clack.md)'s global-flag set): * 1. an explicit `--color` / `--no-color` flag (a per-invocation override); - * 2. `NO_COLOR` — ANY non-empty value ⇒ OFF (the no-color.org accessibility contract; wins over FORCE_COLOR); + * 2. `NO_COLOR` — ANY non-empty value ⇒ OFF (the no-color.org accessibility contract); * 3. `FORCE_COLOR` — `0`/`false` ⇒ OFF (the `supports-color` convention; the value is checked, unlike NO_COLOR); - * 4. default ON (a truthy `FORCE_COLOR` is consistent with this — it can only opt OUT here, since color already - * defaults on and is consulted only on a TTY where it is on). - * `NO_COLOR` intentionally beats `FORCE_COLOR`: a user who opts OUT of color must win over a tool/CI opting in. + * 4. default ON. + * Both env vars are opt-OUT signals here: a truthy `FORCE_COLOR` has NO independent effect (color already + * defaults on, and is consulted only on a TTY where it is on), so there is no `NO_COLOR`-vs-`FORCE_COLOR` + * conflict to resolve — the flag overrides both, and either env opt-out disables. */ function resolveColor( raw: RawGlobalOptions, From 11ccdc200e352ddfb3c33aaf15ee59b896fcd3a4 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 00:46:01 +0300 Subject: [PATCH 31/50] =?UTF-8?q?test(cli):=202.5-close=20Step=2011=20(Bat?= =?UTF-8?q?ch=20A)=20=E2=80=94=20fix=20the=20media-gc=20flake=20+=20a=202.?= =?UTF-8?q?5.A=20git=5Fstatus=20union=20pin?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - media-gc.test.ts: the "sweeps a row-less CAS orphan" case was non-deterministic — with orphanMinAgeMs:0 and no injected clock, `settledBefore` (Date.now() − 0) can land BELOW the just-put orphan's filesystem mtimeMs (coarse/rounded-up mtime), so `mtimeMs <= settledBefore` is false and the orphan is skipped as a concurrent writer. Inject a fixed +60s clock so the orphan is unambiguously settled. 5× stable. - session-host.test.ts: add the 2.5.A UNION pin — a chat session dispatching git_status end-to-end through the process arm (the arm + the session→host dispatch were each covered separately, never together). A real temp git repo + a git_status tool-call cassette; asserts the agent:tool_call annotation + the post-tool answer streams (the process arm ran it and the result folded back). The other Batch-A pins stay covered/tracked: the persister failed-turn REAL-token/cost fold is already pinned (persister.test.ts — "flushes the running cost on a failed turn" + 4 sibling cases), and the 2.I read-command pins remain in deferred-tasks (transitively covered by the process arm + read tests). Refs: docs/roadmap/phase-2.5-close-plan.md Step 11 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/chat/session-host.test.ts | 30 +++++++++++++++++++++++++- apps/cli/src/engine/media-gc.test.ts | 7 +++++- 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/apps/cli/src/chat/session-host.test.ts b/apps/cli/src/chat/session-host.test.ts index bc7e07a0..3ea3f1e7 100644 --- a/apps/cli/src/chat/session-host.test.ts +++ b/apps/cli/src/chat/session-host.test.ts @@ -1,4 +1,5 @@ -import { existsSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'; +import { execFileSync } from 'node:child_process'; +import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; @@ -183,6 +184,33 @@ describe('buildChatSession', () => { expect(tokens).toContain('the answer'); // the post-tool answer reached the stream }); + it('dispatches git_status through the process arm end-to-end (2.5.A union pin: session→host→process)', async () => { + // The process arm and the session→host dispatch are each covered separately (assemble.test.ts / the + // read_file case above); this pins the UNION for a real process-arm tool. git_status is granted to the + // default agent, pre-approved (no confirm gate), and takes no model-controlled args, so it reaches the host. + const repo = mkdtempSync(join(tmpdir(), 'relavium-git-')); + execFileSync('git', ['init', '-q'], { cwd: repo }); // `git status` needs no user identity (unlike commits) + try { + const built = await build({ + cwd: repo, + providers: scriptedResolver([callWithArgs('c1', 'git_status', {}), textTurn('clean')]), + }); + built.session.start(); + await built.session.sendMessage('what changed?'); + built.session.cancel(); + const events = await drainHandle(built.handle.events); + + // The git_status tool call was annotated on the stream (the session routed it to the host)… + const toolCall = events.find((e) => e.type === 'agent:tool_call'); + expect(toolCall?.type === 'agent:tool_call' && toolCall.toolId).toBe('git_status'); + // …the process arm ran it and the result folded back, so the post-tool answer streamed (the loop completed). + const tokens = events.flatMap((e) => (e.type === 'agent:token' ? [e.token] : [])).join(''); + expect(tokens).toContain('clean'); + } finally { + rmSync(repo, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); + } + }); + it('enforces [chat].max_turns: an over-cap sendMessage settles loudly as turn_limit with no provider call', async () => { // Two ENGAGED (successful) turns reach the cap of 2; the 3rd is blocked as turn_limit WITHOUT a provider // call. Only an engaged turn counts toward the cap (F7, ADR-0055) — so the cap is reached by real turns, diff --git a/apps/cli/src/engine/media-gc.test.ts b/apps/cli/src/engine/media-gc.test.ts index 2f992a58..538b88df 100644 --- a/apps/cli/src/engine/media-gc.test.ts +++ b/apps/cli/src/engine/media-gc.test.ts @@ -290,7 +290,12 @@ describe('sweepHostMediaBestEffort (the run/gate run-end wrapper — real db + C db: client.db, casRoot, currentRunId: 'run-1', - orphanMinAgeMs: 0, // treat the just-written blob as settled for the test + orphanMinAgeMs: 0, + // Inject a clock a minute in the FUTURE so `settledBefore` (now − 0) is unambiguously past the orphan's + // filesystem `mtimeMs`. Without it (`now = Date.now`), a coarse/rounded-up mtime can land above the + // just-captured `now`, so `mtimeMs <= settledBefore` is false and the orphan is skipped as a concurrent + // writer — the source of this test's flake. + now: () => Date.now() + 60_000, }); expect(report?.orphanSweepRan).toBe(true); expect(report?.orphansDeleted).toBe(1); From 1d521116a4c90d4362e03129f4c58f6f0f179b50 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 00:58:58 +0300 Subject: [PATCH 32/50] =?UTF-8?q?test(cli):=202.5-close=20Step=2011=20Opus?= =?UTF-8?q?-review=20fixes=20=E2=80=94=20genuinely=20lock=20the=20git=5Fst?= =?UTF-8?q?atus=20union=20pin?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two review findings on the 2.5.A git_status union pin (session-host.test.ts): - MEDIUM: the pin under-asserted. It only checked the agent:tool_call annotation + the cassette-driven 'clean' token — neither proves git_status ran to a clean result. The reviewer's proposed `success === true` check is itself inert for cleanliness: the registry stamps success:true on ANY resolved dispatch, and git_status's spawn RESOLVES a ProcessResult for both exit 0 and exit 128 ("not a repository"), so success cannot distinguish them. The genuine lock is the tool-result outputSummary, which carries git's captured `{"exitCode":0,…}` stdout — added that assertion (keeping success as the weaker "git actually started" check) with an honest comment on what each proves. - NIT: moved `git init` inside the try so the mkdtemp'd dir can't leak if init throws (git absent / hostile template hook) — the finally rmSync now covers it. Verified: 52/52 pass, stable across 3 runs; typecheck/eslint/prettier green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 11, Batch A) Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/chat/session-host.test.ts | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/apps/cli/src/chat/session-host.test.ts b/apps/cli/src/chat/session-host.test.ts index 3ea3f1e7..af57cb48 100644 --- a/apps/cli/src/chat/session-host.test.ts +++ b/apps/cli/src/chat/session-host.test.ts @@ -189,8 +189,8 @@ describe('buildChatSession', () => { // read_file case above); this pins the UNION for a real process-arm tool. git_status is granted to the // default agent, pre-approved (no confirm gate), and takes no model-controlled args, so it reaches the host. const repo = mkdtempSync(join(tmpdir(), 'relavium-git-')); - execFileSync('git', ['init', '-q'], { cwd: repo }); // `git status` needs no user identity (unlike commits) try { + execFileSync('git', ['init', '-q'], { cwd: repo }); // `git status` needs no user identity (unlike commits) const built = await build({ cwd: repo, providers: scriptedResolver([callWithArgs('c1', 'git_status', {}), textTurn('clean')]), @@ -203,7 +203,19 @@ describe('buildChatSession', () => { // The git_status tool call was annotated on the stream (the session routed it to the host)… const toolCall = events.find((e) => e.type === 'agent:tool_call'); expect(toolCall?.type === 'agent:tool_call' && toolCall.toolId).toBe('git_status'); - // …the process arm ran it and the result folded back, so the post-tool answer streamed (the loop completed). + // …the process arm produced a git_status RESULT that folded back. `success` here means only "the dispatch + // did not THROW" (the registry stamps it true for any resolved spawn) — it rules out git-not-on-PATH but, + // crucially, does NOT distinguish a clean run from an exit-128 "not a repository" (a non-zero git exit still + // RESOLVES a ProcessResult, so success stays true). The real lock is the summary: the process arm captured + // git's `{exitCode:0,…}` stdout, proving it ran to a CLEAN exit against the repo we init'd (not a silent + // wrong-cwd/uninit'd error the model could narrate 'clean' over from the cassette regardless). + const result = events.find((e) => e.type === 'agent:tool_result'); + expect(result?.type === 'agent:tool_result' && result.toolId).toBe('git_status'); + expect(result?.type === 'agent:tool_result' && result.success).toBe(true); + expect(result?.type === 'agent:tool_result' && result.outputSummary).toContain( + '"exitCode":0', + ); + // …and the post-tool answer streamed, so the loop completed after the fold. const tokens = events.flatMap((e) => (e.type === 'agent:token' ? [e.token] : [])).join(''); expect(tokens).toContain('clean'); } finally { From f32803a33d5461710a8d4b4fbd2a06b52c0aaf51 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 01:15:39 +0300 Subject: [PATCH 33/50] =?UTF-8?q?feat(cli):=202.5-close=20Step=2012=20(Bat?= =?UTF-8?q?ch=20B)=20=E2=80=94=202.5.H=20TUI=20polish=20(persistent=20abor?= =?UTF-8?q?t=20hint=20+=20line-bounded=20reasoning=20panel)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two chat-TUI live-region polishes, both landed in the pure, unit-tested projection formatters (the React view only arranges them): - Persistent mid-stream abort hint (EA7). The abort affordance showed on the pre-first-token status line ("Working… {elapsed} · Esc to stop") but vanished once answer content streamed — even though Esc aborts the WHOLE turn. New pure `streamingAbortHint(busy)` returns a compact "Esc to stop" only for a streaming CONTENT line; a STATUS line (compaction/shell/pre-token) already carries its inline hint, so it returns undefined (no double-print). Rendered as a dim line beneath the streaming content in ChatView. - Bound the EXPANDED reasoning panel by rendered ROWS. The panel body is capped at MAX_LIVE_TOKEN_CHARS (4000) CHARS, which can wrap to 60+ rows on a short/ narrow terminal and flicker in the live region. `formatReasoningPanel` now tails the expanded body to the last MAX_REASONING_PANEL_LINES (12) RENDERED rows — each logical line counts as ceil(len/columns) wrapped rows (min 1), so both many short lines AND one very long line are bounded; a single over-budget line is head-sliced to its tail. The `…` head-elision marker now ORs the store char-cap flag with this row tail. A new optional `columns` prop threads the live width (ChatApp reads process.stdout.columns; the Home passes its resize-tracked size.cols); absent ⇒ an 80-col fallback so the bound still applies headless/in tests. Kept OUT (per the batch scope): Ctrl+T-during-approval — it touches the approval floor and lands with the Step 14 security batch. Tests: +7 projection cases (streamingAbortHint status-vs-content; reasoning row-tail for short lines / wrapped rows / single-over-budget head-slice / marker OR / 80-col fallback). 54 projection + 656 render+chat tests pass; typecheck, eslint, prettier, build, fence, engine-deps all green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 12, Batch B); 2.5.H Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/render/tui/chat-ink.tsx | 35 ++++-- .../src/render/tui/chat-projection.test.ts | 102 ++++++++++++++++++ apps/cli/src/render/tui/chat-projection.ts | 75 ++++++++++++- apps/cli/src/render/tui/home-app.tsx | 4 + 4 files changed, 207 insertions(+), 9 deletions(-) diff --git a/apps/cli/src/render/tui/chat-ink.tsx b/apps/cli/src/render/tui/chat-ink.tsx index 9078ffda..93c3f899 100644 --- a/apps/cli/src/render/tui/chat-ink.tsx +++ b/apps/cli/src/render/tui/chat-ink.tsx @@ -94,6 +94,7 @@ import { formatTurnSummary, reasoningLabelActive, sanitizeInline, + streamingAbortHint, stripTerminalControls, } from './chat-projection.js'; import type { ReasoningEffort } from '@relavium/shared'; @@ -216,6 +217,11 @@ interface ChatViewProps { /** The in-flight `!`-shell command line (2.5.D) — when set, the busy indicator labels WHAT is running (a `!`- * command emits no session tokens, so without this the spinner would be bare) + how to cancel (Esc). */ readonly busyCommand?: string | undefined; + /** Live terminal width, to bound the EXPANDED reasoning panel to the last N rendered rows (2.5.H) so a full + * 4000-char buffer cannot wrap into a flickering, screen-filling panel on a short terminal. Render-only + + * cosmetic (parity with `nowMs`); the owner passes `process.stdout.columns` (ChatApp) or its resize-tracked + * width (the Home). Absent ⇒ the formatter's 80-col fallback. `| undefined` for the createElement passthrough. */ + readonly columns?: number | undefined; } /** @@ -246,6 +252,7 @@ export function ChatView(props: Readonly): ReactElement { liveReasoning: state.liveReasoning, liveReasoningTruncated: state.liveReasoningTruncated, visible: reasoningVisible, + columns: props.columns, }) : undefined; // The pre-token busy line reads "Thinking…" ONLY while the model is plausibly reasoning (reasoning streamed AND no @@ -274,12 +281,25 @@ export function ChatView(props: Readonly): ReactElement { elapsedMs, reasoningActive, }); - return line.dim ? ( - - {line.text} - - ) : ( - {line.text} + // A STATUS line (compaction / shell / pre-token) already carries its inline "· Esc to …" hint; a streaming + // CONTENT line has no room for it, so surface the abort affordance on a compact dim line beneath it — `Esc` + // aborts the whole turn (EA7), so the hint must persist for the ENTIRE turn, not just the pre-token wait. + const abortHint = streamingAbortHint(line); + return ( + + {line.dim ? ( + + {line.text} + + ) : ( + {line.text} + )} + {abortHint !== undefined && ( + + {abortHint} + + )} + ); }; return ( @@ -1067,6 +1087,9 @@ export function ChatApp(props: Readonly): ReactElement { approval={approval} attachments={attachments} busyCommand={busyCommand} + // Live terminal width for the reasoning-panel row bound (2.5.H) — read fresh each render (parity with + // `nowMs={Date.now()}`); the frame loop re-renders, so a resize is picked up on the next tick. + columns={process.stdout.columns} paletteOpen={ palette !== undefined || search !== undefined || diff --git a/apps/cli/src/render/tui/chat-projection.test.ts b/apps/cli/src/render/tui/chat-projection.test.ts index 3fb46caa..da5cd472 100644 --- a/apps/cli/src/render/tui/chat-projection.test.ts +++ b/apps/cli/src/render/tui/chat-projection.test.ts @@ -12,7 +12,9 @@ import { formatSessionFooterWithMode, formatToolCall, formatTurnSummary, + MAX_REASONING_PANEL_LINES, reasoningLabelActive, + streamingAbortHint, stripTerminalControls, } from './chat-projection.js'; import { formatDuration, formatTokens } from './format.js'; @@ -327,6 +329,30 @@ describe('chat-projection', () => { }); }); + describe('streamingAbortHint (2.5.H / EA7 — the abort affordance persists during streaming)', () => { + const base = { + spinner: '⠋', + compacting: false, + liveTokens: '', + liveTokensTruncated: false, + } as const; + + it('returns the standalone hint for a streaming CONTENT line (which has no inline hint)', () => { + const content = formatBusyLine({ ...base, liveTokens: 'the answer streams' }); + expect(content.dim).toBe(false); + expect(streamingAbortHint(content)).toBe('Esc to stop'); + }); + + it('returns undefined for every STATUS line — they already carry their own inline hint (no double-print)', () => { + // Pre-token ("· Esc to stop"), compaction ("· Esc to cancel"), and shell ("· Esc to cancel") are all dim. + expect(streamingAbortHint(formatBusyLine({ ...base, elapsedMs: 1000 }))).toBeUndefined(); + expect(streamingAbortHint(formatBusyLine({ ...base, compacting: true }))).toBeUndefined(); + expect( + streamingAbortHint(formatBusyLine({ ...base, busyCommand: 'npm test' })), + ).toBeUndefined(); + }); + }); + describe('reasoningLabelActive (2.5.H — Thinking… only when no tool is executing)', () => { const call = (resolved: boolean) => ({ id: 't', toolId: 'read_file', resolved }); @@ -380,6 +406,82 @@ describe('chat-projection', () => { expect(panel.body).not.toMatch(/\x1b/); expect(panel.body).toBe('line1\nline2'); // ANSI stripped, the newline (multi-line prose) kept }); + + describe('bounds the expanded body to the last N rendered rows (2.5.H)', () => { + it('keeps every line when the body fits within the row budget', () => { + const body = ['a', 'b', 'c'].join('\n'); + const panel = formatReasoningPanel({ + liveReasoning: body, + liveReasoningTruncated: false, + visible: true, + columns: 80, + }); + expect(panel.body).toBe(body); // under budget ⇒ no tail, no marker + }); + + it('tails MANY short lines to the last N and prefixes the elision marker', () => { + // 30 one-char lines = 30 rendered rows (each short line is its own row) — well over the 12-row budget. + const lines = Array.from({ length: 30 }, (_, i) => String(i)); + const panel = formatReasoningPanel({ + liveReasoning: lines.join('\n'), + liveReasoningTruncated: false, + visible: true, + columns: 80, + }); + const kept = lines.slice(lines.length - MAX_REASONING_PANEL_LINES); // the most-recent N + expect(panel.body).toBe(`…${kept.join('\n')}`); + }); + + it('counts WRAPPED rows: one long logical line spends multiple rows of the budget (narrow terminal)', () => { + // At width 10, a 25-char line wraps to ceil(25/10)=3 rows. With three such lines (9 rows) plus a fourth + // (→12) the budget is exactly full; a fifth older line (would be 15) is dropped. + const long = (tag: string): string => `${tag}`.padEnd(25, '.'); + const lines = [long('oldest'), long('l2'), long('l3'), long('l4'), long('newest')]; + const panel = formatReasoningPanel({ + liveReasoning: lines.join('\n'), + liveReasoningTruncated: false, + visible: true, + columns: 10, + }); + // 4 lines × 3 rows = 12 rows = the budget; the oldest is dropped, so the marker shows. + expect(panel.body).toBe(`…${lines.slice(1).join('\n')}`); + }); + + it('slices the HEAD of a single line that alone exceeds the whole budget (keeps its tail)', () => { + // One 200-char line at width 10 = 20 rows > the 12-row budget; keep only the last 12×10 = 120 chars. + const line = 'x'.repeat(200); + const panel = formatReasoningPanel({ + liveReasoning: line, + liveReasoningTruncated: false, + visible: true, + columns: 10, + }); + expect(panel.body).toBe(`…${'x'.repeat(MAX_REASONING_PANEL_LINES * 10)}`); + }); + + it('ORs the row-tail elision with the store char-cap marker (one leading marker either way)', () => { + const lines = Array.from({ length: 30 }, (_, i) => String(i)); + const panel = formatReasoningPanel({ + liveReasoning: lines.join('\n'), + liveReasoningTruncated: true, // the store already elided the head too + visible: true, + columns: 80, + }); + expect(panel.body?.startsWith('…')).toBe(true); + expect(panel.body?.startsWith('……')).toBe(false); // exactly one marker, not doubled + }); + + it('falls back to an 80-col assumption when no width is passed (headless/test render)', () => { + // A 400-char single line: at the 80-col fallback that is 5 rows (< budget) ⇒ kept whole, no tail. + const line = 'y'.repeat(400); + const panel = formatReasoningPanel({ + liveReasoning: line, + liveReasoningTruncated: false, + visible: true, + }); + expect(panel.body).toBe(line); + }); + }); }); describe('formatSessionFooter', () => { diff --git a/apps/cli/src/render/tui/chat-projection.ts b/apps/cli/src/render/tui/chat-projection.ts index 374ec708..130f09c7 100644 --- a/apps/cli/src/render/tui/chat-projection.ts +++ b/apps/cli/src/render/tui/chat-projection.ts @@ -266,6 +266,19 @@ export function formatBusyLine(input: { return { text: `${spinner} ${input.liveTokensTruncated ? '…' : ''}${content}`, dim: false }; } +/** + * The compact mid-stream abort hint (2.5.H / EA7). The pre-first-token STATUS line carries "· Esc to stop" + * inline, and the compaction/shell status lines carry their own "· Esc to cancel" — but once answer CONTENT + * streams the busy line becomes full-width, wrapping text with no room for the affordance, even though `Esc` + * still aborts the whole turn. Returns the standalone hint the caller renders as a dim line BENEATH the streaming + * content, or `undefined` for a STATUS line (`busy.dim`) that already shows its inline hint — so the affordance + * is visible for the ENTIRE turn without ever double-printing. Keyed off the same `dim` flag the caller maps, so + * there is one source of truth for status-vs-content. + */ +export function streamingAbortHint(busy: BusyLine): string | undefined { + return busy.dim ? undefined : 'Esc to stop'; +} + /** * Whether the pre-token busy line should read "Thinking…" (vs "Working…") — the turn streamed reasoning AND no tool * call is currently executing (2.5.H). During a tool round the model idle-waits on the tool, so the label falls @@ -290,23 +303,79 @@ export interface ReasoningPanel { readonly body?: string; } +/** + * The max terminal ROWS the EXPANDED reasoning body may occupy (2.5.H). The store already bounds the reasoning + * buffer to {@link MAX_LIVE_TOKEN_CHARS} (4000) CHARACTERS, but on a short/narrow terminal that wraps to many + * dozens of rows in the live (non-``) region — a churning, flickering panel that buries the answer below + * it. Tailing the DISPLAYED body to the last N rendered rows (see {@link formatReasoningPanel}) keeps the panel + * compact regardless of terminal size, while the FULL reasoning is still persisted from the raw events. + */ +export const MAX_REASONING_PANEL_LINES = 12; + +/** The assumed width when the caller passes no live column count (a headless/test render, or a non-TTY stdout with + * no `.columns`). 80 is the conventional terminal width + the 80×24 degrade floor the harness pins. */ +const REASONING_PANEL_FALLBACK_COLUMNS = 80; + +/** + * Tail `text` to its last {@link MAX_REASONING_PANEL_LINES} RENDERED rows at the given width — a long logical line + * counts as `⌈len / columns⌉` wrapped rows (never < 1), so both many short lines AND one very long line are bounded. + * When the single most-recent logical line alone exceeds the row budget its HEAD is sliced so the tail still fits. + * Returns the kept body + whether anything was dropped (`tailed`), so the caller can show the head-elision marker. + */ +function tailToRenderedRows( + text: string, + columns: number | undefined, +): { body: string; tailed: boolean } { + const width = + columns !== undefined && columns > 0 ? Math.floor(columns) : REASONING_PANEL_FALLBACK_COLUMNS; + const rowsOf = (line: string): number => Math.max(1, Math.ceil(line.length / width)); + const lines = text.split('\n'); + const kept: string[] = []; + let rows = 0; + for (let i = lines.length - 1; i >= 0; i -= 1) { + const line = lines[i] ?? ''; + if (rows + rowsOf(line) > MAX_REASONING_PANEL_LINES) { + // The tail is full. If we have kept nothing yet, this single (oldest-included) line is itself taller than the + // whole budget — keep only its last budget×width chars so the most recent reasoning still shows. Otherwise + // stop: the already-kept newer lines fill the budget and older ones are dropped. + if (kept.length === 0) { + kept.unshift(line.slice(line.length - MAX_REASONING_PANEL_LINES * width)); + } + return { body: kept.join('\n'), tailed: true }; + } + kept.unshift(line); + rows += rowsOf(line); + } + return { body: kept.join('\n'), tailed: false }; +} + /** * Project the in-flight reasoning into the collapsible panel (2.5.H). Collapsed (default): a dim header with the * Ctrl+T toggle hint, so the user knows thinking is available without it flooding the view. Expanded: the header + * the reasoning BODY — terminal-sanitized at this display boundary (newline-preserving, it is multi-line prose), - * with a leading `…` elision marker when the bounded buffer's head scrolled out (parity with the answer stream). + * bounded to the last {@link MAX_REASONING_PANEL_LINES} rendered rows so a full 4000-char buffer cannot wrap into a + * flickering, screen-filling panel on a short terminal, with a leading `…` elision marker when the buffer's head + * scrolled out — via the store's char cap (`liveReasoningTruncated`) OR this row tail (parity with the answer stream). */ export function formatReasoningPanel(input: { readonly liveReasoning: string; readonly liveReasoningTruncated: boolean; readonly visible: boolean; + /** Live terminal width in columns, for bounding the expanded body to the last N RENDERED rows. Render-only + + * cosmetic (parity with `nowMs`): the owner passes `process.stdout.columns` / its resize-tracked width. Absent + * or ≤0 ⇒ the {@link REASONING_PANEL_FALLBACK_COLUMNS} default, so the row bound still applies headless/in tests. */ + readonly columns?: number | undefined; }): ReasoningPanel { const header = `✻ Reasoning · Ctrl+T ${input.visible ? 'hide' : 'show'}`; if (!input.visible) { return { header }; } - const body = `${input.liveReasoningTruncated ? '…' : ''}${stripTerminalControls(input.liveReasoning)}`; - return { header, body }; + const { body, tailed } = tailToRenderedRows( + stripTerminalControls(input.liveReasoning), + input.columns, + ); + const elided = input.liveReasoningTruncated || tailed; + return { header, body: `${elided ? '…' : ''}${body}` }; } /** diff --git a/apps/cli/src/render/tui/home-app.tsx b/apps/cli/src/render/tui/home-app.tsx index 25be8252..3f0b50f4 100644 --- a/apps/cli/src/render/tui/home-app.tsx +++ b/apps/cli/src/render/tui/home-app.tsx @@ -57,6 +57,8 @@ function ChatRegion( * re-renders on controller state changes. So the live-turn timer (2.5.H) MUST read the clock here, per frame — * a number prop frozen at the parent's last render would stick the elapsed at 0s for the whole pre-token wait. */ now: () => number; + /** The live terminal width (resize-tracked in `RootApp`) — bounds the reasoning panel to N rendered rows (2.5.H). */ + cols: number; shellBusy: boolean; submitBusy: boolean; shellCommand: string | undefined; @@ -83,6 +85,7 @@ function ChatRegion( approval={approval} attachments={props.attachments} busyCommand={props.shellCommand} + columns={props.cols} paletteOpen={ props.palette !== undefined || props.search !== undefined || @@ -137,6 +140,7 @@ export function RootApp(props: Readonly): ReactElement { modelPicker={state.modelPicker} effortPicker={state.effortPicker} now={props.nowMs} + cols={size.cols} shellBusy={state.shellBusy} submitBusy={state.submitBusy} shellCommand={state.shellCommand} From 73de915158c018da7e729781d88806e7e7b3fb98 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 01:25:41 +0300 Subject: [PATCH 34/50] =?UTF-8?q?fix(cli):=202.5-close=20Step=2012=20Opus-?= =?UTF-8?q?review=20fixes=20=E2=80=94=20width-count=20honesty=20+=20div-by?= =?UTF-8?q?-zero=20floor?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review of the 2.5.H reasoning-panel row bound — no blockers; applied the correctness/honesty items: - MEDIUM: `tailToRenderedRows` counts `.length` (UTF-16 units), not ink's display width, so a wide-glyph (CJK/emoji) reasoning stream under-counts rows (renders up to ~2× the budget) and the prepended `…` marker can add one more row. The code claimed a hard "last N rendered rows" bound. Documented it honestly as an APPROXIMATE, cosmetic anti-flicker guard (matching the store's existing `.length`-based char cap; no `string-width` runtime dependency wanted) and softened the `formatReasoningPanel` JSDoc accordingly. - NIT (latent div-by-zero): tightened the width guard from `columns > 0` to `Math.floor(columns) >= 1` so a fractional 0 --- apps/cli/src/render/tui/chat-ink.tsx | 2 ++ .../src/render/tui/chat-projection.test.ts | 14 ++++++++++++++ apps/cli/src/render/tui/chat-projection.ts | 19 +++++++++++++++---- 3 files changed, 31 insertions(+), 4 deletions(-) diff --git a/apps/cli/src/render/tui/chat-ink.tsx b/apps/cli/src/render/tui/chat-ink.tsx index 93c3f899..6de2a1bc 100644 --- a/apps/cli/src/render/tui/chat-ink.tsx +++ b/apps/cli/src/render/tui/chat-ink.tsx @@ -1089,6 +1089,8 @@ export function ChatApp(props: Readonly): ReactElement { busyCommand={busyCommand} // Live terminal width for the reasoning-panel row bound (2.5.H) — read fresh each render (parity with // `nowMs={Date.now()}`); the frame loop re-renders, so a resize is picked up on the next tick. + // `stdout.columns` is typed `number` but is `undefined` at runtime off a TTY — the formatter's 80-col + // fallback covers that (moot here anyway: ChatApp only mounts on a TTY via the driveInk gate). columns={process.stdout.columns} paletteOpen={ palette !== undefined || diff --git a/apps/cli/src/render/tui/chat-projection.test.ts b/apps/cli/src/render/tui/chat-projection.test.ts index da5cd472..4d4eabd3 100644 --- a/apps/cli/src/render/tui/chat-projection.test.ts +++ b/apps/cli/src/render/tui/chat-projection.test.ts @@ -419,6 +419,20 @@ describe('chat-projection', () => { expect(panel.body).toBe(body); // under budget ⇒ no tail, no marker }); + it('keeps everything with NO marker when the rows sum to EXACTLY the budget', () => { + // Exactly MAX one-char lines at width 80 = exactly 12 rendered rows — the boundary case: the loop ends + // naturally (nothing dropped), so `tailed` is false and no leading marker is added. + const lines = Array.from({ length: MAX_REASONING_PANEL_LINES }, (_, i) => String(i)); + const body = lines.join('\n'); + const panel = formatReasoningPanel({ + liveReasoning: body, + liveReasoningTruncated: false, + visible: true, + columns: 80, + }); + expect(panel.body).toBe(body); // == budget ⇒ kept whole, no `…` + }); + it('tails MANY short lines to the last N and prefixes the elision marker', () => { // 30 one-char lines = 30 rendered rows (each short line is its own row) — well over the 12-row budget. const lines = Array.from({ length: 30 }, (_, i) => String(i)); diff --git a/apps/cli/src/render/tui/chat-projection.ts b/apps/cli/src/render/tui/chat-projection.ts index 130f09c7..ca21e274 100644 --- a/apps/cli/src/render/tui/chat-projection.ts +++ b/apps/cli/src/render/tui/chat-projection.ts @@ -321,13 +321,23 @@ const REASONING_PANEL_FALLBACK_COLUMNS = 80; * counts as `⌈len / columns⌉` wrapped rows (never < 1), so both many short lines AND one very long line are bounded. * When the single most-recent logical line alone exceeds the row budget its HEAD is sliced so the tail still fits. * Returns the kept body + whether anything was dropped (`tailed`), so the caller can show the head-elision marker. + * + * The row count is APPROXIMATE, not exact: `.length` (UTF-16 units) stands in for ink's display-width wrap, so a + * wide-glyph (CJK/emoji) line under-counts and a combining-mark line over-counts — the panel can render up to ~2× + * on wide text; the prepended `…` marker in {@link formatReasoningPanel} can add one more row. This matches the + * store's existing `.length`-based 4000-char cap and avoids a `string-width` runtime dependency; the bound is a + * cosmetic anti-flicker guard, so an off-by-a-row on unusual scripts is acceptable. */ function tailToRenderedRows( text: string, columns: number | undefined, ): { body: string; tailed: boolean } { + // `>= 1` (not `> 0`): a fractional 0 0 ? Math.floor(columns) : REASONING_PANEL_FALLBACK_COLUMNS; + columns !== undefined && Math.floor(columns) >= 1 + ? Math.floor(columns) + : REASONING_PANEL_FALLBACK_COLUMNS; const rowsOf = (line: string): number => Math.max(1, Math.ceil(line.length / width)); const lines = text.split('\n'); const kept: string[] = []; @@ -353,9 +363,10 @@ function tailToRenderedRows( * Project the in-flight reasoning into the collapsible panel (2.5.H). Collapsed (default): a dim header with the * Ctrl+T toggle hint, so the user knows thinking is available without it flooding the view. Expanded: the header + * the reasoning BODY — terminal-sanitized at this display boundary (newline-preserving, it is multi-line prose), - * bounded to the last {@link MAX_REASONING_PANEL_LINES} rendered rows so a full 4000-char buffer cannot wrap into a - * flickering, screen-filling panel on a short terminal, with a leading `…` elision marker when the buffer's head - * scrolled out — via the store's char cap (`liveReasoningTruncated`) OR this row tail (parity with the answer stream). + * bounded to ~the last {@link MAX_REASONING_PANEL_LINES} rendered rows (an approximate width count — see + * {@link tailToRenderedRows}) so a full 4000-char buffer cannot wrap into a flickering, screen-filling panel on a + * short terminal, with a leading `…` elision marker when the buffer's head scrolled out — via the store's char cap + * (`liveReasoningTruncated`) OR this row tail (parity with the answer stream). */ export function formatReasoningPanel(input: { readonly liveReasoning: string; From c5a7d36b37e59e1fdf21e2db14cba5cb4d93a7c9 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 01:35:18 +0300 Subject: [PATCH 35/50] =?UTF-8?q?fix(cli):=202.5-close=20Step=2012=20Sonne?= =?UTF-8?q?t-review=20fixes=20=E2=80=94=20trailing-newline=20row=20miscoun?= =?UTF-8?q?t?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet review of the 2.5.H reasoning-panel row bound (caught a case the Opus round missed): - MEDIUM: a live reasoning buffer cut off right after a line break (very common between stream chunks) ends in '\n', which `split('\n')` turns into a phantom empty last segment. `rowsOf('')` clamps to 1, so that phantom spent a full budget row and could prematurely DROP a real line + flash the `…` marker for content that hasn't arrived yet (self-correcting on the next token, but a real untested gap). Fix: strip exactly ONE trailing '\n' before splitting — the not-yet-filled next line costs zero rows, while an intentional trailing blank line ("\n\n") keeps its blank (only the final cursor newline is dropped). - LOW: the `columns` prop JSDoc said "Absent or ≤0 ⇒ fallback" but the guard is `Math.floor(columns) >= 1` (any < 1 falls back). Corrected the doc to "< 1". - LOW: the div-by-zero guard the prior fix commit added was untested. Added a case pinning columns ∈ {0, 0.5, -5, NaN} → 80-col fallback (no crash). Tests: +4 cases (trailing-newline no-drop; intentional trailing blank kept; fractional/non-positive width fallback). 58 projection cases pass; typecheck/eslint/prettier green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 12, Batch B); 2.5.H Co-Authored-By: Claude Opus 4.8 (1M context) --- .../src/render/tui/chat-projection.test.ts | 39 +++++++++++++++++++ apps/cli/src/render/tui/chat-projection.ts | 9 ++++- 2 files changed, 46 insertions(+), 2 deletions(-) diff --git a/apps/cli/src/render/tui/chat-projection.test.ts b/apps/cli/src/render/tui/chat-projection.test.ts index 4d4eabd3..fa28cdd1 100644 --- a/apps/cli/src/render/tui/chat-projection.test.ts +++ b/apps/cli/src/render/tui/chat-projection.test.ts @@ -495,6 +495,45 @@ describe('chat-projection', () => { }); expect(panel.body).toBe(line); }); + + it('falls back to 80 cols for a non-positive/fractional width (guards the div-by-zero floor)', () => { + // The width guard is `Math.floor(columns) >= 1` — 0 / 0.5 / negative / NaN all take the 80-col fallback + // (no divide-by-zero, no negative slice). A 400-char line = 5 rows at 80 ⇒ kept whole, proving the width. + const line = 'y'.repeat(400); + for (const columns of [0, 0.5, -5, Number.NaN]) { + const panel = formatReasoningPanel({ + liveReasoning: line, + liveReasoningTruncated: false, + visible: true, + columns, + }); + expect(panel.body).toBe(line); // 80-col fallback ⇒ under budget ⇒ no crash, no tail + } + }); + + it('does NOT count a trailing newline as a rendered row (no premature drop between stream chunks)', () => { + // A live buffer cut off right after a line break ends in '\n'. The MAX real lines alone fit the budget + // exactly; the trailing '\n' must not spend a phantom row that drops the oldest real line + flashes `…`. + const lines = Array.from({ length: MAX_REASONING_PANEL_LINES }, (_, i) => String(i)); + const panel = formatReasoningPanel({ + liveReasoning: `${lines.join('\n')}\n`, // trailing newline (mid-stream cut) + liveReasoningTruncated: false, + visible: true, + columns: 80, + }); + expect(panel.body).toBe(lines.join('\n')); // every real line kept, no leading marker + }); + + it('keeps an INTENTIONAL trailing blank line (only the final cursor newline is dropped)', () => { + // "a\n\n" is a line + a deliberate blank line + the cursor newline; stripping ONE '\n' keeps the blank. + const panel = formatReasoningPanel({ + liveReasoning: 'a\n\n', + liveReasoningTruncated: false, + visible: true, + columns: 80, + }); + expect(panel.body).toBe('a\n'); // the intentional blank line survives; only the trailing cursor newline goes + }); }); }); diff --git a/apps/cli/src/render/tui/chat-projection.ts b/apps/cli/src/render/tui/chat-projection.ts index ca21e274..c5ef06c8 100644 --- a/apps/cli/src/render/tui/chat-projection.ts +++ b/apps/cli/src/render/tui/chat-projection.ts @@ -339,7 +339,12 @@ function tailToRenderedRows( ? Math.floor(columns) : REASONING_PANEL_FALLBACK_COLUMNS; const rowsOf = (line: string): number => Math.max(1, Math.ceil(line.length / width)); - const lines = text.split('\n'); + // Drop ONE trailing newline before splitting: a live buffer cut off right after a line break (very common + // between stream chunks) ends in '\n', which `split` turns into a phantom empty last segment. That segment + // would cost a full budget row (`rowsOf('')` clamps to 1) and could prematurely drop a REAL line + flash the + // elision marker for content that hasn't arrived yet. Stripping exactly one '\n' treats the not-yet-filled + // next line as zero rows while KEEPING any intentional trailing blank line (a "\n\n" loses only the last). + const lines = (text.endsWith('\n') ? text.slice(0, -1) : text).split('\n'); const kept: string[] = []; let rows = 0; for (let i = lines.length - 1; i >= 0; i -= 1) { @@ -374,7 +379,7 @@ export function formatReasoningPanel(input: { readonly visible: boolean; /** Live terminal width in columns, for bounding the expanded body to the last N RENDERED rows. Render-only + * cosmetic (parity with `nowMs`): the owner passes `process.stdout.columns` / its resize-tracked width. Absent - * or ≤0 ⇒ the {@link REASONING_PANEL_FALLBACK_COLUMNS} default, so the row bound still applies headless/in tests. */ + * or < 1 ⇒ the {@link REASONING_PANEL_FALLBACK_COLUMNS} default, so the row bound still applies headless/in tests. */ readonly columns?: number | undefined; }): ReasoningPanel { const header = `✻ Reasoning · Ctrl+T ${input.visible ? 'hide' : 'show'}`; From 2e6527ba8720aaad8ea27528042a35e7360dc3b1 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 04:47:34 +0300 Subject: [PATCH 36/50] =?UTF-8?q?fix(core):=202.5-close=20Step=2013=20(Bat?= =?UTF-8?q?ch=20C)=20=E2=80=94=20AgentParseError=20line/col=20({{ctx.*}}?= =?UTF-8?q?=20deferred)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Attach actionable line/column diagnostics to a `.agent.yaml` YAML syntax fault, the agent sibling of parser.ts's `syntaxErrorFrom`: - AgentParseError gains optional 1-based `line`/`column` fields (parity with WorkflowSyntaxError), populated for an `agent_syntax` YAMLParseError when the loader reports a position (skipped for a bare stream error at offset -1). A new `agentSyntaxErrorFrom` normalizes the decode throw: a positioned YAMLParseError → the YAML RULE message + folded `(source — line L, column C)` locator + the structured fields; an anchor/alias ReferenceError (maxAliasCount:0) → a clear source-free label with NO bogus position; else the generic fallback. - The position is folded into the message (via a shared `locate()` suffix mirroring the config loader's `(line L, column C)` shape) so every existing `.message`-surfacing consumer (catalog `parseReason`, authoring, agent run) shows it end-to-end — the structured fields are additionally exposed for machine/--json readers. - Echoing the YAML rule is secret-free by construction: `prettyErrors: false` (yaml-decode.ts) makes the message the rule ALONE — no source snippet, no authored key/value (verified: a duplicate-key fault does not echo the value). A regression test pins that an authored (secret-shaped) value never leaks. The Step 13 `{{ctx.*}}` session-interpolation half is DEFERRED to Phase 2.6 / 2.6.D per the maintainer's 2.5-close decision (2026-07-08): it is governed by ADR-0060 (Proposed) which requires a new SessionContext taint-provenance marker + a mandatory pre-Accept security review — not a plain resolveTemplate reuse, and not appropriate to pull forward into the close-out. Recorded in deferred-tasks.md. Tests: +2 agent-parser cases (line/col attach + secret-free syntax message) and the alias case; existing syntax test extended. 982 core tests pass; catalog/list consumer suites green; typecheck/eslint/prettier/build green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 13, Batch C); ADR-0060 (deferral) Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/roadmap/deferred-tasks.md | 13 ++++- packages/core/src/agent-parser.test.ts | 38 ++++++++++++- packages/core/src/agent-parser.ts | 76 +++++++++++++++++++++++--- 3 files changed, 116 insertions(+), 11 deletions(-) diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index 0692cd1a..cdf33071 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -499,8 +499,17 @@ Severity is the review's verified rating. Check an item off in the PR that resol `#context.variables` the way the workflow `AgentRunner` interpolates an `agent` node's prompt. So `relavium agent run --input k=v` (2.Q) carries the variables in `SessionContext` (visible on `session:started`) but a `{{ctx.k}}` placeholder in the agent's prompt is sent to the model **literally**. Wire a `resolveTemplate` - pass over the session prompt against a `RunScope` built from `context.variables` (deliberately deferred — - no surface needed it before 2.Q). *(medium · packages/core/src/engine/agent-session.ts)* + pass over the session prompt against a `RunScope` built from `context.variables`. **Governed by + [ADR-0060](../decisions/0060-session-ctx-prompt-interpolation.md) (Proposed, Phase-2.6 / workstream 2.6.D)** — + it is NOT a plain `resolveTemplate` reuse: the safe implementation requires a **new per-variable + provenance/taint marker on `SessionContext`** (today a flat record) so `--input`-derived (untrusted) values + can never reach the `system` position, plus the ADR's **mandatory security review of the session-prompt taint + path before Accept**. **2.5-close decision (2026-07-08):** the phase-2.5 close-plan (Step 13, Batch C) proposed + landing this here; the maintainer chose to **DEFER to Phase 2.6 / 2.6.D** — where ADR-0060 is finalized + (Proposed→Accepted) with its taint-provenance marker + security review — rather than pull an unaccepted, + security-critical Phase-2.6 ADR forward into the consolidation close-out. Only the sibling + `AgentParseError` line/col half of Step 13 landed. *(medium · packages/core/src/engine/agent-session.ts; + ADR-0060)* ## Phase 2.5.D (`@`-mention / input ergonomics) follow-ups diff --git a/packages/core/src/agent-parser.test.ts b/packages/core/src/agent-parser.test.ts index b6f7e933..65a02e64 100644 --- a/packages/core/src/agent-parser.test.ts +++ b/packages/core/src/agent-parser.test.ts @@ -44,13 +44,47 @@ describe('parseAgent', () => { } }); - it('rejects a YAML syntax fault as agent_syntax', () => { + it('rejects a YAML syntax fault as agent_syntax, attaching the fault line/column', () => { try { - parseAgent('id: x\n : : :'); + parseAgent('id: x\n : : :', { source: 'agents/broken.agent.yaml' }); expect.unreachable('should have thrown'); } catch (err) { if (!(err instanceof AgentParseError)) throw err; expect(err.code).toBe('agent_syntax'); + // The fault is on line 2 — line/column are attached (1-based) AND folded into the message for actionable + // diagnostics; the source label rides along. The YAML rule is the message body (secret-free, prettyErrors:false). + expect(err.line).toBe(2); + expect(typeof err.column).toBe('number'); + expect(err.message).toContain(`line ${err.line}, column ${err.column}`); + expect(err.message).toContain('agents/broken.agent.yaml'); + } + }); + + it('does not echo an authored value in a syntax-fault message (secret-free)', () => { + // A duplicate-key fault: the YAML rule ("Map keys must be unique") names no key/value, so no authored + // content (here a plausible secret-shaped value) can ride the message. + try { + parseAgent('id: sk-live-not-a-real-secret-000\nid: other\nmodel: m'); + expect.unreachable('should have thrown'); + } catch (err) { + if (!(err instanceof AgentParseError)) throw err; + expect(err.code).toBe('agent_syntax'); + expect(err.message).not.toContain('sk-live-not-a-real-secret-000'); + } + }); + + it('rejects an anchor/alias with a clear message and no bogus position (maxAliasCount: 0)', () => { + // Aliases are disabled in the hardened profile → a ReferenceError (not a positioned YAMLParseError), so the + // message is the clear source-free label and no misleading line/column is attached. + try { + parseAgent('id: &x tiny\nmodel: *x\nprovider: anthropic\nsystem_prompt: hi'); + expect.unreachable('should have thrown'); + } catch (err) { + if (!(err instanceof AgentParseError)) throw err; + expect(err.code).toBe('agent_syntax'); + expect(err.message).toContain('anchors and aliases are not supported'); + expect(err.line).toBeUndefined(); + expect(err.column).toBeUndefined(); } }); diff --git a/packages/core/src/agent-parser.ts b/packages/core/src/agent-parser.ts index 1c46a119..e46b8f63 100644 --- a/packages/core/src/agent-parser.ts +++ b/packages/core/src/agent-parser.ts @@ -10,7 +10,7 @@ * live in the keychain, ADR-0006), but the error still echoes only field *paths*, never an authored value. */ -import { LineCounter } from 'yaml'; +import { LineCounter, YAMLParseError } from 'yaml'; import type { ZodIssue } from 'zod'; import { AgentSchema, type Agent } from '@relavium/shared'; @@ -29,30 +29,92 @@ export interface ParseAgentOptions { /** * A typed, secret-free agent parse failure — a YAML syntax fault or a schema rejection. Narrow on `code`, * never on `message`. `fields` lists the failing field *paths* (key names / indices only, no authored values). + * For an `agent_syntax` YAML fault the `line`/`column` (1-based) of the fault are attached when the loader + * reports a position — parity with `WorkflowSyntaxError` (errors.ts) — and are also folded into the message. */ export class AgentParseError extends Error { readonly code: 'agent_syntax' | 'agent_validation'; readonly fields: readonly string[]; + /** 1-based line of a YAML syntax fault, when the loader reports a position (`agent_syntax` only). */ + readonly line?: number; + /** 1-based column of a YAML syntax fault, when the loader reports a position (`agent_syntax` only). */ + readonly column?: number; constructor( code: 'agent_syntax' | 'agent_validation', message: string, fields: readonly string[] = [], - opts?: { readonly cause?: unknown }, + opts?: { readonly cause?: unknown; readonly line?: number; readonly column?: number }, ) { super(message, opts?.cause === undefined ? undefined : { cause: opts.cause }); this.name = 'AgentParseError'; this.code = code; this.fields = fields; + if (opts?.line !== undefined) { + this.line = opts.line; + } + if (opts?.column !== undefined) { + this.column = opts.column; + } } } +/** A secret-free ` (source — line L, column C)` locator suffix from a workspace-relative source label and/or a + * YAML position. Only field-name/integer context — never an authored value or a source snippet. Empty when + * neither is known. The ` (line L, column C)` shape mirrors the config loader's TOML suffix (config/load.ts). */ +function locate(source: string | undefined, pos?: { line: number; col: number }): string { + const parts = [ + source, + pos === undefined ? undefined : `line ${pos.line}, column ${pos.col}`, + ].filter((part): part is string => part !== undefined); + return parts.length === 0 ? '' : ` (${parts.join(' — ')})`; +} + +/** + * Normalize a YAML decode throw to a typed, secret-free `agent_syntax` error, attaching the fault's line/column + * (1-based) when the loader reports a position — the agent sibling of parser.ts's `syntaxErrorFrom`, so agent + * files get the same actionable diagnostics as workflow files. `prettyErrors: false` (yaml-decode.ts) keeps the + * `YAMLParseError` message the RULE alone — no source snippet, no authored key/value — so echoing it is safe. + */ +function agentSyntaxErrorFrom( + err: unknown, + source: string | undefined, + lineCounter: LineCounter, +): AgentParseError { + if (err instanceof YAMLParseError) { + // `err.pos[0]` is -1 when the loader has no offset for the fault (a bare stream error); `linePos(-1)` would + // return a nonsensical {line:0, col:-1}, so skip it and emit a positionless error (mirrors parser.ts). + const posOffset = err.pos[0]; + const pos = posOffset >= 0 ? lineCounter.linePos(posOffset) : undefined; + return new AgentParseError('agent_syntax', `${err.message}${locate(source, pos)}`, [], { + cause: err, + ...(pos === undefined ? {} : { line: pos.line, column: pos.col }), + }); + } + if (err instanceof Error && /alias/i.test(err.message)) { + // `maxAliasCount: 0` throws a plain `ReferenceError` ("Alias resolution is disabled") with no position — + // surface a clear, source-free message rather than the generic fallback (parity with parser.ts). + return new AgentParseError( + 'agent_syntax', + `anchors and aliases are not supported${locate(source)}`, + ); + } + return new AgentParseError( + 'agent_syntax', + `the agent file is not valid YAML${locate(source)}`, + [], + { + cause: err, + }, + ); +} + /** * Parse + validate an agent YAML string. Throws {@link AgentParseError} (`agent_syntax` on a YAML fault, * `agent_validation` on a schema failure) — an invalid file never yields an `AgentDefinition`. */ export function parseAgent(yamlText: string, opts?: ParseAgentOptions): AgentDefinition { const source = opts?.source; - const label = source === undefined ? '' : ` (${source})`; + const label = locate(source); if (yamlText.length > MAX_SOURCE_CHARS) { throw new AgentParseError( @@ -61,13 +123,13 @@ export function parseAgent(yamlText: string, opts?: ParseAgentOptions): AgentDef ); } + // Hoisted (not inlined) so a YAML fault's byte offset can be resolved to a line/column AFTER the throw. + const lineCounter = new LineCounter(); let raw: unknown; try { - raw = decodeHardenedYaml(yamlText, new LineCounter()); + raw = decodeHardenedYaml(yamlText, lineCounter); } catch (err) { - throw new AgentParseError('agent_syntax', `the agent file is not valid YAML${label}`, [], { - cause: err, - }); + throw agentSyntaxErrorFrom(err, source, lineCounter); } const result = AgentSchema.safeParse(raw); From a8dd337073aaca31a58f9c5f48feb95abb8ba0ba Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 04:54:50 +0300 Subject: [PATCH 37/50] =?UTF-8?q?fix(core):=202.5-close=20Step=2013=20Opus?= =?UTF-8?q?-review=20fix=20=E2=80=94=20keep=20the=20alias-branch=20cause?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opus review of the AgentParseError line/col change — no blockers; the security claim (echoed YAML rule is secret-free) held under a 14-case empirical sweep. Applied the one code finding (LOW): the anchor/alias branch of `agentSyntaxErrorFrom` called the 2-arg AgentParseError ctor and swallowed the underlying ReferenceError — diverging from parser.ts's sibling alias branch (which attaches `cause`) and from error-handling.md's no-swallow rule. Attach the cause (a fixed, non-secret "Alias resolution is disabled") to match. The two NITs (the `—` em-dash glyph; a commit-message wording nuance on --json exposure) are acceptable as-is per the review. Tests: 8 agent-parser cases pass; typecheck/eslint/prettier green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 13, Batch C) Co-Authored-By: Claude Opus 4.8 (1M context) --- packages/core/src/agent-parser.ts | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/packages/core/src/agent-parser.ts b/packages/core/src/agent-parser.ts index e46b8f63..d84265ba 100644 --- a/packages/core/src/agent-parser.ts +++ b/packages/core/src/agent-parser.ts @@ -92,10 +92,13 @@ function agentSyntaxErrorFrom( } if (err instanceof Error && /alias/i.test(err.message)) { // `maxAliasCount: 0` throws a plain `ReferenceError` ("Alias resolution is disabled") with no position — - // surface a clear, source-free message rather than the generic fallback (parity with parser.ts). + // surface a clear, source-free message rather than the generic fallback (parity with parser.ts). Keep the + // `cause` (the fixed, non-secret ReferenceError) so the root cause is never swallowed (error-handling.md). return new AgentParseError( 'agent_syntax', `anchors and aliases are not supported${locate(source)}`, + [], + { cause: err }, ); } return new AgentParseError( From 9132ff99e9670016fdf9535bb5f2a7205f0e7a97 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 05:15:51 +0300 Subject: [PATCH 38/50] =?UTF-8?q?test(core,cli):=202.5-close=20Step=2013?= =?UTF-8?q?=20Sonnet-review=20fixes=20=E2=80=94=20cause=20+=20locator=20+?= =?UTF-8?q?=20surfacing=20gap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet review of the AgentParseError line/col change — no blockers; the security sweep held (12 more adversarial decode cases + a full enumeration of the yaml throw surface: no authored key/value ever rides the message). Applied: - MEDIUM-1 (regression test for the Opus-round cause fix): added `err.cause` assertions — the alias branch preserves the ReferenceError, the syntax branch the YAMLParseError. Reverting the cause fix now fails a test (testing.md). - LOW-1: pinned the pos-only `locate()` branch — a syntax fault with NO source ends in a clean `(line L, column C)`, no stray `—` separator, no `undefined`. - LOW-2: added a catalog end-to-end test — a broken `.agent.yaml` with a real YAML SYNTAX fault surfaces the line/column through `discoverCatalog`/parseReason (proving Step 13's diagnostic reaches the catalog surface, secret-free). - MEDIUM-2 (pre-existing, DEFERRED — recorded in deferred-tasks.md): a malformed `.agent.yaml` on the `chat --agent` / `agent run` surfaces reaches the top-level handler as exit 1 + a generic "internal error" (the diagnostic only actually surfaces via list/create/import). `resolveChatAgent` DELIBERATELY surfaces the raw AgentParseError — pinned by agent-source.test.ts "not a silent default or CliError" — so re-tagging is a cross-surface DESIGN decision conflicting with an existing test, not a mechanical fix; it predates this diff and is out of the fix(core) close-out scope. Documented with candidate fixes (wrap at the caller vs. teach the top-level renderError) + the absolute-vs-relative `source` note. Clarified the resolveChatAgent docstring to record the deliberate raw-surface. Also marked the original "AgentParseError line/col" deferred entry DONE. Tests: 9 agent-parser + 22 catalog/agent-source pass; core+cli typecheck, eslint, prettier green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 13, Batch C); deferred-tasks.md Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/chat/agent-source.ts | 3 ++- apps/cli/src/workflows/catalog.test.ts | 11 +++++++++++ docs/roadmap/deferred-tasks.md | 26 +++++++++++++++++++++++--- packages/core/src/agent-parser.test.ts | 17 +++++++++++++++++ 4 files changed, 53 insertions(+), 4 deletions(-) diff --git a/apps/cli/src/chat/agent-source.ts b/apps/cli/src/chat/agent-source.ts index 4863fcde..d608843a 100644 --- a/apps/cli/src/chat/agent-source.ts +++ b/apps/cli/src/chat/agent-source.ts @@ -20,7 +20,8 @@ export interface ResolveChatAgentOptions { * discovered under `/agents/`) parsed by the same strict core {@link parseAgent} a * workflow uses, or — when omitted — the {@link buildDefaultChatAgent built-in default agent} over * `[chat].default_model`. The host owns the file read ({@link resolveYamlSource}); the parser stays pure. - * A missing ref is a clean exit-2 invocation error; an invalid `.agent.yaml` is a field-named parse error. + * A missing ref is a clean exit-2 invocation error; an invalid `.agent.yaml` surfaces the raw, field-named + * {@link AgentParseError} (deliberately NOT re-tagged as a CliError — see agent-source.test.ts). */ export function resolveChatAgent( agentRef: string | undefined, diff --git a/apps/cli/src/workflows/catalog.test.ts b/apps/cli/src/workflows/catalog.test.ts index 08889207..42b07f57 100644 --- a/apps/cli/src/workflows/catalog.test.ts +++ b/apps/cli/src/workflows/catalog.test.ts @@ -116,6 +116,17 @@ describe('discoverCatalog', () => { expect(broken?.error).toBeTruthy(); }); + it('surfaces the YAML syntax line/column in a broken agent entry (Step 13, end-to-end via parseReason)', () => { + write('agents', 'syntax.agent.yaml', 'id: x\n : : :'); // a genuine YAML SYNTAX fault (not a schema miss) + + const entries = discoverCatalog({ projectConfigDir: configDir, cwd: proj, kind: 'agents' }); + const broken = entries.find((e) => e.slug === 'syntax'); + expect(broken?.valid).toBe(false); + // parseReason passes the AgentParseError.message through — so the position Step 13 attached is visible on + // the catalog surface (the discipline: field paths / integers only, never an authored value). + expect(broken?.error).toMatch(/line \d+, column \d+/); + }); + it('throws an exit-2 invocation error when the catalog path is not a directory (ENOTDIR)', () => { // A regular file where the `workflows/` directory is expected → readdirSync raises ENOTDIR (not ENOENT), // which is a real fault, not an empty catalog. Cross-platform (no chmod/perms needed). diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index cdf33071..a3e0ce5c 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -671,9 +671,29 @@ Severity is the review's verified rating. Check an item off in the PR that resol *commands* (`relavium list` / `loadLatestRunPerWorkflow` still return the full set) — genuinely unneeded at single-user CLI scale; add a cursor API before the desktop/cloud surfaces drive these reads at volume. *(low → scale · packages/db/src/run-history-store.ts; database-schema.md)* -- [ ] **`AgentParseError` carries no line/column on a YAML syntax fault** — parity gap with `parseWorkflow` - (which threads `LineCounter` positions into `WorkflowSyntaxError`). Add line/col before the 2.J authoring - commands surface agent-parse errors to a user. *(low · packages/core/src/agent-parser.ts; before 2.J)* +- [x] **`AgentParseError` line/column — DONE (2.5-close Step 13, `fix(core)`).** `agentSyntaxErrorFrom` (the + agent sibling of parser.ts's `syntaxErrorFrom`) now threads `LineCounter` positions into a positioned + `YAMLParseError`: optional 1-based `line`/`column` fields (parity with `WorkflowSyntaxError`) plus a folded + `(source — line L, column C)` locator in the message so the position reaches every `.message`-surfacing + consumer. The echoed YAML rule is secret-free (`prettyErrors: false`; verified across a 14-case + 12-case + adversarial sweep — no authored key/value ever rides the message). *(packages/core/src/agent-parser.ts)* +- [ ] **`AgentParseError` diagnostics are invisible on the `chat --agent` / `agent run` surfaces** (surfaced by + the 2.5-close Step 13 Sonnet review, 2026-07-08). `resolveChatAgent` (agent-source.ts) deliberately surfaces + the RAW `AgentParseError` — pinned by `agent-source.test.ts` "surfaces an invalid .agent.yaml as a field-named + AgentParseError (not a silent default or CliError)" — and neither `buildChatSession` nor `agent-run.ts` catches + it, so a malformed-but-existing `.agent.yaml` reaches the top-level `renderError`/`toUserFacing` and is reduced + to exit **1** + the generic "An unexpected internal error occurred" (human AND `--json`). So the field-named, + position-enriched diagnostic Step 13 built only actually reaches a user via `list --agents` / `create` / + `import` (`catalog.ts` / `authoring.ts`, which DO catch it). **Pre-existing** (predates Step 13; the diff only + changed message CONTENT, not this catch-layer gap) and it **conflicts with the deliberate `isCliError === false` + test above**, so re-tagging is a cross-surface DESIGN decision, not a mechanical fix — deferred out of the + `fix(core)` close-out. Fix options (need a maintainer call on which): (a) wrap the `AgentParseError` into a + `CliError('invalid_invocation', err.message, { cause })` at `resolveChatAgent` / the `agent-run`/`chat` callers + (revising the pinned test), or (b) teach the top-level `renderError`/`toUserFacing` to render a typed + `AgentParseError` as an exit-2 invocation fault. Also relativize the `source` label at the `chat`/`agent run` + call site (`agent-source.ts` passes the ABSOLUTE `source.path`, unlike the catalog's workspace-relative `rel`; + no secret leak — it is the user-typed path — but it contradicts the parser docstring's "workspace-relative"). + *(medium · apps/cli/src/chat/agent-source.ts + session-host.ts + commands/agent-run.ts + run.ts)* - [ ] **Residual read-command test pins.** A few low-risk coverage gaps remain after the PR's test additions: `pendingHumanGates` with `expiresAt` present and with multiple simultaneous gates; `list --json` no-project stderr + invalid-entry `error` machine-contract; the `status` "no node activity" fallback. The production code diff --git a/packages/core/src/agent-parser.test.ts b/packages/core/src/agent-parser.test.ts index 65a02e64..82a62126 100644 --- a/packages/core/src/agent-parser.test.ts +++ b/packages/core/src/agent-parser.test.ts @@ -1,3 +1,4 @@ +import { YAMLParseError } from 'yaml'; import { describe, expect, it } from 'vitest'; import { AgentParseError, parseAgent } from './agent-parser.js'; @@ -57,6 +58,21 @@ describe('parseAgent', () => { expect(typeof err.column).toBe('number'); expect(err.message).toContain(`line ${err.line}, column ${err.column}`); expect(err.message).toContain('agents/broken.agent.yaml'); + expect(err.cause).toBeInstanceOf(YAMLParseError); // the root cause is preserved (error-handling.md) + } + }); + + it('folds a bare (line L, column C) locator — no source label, no stray separator', () => { + // The pos-only branch of `locate()`: a syntax fault with NO source option ⇒ the message ends in a clean + // ` (line L, column C)` (the config-loader shape), with no leading `—` dash and no `(undefined` artifact. + try { + parseAgent('id: x\n : : :'); // no `source` + expect.unreachable('should have thrown'); + } catch (err) { + if (!(err instanceof AgentParseError)) throw err; + expect(err.message).toMatch(/\(line 2, column \d+\)$/); + expect(err.message).not.toContain('—'); // no source ⇒ no em-dash separator + expect(err.message).not.toContain('undefined'); } }); @@ -85,6 +101,7 @@ describe('parseAgent', () => { expect(err.message).toContain('anchors and aliases are not supported'); expect(err.line).toBeUndefined(); expect(err.column).toBeUndefined(); + expect(err.cause).toBeInstanceOf(ReferenceError); // the root cause is preserved, not swallowed } }); From 1dce56cebf62b3c030e2eda22bd13472b7fb88d1 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 05:26:34 +0300 Subject: [PATCH 39/50] =?UTF-8?q?feat(cli):=202.5-close=20Step=2014=20(Bat?= =?UTF-8?q?ch=20D-5)=20=E2=80=94=20bidi/format-control=20strip=20(Trojan-S?= =?UTF-8?q?ource=20floor)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extend the shared render sanitizer to strip Unicode bidirectional/directional format controls at every display boundary (CVE-2021-42574 floor): - `stripTerminalControls` now also strips the standard Trojan-Source family — the embeddings/overrides U+202A–202E, the isolates U+2066–2069, and the marks LRM (U+200E) / RLM (U+200F) / ALM (U+061C). These live above the C0/C1 range the existing BARE_CONTROLS strips, so they previously survived — yet they reorder how a terminal renders a line, letting streamed/pasted text visually differ from its logical bytes (a spoofed path/command in an approval prompt). Every surface (chat, Home, run TUI, commands) imports this one primitive, so the floor is uniform. The regex is written with `\u` escapes only — never literal bidi bytes — so the source itself carries no Trojan-Source hazard. - ZWJ/ZWNJ (U+200D/U+200C) are deliberately PRESERVED — they are legitimate in emoji sequences and Indic/Arabic/Persian shaping; an over-broad strip would mangle names/emoji. (Pinned by a test.) - Reconciled the pre-existing provider.ts `BIDI_ZERO_WIDTH`: it is now a stricter SUPERSET (it additionally strips the zero-width/BOM family the general floor keeps, which a structured URL must reject) rather than a workaround for a floor that "MISSES bidi" — corrected its now-stale comment (the earlier "bidi is legitimate for RTL text" rationale is superseded: legitimate RTL renders from inherent character directionality, not these override controls). Tests: +4 projection cases (strip the full family; flatten an RLO spoof; preserve ZWJ/ZWNJ; strip through sanitizeInline). 62 projection + 484 render tests pass; 36 provider tests pass; typecheck/eslint/prettier green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 14, Batch D); ADR-0057 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/commands/provider.ts | 16 ++++--- .../src/render/tui/chat-projection.test.ts | 44 +++++++++++++++++++ apps/cli/src/render/tui/chat-projection.ts | 20 +++++++-- 3 files changed, 71 insertions(+), 9 deletions(-) diff --git a/apps/cli/src/commands/provider.ts b/apps/cli/src/commands/provider.ts index 273624e9..509a7e86 100644 --- a/apps/cli/src/commands/provider.ts +++ b/apps/cli/src/commands/provider.ts @@ -122,12 +122,16 @@ async function verifyProvider( return { verified: result.ok, detail: result.ok ? null : cleanDetail(result.detail) }; } -/** The Unicode bidi-override + zero-width "Trojan Source" family the ASCII-only {@link stripTerminalControls} - * MISSES: LRE/RLE/PDF/LRO/RLO (U+202A–E), the isolates LRI/RLI/FSI/PDI (U+2066–9), the zero-width + directional - * marks (U+200B–F), the word-joiner (U+2060), and the BOM (U+FEFF). None has a legitimate use in a URL or a probe - * detail; left in, they visually reorder/hide text (a spoof), so they are stripped from any provider-supplied - * value echoed inline here. NOT added to the shared `stripTerminalControls` — that also renders chat bodies, where - * bidi controls ARE legitimate for RTL text. */ +/** The Unicode bidi-override + zero-width "Trojan Source" family stripped from any provider-supplied value + * echoed inline here - none has a legitimate use in a URL or a probe detail, and left in they visually + * reorder/hide text (a spoof). This is a stricter SUPERSET of the shared render floor: the reordering bidi + * controls (the isolates + embeddings/overrides + LRM/RLM/ALM) are now stripped by {@link stripTerminalControls} + * itself (the Trojan-Source floor added in 2.5-close Step 14, which `stripInline` runs first), so the additive + * coverage this constant contributes is the ZERO-WIDTH family that floor deliberately KEEPS for chat bodies - + * ZWSP, ZWNJ/ZWJ (legitimate in emoji + Indic/Arabic shaping), the word-joiner, and the BOM - which a + * structured URL/probe field must nonetheless reject. The overlap with the shared floor is harmless + * defense-in-depth. (Legitimate RTL text needs no explicit control chars - it renders from the letters own + * directionality - so stripping these controls never harms Arabic/Hebrew display.) */ const BIDI_ZERO_WIDTH = /[\u200b-\u200f\u2060\u2066-\u2069\u202a-\u202e\ufeff]/gu; /** C0/C1 control bytes + the {@link BIDI_ZERO_WIDTH} spoof family, as a NON-global tester (safe for `.test`) \u2014 none diff --git a/apps/cli/src/render/tui/chat-projection.test.ts b/apps/cli/src/render/tui/chat-projection.test.ts index fa28cdd1..089b59da 100644 --- a/apps/cli/src/render/tui/chat-projection.test.ts +++ b/apps/cli/src/render/tui/chat-projection.test.ts @@ -14,6 +14,7 @@ import { formatTurnSummary, MAX_REASONING_PANEL_LINES, reasoningLabelActive, + sanitizeInline, streamingAbortHint, stripTerminalControls, } from './chat-projection.js'; @@ -662,5 +663,48 @@ describe('chat-projection', () => { expect(stripTerminalControls('a\x1bPpayload\x1b\\b')).toBe('ab'); expect(stripTerminalControls('a\x1b_apc\x07b')).toBe('ab'); }); + + it('strips Unicode bidi/format controls - the Trojan-Source floor (Step 14)', () => { + // The standard Trojan-Source family: overrides/embeddings U+202A-202E, isolates U+2066-2069, and the + // marks LRM/RLM/ALM. None is in the C0/C1 range BARE_CONTROLS strips, so each would otherwise survive. + const bidi = [ + '\u202A', + '\u202B', + '\u202C', + '\u202D', + '\u202E', + '\u2066', + '\u2067', + '\u2068', + '\u2069', + '\u200E', + '\u200F', + '\u061C', + ]; + for (const c of bidi) { + expect(stripTerminalControls(`a${c}b`)).toBe('ab'); + } + }); + + it('flattens a Trojan-Source-style RLO spoof to its logical byte order', () => { + // RLO then PDF would render the middle reversed as a visual spoof; stripped, only the logical text + // survives, so what the user SEES equals the bytes (an approval-prompt path cannot lie). + const spoof = `rm -rf \u202E/nimda\u202C safe`; + const clean = stripTerminalControls(spoof); + expect(clean).toBe('rm -rf /nimda safe'); // both controls gone; logical order preserved + expect(clean).not.toMatch(/[\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]/); + }); + + it('preserves ZWJ / ZWNJ (legitimate in emoji sequences + Indic/Arabic shaping)', () => { + // ZWJ (U+200D) joins emoji; ZWNJ (U+200C) is a real letter-shaping control - neither reorders text, so + // both must survive (an over-broad strip would mangle names / emoji). + expect(stripTerminalControls('a\u200Db')).toBe('a\u200Db'); + expect(stripTerminalControls('a\u200Cb')).toBe('a\u200Cb'); + }); + + it('strips bidi controls through sanitizeInline too (shared primitive, all surfaces)', () => { + // sanitizeInline builds on stripTerminalControls, so an id/model/path field is bidi-safe everywhere. + expect(sanitizeInline('git\u202Estatus')).toBe('gitstatus'); + }); }); }); diff --git a/apps/cli/src/render/tui/chat-projection.ts b/apps/cli/src/render/tui/chat-projection.ts index c5ef06c8..985ad634 100644 --- a/apps/cli/src/render/tui/chat-projection.ts +++ b/apps/cli/src/render/tui/chat-projection.ts @@ -31,14 +31,28 @@ const ESC_SEQUENCES = new RegExp(`${ESC_STRING_SEQ}|${ESC_CSI}|${ESC_2BYTE}`, 'g const BARE_CONTROLS = /[\x00-\x08\x0b-\x1f\x7f-\x9f]/g; /* eslint-enable no-control-regex */ +/** + * Unicode bidirectional / directional FORMAT controls — the Trojan-Source floor (CVE-2021-42574; 2.5-close + * Step 14). These code points live ABOVE the C0/C1 range {@link BARE_CONTROLS} strips, so they survive it — yet + * they REORDER how a terminal visually renders a line, letting streamed model output or pasted input display in + * an order that differs from its logical bytes (a spoofed path/command in an approval prompt, a hidden argument, + * a reversed URL). The set is the standard Trojan-Source family: the embeddings/overrides U+202A–202E + * (LRE/RLE/PDF/LRO/RLO), the isolates U+2066–2069 (LRI/RLI/FSI/PDI), and the marks LRM (U+200E) / RLM (U+200F) / + * ALM (U+061C). ZWJ/ZWNJ (U+200D/U+200C) are deliberately NOT stripped — they are legitimate in emoji sequences + * and in Indic/Arabic/Persian shaping. Not in `no-control-regex`'s C0/C1 range, so no eslint-disable is needed. + * Written with `\u` escapes (never literal bidi bytes) so the source itself carries no Trojan-Source hazard. + */ +const BIDI_CONTROLS = /[\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]/g; + /** * Strip terminal control sequences from text that will be written to a terminal — so model output (or pasted * input) cannot inject ANSI/OSC escapes (colors, a cursor jump, a window-title/clipboard/hyperlink write, a - * `\r` line-overwrite). Applied at the **display** boundary only; the PERSISTED transcript keeps the raw text - * (it is user/model data, not displayed back through a shell). Keeps printable text plus tabs and newlines. + * `\r` line-overwrite) NOR spoof the visual line order with Unicode bidi controls (the Trojan-Source floor). + * Applied at the **display** boundary only; the PERSISTED transcript keeps the raw text (it is user/model data, + * not displayed back through a shell). Keeps printable text plus tabs and newlines. */ export function stripTerminalControls(text: string): string { - return text.replace(ESC_SEQUENCES, '').replace(BARE_CONTROLS, ''); + return text.replace(ESC_SEQUENCES, '').replace(BARE_CONTROLS, '').replace(BIDI_CONTROLS, ''); } /** From abfbdbe105902b1e014e9ee64dafc88b8c9788d3 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 05:29:37 +0300 Subject: [PATCH 40/50] =?UTF-8?q?feat(cli):=202.5-close=20Step=2014=20(Bat?= =?UTF-8?q?ch=20D-4)=20=E2=80=94=20whitelist=20the=20view-only=20reasoning?= =?UTF-8?q?=20toggle=20in=20the=20approval=20swallow?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit During a pending per-tool approval the fail-closed key-swallow (ADR-0057) owns the keyboard and drops every key except [y]/[a]/[n]/[esc], so a user could not expand the model's reasoning to inform the decision. Whitelist the ONE view-only chord — Ctrl+T (toggle-reasoning) — through `reduceApprovalKey`: - Security-verified side-effect-free before bypassing the swallow: `toggleReasoning` (chat-store.ts) only flips a local `reasoningVisible` boolean and repaints — zero effect on `approval`, session state, or the pending decision. It cannot answer, cancel, or mutate anything a governed dispatch gates on. - The whitelist is TIGHT: matched only for the exact Ctrl-without-meta 't' chord, placed AFTER the `escape` (abort wins) and BEFORE the `bare` gate. A bare 't', Ctrl+Meta+T, or Alt+T (ESC-prefixed) all stay swallowed — the approval input set is exactly {y/1, a/2, n/r/3, esc} plus this view-only toggle. The most-permissive approve/reject can still never be picked by a modified chord. - Both surfaces (chat ChatApp + the in-Home chat) route approval keys through the shared `reduceChatKey`/`toggle-reasoning` case, so one change fixes both. Tests: +1 chat-input case (Ctrl+T → toggle-reasoning during approval; bare/meta variants stay swallowed). 132 chat-input + home-controller tests pass; typecheck/eslint/prettier green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 14, Batch D); ADR-0057 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/render/tui/chat-input.test.ts | 18 +++++++++++++++++- apps/cli/src/render/tui/chat-input.ts | 12 ++++++++++-- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/apps/cli/src/render/tui/chat-input.test.ts b/apps/cli/src/render/tui/chat-input.test.ts index f2179abf..1e98438d 100644 --- a/apps/cli/src/render/tui/chat-input.test.ts +++ b/apps/cli/src/render/tui/chat-input.test.ts @@ -211,8 +211,24 @@ describe('reduceChatKey — approval-prompt intercept (in-flight key-swallow byp }); }); + it('whitelists ONLY the view-only reasoning toggle (Ctrl+T) through the swallow (Step 14)', () => { + // Ctrl+T is a pure store-view flip (toggleReasoning: no session/approval/decision effect), so a user may + // expand the model's reasoning to INFORM the pending decision WITHOUT it counting as an answer. + expect(reduceChatKey('t', { ctrl: true }, '', true, PENDING)).toEqual({ + kind: 'toggle-reasoning', + }); + // The whitelist is TIGHT — only the exact Ctrl-without-meta 't' chord: a bare 't' is a non-answer key + // (swallowed), and Ctrl+Meta+T / Alt+T do NOT toggle (so an ESC-prefixed Alt-chord can't sneak through). + expect(reduceChatKey('t', KEY, '', true, PENDING)).toEqual({ kind: 'none' }); // bare t: swallowed + expect(reduceChatKey('t', { ctrl: true, meta: true }, '', true, PENDING)).toEqual({ + kind: 'none', + }); + expect(reduceChatKey('t', { meta: true }, '', true, PENDING)).toEqual({ kind: 'none' }); // Alt+T: swallowed + }); + it('SWALLOWS every other key while an approval is pending (no deadlock, no stray edit)', () => { - // Even Ctrl-C / Return / a printable are ignored during the approval — only y/a/n/1/2/3/Esc act. + // Even Ctrl-C / Return / a printable are ignored during the approval — only y/a/n/1/2/3/Esc answer, plus the + // view-only Ctrl+T toggle (tested above). Nothing else acts. expect(reduceChatKey('c', { ctrl: true }, '', true, PENDING)).toEqual({ kind: 'none' }); expect(reduceChatKey('', { return: true }, '', true, PENDING)).toEqual({ kind: 'none' }); expect(reduceChatKey('z', KEY, '', true, PENDING)).toEqual({ kind: 'none' }); diff --git a/apps/cli/src/render/tui/chat-input.ts b/apps/cli/src/render/tui/chat-input.ts index 7fc6692a..fa311085 100644 --- a/apps/cli/src/render/tui/chat-input.ts +++ b/apps/cli/src/render/tui/chat-input.ts @@ -81,11 +81,19 @@ export type ChatKeyAction = /** * The approval-prompt keystroke intercept (accept-edits / auto's protected-path fallback), extracted so * {@link reduceChatKey} stays flat: `[y]`/`1` approve once, `[a]`/`2` approve always, `[n]`/`r`/`3` reject, - * `Esc` aborts the whole turn (and this pending approval); every other key is ignored. It bypasses the - * running-swallow so a pending approval can never deadlock. + * `Esc` aborts the whole turn (and this pending approval); the VIEW-ONLY reasoning toggle (Ctrl+T) is the one + * non-answer chord let through (Step 14); every other key is ignored. It bypasses the running-swallow so a + * pending approval can never deadlock. */ function reduceApprovalKey(char: string, key: ChatKey): ChatKeyAction { if (key.escape === true) return { kind: 'abort' }; // Esc aborts regardless of modifiers + // The collapsible "thinking" panel toggle (Ctrl+T) is whitelisted through the fail-closed swallow (Step 14): + // it is a PURE store-view flip (`toggleReasoning` only flips `reasoningVisible` + repaints — zero session / + // approval / decision effect), so a user can expand the model's reasoning to INFORM the pending decision + // without it counting as an answer. It is the ONLY non-answer chord allowed — a mode cycle / edit / submit + // stays swallowed, so the approval input set is exactly {y/1, a/2, n/r/3, esc} plus this view-only toggle. + // Matched BEFORE the `bare` gate (Ctrl+T is a modified chord) but AFTER `escape` (abort wins). + if (key.ctrl === true && key.meta !== true && char === 't') return { kind: 'toggle-reasoning' }; // Only an UNMODIFIED key answers the prompt. Otherwise a now-bound editor chord (Ctrl+A line-start, Ctrl+W kill, // Alt+B word-left) OR a meta+digit (Alt+1 — reachable via ink's ESC-prefixed parsing, `\x1b1` → char '1', // meta:true) would, during a pending approval, silently pick the most-permissive, session-persistent approve / From 3e0af067084061311e8529d8762d70be0671c169 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 05:47:40 +0300 Subject: [PATCH 41/50] =?UTF-8?q?feat(cli):=202.5-close=20Step=2014=20(Bat?= =?UTF-8?q?ch=20D-1)=20=E2=80=94=20[c]=20reject-with-typed-reason=20at=20t?= =?UTF-8?q?he=20approval=20prompt?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a `[c]` "reject with reason" branch to the per-tool approval prompt (ADR-0057 deferred follow-up): instead of a bare "denied by the user", the user types WHY, and the reason rides the EXISTING `ToolApprovalDecision.reject.reason` seam into the denial the turn records/shows. The engine + type plumbing already existed (`ApprovalAnswer.reject.reason` → `toDecision` → `ToolApprovalDecision.reject.reason` → registry → `ToolDeniedByUserError`); this is the missing UI capture. - `[c]` at the prompt returns a new `reject-with-reason` action (chat-input.ts) — a pure UI transition that OPENS a small keyboard-owning reason-input sub-mode; it does NOT itself answer. The bare `[n]`/`[r]`/`3` reject (no reason) is unchanged, and a modified `c` (Ctrl+C / Alt+C) stays swallowed (fail-closed). - The capture sub-mode is wired into BOTH surfaces (chat `ChatApp` + the in-Home chat) — checked FIRST in the key handler: Esc CANCELS back to the [y]/[a]/[n] prompt (the approval stays pending — NOT an abort), plain Enter submits the reject with the reason, every other key edits the buffer (reusing the tested editor primitives). A stale capture (the approval settled out-of-band) is dropped. The reasonDraft is chat-scoped and reset on every session swap. - The reason is sanitized + bounded before it leaves the UI (`sanitizeApprovalReason`: strip terminal/bidi controls, collapse to one line, trim, cap at 300 chars) so a pasted wall of text or a control/bidi spoof can't blow up or spoof the error line, logs, or `--json`. An empty reason degrades to a plain reject. - SECURITY: the fail-closed floor is UNCHANGED — `[c]`+Enter is a reject with extra data; a governed dispatch still cannot proceed without an explicit decision, and ADR-0057's "a user deny is final" (`tool_denied`, non-retryable) is untouched. Tests: +1 chat-input reducer case, +4 sanitizeApprovalReason cases, +2 home-controller INTEGRATION cases (open on a real pending approval → type → reject-with-reason; and Esc-cancels-keeps-pending → later [n] plain-rejects). 672 render+chat tests pass; typecheck/eslint/prettier/build/fence/engine-deps green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 14, Batch D); ADR-0057 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/render/tui/chat-ink.tsx | 72 +++++++++++++++++-- apps/cli/src/render/tui/chat-input.test.ts | 11 +++ apps/cli/src/render/tui/chat-input.ts | 10 ++- .../src/render/tui/chat-projection.test.ts | 28 ++++++++ apps/cli/src/render/tui/chat-projection.ts | 19 +++++ apps/cli/src/render/tui/home-app.tsx | 4 ++ .../src/render/tui/home-controller.test.ts | 68 ++++++++++++++++++ apps/cli/src/render/tui/home-controller.ts | 42 +++++++++++ 8 files changed, 248 insertions(+), 6 deletions(-) diff --git a/apps/cli/src/render/tui/chat-ink.tsx b/apps/cli/src/render/tui/chat-ink.tsx index 6de2a1bc..f681fa08 100644 --- a/apps/cli/src/render/tui/chat-ink.tsx +++ b/apps/cli/src/render/tui/chat-ink.tsx @@ -36,6 +36,7 @@ import { emptyEditor, insertAtCursor, reduceChatKey, + reduceEditorMotion, type EditorState, } from './chat-input.js'; import { @@ -93,6 +94,7 @@ import { formatToolCall, formatTurnSummary, reasoningLabelActive, + sanitizeApprovalReason, sanitizeInline, streamingAbortHint, stripTerminalControls, @@ -222,6 +224,9 @@ interface ChatViewProps { * cosmetic (parity with `nowMs`); the owner passes `process.stdout.columns` (ChatApp) or its resize-tracked * width (the Home). Absent ⇒ the formatter's 80-col fallback. `| undefined` for the createElement passthrough. */ readonly columns?: number | undefined; + /** The in-flight `[c]` typed-reason capture buffer (Step 14) — when set (only while `approval` is pending), the + * approval prompt shows the reason input instead of the `[y]/[a]/[n]` hint. `| undefined` ⇒ the normal prompt. */ + readonly reasonDraft?: EditorState | undefined; } /** @@ -379,11 +384,24 @@ export function ChatView(props: Readonly): ReactElement { : '' }?`} - - {approval.cacheable - ? '[y] yes [a] always [n] no [esc] abort' - : '[y] yes [n] no [esc] abort'} - + {props.reasonDraft !== undefined ? ( + // The `[c]` typed-reason capture (Step 14): a hint + the buffer echoed SANITIZED (bidi/control-stripped, + // one line) with a block cursor. Enter denies WITH this reason; Esc cancels back to the choices below. + + + reason to deny · Enter to send · Esc to cancel + + + {`> ${sanitizeInline(props.reasonDraft.text)}█`} + + + ) : ( + + {approval.cacheable + ? '[y] yes [a] always [n] no [c] reason [esc] abort' + : '[y] yes [n] no [c] reason [esc] abort'} + + )} )} @@ -463,6 +481,15 @@ export function ChatApp(props: Readonly): ReactElement { effortPickerRef.current = next; setEffortPicker(next); }; + // The `[c]` typed-reason capture (Step 14) — a small keyboard-owning buffer opened FROM a pending approval to + // record WHY the user denies. `undefined` ⇒ closed (the normal [y]/[a]/[n] prompt). React-local + ref-shadowed + // (survives a coalesced stdin chunk, like the other submodes); only ever open while an approval is pending. + const [reasonDraft, setReasonDraft] = useState(undefined); + const reasonDraftRef = useRef(undefined); + const applyReasonDraft = (next: EditorState | undefined): void => { + reasonDraftRef.current = next; + setReasonDraft(next); + }; // A monotonic submit generation: bumped every time the compose buffer is submitted (cleared). An async mention // read captures it at accept time and DROPS its inject if a submit has since happened — so a slow read that // resolves after Enter can never splice the file into the (now-empty) buffer meant for the NEXT message. @@ -842,6 +869,35 @@ export function ChatApp(props: Readonly): ReactElement { props.store.getSnapshot().state.status === 'running' || shellBusyRef.current || submitBusyRef.current; + // The `[c]` typed-reason capture (Step 14) owns the keyboard while open — checked FIRST. It ONLY opens from a + // pending approval, so if that approval settled out-of-band (an external abort while typing) the capture is + // stale: drop it. Else `Esc` CANCELS back to the [y]/[a]/[n] prompt (the approval is STILL pending — not an + // abort); plain `Enter` submits the reject with the sanitized+bounded reason; every other key edits the buffer + // (Ctrl-C / a non-edit chord is a harmless no-op, matching the approval floor's keyboard ownership). Read the + // REF so a coalesced same-chunk key sees a just-applied open/edit. The floor is unchanged — this only enriches + // a reject; a governed dispatch still cannot proceed without an explicit decision. + const openReason = reasonDraftRef.current; + if (openReason !== undefined) { + if (props.store.getSnapshot().approval === undefined) { + applyReasonDraft(undefined); // the approval vanished — discard the orphaned capture + return; + } + if (key.escape === true) { + applyReasonDraft(undefined); // cancel the reason; the approval stays pending ([y]/[a]/[n] again) + return; + } + if (key.return === true && key.shift !== true) { + applyReasonDraft(undefined); + const reason = sanitizeApprovalReason(openReason.text); + props.store.answerApproval( + reason === undefined ? { outcome: 'reject' } : { outcome: 'reject', reason }, + ); + return; + } + const edit = reduceEditorMotion(char, key); + if (edit !== undefined) applyReasonDraft(applyEditorAction(openReason, edit)); + return; + } // The open `/models` reseat picker owns every key (ADR-0059) — checked FIRST (mutually exclusive with the other // submodes; it only opens at an idle prompt). Read the REF so a coalesced same-chunk key sees a just-applied // open/close/accept; on accept it triggers the reseat + ends the loop (see routeModelPickerKey). @@ -1067,6 +1123,11 @@ export function ChatApp(props: Readonly): ReactElement { case 'reject': props.store.answerApproval({ outcome: 'reject' }); return; + case 'reject-with-reason': + // `[c]` — open the typed-reason capture; the next keys fill it, then submit rejects WITH the reason + // (Step 14). The approval stays pending until then; the capture handler above owns the keyboard while open. + applyReasonDraft(emptyEditor()); + return; case 'none': return; } @@ -1092,6 +1153,7 @@ export function ChatApp(props: Readonly): ReactElement { // `stdout.columns` is typed `number` but is `undefined` at runtime off a TTY — the formatter's 80-col // fallback covers that (moot here anyway: ChatApp only mounts on a TTY via the driveInk gate). columns={process.stdout.columns} + reasonDraft={reasonDraft} paletteOpen={ palette !== undefined || search !== undefined || diff --git a/apps/cli/src/render/tui/chat-input.test.ts b/apps/cli/src/render/tui/chat-input.test.ts index 1e98438d..cec61d48 100644 --- a/apps/cli/src/render/tui/chat-input.test.ts +++ b/apps/cli/src/render/tui/chat-input.test.ts @@ -193,6 +193,17 @@ describe('reduceChatKey — approval-prompt intercept (in-flight key-swallow byp expect(reduceChatKey('', { escape: true }, '', true, PENDING)).toEqual({ kind: 'abort' }); }); + it('maps [c] to reject-with-reason — OPENS the typed-reason capture (Step 14), a distinct action from reject', () => { + // `[c]` does NOT itself answer the prompt — it opens the reason-input sub-mode; the surface then captures the + // text and submits a `{ outcome: 'reject', reason }`. The bare `[n]`/`[r]`/`3` reject (no reason) is unchanged. + expect(reduceChatKey('c', KEY, '', true, PENDING)).toEqual({ kind: 'reject-with-reason' }); + expect(reduceChatKey('n', KEY, '', true, PENDING)).toEqual({ kind: 'reject' }); // still a bare reject + // A modified `c` (Ctrl+C / Alt+C) is NOT the reason chord — it stays swallowed (fail-closed; Ctrl+C never + // silently rejects, and an ESC-prefixed Alt+C cannot open the capture either). + expect(reduceChatKey('c', { ctrl: true }, '', true, PENDING)).toEqual({ kind: 'none' }); + expect(reduceChatKey('c', { meta: true }, '', true, PENDING)).toEqual({ kind: 'none' }); + }); + it('a modified CHORD (Ctrl+A/Y/N, Alt+A, Alt+1/2/3) does NOT answer a pending approval (fail-closed floor)', () => { // Ctrl+A is now a line-start motion; during a pending approval it must be SWALLOWED, never silently taken as // the persistent approve-always — that would subvert the ADR-0057 fail-closed confirmAction floor. The DIGIT diff --git a/apps/cli/src/render/tui/chat-input.ts b/apps/cli/src/render/tui/chat-input.ts index fa311085..4370562c 100644 --- a/apps/cli/src/render/tui/chat-input.ts +++ b/apps/cli/src/render/tui/chat-input.ts @@ -76,7 +76,11 @@ export type ChatKeyAction = /** An approval-prompt decision (accept-edits / auto's protected-path fallback): `[y]` once / `[a]` always. */ | { readonly kind: 'approve'; readonly scope: 'once' | 'always' } /** An approval-prompt rejection (`[n]`). */ - | { readonly kind: 'reject' }; + | { readonly kind: 'reject' } + /** `[c]` at the approval prompt — START capturing a typed denial reason (Step 14). The surface opens a small + * reason-input buffer; on submit it rejects with `{ outcome: 'reject', reason }`, so the model/turn outcome + * carries WHY (not a bare "denied by the user"). A pure UI transition — it does not itself answer the prompt. */ + | { readonly kind: 'reject-with-reason' }; /** * The approval-prompt keystroke intercept (accept-edits / auto's protected-path fallback), extracted so @@ -103,6 +107,10 @@ function reduceApprovalKey(char: string, key: ChatKey): ChatKeyAction { if (char === 'y' || char === '1') return { kind: 'approve', scope: 'once' }; if (char === 'a' || char === '2') return { kind: 'approve', scope: 'always' }; if (char === 'n' || char === 'r' || char === '3') return { kind: 'reject' }; + // `[c]` opens the typed-reason capture (Step 14) — a reject that records WHY the user denied, instead of a bare + // "denied by the user". A separate keyboard-owning sub-mode captures the text (the surface owns its buffer), so + // it does not collide with the `[n]`/`[r]`/`3` bare reject here. + if (char === 'c') return { kind: 'reject-with-reason' }; return { kind: 'none' }; } diff --git a/apps/cli/src/render/tui/chat-projection.test.ts b/apps/cli/src/render/tui/chat-projection.test.ts index 089b59da..29e82517 100644 --- a/apps/cli/src/render/tui/chat-projection.test.ts +++ b/apps/cli/src/render/tui/chat-projection.test.ts @@ -12,8 +12,10 @@ import { formatSessionFooterWithMode, formatToolCall, formatTurnSummary, + MAX_APPROVAL_REASON_CHARS, MAX_REASONING_PANEL_LINES, reasoningLabelActive, + sanitizeApprovalReason, sanitizeInline, streamingAbortHint, stripTerminalControls, @@ -707,4 +709,30 @@ describe('chat-projection', () => { expect(sanitizeInline('git\u202Estatus')).toBe('gitstatus'); }); }); + + describe('sanitizeApprovalReason ([c] typed denial reason, Step 14)', () => { + it('returns undefined for an empty / whitespace-only reason (degrades to a plain reject)', () => { + expect(sanitizeApprovalReason('')).toBeUndefined(); + expect(sanitizeApprovalReason(' \t ')).toBeUndefined(); + }); + + it('trims + collapses to one line and keeps a normal reason', () => { + expect(sanitizeApprovalReason(' use the project config instead ')).toBe( + 'use the project config instead', + ); + expect(sanitizeApprovalReason('line1\nline2')).toBe('line1 line2'); // newline collapsed to a space + }); + + it('strips terminal + bidi controls (secret-free-shaped, spoof-free) from the reason', () => { + const RLO = String.fromCharCode(0x202e); + const ESC = String.fromCharCode(0x1b); + expect(sanitizeApprovalReason(`no ${RLO}spoof`)).toBe('no spoof'); // bidi gone + expect(sanitizeApprovalReason(`no ${ESC}[31mansi`)).toBe('no ansi'); // ANSI gone + }); + + it('caps the reason at MAX_APPROVAL_REASON_CHARS (a pasted wall of text cannot blow up an error line)', () => { + const long = 'x'.repeat(MAX_APPROVAL_REASON_CHARS + 50); + expect(sanitizeApprovalReason(long)).toHaveLength(MAX_APPROVAL_REASON_CHARS); + }); + }); }); diff --git a/apps/cli/src/render/tui/chat-projection.ts b/apps/cli/src/render/tui/chat-projection.ts index 985ad634..cefc7955 100644 --- a/apps/cli/src/render/tui/chat-projection.ts +++ b/apps/cli/src/render/tui/chat-projection.ts @@ -65,6 +65,25 @@ export function sanitizeInline(text: string): string { return stripTerminalControls(text).replace(/[\t\n]+/g, ' '); } +/** The max length of a typed denial reason (`[c]`, Step 14) — bounds what rides the `ToolDeniedByUserError` into + * the turn outcome / `--json` / logs, so a pasted wall of text can't blow up one error line. */ +export const MAX_APPROVAL_REASON_CHARS = 300; + +/** + * Sanitize + bound a USER-typed denial reason for the approval seam (`[c]`, Step 14): strip terminal/bidi controls + * and collapse to a single line ({@link sanitizeInline}), trim, and cap the length. Returns `undefined` for an + * empty/whitespace reason so the answer degrades to a plain reject (no dangling `": "` in the denial). The value + * flows into `ToolApprovalDecision.reject.reason`; it is the user's own text (not a secret they'd hide from + * themselves), but bounding + control-stripping keep it safe for the error line, logs, and `--json`. + */ +export function sanitizeApprovalReason(text: string): string | undefined { + const clean = sanitizeInline(text).trim(); + if (clean.length === 0) return undefined; + return clean.length > MAX_APPROVAL_REASON_CHARS + ? clean.slice(0, MAX_APPROVAL_REASON_CHARS) + : clean; +} + /** * Error codes whose `errorMessage` is safe to render in-chat. This leans on a load-bearing project-wide * contract, not a two-string allowlist: EVERY `tool_denied` / `tool_unavailable` message across the engine is, diff --git a/apps/cli/src/render/tui/home-app.tsx b/apps/cli/src/render/tui/home-app.tsx index 3f0b50f4..5b4bfb8e 100644 --- a/apps/cli/src/render/tui/home-app.tsx +++ b/apps/cli/src/render/tui/home-app.tsx @@ -64,6 +64,8 @@ function ChatRegion( shellCommand: string | undefined; historyEntries: readonly string[]; attachments: readonly PendingAttachment[]; + /** The in-flight `[c]` typed-reason capture buffer (Step 14) — shows the reason input in the approval prompt. */ + reasonDraft: EditorState | undefined; }>, ): ReactElement { const { state, tick, color, mode, reasoningEffort, reasoningVisible, approval } = @@ -86,6 +88,7 @@ function ChatRegion( attachments={props.attachments} busyCommand={props.shellCommand} columns={props.cols} + reasonDraft={props.reasonDraft} paletteOpen={ props.palette !== undefined || props.search !== undefined || @@ -141,6 +144,7 @@ export function RootApp(props: Readonly): ReactElement { effortPicker={state.effortPicker} now={props.nowMs} cols={size.cols} + reasonDraft={state.reasonDraft} shellBusy={state.shellBusy} submitBusy={state.submitBusy} shellCommand={state.shellCommand} diff --git a/apps/cli/src/render/tui/home-controller.test.ts b/apps/cli/src/render/tui/home-controller.test.ts index 7ed4d157..bcc88d45 100644 --- a/apps/cli/src/render/tui/home-controller.test.ts +++ b/apps/cli/src/render/tui/home-controller.test.ts @@ -281,6 +281,74 @@ describe('createHomeController (2.5.B lifecycle / ADR-0054)', () => { expect(c.getSnapshot().input.text).toBe(''); }); + it('[c] typed-reason capture: opens on a pending approval, records the reason, rejects WITH it (Step 14)', async () => { + const made = makeSession(); + const c = createHomeController({ + doctorProbes: STUB_DOCTOR_PROBES, + startChat: () => Promise.resolve(made.session), + homeStore, + onExit: vi.fn(), + onError: vi.fn(), + }); + type(c, 'hi'); + c.handleKey('', ENTER); // start the chat + await flush(); + expect(c.getSnapshot().mode).toBe('chat'); + + // Inject a pending per-tool approval (a governed write) into the live session store. + const request = { + toolId: 'write_file', + action: 'fs_write', + preview: { path: 'notes.md' }, + } as const; + const pending = made.store.requestApproval(request, true); + + // `[c]` opens the typed-reason capture (an empty buffer) — the [y]/[a]/[n] prompt is replaced. + c.handleKey('c', {}); + expect(c.getSnapshot().reasonDraft).toEqual({ text: '', cursor: 0 }); + + // Type a reason: a char that would OTHERWISE be an approval answer ('n'/'y') is now ordinary buffer text. + type(c, 'use the project config instead'); + expect(c.getSnapshot().reasonDraft?.text).toBe('use the project config instead'); + + // Enter submits the reject WITH the sanitized reason; the capture closes. + c.handleKey('', ENTER); + expect(c.getSnapshot().reasonDraft).toBeUndefined(); + await expect(pending).resolves.toEqual({ + outcome: 'reject', + reason: 'use the project config instead', + }); + }); + + it('[c] then Esc cancels the reason capture — the approval stays pending, a later [n] plain-rejects (Step 14)', async () => { + const made = makeSession(); + const c = createHomeController({ + doctorProbes: STUB_DOCTOR_PROBES, + startChat: () => Promise.resolve(made.session), + homeStore, + onExit: vi.fn(), + onError: vi.fn(), + }); + type(c, 'hi'); + c.handleKey('', ENTER); + await flush(); + const request = { + toolId: 'write_file', + action: 'fs_write', + preview: { path: 'notes.md' }, + } as const; + const pending = made.store.requestApproval(request, true); + + c.handleKey('c', {}); // open the capture + expect(c.getSnapshot().reasonDraft).toBeDefined(); + c.handleKey('', { escape: true }); // Esc CANCELS the reason (not an abort) — back to the choices + expect(c.getSnapshot().reasonDraft).toBeUndefined(); + expect(made.store.getSnapshot().approval).toBeDefined(); // the approval is STILL pending + + c.handleKey('n', {}); // a plain reject (no reason) now settles it + await expect(pending).resolves.toEqual({ outcome: 'reject' }); + }); + it('`@` completion: Esc restores the typed keystrokes; a mid-word `@` stays literal (2.5.D step 4)', async () => { const mentionReader: MentionReader = { list: () => Promise.resolve([{ name: 'app.ts', type: 'file' as const, path: 'app.ts' }]), diff --git a/apps/cli/src/render/tui/home-controller.ts b/apps/cli/src/render/tui/home-controller.ts index cf232304..888bec27 100644 --- a/apps/cli/src/render/tui/home-controller.ts +++ b/apps/cli/src/render/tui/home-controller.ts @@ -26,6 +26,7 @@ import { emptyEditor, insertAtCursor, reduceChatKey, + reduceEditorMotion, type ChatKey, type ChatKeyAction, type EditorState, @@ -42,6 +43,7 @@ import { type ReverseSearchState, } from './input-history.js'; import type { ChatStoreController } from './chat-store.js'; +import { sanitizeApprovalReason } from './chat-projection.js'; import { foldMentionKey, mentionOpensAt, @@ -175,6 +177,9 @@ export interface HomeControllerState { * with the effort setter wired + a reasoning-capable model); a keyboard-owning overlay like the mention/search * submodes. On accept it pushes the per-turn session override via `active.onSetEffort` (no reseat). */ readonly effortPicker: EffortPickerState | undefined; + /** The in-flight `[c]` typed-reason capture (Step 14) — `undefined` ⇒ closed. CHAT-scoped; opened FROM a pending + * approval to record WHY the user denies. A keyboard-owning submode; on submit it rejects with the reason. */ + readonly reasonDraft: EditorState | undefined; } /** @@ -260,6 +265,7 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { notice: undefined, modelPicker: undefined, effortPicker: undefined, + reasonDraft: undefined, }; // Per-session command history for the in-Home chat (2.5.D step 3) — accumulates submitted lines across the Home // process; Up/Down recall, Ctrl+R reverse-searches. Not persisted (a chat-resume starts fresh). @@ -347,6 +353,7 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { search: undefined, // ditto a reverse-search submode mention: undefined, // ditto an `@`-completion submode effortPicker: undefined, // ditto the `/effort` overlay (chat-scoped — closed on return to the bare Home) + reasonDraft: undefined, modelPicker: undefined, // ditto the `/models` picker (Home-only, so never open here — reset for hygiene) shellBusy: false, // a `!`-command in flight when the chat ended must not leave the returned Home gated submitBusy: false, // ditto a submit/compaction in flight — the returned Home must not be left gated @@ -409,6 +416,7 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { mention: undefined, modelPicker: undefined, effortPicker: undefined, + reasonDraft: undefined, shellBusy: false, submitBusy: false, // the swap is done — un-gate the fresh chat shellCommand: undefined, @@ -482,6 +490,7 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { mention: undefined, modelPicker: undefined, effortPicker: undefined, + reasonDraft: undefined, shellBusy: false, submitBusy: false, // the swap is done — un-gate the reseated chat shellCommand: undefined, @@ -554,6 +563,7 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { mention: undefined, modelPicker: undefined, effortPicker: undefined, + reasonDraft: undefined, historyEntries: history.entries, }); // Track the in-flight build so a signal (or a mid-build exit) during `loading` can reclaim its just-spawned @@ -1304,6 +1314,11 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { case 'reject': active.store.answerApproval({ outcome: 'reject' }); return; + case 'reject-with-reason': + // `[c]` — open the typed-reason capture (Step 14); the next keys fill it (handleChatKey routes them), + // then submit rejects WITH the reason. The approval stays pending until then (parity with `relavium chat`). + set({ reasonDraft: emptyEditor() }); + return; case 'none': return; } @@ -1311,6 +1326,33 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { const handleChatKey = (active: HomeChatSession, input: string, key: ChatKey): void => { if (tearingDown === active) return; // a key arriving mid-teardown must not drive sendMessage on a cancelled session + // The `[c]` typed-reason capture (Step 14) owns the keyboard while open — checked FIRST. It ONLY opens from a + // pending approval; if that approval settled out-of-band, the capture is stale → drop it. Else Esc CANCELS back + // to the [y]/[a]/[n] prompt (still pending — not an abort); plain Enter rejects WITH the sanitized+bounded + // reason; every other key edits the buffer. Parity with the standalone ChatApp; the floor is unchanged (this + // only enriches a reject — a governed dispatch still cannot proceed without an explicit decision). + const openReason = state.reasonDraft; + if (openReason !== undefined) { + if (active.store.getSnapshot().approval === undefined) { + set({ reasonDraft: undefined }); // the approval vanished — discard the orphaned capture + return; + } + if (key.escape === true) { + set({ reasonDraft: undefined }); // cancel the reason; the approval stays pending + return; + } + if (key.return === true && key.shift !== true) { + const reason = sanitizeApprovalReason(openReason.text); + set({ reasonDraft: undefined }); + active.store.answerApproval( + reason === undefined ? { outcome: 'reject' } : { outcome: 'reject', reason }, + ); + return; + } + const edit = reduceEditorMotion(input, key); + if (edit !== undefined) set({ reasonDraft: applyEditorAction(openReason, edit) }); + return; + } // Busy = a streaming turn OR a `!`-shell command in flight (`state.shellBusy` — the session has no store status // for it). A gated keystroke can't reach `sendMessage` → no `SessionStateError` crash. const running = From a8cb4416fdb5e8269fc46155a433c9048369bf8d Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 05:51:36 +0300 Subject: [PATCH 42/50] =?UTF-8?q?refactor(cli):=202.5-close=20Step=2014=20?= =?UTF-8?q?(Batch=20D-2)=20=E2=80=94=20consolidate=20the=20non-TTY=20fail-?= =?UTF-8?q?closed=20approval=20policy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The no-TTY approval policy (deny a governed dispatch when there is no user to approve — never a hang, never an auto-approve) already shipped in 2.5.E (the "High 9" fix) and is covered end-to-end (agent-run.test.ts "DENIES a governed dispatch in the non-interactive one-shot"; chat.test.ts chatIsInteractive → reject-immediately). But it was hand-rolled as an inline reject lambda in TWO places (chat.ts + agent-run.ts) with slightly divergent reason strings — a drift risk on a security-critical floor. Step 14 "define + enforce one policy": extract the canonical `nonInteractiveApprovalPrompt(context)` into chat-mode.ts (the home of `ApprovalPrompt`) — every request rejects regardless of tool/mode, with the surface named in the secret-free reason — and use it from both the non-interactive chat driver and the one-shot `agent run`. Behavior is byte-identical; the policy now has ONE home. Tests: +2 chat-mode cases (denies any request/tool — never a hang/auto-approve; names the surface in the reason). 110 chat-mode/chat/agent-run tests pass; typecheck/eslint/prettier green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 14, Batch D); ADR-0057 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/chat/chat-mode.test.ts | 28 ++++++++++++++++++++++++++++ apps/cli/src/chat/chat-mode.ts | 14 ++++++++++++++ apps/cli/src/commands/agent-run.ts | 9 ++++----- apps/cli/src/commands/chat.ts | 7 ++----- 4 files changed, 48 insertions(+), 10 deletions(-) diff --git a/apps/cli/src/chat/chat-mode.test.ts b/apps/cli/src/chat/chat-mode.test.ts index f11be212..6a26d3b3 100644 --- a/apps/cli/src/chat/chat-mode.test.ts +++ b/apps/cli/src/chat/chat-mode.test.ts @@ -9,6 +9,7 @@ import { governedToolIds, isGovernedTool, nextMode, + nonInteractiveApprovalPrompt, parseMode, type ApprovalAnswer, type ApprovalPrompt, @@ -91,6 +92,33 @@ describe('isGovernedTool — mirrors the registry governedAction (advertise-filt }); }); +describe('nonInteractiveApprovalPrompt — the fail-closed no-TTY policy (Step 14)', () => { + it('DENIES every request (any tool / mode) — never a hang, never an auto-approve', async () => { + const prompt = nonInteractiveApprovalPrompt('in a one-shot agent run'); + // A governed write and a process action both reject — the outcome does not depend on the request. + await expect(prompt(req(), true)).resolves.toEqual({ + outcome: 'reject', + reason: 'interactive approval is unavailable in a one-shot agent run', + }); + const proc = await prompt( + req({ toolId: 'run_command', action: 'process', preview: {} }), + false, + ); + expect(proc.outcome).toBe('reject'); // still a reject — no tool is ever auto-approved off a TTY + }); + + it('names the surface in the (secret-free) reason so the two call sites stay distinguishable', async () => { + const driver = await nonInteractiveApprovalPrompt('on this non-interactive driver')( + req(), + true, + ); + expect(driver).toEqual({ + outcome: 'reject', + reason: 'interactive approval is unavailable on this non-interactive driver', + }); + }); +}); + describe('ApprovalCache — session once/always memory', () => { it('remembers an always-approval by tool id; once caches nothing', () => { const cache = new ApprovalCache(); diff --git a/apps/cli/src/chat/chat-mode.ts b/apps/cli/src/chat/chat-mode.ts index b137107a..cc9e4129 100644 --- a/apps/cli/src/chat/chat-mode.ts +++ b/apps/cli/src/chat/chat-mode.ts @@ -91,6 +91,20 @@ export type ApprovalPrompt = ( signal?: AbortSignalLike, ) => Promise; +/** + * The canonical fail-closed {@link ApprovalPrompt} for a NON-INTERACTIVE surface (Step 14 — the one home for the + * no-TTY policy, previously hand-rolled in `chat.ts` and `agent-run.ts`). When there is no user at a TTY to answer + * an interactive consent prompt — a plain non-TTY / `--json` chat, or a one-shot `agent run` — a governed dispatch + * must be DENIED immediately: never a hang (an unanswerable `store.requestApproval` promise), and never an + * auto-approve (that would be the "bypass all permissions" valve ADR-0057 forbids). So every request resolves to a + * reject, regardless of the tool / mode; the `context` names the surface in the secret-free denial reason. This is + * the ADR-0057 fail-closed floor with no interactive answerer — the model-facing outcome is the same `tool_denied`. + */ +export function nonInteractiveApprovalPrompt(context: string): ApprovalPrompt { + const reason = `interactive approval is unavailable ${context}`; + return () => Promise.resolve({ outcome: 'reject', reason }); +} + /** * The session-scoped, IN-MEMORY once/always cache (ADR-0057 — NOT persisted across resume, so a `chat-resume` * re-prompts). `always` = a tool id approved for the remainder of this session instance; `once` caches diff --git a/apps/cli/src/commands/agent-run.ts b/apps/cli/src/commands/agent-run.ts index 17183757..a212c726 100644 --- a/apps/cli/src/commands/agent-run.ts +++ b/apps/cli/src/commands/agent-run.ts @@ -3,6 +3,7 @@ import { StringDecoder } from 'node:string_decoder'; import type { SessionStreamHandleEvent } from '@relavium/core'; +import { nonInteractiveApprovalPrompt } from '../chat/chat-mode.js'; import { applyChatMode, makeChatModeEnv } from '../chat/chat-mode-host.js'; import { cassetteResolver, loadCassette } from '../chat/fixture.js'; import { buildChatSession, type BuiltChatSession } from '../chat/session-host.js'; @@ -164,11 +165,9 @@ async function runOneShotTurn( session: built.session, tools: built.tools, workspaceDir: built.context.workingDir, - prompt: () => - Promise.resolve({ - outcome: 'reject', - reason: 'interactive approval is unavailable in a one-shot agent run', - }), + // The one canonical no-TTY fail-closed policy (Step 14) — a one-shot has no user to approve, so every + // governed dispatch is denied (never a hang / auto-approve). Shared with the non-interactive chat driver. + prompt: nonInteractiveApprovalPrompt('in a one-shot agent run'), }); applyChatMode(modeEnv, 'ask'); built.session.start(); diff --git a/apps/cli/src/commands/chat.ts b/apps/cli/src/commands/chat.ts index ccf2a392..a36654bb 100644 --- a/apps/cli/src/commands/chat.ts +++ b/apps/cli/src/commands/chat.ts @@ -38,6 +38,7 @@ import { CHAT_MODES, MODE_DESCRIPTION, MODE_LABEL, + nonInteractiveApprovalPrompt, parseMode, type ApprovalPrompt, type ChatMode, @@ -694,11 +695,7 @@ export function createChatModeControl( const interactive = opts?.interactive ?? true; const prompt: ApprovalPrompt = interactive ? store.requestApproval - : () => - Promise.resolve({ - outcome: 'reject', - reason: 'interactive approval is unavailable on this non-interactive driver', - }); + : nonInteractiveApprovalPrompt('on this non-interactive driver'); const modeEnv = makeChatModeEnv({ session: built.session, tools: built.tools, From 8f7d9544895d862f4ec0950099abb9fe2ec9118d Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 06:05:23 +0300 Subject: [PATCH 43/50] =?UTF-8?q?feat(core,cli):=202.5-close=20Step=2014?= =?UTF-8?q?=20(Batch=20D-3)=20=E2=80=94=20SCOPE-denial=20conversational=20?= =?UTF-8?q?recovery?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make a pure SCOPE denial recoverable so the interactive chat turn can adapt instead of dying — the "conversational recovery" of ADR-0057, scoped tightly to fs/media (the maintainer's 2.5-close D14-c decision). Taxonomy change (`changes the denial taxonomy`): the `recoverable` flag moves from `ToolExecutionError` onto the base `ToolDispatchError` (default FALSE — the safe direction; a forgotten opt-in just stays fatal). It now gates the chat within-turn recovery for TWO disjoint sources: an idempotent host execution failure (unchanged), AND a scope denial refused BEFORE any side effect. Exactly two `tool_denied`s opt in: - `ToolPolicyError('media_scope_denied')` — read_media's idempotent scope denial. - the CLI fs arm's PURE scope-tier escape (`FsScopeDeniedError` "the path escapes the allowed filesystem scope") — the canonical "path outside the workspace". `isRecoverableToolError` (agent-turn.ts) simplifies to `recoverToolFailures && err.recoverable` (plus the always-correctable unknown_tool/invalid_args). A recoverable scope denial is fed back as an `isError` tool_result so the model adapts to an in-bounds path; the floor STILL denies every attempt (no bypass). Kept FATAL (recoverable=false — no capability leak, no probe oracle, no re-execution): a user reject / fail-closed approval, every other guardrail reason (not_granted / command / domain / gate), an egress SSRF denial, and the fs CONFIDENTIALITY (secret-store read) / protected-path / symlink / hard-link refusals. Only on the chat surface (recoverToolFailures) — a WORKFLOW run stays fail-fast/deterministic. Tests: +2 agent-turn (recovers media_scope_denied when recoverToolFailures; does NOT recover a not_granted guardrail denial even then), +2 fs (the scope escape is recoverable:true; the confidentiality-floor read is recoverable:false). 985 core + 1626 cli tests pass; typecheck/eslint/prettier/build/engine-deps green. Canonical taxonomy home updated (tool-registry.md). Refs: docs/roadmap/phase-2.5-close-plan.md (Step 14, Batch D); ADR-0057 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/engine/tool-host/errors.ts | 7 ++- apps/cli/src/engine/tool-host/fs.test.ts | 23 +++++++- apps/cli/src/engine/tool-host/fs.ts | 22 +++++++- docs/reference/shared-core/tool-registry.md | 26 ++++++--- packages/core/src/engine/agent-turn.test.ts | 61 +++++++++++++++++++++ packages/core/src/engine/agent-turn.ts | 20 +++---- packages/core/src/tools/errors.ts | 40 +++++++++----- 7 files changed, 158 insertions(+), 41 deletions(-) diff --git a/apps/cli/src/engine/tool-host/errors.ts b/apps/cli/src/engine/tool-host/errors.ts index a1cde7d2..134df8de 100644 --- a/apps/cli/src/engine/tool-host/errors.ts +++ b/apps/cli/src/engine/tool-host/errors.ts @@ -29,8 +29,11 @@ export class HostDeniedError extends ToolDispatchError { readonly code = 'tool_denied'; readonly runErrorCode: ErrorCode = 'tool_denied'; readonly retryable = false; - constructor(message: string) { - super(message, undefined, undefined); + /** `recoverable` (the inherited base flag) is FALSE by default (fatal — the safe direction): a host denial ends + * the turn. A specific SCOPE denial refused before any side effect (an fs scope-tier escape — Step 14) opts in + * by passing `true`; egress SSRF / a confidentiality refusal stays fatal. */ + constructor(message: string, recoverable = false) { + super(message, undefined, undefined, recoverable); this.name = new.target.name; } } diff --git a/apps/cli/src/engine/tool-host/fs.test.ts b/apps/cli/src/engine/tool-host/fs.test.ts index d85b3ee8..f557bde6 100644 --- a/apps/cli/src/engine/tool-host/fs.test.ts +++ b/apps/cli/src/engine/tool-host/fs.test.ts @@ -85,16 +85,33 @@ describe('createNodeFsCapability — read (jailed)', () => { expect((await sandboxed().readFile('sub/x.txt', {})).content).toBe('deep'); }); - it('rejects a `..` traversal that escapes the workspace — FATAL tool_denied, not retryable', async () => { + it('rejects a `..` traversal that escapes the workspace — tool_denied, not retryable, but RECOVERABLE (Step 14)', async () => { await writeFile(join(outside, 'secret.txt'), 'TOP SECRET'); const err: unknown = await sandboxed() .readFile('../outside/secret.txt', {}) .catch((e: unknown) => e); expect(err).toBeInstanceOf(FsScopeDeniedError); - // A deterministic scope denial must be fatal so it never burns the node-retry budget (error-handling.md). if (err instanceof FsScopeDeniedError) { expect(err.code).toBe('tool_denied'); - expect(err.retryable).toBe(false); + expect(err.retryable).toBe(false); // never burns the node-retry budget (error-handling.md) + // The PURE scope-tier escape is refused before any side effect, so it is fed back for conversational + // recovery on the chat surface (the model adapts to an in-bounds path); the floor still denies every try. + expect(err.recoverable).toBe(true); + } + }); + + it('a confidentiality-floor read refusal is FATAL (recoverable:false) — no secret-store probe oracle (Step 14)', async () => { + // A secret/credential store read (.ssh here) is refused as a FsScopeDeniedError like a scope escape, but it + // must NOT be fed back to the model (that would leak a "this path is a secret store" oracle) — it stays fatal. + await mkdir(join(workspace, '.ssh'), { recursive: true }); + await writeFile(join(workspace, '.ssh', 'id_rsa'), 'PRIVATE'); + const err: unknown = await sandboxed() + .readFile('.ssh/id_rsa', {}) + .catch((e: unknown) => e); + expect(err).toBeInstanceOf(FsScopeDeniedError); + if (err instanceof FsScopeDeniedError) { + expect(err.code).toBe('tool_denied'); + expect(err.recoverable).toBe(false); // the confidentiality boundary stays fatal (unlike a scope escape) } }); diff --git a/apps/cli/src/engine/tool-host/fs.ts b/apps/cli/src/engine/tool-host/fs.ts index 27baa2cc..3e3efcb3 100644 --- a/apps/cli/src/engine/tool-host/fs.ts +++ b/apps/cli/src/engine/tool-host/fs.ts @@ -93,8 +93,19 @@ export class FsCapabilityError extends HostCapabilityError {} * A **deterministic scope/security denial** — a path escapes the tier, a UNC path, or a refusal to write through * a symlink — mapping to the **fatal**, non-retryable `tool_denied` (the shared {@link HostDeniedError}): a * denied path just re-denies, so it must NOT burn the node-retry budget (error-handling.md §tool-dispatch codes). + * + * Recoverability (Step 14): fatal by default (`recoverable=false`, inherited). ONE throw opts in — the pure + * **scope-tier escape** ("the path escapes the allowed filesystem scope") passes `recoverable: true`, so on the + * interactive chat surface it is fed back as a correctable tool result and the model can adapt to an in-bounds + * path (conversational recovery). The **confidentiality** refusal (a secret/credential store read), the protected + * -path write, and the symlink/hard-link refusals stay fatal — feeding those back would leak a probe oracle or + * risk nothing useful. */ -export class FsScopeDeniedError extends HostDeniedError {} +export class FsScopeDeniedError extends HostDeniedError { + constructor(message: string, opts?: { recoverable?: boolean }) { + super(message, opts?.recoverable ?? false); + } +} /** * Build a node-backed {@link FsCapability} jailed to `config`'s scope tier. The returned object is the value a @@ -784,10 +795,15 @@ async function jailWriteTarget( return { realDir, finalTarget: join(realDir, basename(lexical)) }; } -/** Throw a fatal scope denial if `real` is not in scope (the authoritative post-realpath jail). */ +/** Throw a scope denial if `real` is not in scope (the authoritative post-realpath jail). RECOVERABLE on the chat + * surface (Step 14) — the pure scope-tier escape is refused before any side effect, so the model may adapt to an + * in-bounds path; the floor still denies every attempt. (The confidentiality / symlink / protected-path denials + * keep the default fatal, so this is the ONLY fs denial fed back.) */ function assertInScope(inScope: ScopeChecker, real: string): void { if (!inScope(real)) { - throw new FsScopeDeniedError('the path escapes the allowed filesystem scope'); + throw new FsScopeDeniedError('the path escapes the allowed filesystem scope', { + recoverable: true, + }); } } diff --git a/docs/reference/shared-core/tool-registry.md b/docs/reference/shared-core/tool-registry.md index f8a8e1df..6a277649 100644 --- a/docs/reference/shared-core/tool-registry.md +++ b/docs/reference/shared-core/tool-registry.md @@ -176,7 +176,7 @@ host is touched once, in the middle. 7. **Bound the model-facing result** (§Result bounding and spill-to-file) from the result via `ctx.limits` + the host `outputStore` — over the ceiling the model gets a preview + a spill handle, the full result still flows to `output_mapping`. 8. **Mark the result untrusted** (§Untrusted-data taint) and hand the structured `tool_call` / `tool_result` data + its taint/secret markers to the bus's single translation point ([ADR-0036](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md)) for `agent:tool_call` / `agent:tool_result` emission. -> **Loop-correctable vs terminal.** `UnknownToolError` and `ToolArgsInvalidError` are **thrown** by the registry; the agent loop (1.O) **catches** them and synthesizes a correctable `isError` `tool_result` (from the secret-free `error.message`) so the model can fix its call, within a **bounded correction budget** it owns — escalating to a node `ErrorCode` only when that budget is spent. A `ToolPolicyError` — and, identically, a `ToolDeniedByUserError` (the per-tool approval denial, ADR-0057) — is structurally fatal (`tool_denied`) and **never** fed back as a correctable result (re-asking a denied tool just burns budget). See [agent-runner.md §the failure ladder](agent-runner.md). A `ToolCancelledError` maps to `cancelled` ahead of all other classifications (cancel wins). +> **Loop-correctable vs terminal.** `UnknownToolError` and `ToolArgsInvalidError` are **thrown** by the registry; the agent loop (1.O) **catches** them and synthesizes a correctable `isError` `tool_result` (from the secret-free `error.message`) so the model can fix its call, within a **bounded correction budget** it owns — escalating to a node `ErrorCode` only when that budget is spent. A `ToolPolicyError` — and, identically, a `ToolDeniedByUserError` (the per-tool approval denial, ADR-0057) — is structurally fatal (`tool_denied`) and **never** fed back as a correctable result (re-asking a denied tool just burns budget), with **one Step-14 exception**: a `recoverable` SCOPE denial (`media_scope_denied` / the fs pure scope-tier escape — refused before any side effect) IS fed back on the interactive chat surface (`recoverToolFailures`) so the model can adapt to an in-bounds path (see the `recoverable` note under the error taxonomy). See [agent-runner.md §the failure ladder](agent-runner.md). A `ToolCancelledError` maps to `cancelled` ahead of all other classifications (cancel wins). ```ts interface ToolDispatchContext { @@ -324,20 +324,28 @@ codes by [sse-event-schema.md](../contracts/sse-event-schema.md#error-code-taxon | Error | When | Run `ErrorCode` | Class | |-------|------|-----------------|-------| | `UnknownToolError` | id not an exact match | `tool_failed` | fatal (loop-correctable first) | -| `ToolPolicyError` | a guardrail / grant denial — `not_granted`, `provider_executed`, `command_not_allowed`, `domain_not_allowed`, `insecure_url`, `gate_required`, `media_scope_denied` (the full `ToolPolicyDenyReason` union; `media_scope_denied` is `read_media`'s scope-set denial, [ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §1) | `tool_denied` | **fatal** (never retried) | +| `ToolPolicyError` | a guardrail / grant denial — `not_granted`, `provider_executed`, `command_not_allowed`, `domain_not_allowed`, `insecure_url`, `gate_required`, `media_scope_denied` (the full `ToolPolicyDenyReason` union; `media_scope_denied` is `read_media`'s scope-set denial, [ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §1) | `tool_denied` | **fatal** — EXCEPT `media_scope_denied` is `recoverable` (a Step-14 SCOPE denial; see the note below) | | `ToolDeniedByUserError` | an interactive **per-tool approval** denial ([ADR-0057](../../decisions/0057-cli-chat-modes-and-per-tool-approval.md) EA3) — `user_rejected` (rejected by the user / mode policy), `no_approval_hook` (fail-closed: a governed dispatch under an active regime with no confirm hook wired), `approval_error` (fail-closed: the hook threw a non-abort error, so consent could not be obtained) | `tool_denied` | **fatal** (never retried; not loop-correctable — re-asking re-prompts/re-denies, like `ToolPolicyError`) | | `ToolArgsInvalidError` | effective args fail `parseArgs` / secret-taint | `validation` | fatal (loop-correctable first) | | `ToolUnavailableError` | the required `ToolHost` capability is absent (host/config gap, not the model's fault) | `tool_unavailable` | **fatal** (names the tool + the unwired arm actionably — never a bare `internal`; EA1, [ADR-0055](../../decisions/0055-cli-host-capability-seam-tool-environment-factory.md)) | | `ToolExecutionError` | the host capability threw a non-cancel error (cause kept off the message, for logs) | `tool_failed` | retryable (node budget) | | *(AbortSignal abort)* | the run was cancelled mid-tool | `cancelled` | fatal (cancel path, not `tool_failed`) | -`ToolExecutionError` additionally carries a `recoverable` flag (stamped at the wrap from `governedAction` — `true` -only for an **idempotent read**: no `fs_write` / `process` / `egress` / `os` side effect). It is STRICTER than -`retryable` (which lets a *fresh node-retry* re-run the whole node): `recoverable` gates the interactive-chat -within-turn recovery ([ADR-0057](../../decisions/0057-cli-chat-modes-and-per-tool-approval.md) `recoverToolFailures`) — -a recoverable failure is fed back to the model as an `isError` tool result so it can adapt, while a governed / -side-effecting failure ends the turn (fail-fast, so the model never re-attempts a non-idempotent side effect). A -WORKFLOW node ignores the flag (fail-fast always). +**`recoverable` (the base `ToolDispatchError` flag; default `false`).** STRICTER than `retryable` (which lets a +*fresh node-retry* re-run the whole node): `recoverable` gates the interactive-chat WITHIN-TURN recovery +([ADR-0057](../../decisions/0057-cli-chat-modes-and-per-tool-approval.md) `recoverToolFailures`) — a recoverable +error is fed back to the model as an `isError` tool result so it can adapt (conversational recovery), while +everything else ends the turn. Two disjoint sources set it (a WORKFLOW node ignores the flag — fail-fast always): +- `ToolExecutionError` — `true` only for an **idempotent read** (no `fs_write` / `process` / `egress` / `os` side + effect), stamped at the wrap from `governedAction`; a governed / side-effecting failure stays `false` (fail-fast, + so the model never re-attempts a non-idempotent side effect — a half-run command, a POST that may have landed). +- A **SCOPE denial** (Step 14) — a `tool_denied` refused BEFORE any side effect, so it is safe to feed back and the + model can adapt to an in-bounds path (the floor still denies every attempt). Exactly two carry it: the engine's + `ToolPolicyError('media_scope_denied')`, and the CLI `fs` arm's **pure scope-tier escape** (`FsScopeDeniedError` + `'the path escapes the allowed filesystem scope'`). Every OTHER `tool_denied` stays fatal (`false`): a + `ToolDeniedByUserError` (user reject / fail-closed), the other `ToolPolicyError` guardrail reasons, an egress + **SSRF** denial, and the fs **confidentiality** (secret-store read) / protected-path / symlink refusals — feeding + those back would re-deny, risk a re-execution, or leak a probe oracle. ## Instantiation diff --git a/packages/core/src/engine/agent-turn.test.ts b/packages/core/src/engine/agent-turn.test.ts index f7bb1173..1e0b1ed4 100644 --- a/packages/core/src/engine/agent-turn.test.ts +++ b/packages/core/src/engine/agent-turn.test.ts @@ -721,6 +721,67 @@ describe('runAgentTurn — tool loop', () => { ).rejects.toMatchObject({ code: 'tool_failed', retryable: true }); }); + it('recovers a RECOVERABLE SCOPE denial (media_scope_denied) when recoverToolFailures is set (Step 14)', async () => { + // A SCOPE denial refused BEFORE any side effect (media_scope_denied / an fs scope-tier escape) is `tool_denied` + // but flagged `recoverable` — on the chat surface it is fed back as an isError result so the model adapts to + // an in-bounds path (conversational recovery), instead of the turn dying. It shares the recoverToolFailures gate. + let dispatched = 0; + const registry = stubRegistry((call) => { + dispatched += 1; + if (dispatched === 1) throw new ToolPolicyError('echo', 'media_scope_denied', 'out of scope'); + const result: ToolResultPart = { type: 'tool_result', toolCallId: call.id, result: 'OK' }; + return { + output: 'OK', + toolResult: markUntrusted(result), + truncated: false, + events: { + call: { toolId: call.name, toolInput: {} }, + result: { toolId: call.name, success: true, outputSummary: 'OK' }, + }, + }; + }); + const provider = scriptedProvider('anthropic', [ + [ + { type: 'tool_call_start', id: 'c1', name: 'echo' }, + { type: 'tool_call_end', id: 'c1' }, + STOP('tool_use'), + ], + [ + { type: 'tool_call_start', id: 'c2', name: 'echo' }, + { type: 'tool_call_end', id: 'c2' }, + STOP('tool_use'), + ], + [{ type: 'text_delta', text: 'recovered' }, STOP()], + ]); + const params = baseParams(provider, { + registry, + limits: { ...DEFAULT_AGENT_TURN_LIMITS, recoverToolFailures: true }, + }); + const result = await runAgentTurn(params); + expect(result.text).toBe('recovered'); // the turn continued past the scope denial + expect( + eventsOf(params).find((e) => e.type === 'agent:tool_result' && !e.success), + ).toBeDefined(); + }); + + it('does NOT recover a NON-scope tool_denied (a guardrail denial) even with recoverToolFailures (Step 14)', async () => { + // The taxonomy split: only a SCOPE denial (recoverable) is fed back — a guardrail/grant denial (`not_granted`, + // and equally a user reject / SSRF / confidentiality) is NOT recoverable, so it ends the turn LOUDLY even on + // the chat surface. Re-issuing the same denied call would just re-deny (or leak a probe oracle). + const registry = stubRegistry(() => { + throw new ToolPolicyError('echo', 'not_granted', 'tool not granted'); // recoverable=false + }); + const provider = scriptedProvider('anthropic', [toolUseTurn('c1')]); + await expect( + runAgentTurn( + baseParams(provider, { + registry, + limits: { ...DEFAULT_AGENT_TURN_LIMITS, recoverToolFailures: true }, + }), + ), + ).rejects.toMatchObject({ code: 'tool_denied', retryable: false }); + }); + it('attaches the turn’s REAL accumulated usage to a failed turn (EA2)', async () => { // The tool-use turn settled an attempt (STOP carries usage 10/5) BEFORE the tool throws, so the // accumulated usage is non-zero — the wrapper attaches it to the thrown AgentTurnError rather than diff --git a/packages/core/src/engine/agent-turn.ts b/packages/core/src/engine/agent-turn.ts index 5b03e369..faba2d7a 100644 --- a/packages/core/src/engine/agent-turn.ts +++ b/packages/core/src/engine/agent-turn.ts @@ -52,7 +52,7 @@ import { type ToolDef as LlmToolDef, } from '@relavium/llm'; -import { ToolDispatchError, ToolExecutionError } from '../tools/errors.js'; +import { ToolDispatchError } from '../tools/errors.js'; import type { ToolCallPart, ToolDispatchContext, ToolRegistry } from '../tools/types.js'; import { unwrapUntrusted } from '../tools/untrusted.js'; import { BudgetExceededError, BudgetPauseError } from './budget-governor.js'; @@ -246,18 +246,18 @@ export function codeForLlmError(error: LlmError): ErrorCode { /** * A tool throw the turn recovers by feeding the model an `isError` tool result (which increments the shared * `maxToolCorrections` budget) instead of ending the turn. Always the model's own syntactic mistakes - * (`unknown_tool` / `invalid_args`); PLUS a host execution failure ONLY when BOTH `limits.recoverToolFailures` - * is set (the interactive chat surface — see {@link AgentTurnLimits.recoverToolFailures}) AND the error is - * flagged {@link ToolExecutionError.recoverable} — i.e. an IDEMPOTENT tool (a read), stamped by the registry - * from `governedAction`. A governed / side-effecting failure (a half-run command, a POST that may have landed) - * is NOT recoverable, so it ends the turn rather than risk a re-execution. A `tool_denied` / `tool_unavailable` - * / `cancelled` is NEVER recoverable here (a security / cancel boundary — it stays fatal so it never loops). + * (`unknown_tool` / `invalid_args`); PLUS — ONLY on the interactive chat surface (`limits.recoverToolFailures`, + * see {@link AgentTurnLimits.recoverToolFailures}) — any error the throwing class flagged `recoverable` + * ({@link ToolDispatchError.recoverable}): an IDEMPOTENT host execution failure (a read, stamped by the registry + * from `governedAction`), OR a SCOPE denial refused BEFORE any side effect (a Step-14 fs scope-tier escape / a + * media scope denial) so the model can adapt to an in-bounds path (conversational recovery). Everything else + * stays fatal: a governed/side-effecting execution failure (a half-run command — a re-execution hazard), and a + * user / guardrail / SSRF / **confidentiality** / process denial + `tool_unavailable` / `cancelled` (a security / + * cancel boundary — re-issuing re-denies or would leak a probe oracle), all carry `recoverable=false`. */ function isRecoverableToolError(err: ToolDispatchError, limits: AgentTurnLimits): boolean { if (err.code === 'unknown_tool' || err.code === 'invalid_args') return true; - return ( - limits.recoverToolFailures === true && err instanceof ToolExecutionError && err.recoverable - ); + return limits.recoverToolFailures === true && err.recoverable; } /** Map a non-correctable tool throw to the node `ErrorCode` (cancel wins; a denial is fatal). */ diff --git a/packages/core/src/tools/errors.ts b/packages/core/src/tools/errors.ts index b8214b8a..8dc2a7fe 100644 --- a/packages/core/src/tools/errors.ts +++ b/packages/core/src/tools/errors.ts @@ -31,12 +31,29 @@ export abstract class ToolDispatchError extends Error { abstract readonly retryable: boolean; /** The tool id involved, when one is known (a name, never a secret). */ readonly toolId?: ToolId; + /** + * Whether this failure is safe to FEED BACK to the model as an `isError` tool result for in-turn CONVERSATIONAL + * recovery (ADR-0057 `recoverToolFailures`, the interactive chat surface only) instead of ending the turn. + * Default **false** (fatal — the safe direction; a forgotten opt-in just stays fatal). Set true by the specific + * throwing class ONLY when re-attempting is safe: an IDEMPOTENT host execution failure (a read — no side-effect + * hazard), or a SCOPE denial refused BEFORE any side effect (a Step-14 fs scope-tier escape / a media scope + * denial) so the model can adapt to an in-bounds path. A user / guardrail / SSRF / **confidentiality** / + * side-effecting denial stays false — re-issuing re-denies, would risk a re-execution, or would leak a + * probe oracle. Distinct from {@link retryable} (a fresh whole-node retry); this is the stricter in-turn re-attempt. + */ + readonly recoverable: boolean; - protected constructor(message: string, toolId: ToolId | undefined, cause: unknown) { + protected constructor( + message: string, + toolId: ToolId | undefined, + cause: unknown, + recoverable = false, + ) { super(message, cause === undefined ? undefined : { cause }); if (toolId !== undefined) { this.toolId = toolId; } + this.recoverable = recoverable; } } @@ -68,7 +85,9 @@ export class ToolPolicyError extends ToolDispatchError { readonly reason: ToolPolicyDenyReason; constructor(toolId: ToolId, reason: ToolPolicyDenyReason, message: string) { - super(message, toolId, undefined); + // A `media_scope_denied` is a SCOPE denial of an idempotent read (read_media) refused before any effect, so + // it is RECOVERABLE — fed back so the model can adapt (Step 14). Every other guardrail reason stays fatal. + super(message, toolId, undefined, reason === 'media_scope_denied'); this.name = 'ToolPolicyError'; this.reason = reason; } @@ -149,21 +168,14 @@ export class ToolExecutionError extends ToolDispatchError { readonly code = 'execution_failed'; readonly runErrorCode: ErrorCode = 'tool_failed'; readonly retryable = true; - /** - * Whether this failure is safe to FEED BACK to the model for a within-turn retry (ADR-0057 - * `recoverToolFailures`, the interactive chat surface) — vs ending the turn. `retryable` (above) means a - * fresh node-retry may re-run the whole node; `recoverable` is the STRICTER "the model may re-attempt this - * specific call in-turn without a side-effect hazard". True ONLY for an IDEMPOTENT tool (a read — no - * `fs_write`/`egress`/`process`/`os` action), stamped by the registry from {@link governedAction}. A - * governed / side-effecting tool defaults to **false**, so a non-idempotent failure (a half-run command, a - * POST that may have reached the server) ends the turn rather than risking a re-execution. - */ - readonly recoverable: boolean; constructor(toolId: ToolId, message: string, cause?: unknown, opts?: { recoverable?: boolean }) { - super(message, toolId, cause); + // `recoverable` (the inherited base flag) is true ONLY for an IDEMPOTENT tool (a read — no + // `fs_write`/`egress`/`process`/`os` action), stamped by the registry from `governedAction`; a governed / + // side-effecting failure (a half-run command, a POST that may have reached the server) stays false so it ends + // the turn rather than risking a re-execution. + super(message, toolId, cause, opts?.recoverable ?? false); this.name = 'ToolExecutionError'; - this.recoverable = opts?.recoverable ?? false; } } From a3d71476c2a72595211a21461d032acaa2488456 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 06:07:07 +0300 Subject: [PATCH 44/50] =?UTF-8?q?docs(adr,security):=202.5-close=20Step=20?= =?UTF-8?q?14=20=E2=80=94=20ADR-0057=20amendment=20note=20+=20security-rev?= =?UTF-8?q?iew.md=20for=20Batch=20D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Append-only ADR-0057 amendment note recording the Step-14 approval/security batch (the deferred `[c]`/non-TTY follow-ups landed, plus the SCOPE-denial recovery taxonomy change, the Ctrl+T approval whitelist, and the Trojan-Source floor) — the core guarantees are unchanged (no ungated governed dispatch, no protected-path / jail escape, read-only `ask` default). Extends security-review.md with a "CLI terminal-render safety + interactive approval" subsection: the bidi/Trojan-Source render floor, the fail-closed approval keyboard + the sanitized/bounded `[c]` reason, and the by-construction safety of the two recoverable scope denials (vs the fatal confidentiality/SSRF/user/guardrail denials). Refs: docs/roadmap/phase-2.5-close-plan.md (Step 14, Batch D); ADR-0057 Co-Authored-By: Claude Opus 4.8 (1M context) --- ...57-cli-chat-modes-and-per-tool-approval.md | 26 +++++++++++++++++ docs/standards/security-review.md | 29 +++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/docs/decisions/0057-cli-chat-modes-and-per-tool-approval.md b/docs/decisions/0057-cli-chat-modes-and-per-tool-approval.md index 00ee3a24..d1bda1a6 100644 --- a/docs/decisions/0057-cli-chat-modes-and-per-tool-approval.md +++ b/docs/decisions/0057-cli-chat-modes-and-per-tool-approval.md @@ -19,6 +19,32 @@ > reject-with-typed-reason prompt, a plain/non-TTY non-interactive approval policy, a live `web_search`/http > egress credential resolver, and the session-level budget pause/resume that rides the same EA4 machine. +> **Amended 2026-07-08 (2.5-close Step 14 — approval/security batch; security-reviewed).** The deferred +> approval follow-ups above landed, each in its own reviewed diff, plus two rendering/recovery hardenings — the +> core guarantees are **unchanged** (no governed dispatch runs ungated; no mode escapes protected-paths / the fs +> jail; the default stays read-only `ask`): +> 1. **`[c]` reject-with-typed-reason.** The approval prompt gains a `[c]` branch that opens a small keyboard-owning +> reason-input sub-mode; on submit it rejects with the (sanitized + 300-char-bounded, terminal/bidi-stripped) +> reason via the EXISTING `ToolApprovalDecision.reject.reason` seam, so the denial records WHY. It only enriches a +> reject — the fail-closed floor and "a user deny is final" (`tool_denied`) are untouched. +> 2. **Non-TTY approval policy = one canonical `nonInteractiveApprovalPrompt`.** The no-TTY fail-closed deny (a +> governed dispatch is denied — never a hang, never an auto-approve — when nothing can answer a consent prompt) +> already shipped as the 2.5.E "High 9" fix; it is now a single named helper shared by the non-interactive chat +> driver and one-shot `agent run` (was hand-rolled in two places). +> 3. **SCOPE-denial conversational recovery (a denial-taxonomy change).** The `recoverable` flag moves onto the base +> `ToolDispatchError` (default false). Exactly two `tool_denied`s — a media scope denial (`media_scope_denied`) +> and the CLI fs arm's **pure scope-tier escape** — now carry `recoverable:true`, so on the interactive chat +> surface (`recoverToolFailures`) they are fed back as a correctable tool result and the model adapts to an +> in-bounds path (the floor still denies every attempt). A user / guardrail / SSRF / **confidentiality** +> (secret-store) / protected-path / symlink denial stays fatal — feeding those back would re-deny, risk a +> re-execution, or leak a probe oracle. Canonical taxonomy: [../reference/shared-core/tool-registry.md](../reference/shared-core/tool-registry.md#error-taxonomy). +> 4. **Ctrl+T through the approval swallow.** The fail-closed key-swallow now whitelists exactly the VIEW-ONLY +> reasoning toggle (Ctrl+T — a pure store repaint, zero session/decision effect), so a user can expand the +> model's reasoning to inform the decision; the approval input set is otherwise unchanged. +> 5. **Trojan-Source (bidi) floor.** The shared render sanitizer (`stripTerminalControls`) now also strips the +> Unicode bidirectional/directional format controls (U+202A–202E, U+2066–2069, LRM/RLM/ALM) so a streamed/pasted +> line can't visually spoof its logical bytes in an approval prompt (CVE-2021-42574); ZWJ/ZWNJ are preserved. + ## Context We want Claude-Code-style chat modes (ask / plan / accept-edits / auto) and per-tool approval for writes diff --git a/docs/standards/security-review.md b/docs/standards/security-review.md index 0a398cfa..aa3a2905 100644 --- a/docs/standards/security-review.md +++ b/docs/standards/security-review.md @@ -326,6 +326,35 @@ security invariants** a review must confirm are: `ToolRegistry` / `AgentSession` implementations (1.O / 1.T / 1.V); it is an engine-layer rule and changes **no** seam shape — `LlmMessage` stays as the seam defines it. +### CLI terminal-render safety + interactive approval (2.5.E / 2.5-close Step 14) + +- **Terminal-control + Trojan-Source floor (a rendering-safety primitive).** Every dynamic + string written to the terminal — streamed model output, a tool id, a path in an approval + prompt, a pasted line — passes the shared `stripTerminalControls` sanitizer at the DISPLAY + boundary (the persisted transcript keeps the raw bytes). It strips ANSI/OSC/DCS escapes, C0/C1 + control bytes, AND the Unicode bidirectional/directional format controls (U+202A–202E, + U+2066–2069, LRM/RLM/ALM — the CVE-2021-42574 family) so untrusted text cannot inject a + cursor/title/clipboard escape NOR visually reorder a line to spoof its logical bytes (a faked + path/command in a consent prompt). ZWJ/ZWNJ are preserved (legitimate in emoji + shaping). The + provider-URL echo strips a stricter zero-width superset. **Never embed literal bidi bytes in + source** — the sanitizer regex uses `\u` escapes. +- **Interactive per-tool approval is fail-closed and cannot be widened by a keystroke.** During a + pending approval the key-swallow answers ONLY `[y]/[a]/[n]/[esc]` plus `[c]` (open a + reject-with-reason capture) and the VIEW-ONLY reasoning toggle (Ctrl+T — a pure repaint, zero + session/decision effect). A modified chord (Ctrl/Alt) never silently picks the most-permissive + approve/reject. The `[c]` reason is a USER-typed reject enrichment only — sanitized (terminal + + bidi stripped, one line) and length-bounded before it rides the secret-free denial; it never + changes the floor or "a user deny is final". Off a TTY (`--json` / piped / one-shot `agent run`) + the one canonical `nonInteractiveApprovalPrompt` DENIES every governed dispatch — never a hang, + never an auto-approve. +- **A recoverable SCOPE denial is safe by construction.** Only two `tool_denied`s are fed back to + the model for in-turn conversational recovery (chat surface only): a media scope denial and the + fs **pure scope-tier escape** — both refused BEFORE any side effect, with a secret-free reason, + so the model adapts to an in-bounds path while the floor still denies every attempt (no bypass, + bounded by the correction budget). The **confidentiality** (secret-store read), protected-path, + symlink/hard-link, egress-SSRF, and user/guardrail denials stay FATAL — feeding those back would + re-deny, risk a re-execution, or leak a probe oracle. + ## Never hand-roll crypto - We **never implement cryptography, TLS, or keychain primitives ourselves**. We use vetted From babd0e4c14a954eeb1456756dd4f1ff0da498472 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 06:27:33 +0300 Subject: [PATCH 45/50] =?UTF-8?q?fix(cli,docs):=202.5-close=20Step=2014=20?= =?UTF-8?q?review=20fixes=20=E2=80=94=20canonical-home=20drift=20+=20doc?= =?UTF-8?q?=20accuracy=20+=20surrogate-safe=20cap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applied the security + Opus review findings (both returned 0 blocker / 0 high; all five guarantees traced + confirmed). Fixes: - MEDIUM (canonical-home drift, rule #8): agent-runner.md's failure-ladder table still called ToolPolicyError unconditionally "not fed back" — corrected to note the Step-14 `media_scope_denied` recoverable exception, linking the tool-registry.md taxonomy note (tool-registry.md was updated in D-3; the linked ladder home was missed). - LOW: two stale "`[c]` … deferred follow-up" comments (chat-store.ts PendingApproval, chat-mode-host.ts MakeChatModeEnv.prompt) — `[c]` landed in D-1; updated both to `[c] reason` + the reject-reason seam. - NIT: `sanitizeApprovalReason` capped by UTF-16 code unit — could split an astral pair into a lone surrogate at the 300-char boundary. Back off one unit when the cap lands on a high surrogate (+ a regression test). - Doc accuracy (security-review finding): the `recoverToolFailures` recovery surface is the chat-read-write host (chat / Home / one-shot `agent run`) — not "interactive chat only"; corrected in tool-registry.md, the ADR-0057 amendment, and security-review.md. Documented the accepted, secret-free scope-boundary probe residual (bounded by maxToolCorrections; the same round-bounded feedback pre-existed for an idempotent not-found read) + the createDirs idempotent-mkdir exception to "before any side effect". - deferred-tasks.md: marked the four now-landed 2.5.E entries DONE (bidi strip, `[c]`, scope-recovery, non-TTY policy); recorded the two optional Step-14 review follow-ups (approval-consent-line zero-width hardening; extract the `[c]` capture to a shared reducer for ChatApp test parity). Accepted as-is (documented): the `createChatModeControl` `interactive` default (true = the ink-REPL convention; the sole production caller always passes the computed flag; off-TTY the worst case is a hang, never an auto-approve). Tests: 67 projection cases pass; typecheck/eslint/prettier/fence green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 14, Batch D); ADR-0057 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/chat/chat-mode-host.ts | 4 +- .../src/render/tui/chat-projection.test.ts | 11 ++++ apps/cli/src/render/tui/chat-projection.ts | 12 +++-- apps/cli/src/render/tui/chat-store.ts | 5 +- ...57-cli-chat-modes-and-per-tool-approval.md | 7 +-- docs/reference/shared-core/agent-runner.md | 2 +- docs/reference/shared-core/tool-registry.md | 22 +++++--- docs/roadmap/deferred-tasks.md | 50 +++++++++++-------- docs/standards/security-review.md | 16 +++--- 9 files changed, 83 insertions(+), 46 deletions(-) diff --git a/apps/cli/src/chat/chat-mode-host.ts b/apps/cli/src/chat/chat-mode-host.ts index a1fe8ac6..1ddf446c 100644 --- a/apps/cli/src/chat/chat-mode-host.ts +++ b/apps/cli/src/chat/chat-mode-host.ts @@ -31,8 +31,8 @@ export interface ChatModeEnv { readonly governed: ReadonlySet; /** The session once/always memory (shared across mode changes — an "always" persists until the session ends). */ readonly cache: ApprovalCache; - /** The REPL's interactive `[y] yes / [a] always / [n] no / [esc] abort` prompt (accept-edits, and auto's - * protected-path fallback). A reject-with-typed-reason (`[c]` comment) is a deferred follow-up. */ + /** The REPL's interactive `[y] yes / [a] always / [n] no / [c] reason / [esc] abort` prompt (accept-edits, and + * auto's protected-path fallback). `[c]` opens the typed-reason capture (Step 14 — a reject carrying WHY). */ readonly prompt: ApprovalPrompt; /** Whether an approval preview targets a protected path — `auto` then falls back to a prompt (ADR-0057). */ readonly isProtectedTarget: (preview: ToolActionPreview) => boolean; diff --git a/apps/cli/src/render/tui/chat-projection.test.ts b/apps/cli/src/render/tui/chat-projection.test.ts index 29e82517..a1edfd22 100644 --- a/apps/cli/src/render/tui/chat-projection.test.ts +++ b/apps/cli/src/render/tui/chat-projection.test.ts @@ -734,5 +734,16 @@ describe('chat-projection', () => { const long = 'x'.repeat(MAX_APPROVAL_REASON_CHARS + 50); expect(sanitizeApprovalReason(long)).toHaveLength(MAX_APPROVAL_REASON_CHARS); }); + + it('does not leave a lone surrogate when the cap lands mid-astral-pair', () => { + // A run of (MAX-1) ASCII then an astral emoji (2 UTF-16 units): the naive slice(0, MAX) would keep the + // high surrogate and drop the low — a `�`. The back-off drops the whole pair (result is MAX-1 chars). + const reason = `${'a'.repeat(MAX_APPROVAL_REASON_CHARS - 1)}😀tail`; + const out = sanitizeApprovalReason(reason); + expect(out).toHaveLength(MAX_APPROVAL_REASON_CHARS - 1); // the split emoji was dropped whole + expect(out).toBe('a'.repeat(MAX_APPROVAL_REASON_CHARS - 1)); + // No unpaired surrogate survived (each code unit is a full BMP code point). + for (const ch of out ?? '') expect(ch.codePointAt(0)).toBeLessThan(0xd800); + }); }); }); diff --git a/apps/cli/src/render/tui/chat-projection.ts b/apps/cli/src/render/tui/chat-projection.ts index cefc7955..a9459975 100644 --- a/apps/cli/src/render/tui/chat-projection.ts +++ b/apps/cli/src/render/tui/chat-projection.ts @@ -79,9 +79,15 @@ export const MAX_APPROVAL_REASON_CHARS = 300; export function sanitizeApprovalReason(text: string): string | undefined { const clean = sanitizeInline(text).trim(); if (clean.length === 0) return undefined; - return clean.length > MAX_APPROVAL_REASON_CHARS - ? clean.slice(0, MAX_APPROVAL_REASON_CHARS) - : clean; + if (clean.length <= MAX_APPROVAL_REASON_CHARS) return clean; + // Cap by code UNIT, but back off one unit if the boundary splits a surrogate pair (a high surrogate with its + // low half beyond the cap) so the truncation never leaves a lone surrogate (a `�` in the error line / --json). + const lastUnit = clean.charCodeAt(MAX_APPROVAL_REASON_CHARS - 1); + const end = + lastUnit >= 0xd800 && lastUnit <= 0xdbff + ? MAX_APPROVAL_REASON_CHARS - 1 + : MAX_APPROVAL_REASON_CHARS; + return clean.slice(0, end); } /** diff --git a/apps/cli/src/render/tui/chat-store.ts b/apps/cli/src/render/tui/chat-store.ts index c5d4d1d1..da921e03 100644 --- a/apps/cli/src/render/tui/chat-store.ts +++ b/apps/cli/src/render/tui/chat-store.ts @@ -32,8 +32,9 @@ import { * rate is capped). */ -/** A pending per-tool approval the REPL renders as a `[y] yes / [a] always / [n] no / [esc] abort` prompt - * (ADR-0057, EA3/EA5). A reject-with-typed-reason (`[c]` comment) is a deferred follow-up. */ +/** A pending per-tool approval the REPL renders as a `[y] yes / [a] always / [n] no / [c] reason / [esc] abort` + * prompt (ADR-0057, EA3/EA5). `[c]` opens the typed-reason capture (Step 14) — a reject carrying WHY, via the + * `ToolApprovalDecision.reject.reason` seam. */ export interface PendingApproval { readonly request: ToolApprovalRequest; /** Whether an "always" answer will be remembered (accept-edits) — the prompt greys it out when false. */ diff --git a/docs/decisions/0057-cli-chat-modes-and-per-tool-approval.md b/docs/decisions/0057-cli-chat-modes-and-per-tool-approval.md index d1bda1a6..ed5db868 100644 --- a/docs/decisions/0057-cli-chat-modes-and-per-tool-approval.md +++ b/docs/decisions/0057-cli-chat-modes-and-per-tool-approval.md @@ -33,9 +33,10 @@ > driver and one-shot `agent run` (was hand-rolled in two places). > 3. **SCOPE-denial conversational recovery (a denial-taxonomy change).** The `recoverable` flag moves onto the base > `ToolDispatchError` (default false). Exactly two `tool_denied`s — a media scope denial (`media_scope_denied`) -> and the CLI fs arm's **pure scope-tier escape** — now carry `recoverable:true`, so on the interactive chat -> surface (`recoverToolFailures`) they are fed back as a correctable tool result and the model adapts to an -> in-bounds path (the floor still denies every attempt). A user / guardrail / SSRF / **confidentiality** +> and the CLI fs arm's **pure scope-tier escape** — now carry `recoverable:true`, so on the `recoverToolFailures` +> surfaces (the chat-read-write host — `relavium chat` / the Home / the one-shot `agent run`) they are fed back as +> a correctable tool result and the model adapts to an in-bounds path (the floor still denies every attempt; a +> WORKFLOW run never sets `recoverToolFailures`, so it stays fatal/deterministic). A user / guardrail / SSRF / **confidentiality** > (secret-store) / protected-path / symlink denial stays fatal — feeding those back would re-deny, risk a > re-execution, or leak a probe oracle. Canonical taxonomy: [../reference/shared-core/tool-registry.md](../reference/shared-core/tool-registry.md#error-taxonomy). > 4. **Ctrl+T through the approval swallow.** The fail-closed key-swallow now whitelists exactly the VIEW-ONLY diff --git a/docs/reference/shared-core/agent-runner.md b/docs/reference/shared-core/agent-runner.md index 6ff77f06..45670177 100644 --- a/docs/reference/shared-core/agent-runner.md +++ b/docs/reference/shared-core/agent-runner.md @@ -46,7 +46,7 @@ The error mapping to the closed `ErrorCode` ([error-handling.md](../../standards | Source | `ErrorCode` | Retryable | Note | |--------|-------------|-----------|------| | abort (`ctx.signal`) / `ToolCancelledError` / chain `cancelled` | `cancelled` | false | precedence over every other classification | -| `ToolPolicyError` | `tool_denied` | false | **not** fed back as a correctable result (re-asking a denied tool burns budget) | +| `ToolPolicyError` | `tool_denied` | false | **not** fed back as a correctable result (re-asking a denied tool burns budget) — EXCEPT `media_scope_denied`, a Step-14 `recoverable` SCOPE denial fed back on the interactive chat surface (`recoverToolFailures`); see the `recoverable` note in [tool-registry.md §error taxonomy](tool-registry.md#error-taxonomy) | | `UnknownToolError` / `ToolArgsInvalidError` | (model-correctable) | — | converted to an `isError` tool result fed back, within a bounded correction budget; after it ⇒ `tool_failed` | | `ToolExecutionError` | `tool_failed` | true | | | absent host capability (`ToolUnavailableError`) | `tool_unavailable` | false | a host/config gap — names the unwired arm actionably, never a bare `internal` (EA1, [ADR-0055](../../decisions/0055-cli-host-capability-seam-tool-environment-factory.md)) | diff --git a/docs/reference/shared-core/tool-registry.md b/docs/reference/shared-core/tool-registry.md index 6a277649..d2b15c8c 100644 --- a/docs/reference/shared-core/tool-registry.md +++ b/docs/reference/shared-core/tool-registry.md @@ -332,20 +332,26 @@ codes by [sse-event-schema.md](../contracts/sse-event-schema.md#error-code-taxon | *(AbortSignal abort)* | the run was cancelled mid-tool | `cancelled` | fatal (cancel path, not `tool_failed`) | **`recoverable` (the base `ToolDispatchError` flag; default `false`).** STRICTER than `retryable` (which lets a -*fresh node-retry* re-run the whole node): `recoverable` gates the interactive-chat WITHIN-TURN recovery -([ADR-0057](../../decisions/0057-cli-chat-modes-and-per-tool-approval.md) `recoverToolFailures`) — a recoverable -error is fed back to the model as an `isError` tool result so it can adapt (conversational recovery), while -everything else ends the turn. Two disjoint sources set it (a WORKFLOW node ignores the flag — fail-fast always): +*fresh node-retry* re-run the whole node): `recoverable` gates the WITHIN-TURN recovery on the +`recoverToolFailures` surfaces (the chat-read-write host — +[ADR-0057](../../decisions/0057-cli-chat-modes-and-per-tool-approval.md): `relavium chat` / the Home / the one-shot +`agent run`; a WORKFLOW node never sets `recoverToolFailures`, so it ignores the flag — fail-fast always) — a +recoverable error is fed back to the model as an `isError` tool result so it can adapt (conversational recovery), +while everything else ends the turn. Two disjoint sources set it: - `ToolExecutionError` — `true` only for an **idempotent read** (no `fs_write` / `process` / `egress` / `os` side effect), stamped at the wrap from `governedAction`; a governed / side-effecting failure stays `false` (fail-fast, so the model never re-attempts a non-idempotent side effect — a half-run command, a POST that may have landed). - A **SCOPE denial** (Step 14) — a `tool_denied` refused BEFORE any side effect, so it is safe to feed back and the model can adapt to an in-bounds path (the floor still denies every attempt). Exactly two carry it: the engine's `ToolPolicyError('media_scope_denied')`, and the CLI `fs` arm's **pure scope-tier escape** (`FsScopeDeniedError` - `'the path escapes the allowed filesystem scope'`). Every OTHER `tool_denied` stays fatal (`false`): a - `ToolDeniedByUserError` (user reject / fail-closed), the other `ToolPolicyError` guardrail reasons, an egress - **SSRF** denial, and the fs **confidentiality** (secret-store read) / protected-path / symlink refusals — feeding - those back would re-deny, risk a re-execution, or leak a probe oracle. + `'the path escapes the allowed filesystem scope'`, thrown by `assertInScope` before any write; the only residual + is an already-accepted parent-symlink-swap race whose sole preceding effect is an idempotent empty `mkdir`, never + the payload write). Every OTHER `tool_denied` stays fatal (`false`): a `ToolDeniedByUserError` (user reject / + fail-closed), the other `ToolPolicyError` guardrail reasons, an egress **SSRF** denial, and the fs + **confidentiality** (secret-store read) / protected-path / symlink refusals — feeding those back would re-deny, + risk a re-execution, or leak a probe oracle. (The scope-escape signal a model can gather across the bounded + `maxToolCorrections` rounds is a secret-free workspace-boundary probe, accepted — the reason interpolates no path + and the same round-bounded feedback already existed for an idempotent not-found read.) ## Instantiation diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index a3e0ce5c..69c0835c 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -527,11 +527,10 @@ Severity is the review's verified rating. Check an item off in the PR that resol expansion is deferred (ADR-0061). *(low · apps/cli/src/render/tui/mention.ts)* - [ ] **`@`-mention of a binary / media file.** The reader fail-closes on a binary file (parity with `read_file`); a durable media-handle injection path (ADR-0031) is a follow-up. *(low · apps/cli/src/render/tui/mention.ts)* -- [ ] **Strip Unicode bidi/format controls at the shared display boundary.** `formatMentionInjection` strips the - bidi range (U+202A–U+202E, U+2066–U+2069, LRM/RLM/ALM) from the injected PATH, but the shared - `stripTerminalControls` / `sanitizeInline` (chat-projection.ts) that render every transcript / candidate / prompt - string still pass bidi controls through — a general spoofing gap beyond `@`. Extending the shared sanitizer would - harden all surfaces at once. *(low · apps/cli/src/render/tui/chat-projection.ts)* +- [x] **Strip Unicode bidi/format controls at the shared display boundary — DONE (2.5-close Step 14, D-5).** The + shared `stripTerminalControls` (chat-projection.ts) now strips the Trojan-Source reordering family (U+202A–202E, + U+2066–2069, LRM/RLM/ALM) at every display boundary; ZWJ/ZWNJ preserved; the source uses `\u` escapes (no literal + bidi bytes). *(apps/cli/src/render/tui/chat-projection.ts)* ## Phase 2.5.E (chat modes + per-tool approval) follow-ups @@ -539,22 +538,31 @@ Severity is the review's verified rating. Check an item off in the PR that resol > reseat-less mode system + per-tool approval + mid-turn abort + the host arms shipped. These bounded pieces > were deliberately deferred (each is additive, none blocks the mode system's security guarantees): -- [ ] **`[c]` reject-with-typed-reason approval prompt.** The `ApprovalAnswer` reject variant already carries an - optional `reason` (surfaced in the `tool_denied` message), but the REPL prompt only wires `[y]`/`[a]`/`[n]`/`[esc]` - — a `[c]` that captures a free-text comment (a rejection with feedback for the model) is a bounded input-mode - addition to `reduceChatKey` + the ink/Home prompt. *(low · apps/cli/src/render/tui/chat-input.ts + chat-ink.tsx + home-controller.ts)* -- [ ] **Conversationally recover from a SCOPE denial in chat.** The PR #63 follow-up made an idempotent host - EXECUTION failure (a file-not-found READ) fed-back-recoverable via `ToolExecutionError.recoverable` - (`recoverToolFailures`), but a `tool_denied` SCOPE denial (reading a path outside the session workspace — a - common launch-cwd gotcha) still ends the turn (informative: the reason is shown). Feeding a *scope* denial back - so the model explains ("that path is outside my workspace — relaunch `chat` from …") would improve UX, but needs - a sub-class split so a SCOPE denial is distinguishable from an approval / protected-path denial (which must stay - fatal — never nag). *(low · packages/core/src/tools/errors.ts `HostDeniedError`; apps/cli/src/engine/tool-host/fs.ts; ADR-0057)* -- [ ] **Plain / non-TTY non-interactive approval policy.** The interactive `[y]/[a]/[n]` prompt is TTY/controller-only. - A non-TTY chat defaults to `ask` (which denies governed WITHOUT prompting, so no deadlock), and `auto` auto-approves - a non-protected target; the only unhandled niche is a non-TTY session switched to `accept-edits` (or `auto` hitting a - protected path) — its `requestApproval` would publish an unanswerable prompt. Add an explicit non-interactive policy - (deny, or a configured default) for the plain/`--json` driver so that niche is deterministic. *(low · apps/cli/src/commands/chat.ts drivePlain/driveJson)* +- [x] **`[c]` reject-with-typed-reason approval prompt — DONE (2.5-close Step 14, D-1).** A `[c]` at the approval + prompt opens a keyboard-owning reason-input sub-mode (both `relavium chat` + the Home); on submit it rejects with + the sanitized + 300-char-bounded reason via the existing `ToolApprovalDecision.reject.reason` seam. The floor is + unchanged. *(apps/cli/src/render/tui/chat-input.ts + chat-ink.tsx + home-controller.ts)* +- [x] **Conversationally recover from a SCOPE denial in chat — DONE (2.5-close Step 14, D-3).** The `recoverable` + flag moved to the base `ToolDispatchError`; exactly two `tool_denied`s opt in — `ToolPolicyError('media_scope_denied')` + and the fs **pure scope-tier escape** (`FsScopeDeniedError` from `assertInScope`) — so on the `recoverToolFailures` + surfaces the model is fed the denial and adapts to an in-bounds path. The confidentiality / protected-path / symlink + / SSRF / user / guardrail denials stay FATAL. *(packages/core/src/tools/errors.ts; apps/cli/src/engine/tool-host/fs.ts; ADR-0057)* +- [x] **Plain / non-TTY non-interactive approval policy — DONE (2.5.E "High 9" + consolidated in 2.5-close Step 14, + D-2).** A non-interactive driver (plain non-TTY / `--json` / one-shot `agent run`) uses the one canonical + `nonInteractiveApprovalPrompt` — every governed dispatch is DENIED (never a hang, never an auto-approve). + *(apps/cli/src/chat/chat-mode.ts; apps/cli/src/commands/chat.ts + agent-run.ts)* +- [ ] **Approval-consent-line zero-width hardening (2.5-close Step 14 security-review, optional).** The shared render + floor strips the Trojan-Source REORDERING controls everywhere (the CVE-2021-42574 vector is closed), but the + highest-trust surface — the approval consent line's target (`formatApprovalTarget` → `sanitizeInline`) — still + passes NON-reordering zero-width chars (ZWSP U+200B / word-joiner U+2060 / BOM U+FEFF), which the provider-URL echo + already strips. Running the stricter zero-width superset on the consent-line target would be defense-in-depth. Not + a CVE gap (those chars cannot reorder a command to masquerade). *(low · apps/cli/src/render/tui/chat-projection.ts)* +- [ ] **Extract the `[c]` reason-capture to a shared pure reducer (2.5-close Step 14 review, test-parity).** The + reason-capture keystroke glue is duplicated inline in `ChatApp` (chat-ink.tsx) and the Home controller; the shared + primitives (`reduceApprovalKey`, `sanitizeApprovalReason`, `reduceEditorMotion`) are unit-tested and the Home path + has an integration test, but the ChatApp inline copy has no direct test (matching the existing no-ChatApp-integration + boundary). Extracting the capture step to one pure reducer (like the mention/effort submodes) would let both + surfaces test the same function. *(low · apps/cli/src/render/tui/chat-ink.tsx + home-controller.ts)* - [ ] **Live `web_search` / `http_request` egress credential resolver.** `assembleToolEnv` accepts an `egressCredentialResolver` and the egress arm attaches it host-side as a Bearer, but the chat/Home session-host does not yet wire it to the keychain — so a `web_search` needing a provider key currently 401s (surfaced, never a crash). diff --git a/docs/standards/security-review.md b/docs/standards/security-review.md index aa3a2905..9407820d 100644 --- a/docs/standards/security-review.md +++ b/docs/standards/security-review.md @@ -348,12 +348,16 @@ security invariants** a review must confirm are: the one canonical `nonInteractiveApprovalPrompt` DENIES every governed dispatch — never a hang, never an auto-approve. - **A recoverable SCOPE denial is safe by construction.** Only two `tool_denied`s are fed back to - the model for in-turn conversational recovery (chat surface only): a media scope denial and the - fs **pure scope-tier escape** — both refused BEFORE any side effect, with a secret-free reason, - so the model adapts to an in-bounds path while the floor still denies every attempt (no bypass, - bounded by the correction budget). The **confidentiality** (secret-store read), protected-path, - symlink/hard-link, egress-SSRF, and user/guardrail denials stay FATAL — feeding those back would - re-deny, risk a re-execution, or leak a probe oracle. + the model for in-turn conversational recovery on the `recoverToolFailures` surfaces (the + chat-read-write host — `relavium chat` / the Home / the one-shot `agent run`; a WORKFLOW run stays + fatal/deterministic): a media scope denial and the fs **pure scope-tier escape** — both refused + BEFORE any side effect, with a **secret-free, path-free** reason, so the model adapts to an + in-bounds path while the floor still denies every attempt (no bypass, bounded by + `maxToolCorrections`). The scope-membership signal a probing model can gather over that bounded + window is an accepted, secret-free workspace-boundary oracle (the same round-bounded feedback + already existed for an idempotent not-found read). The **confidentiality** (secret-store read), + protected-path, symlink/hard-link, egress-SSRF, and user/guardrail denials stay FATAL — feeding + those back would re-deny, risk a re-execution, or leak a probe oracle. ## Never hand-roll crypto From 9aebbe551241759cd9b197e59bed05ba986e3598 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 06:42:14 +0300 Subject: [PATCH 46/50] =?UTF-8?q?docs,test:=202.5-close=20Step=2014=20Sonn?= =?UTF-8?q?et-review=20fixes=20=E2=80=94=20propagate=20the=20recoverToolFa?= =?UTF-8?q?ilures-surface=20phrasing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sonnet review of Batch D (0 blocker / 0 high; it independently re-traced the whole taxonomy + ran the full suites green). Applied: - MEDIUM (canonical-home consistency): the babd0e4 fixup corrected "interactive chat surface [only]" → the precise "recoverToolFailures surfaces (chat / Home / one-shot agent run)" in 3 docs but left 5 other spots (incl. the agent-runner.md row it wrote) saying "interactive chat surface". Propagated the precise phrasing to all five: ToolDispatchError.recoverable JSDoc (errors.ts), isRecoverableToolError JSDoc (agent-turn.ts), FsScopeDeniedError class doc (fs.ts), the agent-runner.md ladder row, and tool-registry.md's "Loop-correctable vs terminal" blockquote. Also fixed a duplicated {@link} the edit introduced in agent-turn.ts. - LOW: pinned `media_scope_denied`'s recoverability at its SOURCE — added `expect(err.recoverable).toBe(true)` to the builtins.test.ts read_media scope-denial case (previously only the agent-turn integration test covered it). Deferred to the close-out Final step (a whole-epic convention, not new to this batch): the commits' `Refs:` cite the untracked/scratch phase-2.5-close-plan.md — the Final phase-doc reconciliation + the PR body provide the durable record. Tests: 52 builtins cases pass (incl. the new assertion); core+cli typecheck, eslint, prettier green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 14, Batch D); ADR-0057 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/engine/tool-host/fs.ts | 5 +++-- docs/reference/shared-core/agent-runner.md | 2 +- docs/reference/shared-core/tool-registry.md | 2 +- packages/core/src/engine/agent-turn.ts | 5 +++-- packages/core/src/tools/builtins.test.ts | 3 +++ packages/core/src/tools/errors.ts | 3 ++- 6 files changed, 13 insertions(+), 7 deletions(-) diff --git a/apps/cli/src/engine/tool-host/fs.ts b/apps/cli/src/engine/tool-host/fs.ts index 3e3efcb3..b6757a04 100644 --- a/apps/cli/src/engine/tool-host/fs.ts +++ b/apps/cli/src/engine/tool-host/fs.ts @@ -96,8 +96,9 @@ export class FsCapabilityError extends HostCapabilityError {} * * Recoverability (Step 14): fatal by default (`recoverable=false`, inherited). ONE throw opts in — the pure * **scope-tier escape** ("the path escapes the allowed filesystem scope") passes `recoverable: true`, so on the - * interactive chat surface it is fed back as a correctable tool result and the model can adapt to an in-bounds - * path (conversational recovery). The **confidentiality** refusal (a secret/credential store read), the protected + * `recoverToolFailures` surface (chat / Home / one-shot `agent run`) it is fed back as a correctable tool result + * and the model can adapt to an in-bounds path (conversational recovery). The **confidentiality** refusal (a + * secret/credential store read), the protected * -path write, and the symlink/hard-link refusals stay fatal — feeding those back would leak a probe oracle or * risk nothing useful. */ diff --git a/docs/reference/shared-core/agent-runner.md b/docs/reference/shared-core/agent-runner.md index 45670177..b030dddf 100644 --- a/docs/reference/shared-core/agent-runner.md +++ b/docs/reference/shared-core/agent-runner.md @@ -46,7 +46,7 @@ The error mapping to the closed `ErrorCode` ([error-handling.md](../../standards | Source | `ErrorCode` | Retryable | Note | |--------|-------------|-----------|------| | abort (`ctx.signal`) / `ToolCancelledError` / chain `cancelled` | `cancelled` | false | precedence over every other classification | -| `ToolPolicyError` | `tool_denied` | false | **not** fed back as a correctable result (re-asking a denied tool burns budget) — EXCEPT `media_scope_denied`, a Step-14 `recoverable` SCOPE denial fed back on the interactive chat surface (`recoverToolFailures`); see the `recoverable` note in [tool-registry.md §error taxonomy](tool-registry.md#error-taxonomy) | +| `ToolPolicyError` | `tool_denied` | false | **not** fed back as a correctable result (re-asking a denied tool burns budget) — EXCEPT `media_scope_denied`, a Step-14 `recoverable` SCOPE denial fed back on the `recoverToolFailures` surfaces (chat / Home / one-shot `agent run`); see the `recoverable` note in [tool-registry.md §error taxonomy](tool-registry.md#error-taxonomy) | | `UnknownToolError` / `ToolArgsInvalidError` | (model-correctable) | — | converted to an `isError` tool result fed back, within a bounded correction budget; after it ⇒ `tool_failed` | | `ToolExecutionError` | `tool_failed` | true | | | absent host capability (`ToolUnavailableError`) | `tool_unavailable` | false | a host/config gap — names the unwired arm actionably, never a bare `internal` (EA1, [ADR-0055](../../decisions/0055-cli-host-capability-seam-tool-environment-factory.md)) | diff --git a/docs/reference/shared-core/tool-registry.md b/docs/reference/shared-core/tool-registry.md index d2b15c8c..4ef74db5 100644 --- a/docs/reference/shared-core/tool-registry.md +++ b/docs/reference/shared-core/tool-registry.md @@ -176,7 +176,7 @@ host is touched once, in the middle. 7. **Bound the model-facing result** (§Result bounding and spill-to-file) from the result via `ctx.limits` + the host `outputStore` — over the ceiling the model gets a preview + a spill handle, the full result still flows to `output_mapping`. 8. **Mark the result untrusted** (§Untrusted-data taint) and hand the structured `tool_call` / `tool_result` data + its taint/secret markers to the bus's single translation point ([ADR-0036](../../decisions/0036-run-loop-substrate-event-bus-and-execution-host.md)) for `agent:tool_call` / `agent:tool_result` emission. -> **Loop-correctable vs terminal.** `UnknownToolError` and `ToolArgsInvalidError` are **thrown** by the registry; the agent loop (1.O) **catches** them and synthesizes a correctable `isError` `tool_result` (from the secret-free `error.message`) so the model can fix its call, within a **bounded correction budget** it owns — escalating to a node `ErrorCode` only when that budget is spent. A `ToolPolicyError` — and, identically, a `ToolDeniedByUserError` (the per-tool approval denial, ADR-0057) — is structurally fatal (`tool_denied`) and **never** fed back as a correctable result (re-asking a denied tool just burns budget), with **one Step-14 exception**: a `recoverable` SCOPE denial (`media_scope_denied` / the fs pure scope-tier escape — refused before any side effect) IS fed back on the interactive chat surface (`recoverToolFailures`) so the model can adapt to an in-bounds path (see the `recoverable` note under the error taxonomy). See [agent-runner.md §the failure ladder](agent-runner.md). A `ToolCancelledError` maps to `cancelled` ahead of all other classifications (cancel wins). +> **Loop-correctable vs terminal.** `UnknownToolError` and `ToolArgsInvalidError` are **thrown** by the registry; the agent loop (1.O) **catches** them and synthesizes a correctable `isError` `tool_result` (from the secret-free `error.message`) so the model can fix its call, within a **bounded correction budget** it owns — escalating to a node `ErrorCode` only when that budget is spent. A `ToolPolicyError` — and, identically, a `ToolDeniedByUserError` (the per-tool approval denial, ADR-0057) — is structurally fatal (`tool_denied`) and **never** fed back as a correctable result (re-asking a denied tool just burns budget), with **one Step-14 exception**: a `recoverable` SCOPE denial (`media_scope_denied` / the fs pure scope-tier escape — refused before any side effect) IS fed back on the `recoverToolFailures` surfaces (chat / Home / one-shot `agent run`) so the model can adapt to an in-bounds path (see the `recoverable` note under the error taxonomy). See [agent-runner.md §the failure ladder](agent-runner.md). A `ToolCancelledError` maps to `cancelled` ahead of all other classifications (cancel wins). ```ts interface ToolDispatchContext { diff --git a/packages/core/src/engine/agent-turn.ts b/packages/core/src/engine/agent-turn.ts index faba2d7a..8b70cc07 100644 --- a/packages/core/src/engine/agent-turn.ts +++ b/packages/core/src/engine/agent-turn.ts @@ -246,8 +246,9 @@ export function codeForLlmError(error: LlmError): ErrorCode { /** * A tool throw the turn recovers by feeding the model an `isError` tool result (which increments the shared * `maxToolCorrections` budget) instead of ending the turn. Always the model's own syntactic mistakes - * (`unknown_tool` / `invalid_args`); PLUS — ONLY on the interactive chat surface (`limits.recoverToolFailures`, - * see {@link AgentTurnLimits.recoverToolFailures}) — any error the throwing class flagged `recoverable` + * (`unknown_tool` / `invalid_args`); PLUS — ONLY on the `recoverToolFailures` surfaces (chat / Home / one-shot + * `agent run`; a workflow node never sets it — see {@link AgentTurnLimits.recoverToolFailures}) — any error the + * throwing class flagged `recoverable` * ({@link ToolDispatchError.recoverable}): an IDEMPOTENT host execution failure (a read, stamped by the registry * from `governedAction`), OR a SCOPE denial refused BEFORE any side effect (a Step-14 fs scope-tier escape / a * media scope denial) so the model can adapt to an in-bounds path (conversational recovery). Everything else diff --git a/packages/core/src/tools/builtins.test.ts b/packages/core/src/tools/builtins.test.ts index 2697b8cd..55595a98 100644 --- a/packages/core/src/tools/builtins.test.ts +++ b/packages/core/src/tools/builtins.test.ts @@ -373,6 +373,9 @@ describe('read_media (1.AF/D12 — scope-set authz + Range gate)', () => { expect(err).toBeInstanceOf(ToolPolicyError); if (err instanceof ToolPolicyError) { expect(err.reason).toBe('media_scope_denied'); // narrow, never an unsafe `as` cast + // A media SCOPE denial is refused before any byte read, so it is RECOVERABLE (Step 14) — pinned at the + // source (the constructor ternary), not only via the agent-turn integration test. + expect(err.recoverable).toBe(true); } }); diff --git a/packages/core/src/tools/errors.ts b/packages/core/src/tools/errors.ts index 8dc2a7fe..cec7498a 100644 --- a/packages/core/src/tools/errors.ts +++ b/packages/core/src/tools/errors.ts @@ -33,7 +33,8 @@ export abstract class ToolDispatchError extends Error { readonly toolId?: ToolId; /** * Whether this failure is safe to FEED BACK to the model as an `isError` tool result for in-turn CONVERSATIONAL - * recovery (ADR-0057 `recoverToolFailures`, the interactive chat surface only) instead of ending the turn. + * recovery (the ADR-0057 `recoverToolFailures` surfaces — `relavium chat` / the Home / one-shot `agent run`; + * a WORKFLOW node never sets the flag) instead of ending the turn. * Default **false** (fatal — the safe direction; a forgotten opt-in just stays fatal). Set true by the specific * throwing class ONLY when re-attempting is safe: an IDEMPOTENT host execution failure (a read — no side-effect * hazard), or a SCOPE denial refused BEFORE any side effect (a Step-14 fs scope-tier escape / a media scope From bd84a38cb630a398622598959ae37a2d97718372 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 06:47:06 +0300 Subject: [PATCH 47/50] =?UTF-8?q?feat(cli):=202.5-close=20Step=2015=20(Bat?= =?UTF-8?q?ch=20E)=20=E2=80=94=20profile-aware=20advertise-filter?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make `wiredToolIds` (the ADR-0055 advertise-filter) read-only aware: with `{ readOnly: true }` it stops advertising a WRITE-class tool (`fsWrite`) whose fs arm IS wired but always denies the write on a read-only host — an always-denied advertisement — the advertise-side complement to the read-only fs arm's dispatch refusal (still authoritative). Read/list fs tools are unaffected; the default (read-write) is unchanged, so the live `chat-read-write` session host is inert. Closes the deferred "profile-unaware advertise-filter" item. Tests: +1 assemble case (readOnly drops write_file, keeps read/list; default keeps write_file). 15 tool-host tests pass; typecheck/eslint/prettier green. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 15, Batch E); ADR-0055 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/engine/tool-host/assemble.test.ts | 10 ++++++++++ apps/cli/src/engine/tool-host/assemble.ts | 14 +++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/apps/cli/src/engine/tool-host/assemble.test.ts b/apps/cli/src/engine/tool-host/assemble.test.ts index 672772d8..ec782869 100644 --- a/apps/cli/src/engine/tool-host/assemble.test.ts +++ b/apps/cli/src/engine/tool-host/assemble.test.ts @@ -190,6 +190,16 @@ describe('wiredToolIds (advertise-filter)', () => { expect(wiredToolIds(['write_file'], fsHost, defs)).toEqual(['write_file']); }); + it('profile-aware (readOnly): drops a WRITE tool but keeps read/list on a read-only host (Step 15)', () => { + // With `{ readOnly: true }` the filter stops advertising an always-denied write (its fs arm is wired but the + // read-only arm refuses the write), the complement to the dispatch refusal — a read/list fs tool is unaffected. + expect( + wiredToolIds(['read_file', 'list_directory', 'write_file'], fsHost, defs, { readOnly: true }), + ).toEqual(['read_file', 'list_directory']); + // Default (read-write) is unchanged — write_file stays advertised. + expect(wiredToolIds(['write_file'], fsHost, defs)).toEqual(['write_file']); + }); + it('routes MCP tools (discovered + the mcp_call built-in) to host.mcp, not host.egress', () => { expect(wiredToolIds(['mcp__server__read', 'mcp_call'], fsHost, defs)).toEqual([]); // no host.mcp ⇒ dropped expect(wiredToolIds(['mcp__server__read', 'mcp_call'], mcpHost, defs)).toEqual([ diff --git a/apps/cli/src/engine/tool-host/assemble.ts b/apps/cli/src/engine/tool-host/assemble.ts index 97d9f02f..a9b35b44 100644 --- a/apps/cli/src/engine/tool-host/assemble.ts +++ b/apps/cli/src/engine/tool-host/assemble.ts @@ -144,12 +144,14 @@ export function wiredToolIds( grantedIds: Iterable, host: ToolHost, defs: readonly ToolDef[], + opts?: { readonly readOnly?: boolean }, ): string[] { + const readOnly = opts?.readOnly ?? false; const byId = new Map(defs.map((d) => [d.id, d])); const out: string[] = []; for (const id of grantedIds) { const def = byId.get(id); - if (def === undefined || requiredArmPresent(def, host)) out.push(id); + if (def === undefined || requiredArmPresent(def, host, readOnly)) out.push(id); } return out; } @@ -159,10 +161,16 @@ export function wiredToolIds( * `policy.egress` *kind*: a discovered MCP tool and the `mcp_call` built-in both route via `host.mcp` (the * `mcp` egress kind is a guardrail label, not the arm), while `http`/`search` egress route via `host.egress`. * An armless tool (os/delegate) is kept and left to the dispatch `tool_unavailable` backstop (EA1). + * + * `readOnly` makes the filter **profile-aware** (Step 14/2.5.I close-out): on a read-only host a WRITE-class tool + * (`fsWrite`) is not merely armless — its `fs` arm IS wired but always DENIES the write, so it must never be + * offered to the model (an always-denied advertisement). Dropping it here is the advertise-side complement to the + * read-only `fs` arm's dispatch refusal (still authoritative). A read/list fs tool is unaffected. */ -function requiredArmPresent(def: ToolDef, host: ToolHost): boolean { +function requiredArmPresent(def: ToolDef, host: ToolHost, readOnly: boolean): boolean { if (def.source === 'mcp') return host.mcp !== undefined; // discovered MCP tools route via host.mcp - if (def.policy.fsScoped) return host.fs !== undefined; + if (def.policy.fsScoped) + return host.fs !== undefined && !(readOnly && def.policy.fsWrite === true); if (def.policy.spawnsProcess) return host.process !== undefined; if (def.policy.egress === 'mcp') return host.mcp !== undefined; // the `mcp_call` built-in also uses host.mcp if (def.policy.egress !== undefined) return host.egress !== undefined; // `http` / `search` → host.egress From 334e367438290a1653d9f81be5fa9d8e15dc5947 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 06:53:07 +0300 Subject: [PATCH 48/50] =?UTF-8?q?feat(cli):=202.5-close=20Step=2015=20(Bat?= =?UTF-8?q?ch=20E)=20=E2=80=94=20in-house=20.gitignore=20matcher=20for=20@?= =?UTF-8?q?-mention?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a dependency-free, ReDoS-safe `.gitignore` / `.relaviumignore` matcher (gitignore.ts) and fold it into the `@`-mention completion candidate filter so `@` no longer surfaces files the project already ignores. A UX/privacy nicety, NOT a security control — the confidentiality floor + listing-gate remain the authoritative fs-capability enforcement (a gitignore file is untrusted project data; the matcher can only HIDE more, never widen what the jail refuses). - The matcher supports the common subset: blank/`#`-comment lines, `!` negation (last match wins), trailing-`/` directory-only, leading-`/`/mid-`/` anchoring vs. any-depth basename match, and the `*` (in-segment) / `**` (cross-dir) / `?` globs. glob→regex is linear (no nested quantifiers → ReDoS-safe). Regex-special chars in a literal pattern are escaped. Documented subset limits (deferred, and only ever UNDER-hide — never a security gap): nested per-dir ignore files (only the workspace-root files are read) + `[a-z]` char classes. - `createMentionReader` lazily reads the root ignore files ONCE (best-effort — a missing/unreadable/refused file degrades to no rules, never a crash), caches the compiled matcher, and drops an ignored entry from each listing. An ignored DIRECTORY is dropped too, so its children are never listed (the per-entry filter naturally respects "an ignored dir hides everything under it"). Tests: +12 gitignore matcher cases (each pattern family + negation + layering + escaping) and +1 mention integration case (real .gitignore + .relaviumignore filter a listing, negation survives). 27 mention/gitignore tests pass; typecheck/eslint/prettier green. Marked the deferred entry done. Refs: docs/roadmap/phase-2.5-close-plan.md (Step 15, Batch E); ADR-0061 Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/cli/src/render/tui/gitignore.test.ts | 87 ++++++++++++++++ apps/cli/src/render/tui/gitignore.ts | 117 ++++++++++++++++++++++ apps/cli/src/render/tui/mention.test.ts | 21 ++++ apps/cli/src/render/tui/mention.ts | 38 +++++-- docs/roadmap/deferred-tasks.md | 12 ++- 5 files changed, 263 insertions(+), 12 deletions(-) create mode 100644 apps/cli/src/render/tui/gitignore.test.ts create mode 100644 apps/cli/src/render/tui/gitignore.ts diff --git a/apps/cli/src/render/tui/gitignore.test.ts b/apps/cli/src/render/tui/gitignore.test.ts new file mode 100644 index 00000000..a9521a95 --- /dev/null +++ b/apps/cli/src/render/tui/gitignore.test.ts @@ -0,0 +1,87 @@ +import { describe, expect, it } from 'vitest'; + +import { compileIgnore, parseIgnoreLines } from './gitignore.js'; + +describe('parseIgnoreLines', () => { + it('drops blank lines + # comments, trims trailing whitespace, tolerates CRLF', () => { + const body = ['# a comment', '', ' ', 'node_modules', 'dist/ ', 'a.txt\r'].join('\n'); + expect(parseIgnoreLines(body)).toEqual(['node_modules', 'dist/', 'a.txt']); + }); +}); + +describe('compileIgnore', () => { + it('an empty / all-comment body ignores nothing', () => { + const m = compileIgnore('# just a comment\n\n'); + expect(m.ignores('anything.ts', false)).toBe(false); + }); + + it('a basename pattern (no slash) matches at ANY depth', () => { + const m = compileIgnore('node_modules\n*.log'); + expect(m.ignores('node_modules', true)).toBe(true); + expect(m.ignores('packages/x/node_modules', true)).toBe(true); + expect(m.ignores('a.log', false)).toBe(true); + expect(m.ignores('deep/nested/b.log', false)).toBe(true); + expect(m.ignores('src/app.ts', false)).toBe(false); + }); + + it('an anchored pattern (leading /) matches only at the root', () => { + const m = compileIgnore('/dist'); + expect(m.ignores('dist', true)).toBe(true); + expect(m.ignores('packages/x/dist', true)).toBe(false); // not at root ⇒ not ignored + }); + + it('a mid-slash pattern is anchored to the root', () => { + const m = compileIgnore('build/output'); + expect(m.ignores('build/output', true)).toBe(true); + expect(m.ignores('sub/build/output', true)).toBe(false); // anchored, not any-depth + }); + + it('a directory-only pattern (trailing /) never matches a FILE of the same name', () => { + const m = compileIgnore('cache/'); + expect(m.ignores('cache', true)).toBe(true); // the directory is ignored + expect(m.ignores('cache', false)).toBe(false); // a file named `cache` is NOT + }); + + it('drops everything UNDER an ignored directory (prefix match)', () => { + const m = compileIgnore('node_modules'); + expect(m.ignores('node_modules/pkg/index.js', false)).toBe(true); + }); + + it('`*` stays within a segment; `**` crosses directories', () => { + const star = compileIgnore('/logs/*.txt'); + expect(star.ignores('logs/a.txt', false)).toBe(true); + expect(star.ignores('logs/sub/a.txt', false)).toBe(false); // `*` does not cross `/` + + const globstar = compileIgnore('logs/**/*.txt'); + expect(globstar.ignores('logs/a.txt', false)).toBe(true); + expect(globstar.ignores('logs/deep/nested/a.txt', false)).toBe(true); + }); + + it('`?` matches exactly one non-slash char', () => { + const m = compileIgnore('/file?.txt'); + expect(m.ignores('fileA.txt', false)).toBe(true); + expect(m.ignores('file.txt', false)).toBe(false); // needs the one extra char + expect(m.ignores('fileAB.txt', false)).toBe(false); // only one + }); + + it('`!` negation re-includes a path an earlier rule ignored (last match wins)', () => { + const m = compileIgnore('*.log\n!keep.log'); + expect(m.ignores('a.log', false)).toBe(true); + expect(m.ignores('keep.log', false)).toBe(false); // re-included by the negation + // Order matters — a later ignore re-ignores. + const reordered = compileIgnore('!keep.log\n*.log'); + expect(reordered.ignores('keep.log', false)).toBe(true); + }); + + it('layers multiple ignore bodies (later precedence, like .gitignore then .relaviumignore)', () => { + const m = compileIgnore('*.tmp', '!important.tmp'); + expect(m.ignores('scratch.tmp', false)).toBe(true); + expect(m.ignores('important.tmp', false)).toBe(false); // the second body's negation wins + }); + + it('escapes regex-special chars in a literal pattern (no accidental wildcard)', () => { + const m = compileIgnore('a.b+c'); // `.` and `+` must be literal, not regex + expect(m.ignores('a.b+c', false)).toBe(true); + expect(m.ignores('axbxc', false)).toBe(false); + }); +}); diff --git a/apps/cli/src/render/tui/gitignore.ts b/apps/cli/src/render/tui/gitignore.ts new file mode 100644 index 00000000..36b419f0 --- /dev/null +++ b/apps/cli/src/render/tui/gitignore.ts @@ -0,0 +1,117 @@ +/** + * A dependency-free `.gitignore` / `.relaviumignore` matcher for the `@`-mention completion (2.5.D, ADR-0061 — + * a deferred follow-up landed in the 2.5 close-out). It trims the candidate list so `@` does not surface files + * the project already ignores (build output, local scratch), matching the developer's mental model. + * + * NOT a security control — the confidentiality floor (a `.ssh`/`.env`/`.aws`/credential-store read/list refusal) + * is enforced SEPARATELY and unconditionally by the `fs` capability's listing-gate, regardless of this matcher. + * A gitignore file is untrusted project data (it can be authored/committed by anyone), so this only ever HIDES + * more entries; it can never widen what the fs jail already refuses. + * + * Supported (the common subset): blank lines + `#` comments; `!` negation (later rules override earlier — + * gitignore precedence); a trailing `/` (directory-only); a leading `/` OR an interior `/` anchors to the ignore + * file's root, else the pattern matches a path's BASENAME at any depth; the `*` (any run of non-`/`), `**` (across + * directories), and `?` (one non-`/`) globs. Deliberately NOT supported (documented subset, never a security gap + * since it only under-hides): nested per-directory `.gitignore` files (only the workspace-root files are read), + * `[a-z]` character classes, and a trailing-space / `\#` / `\!` escape. An unmatched candidate is simply shown. + */ + +/** One compiled ignore rule. */ +interface IgnoreRule { + readonly re: RegExp; + /** `!pattern` — a negation re-includes a path an earlier rule ignored (last match wins). */ + readonly negate: boolean; + /** A trailing `/` restricts the rule to directories. */ + readonly dirOnly: boolean; +} + +/** The compiled matcher: does `relPath` (a workspace-relative POSIX path) match an ignore rule? */ +export interface IgnoreMatcher { + ignores(relPath: string, isDir: boolean): boolean; +} + +/** Split an ignore-file body into its non-comment, non-blank pattern lines (CRLF-tolerant, trailing-ws trimmed). */ +export function parseIgnoreLines(text: string): string[] { + return text + .split('\n') + .map((line) => line.replace(/\r$/, '').replace(/\s+$/, '')) + .filter((line) => line.length > 0 && !line.startsWith('#')); +} + +/** Escape a literal char for embedding in a RegExp source. */ +function escapeRe(ch: string): string { + return /[.+^${}()|[\]\\]/.test(ch) ? `\\${ch}` : ch; +} + +/** + * Translate a gitignore glob (the pattern with any leading `/` and trailing `/` already stripped) to a RegExp + * source matching a whole path. `anchored` ⇒ match from the path start; else match at any segment boundary (so an + * unanchored `foo` matches `foo` and `a/foo`). The trailing `(?:/|$)` lets a pattern match a path PREFIX, so a + * directory pattern also ignores everything under it. + */ +function globToRegExp(glob: string, anchored: boolean): RegExp { + let src = ''; + for (let i = 0; i < glob.length; i += 1) { + const ch = glob[i]; + if (ch === '*') { + if (glob[i + 1] === '*') { + i += 1; // consumed the second '*' + if (glob[i + 1] === '/') { + i += 1; // consumed the '/' + src += '(?:.*/)?'; // `**/` — zero or more leading directories + } else { + src += '.*'; // `**` (crosses `/`) + } + } else { + src += '[^/]*'; // a single `*` stays within a segment + } + } else if (ch === '?') { + src += '[^/]'; + } else if (ch !== undefined) { + src += escapeRe(ch); + } + } + const head = anchored ? '^' : '(?:^|/)'; + return new RegExp(`${head}${src}(?:/|$)`); +} + +/** Compile a single pattern line into a rule, or `undefined` if it is empty after stripping the `!`. */ +function compileRule(raw: string): IgnoreRule | undefined { + const negate = raw.startsWith('!'); + let pattern = negate ? raw.slice(1) : raw; + const dirOnly = pattern.endsWith('/'); + if (dirOnly) pattern = pattern.slice(0, -1); + // Anchored if a `/` is at the start or the middle (a trailing `/` was already stripped above). + const anchored = pattern.startsWith('/') || pattern.includes('/'); + if (pattern.startsWith('/')) pattern = pattern.slice(1); + if (pattern.length === 0) return undefined; + return { re: globToRegExp(pattern, anchored), negate, dirOnly }; +} + +/** The always-ignored matcher (no rules) — nothing is ignored. */ +const NEVER: IgnoreMatcher = { ignores: () => false }; + +/** + * Compile one or more ignore-file bodies (e.g. `.gitignore` then `.relaviumignore`) into a matcher. The bodies + * are concatenated in order, so a later file's rules take precedence (last match wins), matching how git layers + * ignore sources. An empty/all-comment input yields the {@link NEVER} matcher. + */ +export function compileIgnore(...texts: string[]): IgnoreMatcher { + const rules = texts + .flatMap((text) => parseIgnoreLines(text)) + .map(compileRule) + .filter((rule): rule is IgnoreRule => rule !== undefined); + if (rules.length === 0) return NEVER; + return { + ignores(relPath, isDir) { + const path = relPath.replace(/^\.\//, '').replace(/\/+$/, ''); // normalize a leading `./` + a trailing `/` + if (path.length === 0) return false; + let ignored = false; + for (const rule of rules) { + if (rule.dirOnly && !isDir) continue; // a `foo/` rule never matches a file + if (rule.re.test(path)) ignored = !rule.negate; // last matching rule wins (negation re-includes) + } + return ignored; + }, + }; +} diff --git a/apps/cli/src/render/tui/mention.test.ts b/apps/cli/src/render/tui/mention.test.ts index 6b69ca66..bf0993fe 100644 --- a/apps/cli/src/render/tui/mention.test.ts +++ b/apps/cli/src/render/tui/mention.test.ts @@ -221,6 +221,27 @@ describe('createMentionReader — over the FsCapability jail (2.5.D step 4)', () expect(out.some((c) => c.name === '..')).toBe(false); // the ROOT has no ascend row }); + it('trims candidates by the workspace .gitignore / .relaviumignore (Step 15)', async () => { + const reader = createMentionReader( + fsMock( + { + '.': [ + { name: 'dist', type: 'directory' }, // /dist (gitignored dir) + { name: 'src', type: 'directory' }, + { name: 'secret.txt', type: 'file' }, // gitignored file + { name: 'app.ts', type: 'file' }, + { name: 'keep.tmp', type: 'file' }, // *.tmp but re-included by the .relaviumignore negation + { name: 'scratch.tmp', type: 'file' }, // *.tmp (relaviumignore) ⇒ dropped + ], + }, + { '.gitignore': '/dist\nsecret.txt', '.relaviumignore': '*.tmp\n!keep.tmp' }, + ), + ); + const out = await reader.list(''); + // dist / secret.txt / scratch.tmp filtered; keep.tmp survives (negation); dirs-first then by name. + expect(out.map((c) => c.name)).toEqual(['src', 'app.ts', 'keep.tmp']); + }); + it('list of a subdir prepends a `..` ascend row, builds nested paths; read goes through fs.readFile', async () => { const reader = createMentionReader( fsMock({ 'src/lib': [{ name: 'app.ts', type: 'file' }] }, { 'src/lib/app.ts': 'hello' }), diff --git a/apps/cli/src/render/tui/mention.ts b/apps/cli/src/render/tui/mention.ts index 8e54811e..fedc3cf6 100644 --- a/apps/cli/src/render/tui/mention.ts +++ b/apps/cli/src/render/tui/mention.ts @@ -1,6 +1,7 @@ import type { FsCapability } from '@relavium/core'; import { dropLastCodePoint } from './chat-input.js'; +import { compileIgnore, type IgnoreMatcher } from './gitignore.js'; import { frameUntrusted } from './injection.js'; // The `@`-injection content bounds + fence nonce — the shared injection primitives ({@link injection.ts}), @@ -189,10 +190,11 @@ export function foldMentionKey(char: string, key: MentionKey, state: MentionStat * The reader — a thin wrapper over the session's FsCapability (the ONE audited jail + floor). * -------------------------------------------------------------------------------------------------- */ -/** Directories always skipped from the completion candidate list (advisory noise — build output, VCS, deps). This - * fixed set is the **v1 advisory trim**; the ADR-0061 `.gitignore` / `.relaviumignore` matcher is a deferred - * follow-up (docs/roadmap/deferred-tasks.md) — NOT a security control (the confidentiality floor is enforced - * SEPARATELY by the fs listing-gate, `.git`/`.ssh`/`.env`/… never appear here regardless of this set). */ +/** Directories always skipped from the completion candidate list (advisory noise — build output, VCS, deps), + * regardless of any `.gitignore`. This fixed set complements the workspace `.gitignore` / `.relaviumignore` + * matcher (Step 15, {@link compileIgnore}) — it guarantees the universal noise dirs are trimmed even in a repo + * with no ignore file. NEITHER is a security control (the confidentiality floor is enforced SEPARATELY by the fs + * listing-gate, so `.git`/`.ssh`/`.env`/… never appear here regardless of this set or the ignore files). */ const NOISE_DIRS: ReadonlySet = new Set([ 'node_modules', 'dist', @@ -227,20 +229,42 @@ function joinRelative(dir: string, name: string): string { return dir.length === 0 ? name : `${dir}/${name}`; } -/** Build a {@link MentionReader} over an `FsCapability` — the read/list go through that one audited boundary. */ +/** Build a {@link MentionReader} over an `FsCapability` — the read/list go through that one audited boundary. The + * completion list is additionally trimmed by the workspace's `.gitignore` / `.relaviumignore` (Step 15) — a + * NOISE/privacy filter (never the security control; the confidentiality floor is enforced by the fs capability). */ export function createMentionReader(fs: FsCapability): MentionReader { + // Lazily read the root ignore files ONCE (best-effort — a missing/unreadable file degrades to no rules), then + // cache the compiled matcher for the reader's lifetime. `.relaviumignore` layers over `.gitignore` (last wins). + let ignorePromise: Promise | undefined; + const loadIgnore = (): Promise => { + ignorePromise ??= Promise.all( + ['.gitignore', '.relaviumignore'].map((name) => + fs.readFile(name, {}).then( + (file) => file.content, + () => '', // missing / unreadable / refused ⇒ no rules from that source (never a crash) + ), + ), + ).then((bodies) => compileIgnore(...bodies)); + return ignorePromise; + }; return { async list(dir) { // `''` lists the workspace root ('.'); the fs capability jails every path + the listing skips a sensitive // store (the listing-gate), so a `.ssh`/`.env`/`.aws` entry is never even offered. - const listing = await fs.listDirectory(dir.length === 0 ? '.' : dir, {}); + const [listing, ignore] = await Promise.all([ + fs.listDirectory(dir.length === 0 ? '.' : dir, {}), + loadIgnore(), + ]); const candidates = listing.entries .filter((entry) => !(entry.type === 'directory' && NOISE_DIRS.has(entry.name))) .map((entry) => ({ name: entry.name, type: entry.type, path: joinRelative(dir, entry.name), - })); + })) + // Drop a gitignored entry (an ignored DIRECTORY is dropped too, so its children are never listed — the + // per-entry filter naturally respects "an ignored dir hides everything under it"). + .filter((candidate) => !ignore.ignores(candidate.path, candidate.type === 'directory')); // Directories first, then case-insensitive by name — a stable, glanceable order. const sorted = [...candidates].sort((a, b) => { if (a.type !== b.type) return a.type === 'directory' ? -1 : 1; diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index 69c0835c..90d0cfd1 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -518,11 +518,13 @@ Severity is the review's verified rating. Check an item off in the PR that resol > untrusted injection) shipped. These bounded pieces were deliberately deferred (each is additive; the > confidentiality floor + jail + injection framing hold without them): -- [ ] **Advisory `.gitignore` / `.relaviumignore` completion trim.** The picker's advisory noise filter is a fixed - `NOISE_DIRS` set (`node_modules`, `dist`, `.git`, …) as the v1 trim; ADR-0061's promised in-house, ReDoS-safe - ignore-file matcher (fold a workspace `.gitignore` / `.relaviumignore` into the candidate filter, no new `ignore` - dependency) is deferred. It is a UX nicety, NOT a security control — the confidentiality floor + listing-gate are - enforced separately by the fs capability regardless. *(low · apps/cli/src/render/tui/mention.ts `NOISE_DIRS`; ADR-0061)* +- [x] **Advisory `.gitignore` / `.relaviumignore` completion trim — DONE (2.5-close Step 15, Batch E).** A + dependency-free, ReDoS-safe in-house matcher ({@link gitignore.ts}) folds the workspace-root `.gitignore` + + `.relaviumignore` into the `@`-mention candidate filter (comments/blanks, `!` negation, dir-only `/`, anchoring, + `*`/`**`/`?` globs; glob→regex is linear — no nested quantifiers), complementing the fixed `NOISE_DIRS` set. A + UX/privacy nicety, NOT a security control — the confidentiality floor + listing-gate remain the authoritative + fs-capability enforcement. Documented subset limits: nested per-dir ignore files + `[a-z]` char classes deferred + (they only UNDER-hide, never a security gap). *(apps/cli/src/render/tui/gitignore.ts + mention.ts; ADR-0061)* - [ ] **`@`-glob / directory expansion.** Single-file injection ships; `@src/**/*.ts` glob / whole-directory expansion is deferred (ADR-0061). *(low · apps/cli/src/render/tui/mention.ts)* - [ ] **`@`-mention of a binary / media file.** The reader fail-closes on a binary file (parity with `read_file`); From 3dffc8c2d40941989e532120910b2eb70769cb82 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 07:04:51 +0300 Subject: [PATCH 49/50] =?UTF-8?q?docs(roadmap):=202.5-close=20Final=20?= =?UTF-8?q?=E2=80=94=20mark=202.5.I/2.5.J=20+=20M2.5-4=20done;=20Phase=202?= =?UTF-8?q?.5=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reconcile every status surface now that the 2.5 close-out has landed: - 2.5.I (regression harness + DB concurrency hardening) and 2.5.J (docs-debt + NO_COLOR/FORCE_COLOR/--color) marked ✅ Done; milestone M2.5-4 reached; Phase 2.5 declared complete — in the phase-2.5 doc, current.md, and CLAUDE.md. - Recorded the doable-now Batch A–E backlog that landed alongside 2.5.I/J (each implement → Opus → Sonnet, a security-review pass on the ADR-0057 approval batch): A test-hardening, B 2.5.H TUI polish, C AgentParseError line/col, D the approval/ security batch (behind the append-only ADR-0057 amendment note), E the profile-aware advertise-filter + the in-house .gitignore matcher. - Recorded the DEFERRED follow-ups in deferred-tasks.md (maintainer call): the two gate.ts-resume items (`relavium budget resume` + secret re-provide — both refactor the security-sensitive cross-process resume path, best landed together with fresh context), the session {{ctx.*}} interpolation (Proposed ADR-0060, Phase-2.6), and extraRoots (blocked on its config source). Marked the four now-landed 2.5.E backlog entries + the bidi-strip + gitignore entries done. This gives the close-out a durable phase-doc record (the per-commit `Refs:` pointed at the now-deleted scratch close-plan). Full acceptance gate green: `pnpm turbo run lint typecheck test build` (24/24 tasks, 1641 CLI tests), `format:check`, `lint:fence-check`, `lint:engine-deps` all pass. Refs: docs/roadmap/phases/phase-2.5-cli-consolidation.md (2.5.I/2.5.J, M2.5-4) Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 13 +++++- docs/roadmap/current.md | 13 +++++- docs/roadmap/deferred-tasks.md | 16 +++++++- .../phases/phase-2.5-cli-consolidation.md | 41 +++++++++++++++++-- 4 files changed, 75 insertions(+), 8 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 47588ed1..e940bbc1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -117,8 +117,17 @@ plus the post-2.5.G model-UX follow-up ([ADR-0059](docs/decisions/0059-cli-mid-s [ADR-0066](docs/decisions/0066-normalized-reasoning-effort-control.md) reasoning-effort). With it **milestone M2.5-2** is reached. The additive lane **2.5.H** (reasoning render + live-turn feedback + an actionable error taxonomy — behind **EA6**, a dual-envelope `agent:reasoning` stream event that *amends* [ADR-0036](docs/decisions/0036-run-loop-substrate-event-bus-and-execution-host.md); -no new top-level ADR) is ✅ **Done (PR #67, 2026-07-07)**, reaching milestone **M2.5-3** with 2.5.E; the remaining additive -lanes 2.5.I / J run in parallel. +no new top-level ADR) is ✅ **Done (PR #67, 2026-07-07)**, reaching milestone **M2.5-3** with 2.5.E; and the consolidation +lanes **2.5.I** (regression harness + DB concurrency hardening: `loadFull` read-txn snapshot, `BEGIN IMMEDIATE` writes with a +deterministic `SQLITE_BUSY` retry, the concurrent chat+run + cassette-chain + perf-budget e2es, an advisory Windows CI lane) and +**2.5.J** (docs-debt: the accurate unencrypted-history posture per ADR-0050 + `NO_COLOR`/`FORCE_COLOR`/`--color` resolution) are +✅ **Done (2.5-close-out, 2026-07-08)** — **reaching milestone M2.5-4, so Phase 2.5 is complete** — landed with the doable-now +Batch A–E backlog (test-hardening; 2.5.H TUI polish; `AgentParseError` line/col; the ADR-0057 approval/security batch — `[c]` +reject-with-reason + non-TTY policy + SCOPE-denial recovery + Ctrl+T-in-approval + the Trojan-Source bidi floor, behind an +append-only ADR-0057 amendment; the profile-aware advertise-filter + the in-house `.gitignore` matcher), each implement → Opus → +Sonnet with a security-review pass on the approval batch. Deferred to a focused follow-up (both refactor the security-sensitive +`gate.ts` resume path): the `relavium budget resume` command + secret re-provide on gate resume; the session `{{ctx.*}}` +interpolation stays with the Proposed [ADR-0060](docs/decisions/0060-session-ctx-prompt-interpolation.md) (Phase-2.6). For live status, per-PR history, milestone dates, and open obligations, see the canonical home [docs/roadmap/current.md](docs/roadmap/current.md); [README.md](README.md) is the public overview. diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 95f668ec..3a84df32 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -202,7 +202,18 @@ host-emit of the reasoning the `@relavium/llm` seam already carries (ADR-0030), secret-free per-`ErrorCode` recovery hint (session-survives; a context-overflow message heuristic → `/compact`· `/trim`). Four steps, each opus + Sonnet reviewed (3 HIGH fixed: a run-path silent-drop, a frozen Home timer, a scrollback elision loss; + a one-shot `agent run` hint-leak). **With 2.5.E this reaches milestone M2.5-3.** The -remaining additive lanes 2.5.I / J run in parallel. See the +consolidation lanes **2.5.I** (regression harness + DB concurrency hardening — `loadFull` read-txn snapshot, +`BEGIN IMMEDIATE` writes with a deterministic `SQLITE_BUSY` retry, the concurrent chat+run + cassette-chain + +perf-budget e2es, an advisory Windows CI lane) and **2.5.J** (docs-debt: the accurate unencrypted-history +posture per ADR-0050, and `NO_COLOR`/`FORCE_COLOR`/`--color` resolution) are ✅ **Done (2.5-close-out, +2026-07-08)** — **reaching milestone M2.5-4, so Phase 2.5 is complete** — landed alongside the doable-now +Batch A–E backlog (test-hardening; 2.5.H TUI polish; `AgentParseError` line/col; the ADR-0057 approval/security +batch — `[c]` reject-with-reason, non-TTY policy, SCOPE-denial recovery, Ctrl+T-in-approval, the Trojan-Source +bidi floor, behind an append-only ADR-0057 amendment; the profile-aware advertise-filter + the in-house +`.gitignore` matcher), each implement → Opus → Sonnet with a security-review pass on the approval batch. Two +`gate.ts`-resume items (`relavium budget resume` + secret re-provide) are deferred to a focused follow-up, and +the session `{{ctx.*}}` interpolation stays with the Proposed ADR-0060 (Phase-2.6); all tracked in +[deferred-tasks.md](deferred-tasks.md). See the [Phase 2.5 workstreams](phases/phase-2.5-cli-consolidation.md). A **post-2.5.G model-UX follow-up** (from six maintainer questions on model/provider/wizard behavior) then landed as a sequenced plan: `/models` key-awareness, the onboarding-wizard live key-validation + retry UX, wizard-dynamic provider docs, **mid-session model switching** diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index 90d0cfd1..bcb8067a 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -573,6 +573,15 @@ Severity is the review's verified rating. Check an item off in the PR that resol abort + approval; the ADR-0028 session budget `pause_for_approval` can now ride the same machine (today a chat cost-cap trip settles the turn loudly as `budget_exceeded` — the REPL is the approval gate). See also the 1.V session-budget follow-up above. *(medium · apps/cli/src/chat + agent-session.ts)* +- [ ] **`relavium budget resume` CLI command (2.5-close Step 15 / Batch E — DEFERRED to a focused follow-up).** The + engine ALREADY supports resuming a budget-paused run (`engine.resume(runId, budgetGateId, decision)`, + budget-governor.ts / checkpoint.ts `isBudgetGate`), and `relavium gate` deliberately EXCLUDES budget gates + (`selectGate` filters `!isBudgetGate`), naming this the "`budget resume` surface." The remaining work is the + documented CLI command — a new manifest entry + dispatch handler + a command core that ~90% overlaps `gate.ts`'s + resume machinery (so the clean form extracts a shared resume core rather than duplicating). Low, dependency-free. + **Why deferred (maintainer call, 2026-07-08):** it modifies the security-sensitive `gate.ts` cross-process resume + path and is coupled to the secret-re-provide follow-up below (both refactor that path), so both are best landed + together with fresh context rather than at the tail of the 2.5-close session. *(low · apps/cli/src/commands/{gate,budget}.ts + manifest.ts + dispatch.ts; ADR-0028)* - [ ] **`project`-tier `extraRoots` allowlist (carried from 2.5.A).** The `project` fs tier behaves as workspace-only until the path-allowlist lands (it can only NARROW the jail, never open a hole). *(low · apps/cli/src/engine/tool-host/assemble.ts)* - [ ] **fs hard-link aliasing — the pnpm virtual-store read exemption (accepted residual, ADR-0057 review record).** @@ -665,7 +674,12 @@ Severity is the review's verified rating. Check an item off in the PR that resol value. The proper fix lets the operator re-supply the secret on resume (e.g. `relavium gate --secret token=…` read from stdin like `provider set-key`, or a keychain/env re-resolution keyed by the input `ref`) so a secret-bearing run becomes resumable. Until then the fail-closed + the - [commands.md](../reference/cli/commands.md) note stand. *(medium · apps/cli/src/commands/gate.ts; ADR-0006)* + [commands.md](../reference/cli/commands.md) note stand. **2.5-close Step 15 / Batch E status (maintainer call, + 2026-07-08):** selected IN by D8 but DEFERRED to a focused follow-up — this RELAXES a fail-closed security + guarantee (allow-with-re-provisioning), demands the stdin-not-argv secret discipline (`provider set-key` pattern) + + a mandatory security-review pass, and is coupled to the `budget resume` command above (both refactor the + `gate.ts` resume path). Best landed together with fresh context, not at the tail of the 2.5-close session. + *(medium · apps/cli/src/commands/gate.ts; ADR-0006)* ### 2.I read-command follow-ups (PR #48 multi-agent review, 2026-06-24) diff --git a/docs/roadmap/phases/phase-2.5-cli-consolidation.md b/docs/roadmap/phases/phase-2.5-cli-consolidation.md index a689fda0..939f38c1 100644 --- a/docs/roadmap/phases/phase-2.5-cli-consolidation.md +++ b/docs/roadmap/phases/phase-2.5-cli-consolidation.md @@ -12,7 +12,11 @@ > render + live-turn feedback + actionable errors, EA6 amending ADR-0036) ✅ **Done (PR #67, 2026-07-07)** — **milestone > M2.5-3 reached** (with 2.5.E). **2.5.G** (onboarding wizard + Home `/models` + the live model catalog) ✅ > **Done (PR #66, 2026-07-07)** behind [ADR-0063](../../decisions/0063-cli-config-write-contract.md)/[ADR-0064](../../decisions/0064-live-model-catalog.md)/[ADR-0065](../../decisions/0065-provider-economics-and-extensibility.md) -> (+ the ADR-0059/0066 model-UX follow-up) — **milestone M2.5-2 reached**. Remaining: the consolidation lanes 2.5.I / J. +> (+ the ADR-0059/0066 model-UX follow-up) — **milestone M2.5-2 reached**. The consolidation lanes **2.5.I** +> (harness + concurrency) and **2.5.J** (docs-debt + `NO_COLOR`/`FORCE_COLOR`/`--color`) ✅ **Done (2.5-close-out, +> 2026-07-08)** — **milestone M2.5-4 reached**, alongside the doable-now Batch A–E backlog (two `gate.ts`-resume +> items — `budget resume` + secret re-provide — deferred to a focused follow-up; `extraRoots` blocked on its +> config source). **Phase 2.5 is complete.** - **Related**: [../README.md](../README.md), [phase-2-cli.md](phase-2-cli.md), [phase-2.6-conversational-authoring.md](phase-2.6-conversational-authoring.md), [phase-3-desktop.md](phase-3-desktop.md), [../../reference/cli/commands.md](../../reference/cli/commands.md), [../../reference/cli/chat-session.md](../../reference/cli/chat-session.md), [../../reference/cli/regression-harness.md](../../reference/cli/regression-harness.md), [../../decisions/README.md](../../decisions/README.md) (ADR-0054–0057) @@ -513,7 +517,16 @@ follow-up) lands with EA2's accuracy surface. truncation is visible; each operational error class renders an actionable recovery hint with the session intact; the seam is not modified. -### 2.5.I — Regression harness and concurrency hardening +### 2.5.I — Regression harness and concurrency hardening — ✅ **Done (2.5-close, 2026-07-08)** + +> **Status:** ✅ **Done (2.5-close-out, 2026-07-08).** `loadFull` is wrapped in a single read transaction +> (torn-read guard); every multi-statement write txn is `BEGIN IMMEDIATE` with a bounded, deterministic +> `SQLITE_BUSY`/`SQLITE_LOCKED` retry (`withBusyRetry`, over the pre-existing WAL + `busy_timeout`); the +> concurrent chat+run e2e (two-connection coexistence + two-process contention), the `Home → chat → resume → +> export` cassette-chain e2e, and the query-shape perf-budget checks (EXPLAIN QUERY PLAN: index-served, no +> filesort) + the 80×24 degrade assertion are green; an advisory Windows CI lane runs the DB + harness suites + +> a headless no-TTY smoke. `regression-harness.md` documents them; ADR-0064 §5 records the DB write-path +> concurrency amendment. Each step went implement → Opus → Sonnet review. **Tasks:** extend the regression harness ([regression-harness.md](../../reference/cli/regression-harness.md)) to cover `Home → chat → resume → export` with an agent/chat replay cassette (the recorded-LLM replay @@ -532,7 +545,27 @@ Windows. performance budgets (Home cold-open at 1000 sessions; 80×24 minimum) hold; the harness gates the backward-compatibility exit criterion. -### 2.5.J — Documentation reconciliation and dead-code cleanup +### 2.5.J — Documentation reconciliation and dead-code cleanup — ✅ **Done (2.5-close, 2026-07-08)** + +> **Status:** ✅ **Done (2.5-close-out, 2026-07-08).** The "encrypted history" wording is corrected to the +> accurate CLI posture (unencrypted, protected by `0700`/`0600` + keychain, ADR-0050) across `uvp.md` / +> `vision.md` / the CLI tutorial + the README milestone spine; color resolution honors `--color` / +> `--no-color` > `NO_COLOR` (any non-empty ⇒ off) > `FORCE_COLOR` (`0`/`false` ⇒ off, the supports-color +> convention) > default-on, orthogonal to the `--json`/CI output mode, with a full test matrix; the roadmap +> status surfaces are reconciled (this doc, `current.md`, `CLAUDE.md`). +> +> **2.5-close-out backlog (Batches A–E), 2026-07-08.** The doable-now deferred items landed alongside 2.5.I/J, +> each implement → Opus → Sonnet (security items with a security-review pass first): **A** test-hardening +> (media-gc flake fix + a 2.5.A `git_status` union pin); **B** 2.5.H TUI polish (persistent Esc-to-stop abort +> hint + a line-bounded reasoning panel); **C** `AgentParseError` line/col (the session `{{ctx.*}}` half +> DEFERRED — governed by the Proposed ADR-0060, Phase-2.6); **D** the ADR-0057 approval/security batch +> (`[c]` reject-with-typed-reason, the consolidated non-TTY fail-closed policy, SCOPE-denial conversational +> recovery, the Ctrl+T-in-approval whitelist, the Trojan-Source bidi floor) — behind an append-only ADR-0057 +> amendment note; **E** the profile-aware advertise-filter + the in-house `.gitignore`/`.relaviumignore` +> matcher. **Deferred to a focused follow-up** (both refactor the security-sensitive `gate.ts` cross-process +> resume path, so best landed together with fresh context): the `relavium budget resume` command and the +> secret re-provide on gate resume; the `project`-tier `extraRoots` allowlist stays blocked on its config +> source. All tracked in [deferred-tasks.md](../deferred-tasks.md). **Tasks:** correct the surface-blind "encrypted history" wording (`docs/uvp.md`, `docs/vision.md`, `docs/tutorials/cli/start-a-chat-session.md`) to the accurate CLI posture — **unencrypted**, protected @@ -554,7 +587,7 @@ state; `NO_COLOR` is honoured. | M2.5-1 Secure base ✅ **(PR #60, 2026-06-28)** | 2.5.A | Root-cause closed (capability gap + merge asymmetry); host seam reviewed | | M2.5-2 Home + entry + onboarding ✅ **(PR #66, 2026-07-07)** | 2.5.B + 2.5.C + 2.5.D + 2.5.F + 2.5.G | First-class entry + ergonomics + onboarding | | M2.5-3 Modes + observability ✅ **(2026-07-07)** | 2.5.E + 2.5.H | Safe reseat-less mode system + per-tool approval + reasoning render + actionable errors | -| M2.5-4 Consolidation | 2.5.I + 2.5.J | Harness + concurrency + docs-debt | +| M2.5-4 Consolidation ✅ **(2.5-close, 2026-07-08)** | 2.5.I + 2.5.J | Harness + concurrency + docs-debt (+ the doable-now Batch A–E backlog; two `gate.ts`-resume items deferred to a follow-up) | ## Sequencing & parallelization From e08b8d0887e609f142d734a7716979f4a1c44c7c Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 8 Jul 2026 08:00:17 +0300 Subject: [PATCH 50/50] =?UTF-8?q?fix(cli,db,docs):=20PR=20#69=20review=20?= =?UTF-8?q?=E2=80=94=20gitignore=20ReDoS+dir-only,=20spawn-hang,=20docs,?= =?UTF-8?q?=20Sonar=20refactors?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address the PR #69 review findings (verified each against current code; fixed the still-valid ones, kept changes minimal): Correctness / security: - gitignore.ts: replace the glob→regex compiler with a LINEAR two-pointer glob matcher — closes THREE findings at once: (1) the ReDoS on untrusted `.gitignore` patterns (a crafted `**`-chain could backtrack super-linearly through the compiled regex; the matcher has no regex, so it is bounded/linear), (2) the directory-only bug (a `cache/` rule now correctly ignores FILES under it — `cache/a.txt` — not just the dir itself), and (3) the two SonarCloud super-linear-regex flags (no regex left). +2 tests (descendant match, and a 64-`**` × 200-segment pattern returns instantly). - concurrency.e2e.test.ts: the `runChild` `error` handler now calls `signalReady()` before rejecting, so a spawn failure can't leave `await child.ready` hanging. - chat-ink.tsx: `process.stdout?.columns` — optional-chain `stdout`, which can be undefined in a headless/redirected-stream harness (`.columns` on it would throw). Docs accuracy: - uvp.md: "API keys … never at rest" → "in the OS keychain — never in history.db or any plaintext store" (the keychain IS at-rest storage; the claim was too absolute). - README.md: Phase 2.5 status → complete (M2.5-4), not "only 2.5.I/J remain". - current.md: fix the MD028 (merge the Status/Last-updated header blockquotes); bump the date. - deferred-tasks.md: the `{@link gitignore.ts}` won't render in Markdown → a proper relative link. - security-review.md: trim the restated denial taxonomy to a summary + link the canonical tool-registry.md §error-taxonomy (one home). Strict-TS / Sonar (maintainability): - perf-budget.e2e.test.ts: replace two `as` casts at the SQLite boundary with runtime guards (`toBindValue` + `detailOf` via `in`-narrowing). - chat-projection.ts: `// NOSONAR` + rationale on the intentional `charCodeAt` (codePointAt would combine a surrogate pair, defeating the lone-surrogate check). - home-controller.ts: extract `routeReasonKey` (cognitive complexity 19 → under 15). - media-reference-store.ts + model-catalog-store.ts: hoist the `db.transaction` bodies into named factory-scoped helpers (function-nesting under the limit). Skipped: the phase-2.5-doc MD028 (pre-existing in the 2.5.G section, not this diff); model-catalog `upsert` "for consistency" (not Sonar-flagged — kept minimal). Gate: `pnpm turbo run lint typecheck test build` (24/24), format:check, fence, engine-deps — all green. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 8 +- apps/cli/src/harness/concurrency.e2e.test.ts | 5 +- apps/cli/src/harness/perf-budget.e2e.test.ts | 31 ++- apps/cli/src/render/tui/chat-ink.tsx | 6 +- apps/cli/src/render/tui/chat-projection.ts | 4 +- apps/cli/src/render/tui/gitignore.test.ts | 21 ++ apps/cli/src/render/tui/gitignore.ts | 153 ++++++++++---- apps/cli/src/render/tui/home-controller.ts | 50 ++--- docs/roadmap/current.md | 4 +- docs/roadmap/deferred-tasks.md | 7 +- docs/standards/security-review.md | 21 +- docs/uvp.md | 2 +- packages/db/src/media-reference-store.ts | 120 +++++------ packages/db/src/model-catalog-store.ts | 206 ++++++++++--------- 14 files changed, 379 insertions(+), 259 deletions(-) diff --git a/README.md b/README.md index e04f69ec..dc78c15f 100644 --- a/README.md +++ b/README.md @@ -133,10 +133,10 @@ proven by a real-spawn end-to-end test; and the **YAML-authoring lifecycle** — scaffolding an agent or a minimal single-agent workflow), `import`, and a share-safe `export` (re-serialized from the validated AST, no provider key by construction). With every in-phase workstream merged, the CLI is cut as **v0.1.1** (the public npm publish is the pending final maintainer step). **Phase 2.5 (CLI -Consolidation) is well underway** — the conversational Home, the slash-command system, reseat-less chat -modes with per-tool approval, context compaction, a live model catalog with provider economics, and -reasoning render have all landed; only the regression-harness/concurrency and documentation-consolidation -lanes remain. For live status and the full roadmap, see +Consolidation) is complete** (milestone **M2.5-4**, 2026-07-08) — the conversational Home, the slash-command +system, reseat-less chat modes with per-tool approval, context compaction, a live model catalog with provider +economics, reasoning render, and the closing consolidation lanes (regression harness + DB-concurrency +hardening + documentation reconciliation) have all landed. For live status and the full roadmap, see [docs/roadmap/current.md](docs/roadmap/current.md) and the [roadmap](docs/roadmap/README.md). diff --git a/apps/cli/src/harness/concurrency.e2e.test.ts b/apps/cli/src/harness/concurrency.e2e.test.ts index a38c3a1f..dd5a2765 100644 --- a/apps/cli/src/harness/concurrency.e2e.test.ts +++ b/apps/cli/src/harness/concurrency.e2e.test.ts @@ -109,7 +109,10 @@ function runChild(args: readonly string[]): SpawnedChild { stderr += chunk.toString('utf8'); }); const done = new Promise<{ code: number; stderr: string }>((resolve, reject) => { - child.on('error', reject); + child.on('error', (err) => { + signalReady(); // a spawn failure (ENOENT/EAGAIN) must ALSO release the ready-wait, else `await child.ready` hangs + reject(err); + }); child.on('close', (code) => { signalReady(); // a child that died before printing READY must not hang the parent's ready-wait resolve({ code: code ?? -1, stderr }); diff --git a/apps/cli/src/harness/perf-budget.e2e.test.ts b/apps/cli/src/harness/perf-budget.e2e.test.ts index 5d8caf72..74d63d98 100644 --- a/apps/cli/src/harness/perf-budget.e2e.test.ts +++ b/apps/cli/src/harness/perf-budget.e2e.test.ts @@ -33,14 +33,37 @@ describe('query-shape perf budgets (2.5.I S5) — the hot reads stay index-serve client.sqlite.close(); }); + /** A valid SQLite bind value — narrowed with a runtime guard (no unsafe cast at the DB boundary). */ + function toBindValue(value: unknown): string | number | bigint | Buffer | null { + if ( + value === null || + typeof value === 'string' || + typeof value === 'number' || + typeof value === 'bigint' || + Buffer.isBuffer(value) + ) { + return value; + } + return null; // EXPLAIN ignores the value; a non-primitive param (drizzle never emits one) binds as NULL + } + + /** Read the `detail` column off an EXPLAIN row via `in`-narrowing — no `as` cast. */ + function detailOf(row: unknown): string { + return typeof row === 'object' && + row !== null && + 'detail' in row && + typeof row.detail === 'string' + ? row.detail + : ''; + } + /** The `EXPLAIN QUERY PLAN` `detail` lines for a drizzle query (its `?`-placeholder SQL + bound params). */ function planFor(query: { toSQL: () => { sql: string; params: unknown[] } }): string[] { const { sql, params } = query.toSQL(); - // The drizzle `params` are valid SQLite bind values (strings/numbers); EXPLAIN ignores their values but the - // `?` placeholders must still be bound. Read the `detail` column defensively at the DB boundary. - const bind = params as ReadonlyArray; + // The `?` placeholders must still be bound (EXPLAIN ignores their values); validate each at the DB boundary. + const bind = params.map(toBindValue); const rows = client.sqlite.prepare(`EXPLAIN QUERY PLAN ${sql}`).all(...bind); - return rows.map((row) => (row as { detail?: string }).detail ?? ''); + return rows.map(detailOf); } /** diff --git a/apps/cli/src/render/tui/chat-ink.tsx b/apps/cli/src/render/tui/chat-ink.tsx index f681fa08..2ce2e672 100644 --- a/apps/cli/src/render/tui/chat-ink.tsx +++ b/apps/cli/src/render/tui/chat-ink.tsx @@ -1151,8 +1151,10 @@ export function ChatApp(props: Readonly): ReactElement { // Live terminal width for the reasoning-panel row bound (2.5.H) — read fresh each render (parity with // `nowMs={Date.now()}`); the frame loop re-renders, so a resize is picked up on the next tick. // `stdout.columns` is typed `number` but is `undefined` at runtime off a TTY — the formatter's 80-col - // fallback covers that (moot here anyway: ChatApp only mounts on a TTY via the driveInk gate). - columns={process.stdout.columns} + // fallback covers that (moot here anyway: ChatApp only mounts on a TTY via the driveInk gate). Optional-chain + // `process.stdout` too: it can be undefined in a headless/redirected-stream harness, and `.columns` on + // `undefined` would throw. + columns={process.stdout?.columns} reasonDraft={reasonDraft} paletteOpen={ palette !== undefined || diff --git a/apps/cli/src/render/tui/chat-projection.ts b/apps/cli/src/render/tui/chat-projection.ts index a9459975..5c30ca4e 100644 --- a/apps/cli/src/render/tui/chat-projection.ts +++ b/apps/cli/src/render/tui/chat-projection.ts @@ -82,7 +82,9 @@ export function sanitizeApprovalReason(text: string): string | undefined { if (clean.length <= MAX_APPROVAL_REASON_CHARS) return clean; // Cap by code UNIT, but back off one unit if the boundary splits a surrogate pair (a high surrogate with its // low half beyond the cap) so the truncation never leaves a lone surrogate (a `�` in the error line / --json). - const lastUnit = clean.charCodeAt(MAX_APPROVAL_REASON_CHARS - 1); + // charCodeAt (NOT codePointAt) is REQUIRED here: we must see the raw UTF-16 unit to detect a LONE high surrogate; + // codePointAt would combine the pair into one code point, defeating the split-detection. + const lastUnit = clean.charCodeAt(MAX_APPROVAL_REASON_CHARS - 1); // NOSONAR — charCodeAt is intentional (lone-surrogate detection) const end = lastUnit >= 0xd800 && lastUnit <= 0xdbff ? MAX_APPROVAL_REASON_CHARS - 1 diff --git a/apps/cli/src/render/tui/gitignore.test.ts b/apps/cli/src/render/tui/gitignore.test.ts index a9521a95..07e93cf8 100644 --- a/apps/cli/src/render/tui/gitignore.test.ts +++ b/apps/cli/src/render/tui/gitignore.test.ts @@ -47,6 +47,27 @@ describe('compileIgnore', () => { expect(m.ignores('node_modules/pkg/index.js', false)).toBe(true); }); + it('a directory-only pattern ignores FILES under the directory (the descendant fix)', () => { + // `cache/` ignores the dir AND everything beneath it — a FILE `cache/a.txt` (isDir=false) must still match, + // even though the rule is directory-only (the matched `cache` segment is a directory, being a mid-path segment). + const m = compileIgnore('cache/'); + expect(m.ignores('cache/a.txt', false)).toBe(true); + expect(m.ignores('cache/sub/b.txt', false)).toBe(true); + expect(m.ignores('cache', true)).toBe(true); // the dir itself + expect(m.ignores('cache', false)).toBe(false); // a FILE named `cache` is NOT dir-only-matched + }); + + it('stays fast on an adversarial many-`**` pattern + a long non-matching path (no ReDoS)', () => { + // A regex `.*` chain would backtrack super-linearly here; the linear two-pointer matcher returns immediately. + const pattern = `${'**/'.repeat(64)}zzz`; + const path = `${'a/'.repeat(200)}b.ts`; + const m = compileIgnore(pattern); + const start = process.hrtime.bigint(); + expect(m.ignores(path, false)).toBe(false); // no `zzz` segment ⇒ no match + const elapsedMs = Number(process.hrtime.bigint() - start) / 1e6; + expect(elapsedMs).toBeLessThan(50); // linear — nowhere near a backtracking stall + }); + it('`*` stays within a segment; `**` crosses directories', () => { const star = compileIgnore('/logs/*.txt'); expect(star.ignores('logs/a.txt', false)).toBe(true); diff --git a/apps/cli/src/render/tui/gitignore.ts b/apps/cli/src/render/tui/gitignore.ts index 36b419f0..713dc2a4 100644 --- a/apps/cli/src/render/tui/gitignore.ts +++ b/apps/cli/src/render/tui/gitignore.ts @@ -6,23 +6,32 @@ * NOT a security control — the confidentiality floor (a `.ssh`/`.env`/`.aws`/credential-store read/list refusal) * is enforced SEPARATELY and unconditionally by the `fs` capability's listing-gate, regardless of this matcher. * A gitignore file is untrusted project data (it can be authored/committed by anyone), so this only ever HIDES - * more entries; it can never widen what the fs jail already refuses. + * more entries; it can never widen what the fs jail already refuses. Because the input is untrusted, matching is + * done with a LINEAR two-pointer glob algorithm (no regex → no super-linear backtracking / ReDoS on a crafted + * pattern), not by compiling patterns to regexes. * * Supported (the common subset): blank lines + `#` comments; `!` negation (later rules override earlier — - * gitignore precedence); a trailing `/` (directory-only); a leading `/` OR an interior `/` anchors to the ignore - * file's root, else the pattern matches a path's BASENAME at any depth; the `*` (any run of non-`/`), `**` (across - * directories), and `?` (one non-`/`) globs. Deliberately NOT supported (documented subset, never a security gap - * since it only under-hides): nested per-directory `.gitignore` files (only the workspace-root files are read), - * `[a-z]` character classes, and a trailing-space / `\#` / `\!` escape. An unmatched candidate is simply shown. + * gitignore precedence); a trailing `/` (directory-only, matched correctly for the dir AND everything under it); + * a leading `/` OR an interior `/` anchors to the ignore file's root, else the pattern matches a path's BASENAME + * at any depth; the `*` (any run of non-`/`), `**` (a whole path segment crossing directories), and `?` (one + * non-`/`) globs. Deliberately NOT supported (documented subset, never a security gap since it only under-hides): + * nested per-directory `.gitignore` files (only the workspace-root files are read), `[a-z]` character classes, + * and a trailing-space / `\#` / `\!` escape. An unmatched candidate is simply shown. */ +/** A doublestar path-segment sentinel (`**`) — matches zero or more whole path segments. */ +const DOUBLESTAR = '**'; + /** One compiled ignore rule. */ interface IgnoreRule { - readonly re: RegExp; + /** The pattern split into `/`-segments; a segment equal to {@link DOUBLESTAR} is the cross-dir wildcard. */ + readonly segments: readonly string[]; + /** `/foo` or `a/b` — the pattern is anchored to the ignore-file root (else it matches a basename at any depth). */ + readonly anchored: boolean; + /** A trailing `/` restricts the rule to directories (and, transitively, everything under them). */ + readonly dirOnly: boolean; /** `!pattern` — a negation re-includes a path an earlier rule ignored (last match wins). */ readonly negate: boolean; - /** A trailing `/` restricts the rule to directories. */ - readonly dirOnly: boolean; } /** The compiled matcher: does `relPath` (a workspace-relative POSIX path) match an ignore rule? */ @@ -34,45 +43,78 @@ export interface IgnoreMatcher { export function parseIgnoreLines(text: string): string[] { return text .split('\n') - .map((line) => line.replace(/\r$/, '').replace(/\s+$/, '')) + .map((line) => line.trimEnd()) // strips a trailing `\r` (CRLF) + trailing spaces; leaves the pattern body .filter((line) => line.length > 0 && !line.startsWith('#')); } -/** Escape a literal char for embedding in a RegExp source. */ -function escapeRe(ch: string): string { - return /[.+^${}()|[\]\\]/.test(ch) ? `\\${ch}` : ch; +/** + * Whether a single glob SEGMENT (`*` = any run of non-`/`, `?` = one non-`/`, else literal) matches one path + * segment — the classic iterative wildcard match with a single star-backtrack. O(n·m) worst, linear typical, and + * critically NON-backtracking-exponential: there is no regex and no nested quantifier, so an adversarial pattern + * cannot make it super-linear beyond the bounded segment lengths. + */ +function segmentMatches(glob: string, seg: string): boolean { + let g = 0; + let s = 0; + let starG = -1; + let starS = 0; + while (s < seg.length) { + if (g < glob.length && (glob[g] === '?' || glob[g] === seg[s])) { + g += 1; + s += 1; + } else if (g < glob.length && glob[g] === '*') { + starG = g; // remember the `*` so we can extend how much it consumes on a later mismatch + starS = s; + g += 1; + } else if (starG !== -1) { + g = starG + 1; // backtrack: let the last `*` absorb one more char + s = starS + 1; + starS += 1; + } else { + return false; + } + } + while (g < glob.length && glob[g] === '*') g += 1; // trailing `*`(s) match the empty remainder + return g === glob.length; } /** - * Translate a gitignore glob (the pattern with any leading `/` and trailing `/` already stripped) to a RegExp - * source matching a whole path. `anchored` ⇒ match from the path start; else match at any segment boundary (so an - * unanchored `foo` matches `foo` and `a/foo`). The trailing `(?:/|$)` lets a pattern match a path PREFIX, so a - * directory pattern also ignores everything under it. + * Match an ANCHORED pattern (its `/`-segments, with `**` sentinels) against the path segments, starting at the + * root — a two-pointer walk where `**` matches zero+ segments (backtracking a WHOLE segment at a time, never a + * character, so this is bounded/linear-ish, never exponential). Returns whether the pattern is fully consumed + * (a match, with prefix semantics — a matched dir also matches everything under it) and, if so, whether it ended + * BEFORE the path end (⇒ the matched thing is a directory, which the dir-only rule needs). */ -function globToRegExp(glob: string, anchored: boolean): RegExp { - let src = ''; - for (let i = 0; i < glob.length; i += 1) { - const ch = glob[i]; - if (ch === '*') { - if (glob[i + 1] === '*') { - i += 1; // consumed the second '*' - if (glob[i + 1] === '/') { - i += 1; // consumed the '/' - src += '(?:.*/)?'; // `**/` — zero or more leading directories - } else { - src += '.*'; // `**` (crosses `/`) - } - } else { - src += '[^/]*'; // a single `*` stays within a segment - } - } else if (ch === '?') { - src += '[^/]'; - } else if (ch !== undefined) { - src += escapeRe(ch); +function anchoredMatch( + segs: readonly string[], + pathSegs: readonly string[], +): { matched: boolean; endsBeforeEnd: boolean } { + let pi = 0; + let si = 0; + let starPi = -1; + let starSi = 0; + while (pi < segs.length) { + const seg = segs[pi]; + if (seg === DOUBLESTAR) { + starPi = pi; // `**` matches zero+ segments; record a resume point and try zero first + starSi = si; + pi += 1; + } else if ( + si < pathSegs.length && + seg !== undefined && + segmentMatches(seg, pathSegs[si] ?? '') + ) { + pi += 1; + si += 1; + } else if (starPi !== -1 && starSi < pathSegs.length) { + starSi += 1; // backtrack: let the last `**` absorb one more whole segment + si = starSi; + pi = starPi + 1; + } else { + return { matched: false, endsBeforeEnd: false }; } } - const head = anchored ? '^' : '(?:^|/)'; - return new RegExp(`${head}${src}(?:/|$)`); + return { matched: true, endsBeforeEnd: si < pathSegs.length }; } /** Compile a single pattern line into a rule, or `undefined` if it is empty after stripping the `!`. */ @@ -85,10 +127,30 @@ function compileRule(raw: string): IgnoreRule | undefined { const anchored = pattern.startsWith('/') || pattern.includes('/'); if (pattern.startsWith('/')) pattern = pattern.slice(1); if (pattern.length === 0) return undefined; - return { re: globToRegExp(pattern, anchored), negate, dirOnly }; + return { segments: pattern.split('/'), anchored, dirOnly, negate }; +} + +/** Whether one compiled rule matches the split path. Unanchored (single-segment) rules match a basename at any depth. */ +function ruleMatches(rule: IgnoreRule, pathSegs: readonly string[], isDir: boolean): boolean { + if (rule.anchored) { + const { matched, endsBeforeEnd } = anchoredMatch(rule.segments, pathSegs); + // dir-only fails ONLY when the match ends exactly at the path end (the LAST segment IS the candidate) and the + // candidate is a file; a match ending earlier means the matched segment is a directory, so dir-only holds. + return matched && (!rule.dirOnly || endsBeforeEnd || isDir); + } + // Unanchored: a single glob segment matches ANY path segment (then it + everything under it — prefix). dir-only + // holds for a non-last matched segment (always a directory) or, on the last segment, only when isDir. + const glob = rule.segments[0] ?? ''; + for (let i = 0; i < pathSegs.length; i += 1) { + if (segmentMatches(glob, pathSegs[i] ?? '')) { + const isLast = i === pathSegs.length - 1; + if (!rule.dirOnly || !isLast || isDir) return true; + } + } + return false; } -/** The always-ignored matcher (no rules) — nothing is ignored. */ +/** The always-false matcher (no rules) — nothing is ignored. */ const NEVER: IgnoreMatcher = { ignores: () => false }; /** @@ -104,12 +166,13 @@ export function compileIgnore(...texts: string[]): IgnoreMatcher { if (rules.length === 0) return NEVER; return { ignores(relPath, isDir) { - const path = relPath.replace(/^\.\//, '').replace(/\/+$/, ''); // normalize a leading `./` + a trailing `/` - if (path.length === 0) return false; + // Normalize a leading `./` + collapse a trailing `/`, then split into segments (dropping empty ones). + const normalized = relPath.startsWith('./') ? relPath.slice(2) : relPath; + const pathSegs = normalized.split('/').filter((seg) => seg.length > 0); + if (pathSegs.length === 0) return false; let ignored = false; for (const rule of rules) { - if (rule.dirOnly && !isDir) continue; // a `foo/` rule never matches a file - if (rule.re.test(path)) ignored = !rule.negate; // last matching rule wins (negation re-includes) + if (ruleMatches(rule, pathSegs, isDir)) ignored = !rule.negate; // last matching rule wins (negation re-includes) } return ignored; }, diff --git a/apps/cli/src/render/tui/home-controller.ts b/apps/cli/src/render/tui/home-controller.ts index 888bec27..8c88201b 100644 --- a/apps/cli/src/render/tui/home-controller.ts +++ b/apps/cli/src/render/tui/home-controller.ts @@ -1324,35 +1324,35 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { } }; - const handleChatKey = (active: HomeChatSession, input: string, key: ChatKey): void => { - if (tearingDown === active) return; // a key arriving mid-teardown must not drive sendMessage on a cancelled session - // The `[c]` typed-reason capture (Step 14) owns the keyboard while open — checked FIRST. It ONLY opens from a - // pending approval; if that approval settled out-of-band, the capture is stale → drop it. Else Esc CANCELS back - // to the [y]/[a]/[n] prompt (still pending — not an abort); plain Enter rejects WITH the sanitized+bounded - // reason; every other key edits the buffer. Parity with the standalone ChatApp; the floor is unchanged (this - // only enriches a reject — a governed dispatch still cannot proceed without an explicit decision). + // The `[c]` typed-reason capture (Step 14) owns the keyboard while open — returns `true` when it handled the key. + // It ONLY opens from a pending approval; if that approval settled out-of-band, the capture is stale → drop it. + // Esc CANCELS back to the [y]/[a]/[n] prompt (still pending — not an abort); plain Enter rejects WITH the + // sanitized+bounded reason; every other key edits the buffer. Parity with the standalone ChatApp; the floor is + // unchanged (this only enriches a reject — a governed dispatch still cannot proceed without an explicit decision). + // Extracted (like routeMentionKey/routeSearchKey) so handleChatKey stays flat. + const routeReasonKey = (active: HomeChatSession, input: string, key: ChatKey): boolean => { const openReason = state.reasonDraft; - if (openReason !== undefined) { - if (active.store.getSnapshot().approval === undefined) { - set({ reasonDraft: undefined }); // the approval vanished — discard the orphaned capture - return; - } - if (key.escape === true) { - set({ reasonDraft: undefined }); // cancel the reason; the approval stays pending - return; - } - if (key.return === true && key.shift !== true) { - const reason = sanitizeApprovalReason(openReason.text); - set({ reasonDraft: undefined }); - active.store.answerApproval( - reason === undefined ? { outcome: 'reject' } : { outcome: 'reject', reason }, - ); - return; - } + if (openReason === undefined) return false; + if (active.store.getSnapshot().approval === undefined) { + set({ reasonDraft: undefined }); // the approval vanished — discard the orphaned capture + } else if (key.escape === true) { + set({ reasonDraft: undefined }); // cancel the reason; the approval stays pending + } else if (key.return === true && key.shift !== true) { + const reason = sanitizeApprovalReason(openReason.text); + set({ reasonDraft: undefined }); + active.store.answerApproval( + reason === undefined ? { outcome: 'reject' } : { outcome: 'reject', reason }, + ); + } else { const edit = reduceEditorMotion(input, key); if (edit !== undefined) set({ reasonDraft: applyEditorAction(openReason, edit) }); - return; } + return true; // while the capture is open it OWNS every key + }; + + const handleChatKey = (active: HomeChatSession, input: string, key: ChatKey): void => { + if (tearingDown === active) return; // a key arriving mid-teardown must not drive sendMessage on a cancelled session + if (routeReasonKey(active, input, key)) return; // the `[c]` typed-reason capture owns the keyboard while open // Busy = a streaming turn OR a `!`-shell command in flight (`state.shellBusy` — the session has no store status // for it). A gated keystroke can't reach `sendMessage` → no `SessionStateError` crash. const running = diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 3a84df32..591a7970 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -1,8 +1,8 @@ # Current state > Status: Living - -> Last updated: 2026-07-03 +> +> Last updated: 2026-07-08 - **Related**: [README.md](README.md), [phases/phase-2.5-cli-consolidation.md](phases/phase-2.5-cli-consolidation.md), [phases/phase-2-cli.md](phases/phase-2-cli.md), [deferred-tasks.md](deferred-tasks.md), [../project-structure.md](../project-structure.md), [../tech-stack.md](../tech-stack.md) diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index bcb8067a..4fec707f 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -519,9 +519,10 @@ Severity is the review's verified rating. Check an item off in the PR that resol > confidentiality floor + jail + injection framing hold without them): - [x] **Advisory `.gitignore` / `.relaviumignore` completion trim — DONE (2.5-close Step 15, Batch E).** A - dependency-free, ReDoS-safe in-house matcher ({@link gitignore.ts}) folds the workspace-root `.gitignore` + - `.relaviumignore` into the `@`-mention candidate filter (comments/blanks, `!` negation, dir-only `/`, anchoring, - `*`/`**`/`?` globs; glob→regex is linear — no nested quantifiers), complementing the fixed `NOISE_DIRS` set. A + dependency-free, ReDoS-safe in-house matcher ([gitignore.ts](../../apps/cli/src/render/tui/gitignore.ts)) folds + the workspace-root `.gitignore` + `.relaviumignore` into the `@`-mention candidate filter (comments/blanks, `!` + negation, dir-only `/`, anchoring, `*`/`**`/`?` globs; a LINEAR two-pointer glob matcher — no regex, so no + backtracking/ReDoS on a crafted pattern), complementing the fixed `NOISE_DIRS` set. A UX/privacy nicety, NOT a security control — the confidentiality floor + listing-gate remain the authoritative fs-capability enforcement. Documented subset limits: nested per-dir ignore files + `[a-z]` char classes deferred (they only UNDER-hide, never a security gap). *(apps/cli/src/render/tui/gitignore.ts + mention.ts; ADR-0061)* diff --git a/docs/standards/security-review.md b/docs/standards/security-review.md index 9407820d..0a49aad4 100644 --- a/docs/standards/security-review.md +++ b/docs/standards/security-review.md @@ -347,17 +347,16 @@ security invariants** a review must confirm are: changes the floor or "a user deny is final". Off a TTY (`--json` / piped / one-shot `agent run`) the one canonical `nonInteractiveApprovalPrompt` DENIES every governed dispatch — never a hang, never an auto-approve. -- **A recoverable SCOPE denial is safe by construction.** Only two `tool_denied`s are fed back to - the model for in-turn conversational recovery on the `recoverToolFailures` surfaces (the - chat-read-write host — `relavium chat` / the Home / the one-shot `agent run`; a WORKFLOW run stays - fatal/deterministic): a media scope denial and the fs **pure scope-tier escape** — both refused - BEFORE any side effect, with a **secret-free, path-free** reason, so the model adapts to an - in-bounds path while the floor still denies every attempt (no bypass, bounded by - `maxToolCorrections`). The scope-membership signal a probing model can gather over that bounded - window is an accepted, secret-free workspace-boundary oracle (the same round-bounded feedback - already existed for an idempotent not-found read). The **confidentiality** (secret-store read), - protected-path, symlink/hard-link, egress-SSRF, and user/guardrail denials stay FATAL — feeding - those back would re-deny, risk a re-execution, or leak a probe oracle. +- **A recoverable SCOPE denial is safe by construction.** On the `recoverToolFailures` chat surfaces, + exactly the two SCOPE denials refused BEFORE any side effect (a media scope denial + the fs pure + scope-tier escape) are fed back to the model with a **secret-free, path-free** reason so it adapts to + an in-bounds path — the floor still denies every attempt (no bypass; the bounded workspace-boundary + signal a probing model could gather is accepted, matching the pre-existing not-found-read feedback). + Every OTHER `tool_denied` — confidentiality (secret-store read), protected-path, symlink/hard-link, + egress-SSRF, and user/guardrail — stays FATAL. The authoritative enumeration of which denials are + `recoverable` and which stay fatal (and why) is the canonical `recoverable` note in + [tool-registry.md §error taxonomy](../reference/shared-core/tool-registry.md#error-taxonomy) — not + restated here. ## Never hand-roll crypto diff --git a/docs/uvp.md b/docs/uvp.md index b4112600..65533cee 100644 --- a/docs/uvp.md +++ b/docs/uvp.md @@ -20,7 +20,7 @@ 2. **No chat-driven assistant lets you keep — and ship — the conversation.** Relavium agent sessions are **persistent, resumable, and exportable**: auto-saved to durable local history (kept on your machine; API keys stay in - the OS keychain, never at rest), resumable on any surface, and one-click + the OS keychain — never in `history.db` or any plaintext store), resumable on any surface, and one-click exportable to a reviewable `.relavium.yaml` scaffold. Competitors' sessions evaporate; Relavium's become committable, re-runnable workflows. See [decisions/0024-agent-first-entry-point-agentsession.md](decisions/0024-agent-first-entry-point-agentsession.md) diff --git a/packages/db/src/media-reference-store.ts b/packages/db/src/media-reference-store.ts index 57042c20..16283985 100644 --- a/packages/db/src/media-reference-store.ts +++ b/packages/db/src/media-reference-store.ts @@ -79,6 +79,64 @@ export function createMediaReferenceStore( db: Db, now: () => number = Date.now, ): MediaReferenceStore { + // The two write-transaction bodies, hoisted out of the `withBusyRetry(db.transaction(…))` nest so the map/loop + // callbacks stay within the function-nesting budget (2.5.I close-out). Each is invoked INSIDE one BEGIN IMMEDIATE. + + /** Capture-then-delete a run's references + refresh the dropped handles' grace clock — atomic under BEGIN IMMEDIATE. */ + function removeRunReferencesTxn(runId: string, ts: number): number { + // Capture the handles this run referenced BEFORE the delete, so the grace window of every handle this sweep + // drops toward zero starts NOW (ADR-0042 §4 — measured from `last_referenced_at`), not from the handle's + // production time. Without this, a long-lived handle losing its last reference is reclaimed on the next sweep + // with zero effective grace. + const affected = db + .selectDistinct({ handle: mediaReferences.handle }) + .from(mediaReferences) + .where(and(eq(mediaReferences.scopeKind, 'run'), eq(mediaReferences.scopeId, runId))) + .all(); + const result = db + .delete(mediaReferences) + .where(and(eq(mediaReferences.scopeKind, 'run'), eq(mediaReferences.scopeId, runId))) + .run(); + const handles = affected.map((row) => row.handle); + // CHUNK the `handle IN (…)` refresh under SQLite's bound-parameter floor (a wide fan-out run can reference many + // handles); one shared `ts` keeps the batches consistent. + for (let i = 0; i < handles.length; i += SQLITE_INARRAY_CHUNK) { + db.update(mediaObjects) + .set({ lastReferencedAt: ts }) + .where(inArray(mediaObjects.handle, handles.slice(i, i + SQLITE_INARRAY_CHUNK))) + .run(); + } + return result.changes; + } + + /** Soft-delete every 0-reference handle past its grace window — one atomic select-then-update snapshot. */ + function reclaimExpiredTxn(cutoff: number, ts: number): string[] { + const referenced = db.select({ handle: mediaReferences.handle }).from(mediaReferences); + const expired = db + .select({ handle: mediaObjects.handle }) + .from(mediaObjects) + .where( + and( + isNull(mediaObjects.deletedAt), // not already reclaimed + lte(mediaObjects.lastReferencedAt, cutoff), // past the grace window + notInArray(mediaObjects.handle, referenced), // zero references (refcount = row count) + ), + ) + .all(); + const handles = expired.map((row) => row.handle); + // Soft-delete EXACTLY the expired handles found above (not a re-run of the 0-ref filter, which would ignore the + // grace window). Serialized under BEGIN IMMEDIATE (2.5.I), so the select-then-update snapshot is consistent even + // across two `relavium` processes. CHUNK the `handle IN (…)` list so a large sweep never exceeds SQLite's + // bound-parameter limit (SQLITE_MAX_VARIABLE_NUMBER, 999 on older builds); one shared `ts` keeps the batches consistent. + for (let i = 0; i < handles.length; i += SQLITE_INARRAY_CHUNK) { + db.update(mediaObjects) + .set({ deletedAt: ts }) + .where(inArray(mediaObjects.handle, handles.slice(i, i + SQLITE_INARRAY_CHUNK))) + .run(); + } + return handles; + } + return { recordObject(input: MediaObjectInput): void { const ts = now(); @@ -171,34 +229,7 @@ export function createMediaReferenceStore( // SELECT + DELETE + UPDATE-loop under ONE BEGIN IMMEDIATE (2.5.I write-path convention): the cursor // refresh must be atomic with the delete so a concurrent GC sweep never reclaims a handle mid-drop. return withBusyRetry(() => - db.transaction( - () => { - // Capture the handles this run referenced BEFORE the delete, so the grace window of every handle this - // sweep drops toward zero starts NOW (ADR-0042 §4 — measured from `last_referenced_at`), not from the - // handle's production time. Without this, a long-lived handle losing its last reference is reclaimed - // on the very next sweep with zero effective grace. - const affected = db - .selectDistinct({ handle: mediaReferences.handle }) - .from(mediaReferences) - .where(and(eq(mediaReferences.scopeKind, 'run'), eq(mediaReferences.scopeId, runId))) - .all(); - const result = db - .delete(mediaReferences) - .where(and(eq(mediaReferences.scopeKind, 'run'), eq(mediaReferences.scopeId, runId))) - .run(); - const handles = affected.map((row) => row.handle); - // CHUNK the `handle IN (…)` refresh under SQLite's bound-parameter floor (a wide fan-out run can - // reference many handles); one shared `ts` keeps the batches consistent. - for (let i = 0; i < handles.length; i += SQLITE_INARRAY_CHUNK) { - db.update(mediaObjects) - .set({ lastReferencedAt: ts }) - .where(inArray(mediaObjects.handle, handles.slice(i, i + SQLITE_INARRAY_CHUNK))) - .run(); - } - return result.changes; - }, - { behavior: 'immediate' }, - ), + db.transaction(() => removeRunReferencesTxn(runId, ts), { behavior: 'immediate' }), ); }, @@ -227,38 +258,7 @@ export function createMediaReferenceStore( // "single-connection ⇒ select-then-update is consistent" note held only WITHIN one process; two `relavium` // processes share this file — ADR-0064 §5.) return withBusyRetry(() => - db.transaction( - () => { - const referenced = db.select({ handle: mediaReferences.handle }).from(mediaReferences); - const expired = db - .select({ handle: mediaObjects.handle }) - .from(mediaObjects) - .where( - and( - isNull(mediaObjects.deletedAt), // not already reclaimed - lte(mediaObjects.lastReferencedAt, cutoff), // past the grace window - notInArray(mediaObjects.handle, referenced), // zero references (refcount = row count) - ), - ) - .all(); - const handles = expired.map((row) => row.handle); - if (handles.length > 0) { - // Soft-delete EXACTLY the expired handles found above (not a re-run of the 0-ref filter, which - // would ignore the grace window). Now serialized under BEGIN IMMEDIATE (2.5.I), so the select-then- - // update snapshot is consistent even across two `relavium` processes. CHUNK the `handle IN (…)` list - // so a large sweep never exceeds SQLite's bound-parameter limit (SQLITE_MAX_VARIABLE_NUMBER, 999 on - // older builds) — a multi-day orphan backlog can easily surpass it; one shared `ts` keeps the batches consistent. - for (let i = 0; i < handles.length; i += SQLITE_INARRAY_CHUNK) { - db.update(mediaObjects) - .set({ deletedAt: ts }) - .where(inArray(mediaObjects.handle, handles.slice(i, i + SQLITE_INARRAY_CHUNK))) - .run(); - } - } - return handles; - }, - { behavior: 'immediate' }, - ), + db.transaction(() => reclaimExpiredTxn(cutoff, ts), { behavior: 'immediate' }), ); }, }; diff --git a/packages/db/src/model-catalog-store.ts b/packages/db/src/model-catalog-store.ts index 139351db..ee5401cc 100644 --- a/packages/db/src/model-catalog-store.ts +++ b/packages/db/src/model-catalog-store.ts @@ -335,6 +335,107 @@ export function createModelCatalogStore(db: Db, deps: ModelCatalogStoreDeps): Mo return row === undefined ? undefined : fromRow(row); }; + // The `replaceProviderModels` write body, hoisted out of the `withBusyRetry(db.transaction(…))` nest so its + // per-row loop/branches stay within the function-nesting budget (2.5.I close-out). Runs INSIDE one BEGIN + // IMMEDIATE, and RETURNS the tallies so they are observed from within the serialized write (a concurrent + // same-provider refresh can never miscount them; drizzle returns the callback value). + const replaceProviderModelsTxn = ( + providerId: string, + rows: ReadonlyArray, + now: number, + ): ReplaceProviderModelsResult => { + // Only LIVE rows are tallied: `added` on a true INSERT, `updated` on an existing-live-row UPDATE (a non-live + // `static`/`user` row hits the provenance `continue` below and is counted in NEITHER), `deactivated` from the + // soft-deactivate UPDATE's `.changes` (its WHERE is already `source='live'`-scoped) — so the counts carry the + // same LIVE-only intent the write enforces, with no separate source filter needed. + let added = 0; + let updated = 0; + for (const input of rows) { + const displayName = input.displayName.trim() === '' ? input.modelId : input.displayName; + // `0` is the NOT-NULL "unknown" sentinel (ADR-0064 §3) — an absent live limit stores as 0. + const contextWindowTokens = input.contextWindowTokens ?? 0; + const maxOutputTokens = input.maxOutputTokens ?? 0; + // Find the existing (provider, model) row (deletedAt IS NULL — the partial-unique scope), whether it is + // active or soft-deactivated. Reuse its id so FK targets stay stable. + const existing = db + .select() + .from(modelCatalog) + .where( + and( + eq(modelCatalog.providerId, providerId), + eq(modelCatalog.modelId, input.modelId), + isNull(modelCatalog.deletedAt), + ), + ) + .get(); + if (existing !== undefined && existing.source !== 'live') { + // A `source='user'` (user pricing, ADR-0065 §1) or `source='static'` (a media-routing seed — + // media_surface/capabilities/rates) row already represents this model. A live refresh must NEVER clobber + // it (that would drop user pricing or regress media routing), so it is left UNTOUCHED and, being non-`live`, + // is also never deactivated below. Counted in neither `added` nor `updated` (provenance-protected). + continue; + } + if (existing === undefined) { + const row: NewModelCatalogRow = { + id: deps.uuid(), + providerId, + modelId: input.modelId, + displayName, + contextWindowTokens, + maxOutputTokens, + source: 'live', + lastRefreshedAt: now, + isActive: true, + createdAt: now, + updatedAt: now, + }; + db.insert(modelCatalog).values(row).run(); + added += 1; + } else { + // Reactivate + refresh the existing live row in place (id/created_at/FK refs preserved); only the discovery + // columns + provenance/freshness are written — pricing/media columns are left as-is. + db.update(modelCatalog) + .set({ + displayName, + contextWindowTokens, + maxOutputTokens, + source: 'live', + lastRefreshedAt: now, + isActive: true, + updatedAt: now, + }) + .where(eq(modelCatalog.id, existing.id)) + .run(); + updated += 1; + } + } + // Soft-deactivate the vanished live rows: every currently-active `source='live'` row of THIS provider whose + // model id is absent from the new list. `isActive=false` with `deletedAt` untouched keeps the partial-unique + // slot occupied so a reappearing model reuses the SAME row (reactivated above). NEVER a hard-DELETE (FK target + // from five tables); NEVER touches `source='user'`/`source='static'`. + const incomingModelIds = rows.map((r) => r.modelId); + const deactivateScope = and( + eq(modelCatalog.providerId, providerId), + eq(modelCatalog.isActive, true), + eq(modelCatalog.source, 'live'), + isNull(modelCatalog.deletedAt), + ); + const deactivateResult = db + .update(modelCatalog) + .set({ isActive: false, updatedAt: now }) + // An empty new list deactivates ALL of the provider's live rows (no `notInArray([])` — its semantics vary; + // the guard makes the "everything vanished" case explicit). + .where( + incomingModelIds.length === 0 + ? deactivateScope + : and(deactivateScope, notInArray(modelCatalog.modelId, incomingModelIds)), + ) + .run(); + // better-sqlite3's `RunResult.changes` = the rows the UPDATE matched (each flips isActive true→false, so every + // matched row is genuinely modified) = the number of live rows soft-deactivated this refresh. + return { added, updated, deactivated: deactivateResult.changes }; + }; + return { resolveMediaSurface: (modelId) => { const row = activeRow(modelId); @@ -483,109 +584,14 @@ export function createModelCatalogStore(db: Db, deps: ModelCatalogStoreDeps): Mo .map(toListing), replaceProviderModels: (providerId, rows, now) => - // The transaction RETURNS the tallies so they are observed from WITHIN the serialized write — a concurrent - // same-provider refresh can never miscount them (an external before/after `listByProvider` diff would read a - // stale `before` and could double-count). drizzle's better-sqlite3 `transaction()` returns the callback value. // `BEGIN IMMEDIATE` — this reads existing rows then writes, so a DEFERRED begin would hit the read→write // lock-upgrade race — plus `withBusyRetry` for residual cross-process contention (ADR-0064 amendment note). + // The body (hoisted above) RETURNS the tallies, observed from WITHIN the serialized write so a concurrent + // same-provider refresh can never miscount them (drizzle's better-sqlite3 `transaction()` returns the value). withBusyRetry(() => - db.transaction( - () => { - // Only LIVE rows are tallied: `added` on a true INSERT, `updated` on an existing-live-row UPDATE (a non-live - // `static`/`user` row hits the provenance `continue` below and is counted in NEITHER), `deactivated` from the - // soft-deactivate UPDATE's `.changes` (its WHERE is already `source='live'`-scoped) — so the counts carry the - // same LIVE-only intent the write enforces, with no separate source filter needed. - let added = 0; - let updated = 0; - for (const input of rows) { - const displayName = - input.displayName.trim() === '' ? input.modelId : input.displayName; - // `0` is the NOT-NULL "unknown" sentinel (ADR-0064 §3) — an absent live limit stores as 0. - const contextWindowTokens = input.contextWindowTokens ?? 0; - const maxOutputTokens = input.maxOutputTokens ?? 0; - // Find the existing (provider, model) row (deletedAt IS NULL — the partial-unique scope), whether it - // is active or soft-deactivated. Reuse its id so FK targets stay stable. - const existing = db - .select() - .from(modelCatalog) - .where( - and( - eq(modelCatalog.providerId, providerId), - eq(modelCatalog.modelId, input.modelId), - isNull(modelCatalog.deletedAt), - ), - ) - .get(); - if (existing !== undefined && existing.source !== 'live') { - // A `source='user'` (user pricing, ADR-0065 §1) or `source='static'` (a media-routing seed — - // media_surface/capabilities/rates) row already represents this model. A live refresh must NEVER - // clobber it (that would drop user pricing or regress media routing), so it is left UNTOUCHED and, - // being non-`live`, is also never deactivated below — the model stays represented by its own row. - // It is counted in neither `added` nor `updated` (provenance-protected — never part of the live delta). - continue; - } - if (existing === undefined) { - const row: NewModelCatalogRow = { - id: deps.uuid(), - providerId, - modelId: input.modelId, - displayName, - contextWindowTokens, - maxOutputTokens, - source: 'live', - lastRefreshedAt: now, - isActive: true, - createdAt: now, - updatedAt: now, - }; - db.insert(modelCatalog).values(row).run(); - added += 1; - } else { - // Reactivate + refresh the existing live row in place (id/created_at/FK refs preserved); only the - // discovery columns + provenance/freshness are written — pricing/media columns are left as-is. - db.update(modelCatalog) - .set({ - displayName, - contextWindowTokens, - maxOutputTokens, - source: 'live', - lastRefreshedAt: now, - isActive: true, - updatedAt: now, - }) - .where(eq(modelCatalog.id, existing.id)) - .run(); - updated += 1; - } - } - // Soft-deactivate the vanished live rows: every currently-active `source='live'` row of THIS provider - // whose model id is absent from the new list. `isActive=false` with `deletedAt` untouched keeps the - // partial-unique slot occupied so a reappearing model reuses the SAME row (reactivated above). NEVER a - // hard-DELETE (FK target from five tables); NEVER touches `source='user'`/`source='static'`. - const incomingModelIds = rows.map((r) => r.modelId); - const deactivateScope = and( - eq(modelCatalog.providerId, providerId), - eq(modelCatalog.isActive, true), - eq(modelCatalog.source, 'live'), - isNull(modelCatalog.deletedAt), - ); - const deactivateResult = db - .update(modelCatalog) - .set({ isActive: false, updatedAt: now }) - // An empty new list deactivates ALL of the provider's live rows (no `notInArray([])` — its semantics - // vary; the guard makes the "everything vanished" case explicit). - .where( - incomingModelIds.length === 0 - ? deactivateScope - : and(deactivateScope, notInArray(modelCatalog.modelId, incomingModelIds)), - ) - .run(); - // better-sqlite3's `RunResult.changes` = the rows the UPDATE matched (each flips isActive true→false, so - // every matched row is genuinely modified) = the number of live rows soft-deactivated this refresh. - return { added, updated, deactivated: deactivateResult.changes }; - }, - { behavior: 'immediate' }, - ), + db.transaction(() => replaceProviderModelsTxn(providerId, rows, now), { + behavior: 'immediate', + }), ), providerRefreshedAt: (providerId) => {