Skip to content

feat(server): persist exact Prism recovery history - #192

Merged
lukemaj merged 5 commits into
fork/v2from
feat/169-prism-recovery-outcomes-v2
Oct 8, 2026
Merged

lukemaj merged 5 commits into
fork/v2from
feat/169-prism-recovery-outcomes-v2

Conversation

@lukemaj

@lukemaj lukemaj commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What: Persist exact Prism recovery decisions and admitted continuation sources for each failed run.
Why: A mutable per-thread recovery row loses history and cannot prove whether an older failure has a successor or whether an opted-out reset was later re-enabled.
So what: Exact-head independent review and cloud CI pass; root can merge, then #174 completes consumer integration proof against the merged reader.

Closes #169. Authorized separate follow-up to merged #183 under D35 and D38.

An admitted retry/reset run carries its immutable forkPrismContinuationSourceRunId. A fork-owned table retains the exact source's conclusive decision without replacing it; the indexed run projection supplies the actual successor. RecoveryHistory.readRecoveryState(sourceRunId) reads {decision, admittedContinuations, pendingRecovery} through the caller's current SqlClient transaction. An earlier opted_out stays immutable while a newly enabled reset reports pending recovery before admission, then the exact admitted successor takes precedence. Each source admits at most one successor, enforced by the transaction guard and partial unique index; the declared array reader orders by ordinal and run ID.

The existing generic EventSink commit-plan transaction admits the run/event and retried fact together, preserving receipt dedupe, rollback and accepted no-op semantics. The reactor processes bounded, fair batches of unresolved exact failed sources, including non-MCP/ineligible threads and older failures behind newer active/completed work. Automatic abandonment remains distinct from human report abandonment. This PR adds no scheduler/report logic, separate link registry, ID parsing or backfill.

Elon record:
Requirements and who asked: User/root D35/D38 requires immutable exact-source outcomes, bidirectional actual admissions, current pending reset and same-transaction consumer visibility.
Deleted: Conclusive overwrite and proposed separate continuation registry; unrelated scheduler logic, report registry, versioning, browser/dev-server work.
Bottleneck: Root-owned merge and174 consumer integration; current-head source review, local focused proof and cloud CI pass.
Checked myself: Read live predecessor diff and approved interfaces, verified remote head/base, identical rebase range-diff, actual failing-before/corrected output, source ownership and numstat.

Proof: current runtime-history correction, exact rebase and retained red/green, bounded-hook correction and retained red/green, current-base commands/results, retained earlier failures.

  • 37 focused recovery tests passed across six files, including the 15-case new real SQLite/EventSink/reactor history proof.
  • Actual Orchestrator retry/reset admission, exhaustion, Stop/retirement and finalization pass with exact source/outcome assertions.
  • Physical SQLite reopens cover opt-out → re-enable → pending reset → admitted continuation; admission/link visibility is checked inside the transaction. Merge scheduled tasks with upstream's scheduler #174 owns its whole scheduler completion-transaction proof.
  • Actual committed non-MCP failure observation, old failure behind newer running/completed work, archived failure, 128-candidate fair batching/history exclusion, concurrent duplicate receipts, rollback/no-op and source/model/existing-ID refusal pass.
  • Scoped actual server (t3), contracts and web tsc --noEmit each exited0. Scoped lint exited0 with one unchanged base layerUnavailable warning. Fork inventory passed with 20 features; git diff --check passed.
  • Resolved runtime-request history is excluded by a fork-owned reader using the existing (thread_id,status) index and pending LIMIT 1; the same140-request oracle failed before this fix. Actual pending input still blocks automatic recovery.
  • Delegated finalization loads only the expected and actual latest run/error IDs; the same130-source oracle failed before this fix and passes with later128+2 fair reactor catch-up.
  • Retained failed checks: two fixture type errors, six persisted-date decoder failures, then one immutable-source lifecycle failure. Local corrections passed the unchanged behavioral assertions; no waiver.
  • All proof commands were serial and load-admitted. First pnpm11 exec unexpectedly auto-reconciled dependencies before compiling; no tracked dependency/config changed, and subsequent checks used existing direct binaries. No further pnpm/install command was run.

Exact candidate head fc89e8c039ffb0bc01efa95a3613ad1ce76fd5eb, intended base d1107646b522926e9e32d127aacf1b9fea1ae9fd. Exclusive branch feat/169-prism-recovery-outcomes-v2. No merge commits or version bumps. Independent exact-head source review passes, reviewer lukemaj, GPT-6-Luna via Codex in T3 Code, no author-proof rerun. review/independent=success is verified on this SHA. All cloud CI checks pass or are workflow-conditional skips on run37851363963; no failure is accepted. The prior98 and6cb review failures remain unwaived. No merge performed.

MODULAR upstream inventory, classified against upstream pin 12069eefd707f78eafc27812027c994eea0613cf, counts against the exact intended PR base:

Upstream-owned edited path Added Removed Hook and primary ownership
apps/server/src/orchestration-v2/Orchestrator.ts 4 1 Three run constructors call the fork source-field helper; the existing commit-plan hook receives planned events. child-threads primary, prism-toolkit shared.
apps/server/src/orchestration-v2/ProjectionStore.ts 5 1 One import and a fork-owned SQL fragment call preserve the admitted source through lifecycle snapshots. child-threads primary, prism-toolkit shared.
packages/contracts/src/orchestrationV2.ts 2 0 One optional Run source field and its comment. child-threads primary, prism-toolkit shared.

Every upstream production hunk is under 15 lines. All behavior resides in fork modules. Existing upstream paths remain allowlisted; prism-toolkit owns the six new history/policy/admission/projection files and their proof. Foundation, child-threads and issues-links ownership is retained.

Tests are fork-owned, not upstream inventory: RecoveryHistory.integration.test.ts adds757 lines because independent cases retain real EventSink/SQLite transactions, database connection reopens, serialized admission and reactor event drains. Recovery.integration.test.ts adds50 lines to existing actual Orchestrator cases for source/outcome/pending assertions. These cover the requested concurrency/restart contracts, with external providers mocked and no sleeps, live database access or generated fixtures.

Limitations: not live-provider verified; no browser, dev server, desktop/mobile build, installation or deployment. No frontend changes here. Initial #183 settings images and integrated UI proof remain explicitly deferred by the user to the authorized planner pass. Website impact none. #174 final consumption remains blocked until this follow-up is reviewed and merged.

Implementation: GPT-6.1-Sol through Codex in visible T3 Code; independent review uses the separate visible T3 reviewer thread. Agent Observer cost publication succeeded. Measurement gap: estimated cost is unknown, 50 sessions include49 without bound usage and1 missing usage; zero attributed responses does not mean zero activity or spend. Evidence is incomplete at the recorded cutoff.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XL labels Oct 8, 2026
@lukemaj

lukemaj commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

What: RecoveryHooks.holdsFinalization now filters runs and turn items to the expected and actual latest run IDs, while still loading every runtime request for the thread.
Why: The unwindowed projection query returns all runtime-request rows, while recovery checks only whether any request is pending.
So what: Finalization work remains proportional to the thread's retained runtime-request history; bound this read before the independent review can pass.

Blocking finding: finalization still loads unbounded runtime-request history

Trigger: A long-lived child thread has accumulated resolved runtime requests and reaches delegated result finalization.

Actual: RecoveryHooks.ts:54-61 requests runtimeRequests without a filter. Since getThreadRecords uses the unwindowed read, ProjectionStore.ts:3074-3082 selects every row for the thread. The projection upserts each request by ID and retains its resolved state (ProjectionStore.ts:2311-2343). RecoveryCoordinator.observe uses that array only for some(request.status === "pending") (RecoveryCoordinator.ts:51-58).

Required: Keep finalization's pending-request check bounded so historical resolved requests are not read and decoded on every result transfer. The added 130-source oracle covers old runs and error items, not runtime-request history.

Review record

Requirements and who asked: User-approved D35/D38 requires conclusive exact-source outcomes and bounded historical catch-up; the user requested an exact independent review of this candidate.
Deleted: No D32 expansion, scheduler/report registry, or separate history redesign. The reactor's approved fair historical scan remains in scope.
Bottleneck: Finalization reads and decodes all retained runtime requests before result transfer.
Checked myself: Confirmed PR head 6cb59a49ac93f9b302e1b12575c4cff4ba9b3943 against base c87ee8c84eb901f5c8abbd24b54a6a7becdecea7; inspected the changed hook, projection query, request lifecycle, and coordinator consumer. Source-only review; no proof rerun.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire — 5.0 KiB — 6.8 KiB ✅
Codex Thread snapshot wire — 3.8 KiB — 4.9 KiB ✅
Codex Live turn WebSocket wire — 1.2 KiB — 2.0 KiB ✅
Codex Live turn WebSocket decoded — 20.9 KiB — 29.3 KiB ✅
Codex Live turn messages — 2 — 8 ✅
Claude Total thread wire — 5.0 KiB — 6.8 KiB ✅
Claude Thread snapshot wire — 3.8 KiB — 4.9 KiB ✅
Claude Live turn WebSocket wire — 1.2 KiB — 2.0 KiB ✅
Claude Live turn WebSocket decoded — 21.2 KiB — 29.3 KiB ✅
Claude Live turn messages — 2 — 8 ✅

Baseline: unavailable · PR result: fc89e8c · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@lukemaj
lukemaj force-pushed the feat/169-prism-recovery-outcomes-v2 branch from abe00b5 to fc89e8c Compare October 8, 2026 22:06
@lukemaj

lukemaj commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

What: PR192 fixes the remaining unbounded resolved-request hydration and publishes rebased successor fc89e8c039ffb0bc01efa95a3613ad1ce76fd5eb.
Why: The retained projection path read every request, although recovery only needs pending existence.
So what: Draft cloud CI and renewed independent review must pass on this exact head;174 still consumes only after reviewed merge.

Elon record:
Requirements and who asked: User bounded D35 exact-source recovery and confirmed independent finding6069830872.
Deleted: Resolved request history from eligibility hydration, duplicate fork call-site projection reads; no upstream query infrastructure, registry or consumer contract change.
Bottleneck: Current-head source review and fresh GitHub CI.
Checked myself: Actual runtime-request lifecycle/SQL, existing migration055(thread_id,status) index, faithful retained-request red/green oracle and current live Git heads.

The fork-owned recoveryProjection.ts supplies only requested runs/error items and at most1 pending request, using the existing index and SQL transaction snapshot. Both hook and reactor use it. No new storage/index, upstream file edit or exported consumer shape. Feature-map prism-toolkit owns the helper;3upstream MODULAR counts remain+4/-1,+5/-1,+2/-0, allhunks<15. New fork history proof is757lines/15cases; helper34lines.

Serial batch, actual sampled1-minute loads before each execution:
-4.02: retained6cb production semantics extracted unchanged into fork reader, with new real140resolved-request event fixture: named bounded test exit1, expected 140 to equal +0.14 unrelated cases deselected only for this reproduction, no accepted skip.
-3.34: corrected indexed pendingLIMIT1 reader, six composed recovery files/one worker:37tests exit0. The same140history oracle and130old-run catch-up pass. Actual pending input blocks retry; resolved history does not block a new independent failure.
-2.97: actual server tsc failed own missingSqlClient closure context. Corrected call-site provision of the already captured SQL service, no infrastructure change.
-3.20: server tsc rerun exit0 (Effect suggestions only).
-4.80: final affected history+actual Orchestrator integration files/one worker:20tests exit0 after the closure fix; other four composed inputs unchanged.
-4.91:4changedTSfiles scopedlint exit0; fork inventory20features exit0; gitdiffcheck0.
-4.34: AgentObserver incremental sync exit0, cost publication follows.
No concurrent proof processes. FINISH/release sent explicitly;174 now owns nextbatch.

Commands use the existing direct vp test run / vp lint binaries with ELECTRON_RUN_AS_NODE unset, TMPDIR=/private/tmp/, --no-file-parallelism --maxWorkers 1, actual apps/server ../../node_modules/.bin/tsc --noEmit, bash scripts/fork-check.sh. Exact6file commands remain in previous proof; contracts/web and unchanged path checks reused.

Newbase d1107646b522926e9e32d127aacf1b9fea1ae9fd added only2lines in .github/workflows/ci.yml to configure apt mirrors before Chromium install. Full delta inspected; clean mechanical rebase, all5patches range-diff identical. git diff abe00b577ce3d32d864d6b7fbccbea2c5bc11fd8..fc89e8c039ffb0bc01efa95a3613ad1ce76fd5eb --name-only contains ONLY .github/workflows/ci.yml; every local test/type/lint/inventory input is identical, so no redundant heavy rerun. Force-with-lease bound verifiedownabe remote only; liveownfc89/base d110 verified.

Red98/6cb reviews and every failed/corrected check remain unwaived; fresh source review and CI requested. No browsers/devservers/install/pnpm/version/mainpush/merge. No live-provider claim. Initial183UIproof remains explicitly deferred to authorized planner;174 owns scheduler whole-completion transaction proof. Cost and final readiness will be recorded separately.

@lukemaj

lukemaj commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Agent work on this PR

Estimated cost unknown · 0 responses · 50 sessions · 8.4 h wall time

Model Responses Tokens Estimated cost

Flags: 4 human corrections · 96 large tool outputs · 118 repeated commands · 11 repeated failures · 120 repeated reads · 17 repeated skill loads · 49 sessions with usage bound to no task · 1 session without usage records

Details: snapshot, prices, coverage, counters
  • Task toolboxmd/chromeria#169: outcome unknown (recorded acceptance only; a finished process never implies it).
  • Proof: Port Prism as a service behind upstream delegation #169
  • Snapshot b8f536804233c2c57b36c9192174874786f79b24f19c3b04a0bbf95823ee078c, records up to 2026-10-08 22:04 UTC.
  • 50 sessions on claude, codex; AgentsMD 14.6.0.
  • Not counted: 6,862 responses (at least $164.48) in sessions shared with other PRs that worked in no single PR's checkout.
  • 5,400 responses in these sessions worked on other PRs and are counted there.
  • Totals reconcile with the measured sessions: yes. Evidence complete: no.
  • Prices: list-price estimate from T3 local rate table (path withheld) as of 2026-09-28, schedule 696aae45933d0a684a015d3f08cfb6f1aa2fef02e7d272b4689e3a692ea04fff. Unknown prices stay unknown, never zero.
    • T3 LiteLLM rate table when present; bundled schedule covers the rest.
  • Native session usage or worker ownership is unavailable.
  • Harness-reported cost: none reported.
  • Usage totals are not billing. Subscription spending is separate and is never posted as spend.
  • Crashed runs are counted separately: 0.

Token counters by model (native counter semantics; never added across semantics):

Selected rates (USD per million tokens). These rates value the report at the selected schedule date; they do not establish historical prices or subscription spending.

Model Input tier Input Cache read Cache write Other output Reasoning

Other output and reasoning are priced without double counting inclusive native output. Missing rates remain unknown.

Local measurement from native records; usage totals are not billing. Updated in place by agent-observer publish.

@lukemaj

lukemaj commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

What: The shared recovery reader now limits finalization and reactor inputs to relevant runs and error items, plus at most one pending runtime request.
Why: This resolves the remaining unbounded runtime-request history read from finding 6069830872.
So what: The exact-head source review is clear; PR #192 remains a draft and this is not a readiness or merge approval.

Review result

The prior finding is resolved on fc89e8c039ffb0bc01efa95a3613ad1ce76fd5eb. recoveryProjection.ts reads the filtered run and turn-item projection and the pending request inside one existing SQL transaction. The pending request query uses status='pending' LIMIT 1 and the existing (thread_id, status) index from migration 055. Both RecoveryHooks and RecoveryReactor use this reader. Their consumer only needs to know whether a pending request exists.

I inspected the 6cb59a49ac..abe00b577c source correction, the full candidate range d1107646b5..fc89e8c039, and the base delta c87ee8c84e..d1107646b5. The base delta is limited to the two CI workflow lines; all five implementation patches range-diff identically. The helper adds no upstream query API, migration, or separate registry. No other supported blocking source finding remains.

The author’s retained red/green and focused proof is linked in the current handoff; I did not rerun tests, type checks, lint, CI, or browser/server proof. Earlier failure statuses on 98c26e29 and 6cb59a49 remain unchanged.

Elon record

Requirements and who asked: The user-approved D35/D38 outcome contract and the user’s request for an exact independent review.
Deleted: No added index, migration, upstream API, or separate recovery registry; resolved request history is excluded from recovery eligibility reads.
Bottleneck: Source review is clear; the draft’s remaining admission stays with its other proof and CI owners.
Checked myself: Verified the live PR head/base and clean author branch; read the exact source correction, both shared-reader call sites, actual schema/index, current diff, base delta, and five-patch range-diff. Source-only review, with no proof rerun.

@lukemaj
lukemaj marked this pull request as ready for review October 8, 2026 22:14
@lukemaj
lukemaj merged commit 4c2b384 into fork/v2 Oct 8, 2026
34 of 36 checks passed
@lukemaj
lukemaj deleted the feat/169-prism-recovery-outcomes-v2 branch October 8, 2026 22:20
lukemaj added a commit that referenced this pull request Oct 14, 2026
* feat(server): publish Prism per-run recovery outcome reader checkpoint

* wip: publish D38 immutable decision and indexed continuation reader

* fix(server): preserve admitted recovery sources and prove durable history

* fix(server): bound Prism finalization history observation

* fix(server): bound Prism pending request hydration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant