refactor(runtime): unify durable model-context projection authority #4283

Problem

Maka durably records tool outcomes across the canonical RuntimeEvent ledger and append-only operational AgentRunEvent ledger, but the history shown to a model is still assembled by several authorities with different lifetimes:

• the durable function_response is committed before the live toModelOutput() projection is computed;
• the live projection is retained only in settledModelOutputs, while later replay starts from the durable result and applies Bash/File-only compatibility projectors;
• active Tool Result pruning and provider-overflow image omission keep replacements only in one send();
• stale Tool Result pruning rewrites replay through a separate policy path;
• history compaction alone has a validated, durable, monotonic successor protocol.

Computer Use and MCP make the split observable: the current continuation can receive richer or differently bounded model-facing content than the next Turn or a restart. A Tool Result or image removed to make a request succeed can also return on a later replay. Besides changing semantics, this repeatedly invalidates an otherwise append-only prompt-cache prefix.

The missing invariant is:

A successful model-visible history is append-only. Any lossy change to already-visible history must first become a durable successor in the append-only operational AgentRunEvent ledger, and no later replay, compaction, branch, or restart may restore the replaced form.

A provider/model change may start a new wire-cache generation, but it must not resurrect content already removed by a durable model-history transition.

This issue complements rather than replaces:

• fix(runtime): RuntimeEvent ledger is recovery-grade (WAL), not source-of-truth — write-path truncation blocks counterfactual replay #615, which owns complete redacted execution facts, counterfactual replay, and request capture;
• [RFC] Provider-native Tool Search for large tool catalogs #1382, which owns provider-native deferred Tool Search and its replay contract.

Plan/Graph backend-generation freshness is a separate lifecycle problem and is out of scope here.

Desired architecture

Use one append-only ledger authority and one effective-history reducer:

durable function_response projection
  + durable projection transitions
  -> one effective model-history reducer
  -> existing contiguous HistoryCompactCheckpoint
  -> pure provider materialization

1. Commit the initial model projection atomically at T2

Before the Tool Result T2 commit, a pure, total projector converts the completed tool result into a closed, versioned, bounded, provider-neutral durable schema. The response fact and its model projection are committed in the same RuntimeEvent transaction.

The durable projection may contain redacted text/JSON and session-owned content-addressed artifact references. It must not contain arbitrary ToolResultOutput, inline unbounded base64, URL/Buffer values, or provider-opaque replay state. Provider-native results continue to use their explicit compatibility contract.

If projection fails, T2 commits one deterministic safe projection/failure sentinel with the completed outcome; it must not leave a second projection write, a pending projection state, or a reason to repeat the tool side effect.

Live continuation, next-Turn replay, restart recovery, branch replay, budgeting, and compaction all consume the same durable projection. settledModelOutputs, replay-time tool-name projectors, and the post-T2 toModelOutput() authority can then be removed after the compatibility cutover.

2. Record sparse lossy changes as append-only transitions

Do not generalize HistoryCompactCheckpoint into a sparse override store. Its existing meaning is replacement of one validated contiguous history prefix, and that meaning remains unchanged.

Instead, active/stale Tool Result pruning and provider-overflow image omission append typed model-projection transitions to the existing append-only operational AgentRunEvent ledger. A transition identifies:

• its target RuntimeEvent / projection part;
• the source projection digest it replaces;
• the replacement projection or session-owned archive reference;
• its predecessor/high-water identity needed for deterministic reduction.

The Session-scoped reducer applies ledger order and source-digest validation to produce effective model history. Once a transition is committed, stale writers, rolling compaction, restart, and branch replay cannot recover the old projection. Ordinary per-request image-budget accounting remains materialization-local; only a lossy rewrite that allowed a rejected provider request to succeed becomes durable.

3. Keep the current checkpoint responsibility narrow

HistoryCompactCheckpoint continues to replace a contiguous effective-history prefix with text summary or compatible provider-native compact state. Its source coverage/digest must be derived from the reducer's effective projection, not raw RuntimeEvents.

Replay, context budgeting, summarization, rolling compaction, and checkpoint validation all read the same reducer output. Raw execution evidence is never summarized or sent merely because a second code path bypassed the reducer.

4. Make copy, artifact lifetime, and compatibility part of the protocol

Projection codecs expose artifact-reference collection and rewriting. Conversation copy/branch must copy and remap all owned references and preserve transition lineage before committing the target history. Cleanup uses reducer-derived reachability; a cleanup failure may delay reclamation but cannot break replay.

Legacy RuntimeEvents are normalized at one compatibility-codec boundary into the same durable projection type. Existing Bash/File behavior may be used there as a bounded fallback. Missing historical MCP/Computer Use information must degrade safely and must not be guessed. New writes use only the new protocol; normal replay does not retain permanent tool-name or old/new dual paths.

Provider-neutral projections may be materialized differently by a new provider profile, but provider-native opaque state remains guarded by its existing connection/model compatibility rules.

Workstreams and PR dependency graph

The architecture has several responsibilities, but it should be delivered as four PR workstreams, not five serial cleanup phases. Each PR is implemented in its own task and isolated worktree. A task that has already delivered a PR is not reused for the next workstream.

PR 1: durable projection foundation
  |-- PR 2: effective-history budgeting and compaction
  `-- PR 3: projection transitions, Tool Result pruning, and copy/artifacts
          `-- PR 4: overflow image omission and final legacy-path deletion

PR 1 — Durable projection foundation (critical path)

Status: ✅ Completed by #4287 (merge commit bd951aa8819c5c03e39d2878f6866ef5aa53a075).

• Define the closed durable projection schema, version, size/privacy constraints, and deterministic projection-failure fallback.
• Compute the projection before T2 and commit it atomically with the completed Tool Result.
• Add the single compatibility codec and effective-history reducer input.
• Make live continuation, next-Turn replay, and cold restart consume the same projection.
• Do not add projection transitions or change HistoryCompactCheckpoint in this PR.

This contract must land first. PR 2 and PR 3 branch from its exact accepted head so they do not invent competing schemas or reducers.

PR 2 — Effective-history budgeting and compaction

• Make context budgeting, summarizer input, rolling compaction, and checkpoint source digests consume the reducer's effective model history.
• Keep existing V2/V3 contiguous checkpoint coverage, selection, and provider-state compatibility unchanged.
• Prove that compaction cannot read raw content that the effective projection removed.

PR 2 depends only on PR 1 and can proceed in parallel with PR 3.

PR 3 — Projection transitions, Tool Result pruning, and copy/artifacts

• Add one typed append-only projection-transition protocol to the existing append-only operational AgentRunEvent ledger.
• Migrate active and stale Tool Result archive replacement to that protocol.
• Include source-digest validation, concurrency ordering, branch/copy ID and artifact remapping, and reducer-derived artifact reachability in the same workstream.
• Delete the migrated active/stale prune maps and replay policy paths in this PR; do not leave a later cleanup task to remove them.

PR 3 depends only on PR 1 and can proceed in parallel with PR 2.

PR 4 — Overflow image omission and final cutover

• Reuse PR 3's transition protocol for historical image omission after a provider overflow rejection.
• Keep ordinary per-request image-budget accounting materialization-local.
• Prove that a successfully omitted historical image does not return on the next Turn, restart, compaction, or branch.
• Delete settledModelOutputs, the overflow omission Turn map, replay-time tool-name projectors, and any remaining post-T2 toModelOutput() authority once their producers are migrated.

PR 4 is stacked on PR 3 and may be developed while PR 3 is under review, but it cannot merge first. It rebases onto the accepted PR 3 head before final review.

Delivery rules

• PR 1 is the only serial foundation. After its contract is accepted, PR 2 and PR 3 run in parallel; PR 4 is the short dependent tail of PR 3.
• Every PR must be independently reviewable and revertible, and must remove the authority it replaces within its own scope.
• No PR may introduce a second durable store, permanent dual-read/dual-write path, or compatibility branch outside the single codec boundary.
• CI and review evidence apply only to the exact PR head. Each workstream owns its focused tests and publication from its own task/worktree.

Acceptance criteria

• A newly committed Tool Result has one durable, bounded model projection in the same T2 transaction as its completed outcome.
• Live continuation, next Turn, and process restart reduce that event to identical provider-neutral model content.
• With no new event or projection transition, request N+1 preserves request N's provider-visible history prefix under the same provider materialization profile and only appends.
• Once a projection transition replaces content, later replay, rolling compaction, a stale concurrent writer, copy/branch, and restart cannot restore it.
• Context budgeting, summarization, and checkpoint digests consume effective model history rather than raw execution facts.
• Existing HistoryCompactCheckpoint remains a contiguous-prefix replacement; sparse transitions are not stored in or selected by its max-coverage lineage.
• Projection failure, T2 failure, transition persistence failure, and crash cuts cannot expose an unreplayable rewrite or repeat a completed tool side effect.
• Computer Use and MCP projections obey explicit redaction, size, artifact ownership, retention, and copy/remap constraints; no unbounded inline image is added to RuntimeEvent JSON.
• Legacy and recovery-synthesized responses enter through one compatibility codec, with deterministic safe degradation and a bounded cutover.
• Focused tests cover current-Turn continuation, next-Turn replay, cold restart, concurrent Turns, rolling compaction, copy/branch, provider-profile switch, artifact cleanup failure, and legacy retry.

Rejected alternatives

• Generalize the current prefix checkpoint with sparse overrides. Prefix replacement and sparse per-event transitions have different coverage, concurrency, copy, and recovery algebra; combining them would add ambiguity rather than remove an authority.
• Keep reseeding per-Turn maps. That preserves the current parallel state machines and makes every new rewrite invent its own recovery contract.
• Persist arbitrary final provider requests or ToolResultOutput. That freezes adapter details, permits non-JSON/unbounded values, and duplicates provider state.
• Recompute historical projections from the current tool implementation on every replay. Tool code and policies evolve, so the past would remain mutable.
• Replace or discard complete redacted execution evidence. Raw evidence and lossy model context have different responsibilities; fix(runtime): RuntimeEvent ledger is recovery-grade (WAL), not source-of-truth — write-path truncation blocks counterfactual replay #615 remains the fact-capture authority.

AI-assisted disclosure: the architecture audit and adversarial review were performed with Codex, Claude, and K3-256K; the maintainer remains responsible for the proposal and implementation.

Execution-identity authority boundary

PR #4286 exposed a neighboring migration boundary that this issue must not accidentally deepen:

• RuntimeEvent is the canonical execution and model-history fact.
• The current schema still resolves immutable source-route metadata through event.runId -> AgentRunHeader because no invocation-opening RuntimeEvent carries that run-scoped metadata yet.
• That lookup is a bounded compatibility seam, not a declaration that mutable AgentRunHeader state is a second long-term semantic authority.

In this issue, “AgentRun ledger” means the existing append-only operational AgentRunEvent ledger where a workstream explicitly needs an accepted projection/checkpoint/attempt fact. It does not authorize new model-history fields, transitions, or lifecycle truth in AgentRunHeader.

Accordingly, all four workstreams above must:

• keep new model-context source and transition facts append-only;
• add no new AgentRunHeader field or independent Header writer;
• distinguish canonical RuntimeEvents, append-only operational AgentRunEvents, and rebuildable indexes/projections in code and review descriptions;
• avoid copying route metadata onto every RuntimeEvent merely to remove the current join;
• remain compatible with retiring independent Header writes later.

The full execution-identity migration is deliberately tracked separately in #4311: introduce an immutable invocation-opening fact, derive a rebuildable RuntimeInvocationIndex, move consumers off independent Header semantics, and then retire AgentRunHeader; runId and invocationId converge only if their supported one-to-one contract is proven across recovery, continuation, copy/import, and persisted data.

This clarification does not add a fifth workstream or change the dependency graph above. It prevents PRs 1–4 from treating a transitional Header record as the destination architecture.