[chadhietala]

Summary

Compaction refusals, task-eval synchronization races, and duplicate background reports are causing CI flakes. This PR is incomplete: the confirmatory holdout failed. The latest correction stops task evals from waiting for a notification already passed by their stream cursor when a successful tool result has established the terminal state. Compaction supplies bounded request context, and interrupted evals retain partial event evidence. The original compaction eviction algorithm and substantive E2E requirements remain unchanged. Related to #3165.

Changes And Boundaries

• Supply the current user request as quoted context only when it is outside the summarized region. Do not duplicate it when it is already present. Bound the supplemental request to 2,000 characters of text plus a truncation marker; include it in prompt-budget accounting.
• Keep reasoning parts and provider metadata out of the summary transcript, preserve the previous checkpoint, and distinguish completed work from outstanding requests.
• Reject provider-filtered summaries even when they contain partial text. This is stricter failure accounting, not a fallback or a new retry policy. Retain only bounded diagnostic codes, never provider explanations or credentials.
• Preserve producer identity across background report redelivery. Distinct reports with identical contents still remain distinct.
• Wait for a task completion/failure notification or an already recorded successful task_cancel result before sending one terminal-view verification. Preserve the existing deadline and task identity/status/error assertions. Match the runtime's actual failed. notification headline.
• Retain partial observed events in eval artifacts without changing the settled-turn behavior of public t.events and t.transcript.
• Make the callback redaction test use a unique secret canary, so a checkout path such as /private/tmp is not mistaken for leaked content. The whole-log secret assertion remains.
• Purchasing-review assignment/report wording remains a candidate. The five-reviewer, overlap, completion, no-redelegation, real-model, refusal, and 98% cache-reuse assertions are unchanged.
• The separate tool-result history-loss fix remains excluded. There are no new refusal retries, model substitutions, skipped failing evals, or relaxed assertions.

Local Validation Before Push

Follow-up 0dfc9d4afbc6ea341c96f594843eb57f469eca17 changes only the task fixture helper and its regression tests. The missing-terminal-evidence regressions failed before the fix. The real-client regression reproduces a correlated send skipping an earlier cancellation notification while retaining the terminal tool result. Before pushing this follow-up, the frozen-dependency checkout passed 8,491 unit tests with one existing skip, the cancellation notification integration test, package and fixture typechecks, formatting, lint, transition declarations, and the invariant guard. The shared checkout's invariant command could not resolve an installed dependency; the frozen checkout passed. The new CI run is pending.

The current code was copied byte-for-byte into an isolated checkout with the committed frozen lockfile. This avoided altering an unrelated local lockfile edit and corrected the shared checkout's missing committed Workflow dependency patch.

• Full package build: passed, with self-import externalization warnings.
• CI=true pnpm --filter eve exec vitest run --config vitest.unit.config.ts --retry=0: 8,484 passed, one existing skip, across 790 files.
• Focused integration validation: 6 passed across invoke-step.integration.test.ts, cancel-notification.integration.test.ts, and durable-session-store.integration.test.ts.
• recursive-session-result.scenario.test.ts: passed. The initial sandboxed run failed at startup with EMFILE; it passed outside the sandbox, including in the frozen-dependency checkout.
• Package typecheck, changed-fixture typecheck, formatting, lint, documentation checks, and invariant guard: passed.
• New regressions failed before their corresponding fixes: missing request context, acceptance of nonempty filtered summaries, premature task verification, mismatched failed-notification wording, and loss of partial eval events.
• No local E2E suites were run.

Real-Provider Local Replays

These are isolated request/compactor experiments, not full E2E runs or the confirmatory holdout:

• Captured failing second-compaction request, ten fixed alternating pairs: 10/10 original requests refused, versus 0/10 with the original user request supplied as quoted context. Every successful response retained the completion marker. This establishes evidence for that captured input, not an overall failure-rate claim.
• An initial implementation added the context even when the request was already in the transcript; it failed 20/20 local flows and was rejected before push.
• The corrected implementation supplements only missing request context: 20/20 complete local two-compaction flows passed, covering both repository inspection and stale-todo cases, with 40 real Anthropic summary requests and no refusals.
• A reconstructed visible-history purchasing-review replay passed five pairs under both old and proposed report instructions. It did not reproduce the CI refusal, so it does not establish a rate improvement.

Historical CI Evidence

All earlier failures remain part of the investigation:

Revision / Experiment	Outcome	Evidence
Input correction only	45/100 baseline failures vs 39/100 candidate failures; one-sided Fisher p = 0.2369455, not significant	Baseline, candidate
Adjacent-commit attribution	25/100 after #3165 vs 43/100 after #3131; p = 0.0054467 for an increase	Before, after
Old compaction wording	9/15 failures, reported reasoning_extraction	Run
7701e671 fixture wording pilot	0/15, but still included the now-excluded history-loss fix and lacked the later default-prompt changes	Run
4fcd9d01f prompt-cache diagnostic	7/20 failures, all later-turn cyber refusals	Run
f404c4521 compaction pilot	Anthropic 3/20 failures; OpenAI 0/20	Anthropic, OpenAI
f404c4521 prompt-cache pilot	Anthropic 1/20 failures, in the final report; OpenAI 0/20	Anthropic, OpenAI
f404c4521 background pilot	0/20 failures across all three local-world evals	Run
df1876272 positive-only compaction wording	Anthropic 1/20 failures; OpenAI 0/20	Anthropic, OpenAI
df1876272 normal CI	Vercel background-export and task-failure timeouts	Background export, task failure

Confirmatory Holdout

The registered holdout completed on September 10, 2026: 52/100 baseline suite trials failed, versus 21/300 candidate suite trials at aa463d6d1dc57f7179fde8c4417a28ba38f81f71. The one-sided suite-level Fisher exact p is 5.533752737638454e-21. The candidate's exact one-sided 95% failure-rate upper bound is 9.923926099%.

Component	Baseline failures / 100	Candidate failures / 300
Compaction / Anthropic	43	11
Compaction / OpenAI	0	0
Prompt cache / Anthropic	13	9
Prompt cache / OpenAI	1	2
Background reports	1	0

All 100 shards and 2,000 component trials completed and passed the source/model/coverage/run-attempt/harness-hash audit. Suite failures are the union by preregistered ordinal, not the sum of component failures. No failed trials were dropped or rerun, no pilots were pooled, and no code changed during the holdout. See the frozen registration and all ten workflow run IDs.

Statistically significant improvement, but acceptance failed: the published design required zero candidate failures, a below-1% upper bound, and green final-head CI. Candidate failures remain. At the frozen holdout SHA, fixture-tasks failed in local-world, Postgres, and Vercel CI; the task-only follow-up above is awaiting CI confirmation and does not address the remaining holdout failures.

The calculations are conditional on independent, stationary trials and the explicitly tested surfaces; shared-provider correlations and time variation limit the inference. The PR remains draft. The flakes are not eliminated.

Checklist

• This change was requested or approved by a maintainer
• I ran the relevant checks from CONTRIBUTING.md
• I added tests and documentation where relevant
• I added a changeset if this touches the published eve package
• DCO sign-off passes for every commit (git commit --signoff)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
eve-docs	Ready	Preview, v0	Sep 10, 2026 7:45pm UTC
eve-docs-4759	Building	Preview, v0	Sep 10, 2026 7:45pm UTC

1 Skipped Deployment

Project	Deployment	Actions	Updated
eve-pkg	Skipped	v0	Sep 10, 2026 7:45pm UTC

[github-actions]

Bundle + Package Summary: apps/fixtures/weather-agent

Key takeaways

• No notable deltas vs main (a24b68c).

Delta vs main (a24b68c)

Area	Metric	Baseline	Current	Delta
Package	Packed tarball	8.70 MB	8.74 MB	+35.4 kB ⚠️
Package	Unpacked publish size	32.78 MB	32.92 MB	+140.8 kB ⚠️
Package	Installed footprint	78.26 MB	78.41 MB	+140.8 kB ⚠️
Package	Published files	3913	3923	+10
Package	Installed files	7830	7840	+10
Package	Installed package instances	33	33	0
Package	Distinct installed package names	32	32	0
Package	Installed dependency edges	51	51	0
Package	Installed optional peer edges	8	8	0
Runtime	Unique function payloads	2	2	0
Runtime	Total function bytes	20.79 MB	21.02 MB	+234.0 kB ⚠️
Runtime	Public routes	18	18	0

Changed function payloads vs main (a24b68c) (2)

Function	Status	Baseline	Current	Delta	Route changes
functions/__server.func	changed	10.39 MB	10.51 MB	+117.0 kB ⚠️	none
functions/.well-known/workflow/v1/flow.func	changed	10.39 MB	10.51 MB	+117.0 kB ⚠️	none

eve init install

Metric	Baseline	Current	Delta
Installed footprint	116.38 MB	116.52 MB	+140.8 kB ⚠️
Installed packages	113	113	0
dependencies	4	4	0
devDependencies	2	2	0
Dependency package bytes	48.97 MB	49.11 MB	+140.8 kB ⚠️
devDependency package bytes	5.04 MB	5.04 MB	0 B ➖

eve init dependency changes vs main (a24b68c)

dependencies

• Changed: ai@^7.0.82 -> ai@^7.0.93, eve@file:eve-0.52.4.tgz -> eve@file:eve-0.53.0.tgz

Build Metadata

• Preset: vercel
• Nitro: nitro@3.0.260903-beta
• Output directory: apps/fixtures/weather-agent/.vercel/output
• Build metadata timestamp: 2026-09-10T19:45:39.524Z
• Route aliases: 18 public, 1 internal (19 total aliases)
• Vercel routes in config: 21
• Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small

Package Drill-Down

Package Details

• Package: eve@0.53.0
• Package directory: packages/eve
• Tarball: 8.74 MB (eve-0.53.0.tgz)
• Unpacked payload: 32.92 MB across 3923 published files
• Installed footprint: 78.41 MB across 7840 installed files
• Installed root package: 31.49 MB
• Installed dependencies: 46.91 MB
• Installed package instances: 33
• Distinct installed package names: 32
• Installed dependency edges: 51
• Installed optional peer edges: 8
• Runtime dependencies: 2
• Peer dependencies: 6 (5 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.
Graph metrics read only package.json files in package directories directly beneath a node_modules boundary, including nested boundaries. Each directory is one package instance; distinct names come from those manifests. Dependency edges count each unique name in dependencies or optionalDependencies per instance; optional peer edges count peerDependencies marked optional.

Heavy installed dependencies

• eve: 31.49 MB (40.2%)
• @rolldown/binding-linux-x64-gnu: 19.31 MB (24.6%)
• ai: 7.08 MB (9.0%)
• zod: 6.74 MB (8.6%)
• undici: 3.51 MB (4.5%)

Publish payload breakdown

Published file size
🔴 dist/src/compiled/shadcn-registry/index.js       [#############...........] 9.76 MB 29.6%
🟠 dist/src/compiled/@photon-ai/chat-adapter-ime... [###.....................] 2.27 MB 6.9%
🟠 dist/src/compiled/@ai-sdk/code-mode/index.js     [#.......................] 1.03 MB 3.1%
🟡 dist/src/compiled/@vercel/blob/index.js          [#.......................] 901.4 kB 2.7%
🟡 dist/src/compiled/_chunks/workflow/signal-exi... [#.......................] 514.6 kB 1.6%
🔴 Other published files                            [########################] 18.45 MB 56.1%

Installed footprint breakdown

Installed package size
🔴 eve                             [########################] 31.49 MB 40.2%
🔴 @rolldown/binding-linux-x64-gnu [###############.........] 19.31 MB 24.6%
🔴 ai                              [#####...................] 7.08 MB 9.0%
🔴 zod                             [#####...................] 6.74 MB 8.6%
🟠 undici                          [###.....................] 3.51 MB 4.5%
🟠 nitro                           [#.......................] 1.89 MB 2.4%
🔴 Other installed packages        [######..................] 8.37 MB 10.7%

Runtime dependencies (2)

Package	Range	Notes
nitro	3.0.260903-beta
undici	8.9.0

Peer dependencies (6)

Package	Range	Notes
@opentelemetry/api	^1.0.0	optional peer
ai	catalog:
braintrust	^3.0.0	optional peer
dd-trace	^6.13.0	optional peer
just-bash	^3.1.0	optional peer
microsandbox	^0.5.0	optional peer

eve init install drill-down

eve init install details

• Command: eve init my-agent
• Package manager: npm
• Installed footprint: 116.52 MB across 9724 installed files
• Installed packages: 113 total (107 transitive-only)
• dependencies: 4 direct packages totaling 49.11 MB
• devDependencies: 2 direct packages totaling 5.04 MB
• Other transitive package files: 62.36 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

• eve: 31.49 MB (27.0%)
• @typescript/typescript-linux-x64: 27.95 MB (24.0%)
• @rolldown/binding-linux-x64-gnu: 19.31 MB (16.6%)
• zod: 10.37 MB (8.9%)
• ai: 7.08 MB (6.1%)

Installed footprint breakdown

Installed package size
🔴 eve                              [########################] 31.49 MB 27.0%
🔴 @typescript/typescript-linux-x64 [#####################...] 27.95 MB 24.0%
🔴 @rolldown/binding-linux-x64-gnu  [###############.........] 19.31 MB 16.6%
🔴 zod                              [########................] 10.37 MB 8.9%
🔴 ai                               [#####...................] 7.08 MB 6.1%
🟠 undici                           [###.....................] 3.51 MB 3.0%
🔴 Other installed packages         [#############...........] 16.80 MB 14.4%

dependencies (4)

Package	Range	Installed size	Share
@vercel/connect	1.0.0	167.9 kB	0.1%
ai	^7.0.93	7.08 MB	6.1%
eve	file:eve-0.53.0.tgz	31.49 MB	27.0%
zod	4.5.4	10.37 MB	8.9%

devDependencies (2)

Package	Range	Installed size	Share
@types/node	24.x	2.54 MB	2.2%
typescript	7.0.2	2.50 MB	2.1%

Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 10.51 MB 50.0%
🔴 functions/__server.func                         [########################] 10.51 MB 50.0%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 10.51 MB

Metric	Value
Public routes	/.well-known/workflow/v1/flow
Runtime	nodejs24.x
Handler	index.mjs
Payload	10.51 MB
Function files	10.51 MB across 114 files
Traced dependencies	0 B
Signal	🟠 Bundled file index.mjs is 2.36 MB (22.5%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                        [###########.............] 2.36 MB 22.5%
🟡 _libs/undici.mjs                 [#####...................] 980.8 kB 9.3%
🟡 _chunks/esm-Fqlolk7e.mjs         [###.....................] 723.3 kB 6.9%
🟡 _chunks/chatgpt-model.mjs        [###.....................] 699.6 kB 6.7%
🟡 _chunks/signal-exit-Dsy-TT0V.mjs [###.....................] 616.2 kB 5.9%
🔴 Other bundled files              [########################] 5.13 MB 48.8%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ],
  "environment": {
    "WORKFLOW_PRECONDITION_GUARD": "1"
  }
}

---

🟠 functions/__server.func • 17 public routes, 1 internal alias • 10.51 MB

Metric	Value
Public routes	/ /.well-known/workflow/v1/webhook/[token] /eve/v1/activity/[token] /eve/v1/callback/[token] /eve/v1/connections/[name]/callback/[attemptId]/[token] /eve/v1/connections/[name]/callback/[token] /eve/v1/health /eve/v1/info /eve/v1/session /eve/v1/session/[parentSessionId]/subagents/[callId]/[childSessionId]/stream /eve/v1/session/[sessionId] /eve/v1/session/[sessionId]/cancel /eve/v1/session/[sessionId]/clear /eve/v1/session/[sessionId]/compact /eve/v1/session/[sessionId]/reset /eve/v1/session/[sessionId]/stream /eve/v1/task-input/[token]
Internal aliases	/__server
Runtime	nodejs24.x
Handler	index.mjs
Payload	10.51 MB
Function files	10.51 MB across 114 files
Traced dependencies	0 B
Signal	🟠 Bundled file index.mjs is 2.36 MB (22.5%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                        [###########.............] 2.36 MB 22.5%
🟡 _libs/undici.mjs                 [#####...................] 980.8 kB 9.3%
🟡 _chunks/esm-Fqlolk7e.mjs         [###.....................] 723.3 kB 6.9%
🟡 _chunks/chatgpt-model.mjs        [###.....................] 699.6 kB 6.7%
🟡 _chunks/signal-exit-Dsy-TT0V.mjs [###.....................] 616.2 kB 5.9%
🔴 Other bundled files              [########################] 5.13 MB 48.8%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}

Build Timing: e2e/fixtures/agent-tools-sandbox

This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.

Build mode: deployable Vercel build with sandbox template prewarm included.

• Build pipeline: 5.07 s -> 4.84 s (-226.8 ms) vs main (a24b68c).
• Timing is informational: shared GitHub runners are too variable for a hard timing budget.

Detailed phase timings vs `main (a24b68c)`

Phase	Baseline	Current	Delta
extension.check	0.7 ms	18.6 ms	+17.9 ms
project.resolve	0.2 ms	10.7 ms	+10.5 ms
workspace.create	0.5 ms	0.5 ms	0.0 ms
host.prepare	470.3 ms	393.4 ms	-76.9 ms
vercel.service-prefix.resolve	1.4 ms	1.3 ms	-0.1 ms
nitro.create	456.9 ms	438.7 ms	-18.2 ms
sandbox.prewarm	351.0 ms	340.7 ms	-10.3 ms
nitro.cache.prepare	0.2 ms	0.2 ms	0.0 ms
nitro.prepare	0.5 ms	0.6 ms	+0.1 ms
nitro.public-assets	0.6 ms	0.6 ms	0.0 ms
nitro.prerender	0.4 ms	0.4 ms	0.0 ms
nitro.bundle	3.71 s	3.57 s	-138.5 ms
nitro.cache.write	0.3 ms	0.3 ms	0.0 ms
vercel.workflow-function.materialize	60.5 ms	54.9 ms	-5.6 ms
agent-summary.emit	0.6 ms	0.5 ms	-0.1 ms
nitro.close	0.1 ms	0.1 ms	0.0 ms
output.publish	8.3 ms	2.5 ms	-5.8 ms
workspace.remove	1.4 ms	1.5 ms	+0.1 ms

[chadhietala]

Fixed Diagnostic Sample: Prompt-Cache Refusals

Recorded before dispatching or observing this sample:

• Source: 4fcd9d01f (fixture-only bounded refusal diagnostics; no prompt, assertion, provider, model, or retry changes).
• Model: matrix anthropic-opus (anthropic/claude-opus-5).
• Fixture: agent-prompt-cache; exact evals parallel-review/0000 parallel-review/0001.
• Sample: 20 complete fresh-process trials, four shards of five; retain all trials and no stopping on green.
• Purpose: identify the provider-reported stop category for ordinary reviewer-launch refusals. This sample selects no final candidate and is not the confirmatory holdout.
• Preserve all requirements: five distinct reviewers launched in one step, overlapping reviews, completion notifications, no repeated delegation, no compaction, and at least 98% provider cache reuse.
• Record filtered calls, incomplete attempts, and failed soft metrics independently of terminal success. Missing evidence, infrastructure failure, or incomplete trials cannot be treated as passes.

The final claim must name the exact eval/model cohorts and frozen source revision. A statistically significant reduction and an upper failure-rate bound are possible; a finite experiment cannot establish a literally zero failure rate across all future CI runs.

[chadhietala]

Candidate Pilot And Confirmatory Design

The completed diagnostic sample at 4fcd9d01f3505c5876fb2143d3b1b990d2421e8b had 7/20 failed trials, distributed 0, 2, 3, 2 across four complete shards. Every failure was a later-turn reviewer-launch content-filter with provider category cyber; the first-turn eval passed in every trial. Run: 34423285570. This is diagnostic evidence, not a holdout.

Before observing the next pilot:

• Candidate: f404c4521b4df29af5febebb9cb295cd66af38ae.
• Changes since that diagnostic: stable producer identity for background workflow report redelivery, and explicit sheet-identifier assignments for the purchasing reviewers. The redelivery unit regression failed before the runtime change and passes afterward. The existing E2E assertions, documents, provider/model settings, cache threshold, and timeouts remain unchanged.
• Exactly 20 fresh-process trials per component, four shards of five. Every trial runs all listed evals. Retain every trial; no stopping on green.

Component	Matrix model	Exact evals
agent-compaction-regressions	anthropic-opus	content-output-file-stub redundant-tool-calls stale-todo-work task-survival
agent-compaction-regressions	openai-sol	content-output-file-stub redundant-tool-calls stale-todo-work task-survival
agent-prompt-cache	anthropic-opus	parallel-review/0000 parallel-review/0001
agent-prompt-cache	openai-sol	parallel-review/0000 parallel-review/0001
agent-background-tools	openai-sol	background-export.update-and-complete empty-delivery-marker scheduled-export.quiet-launch

The background-tools fixture intentionally uses its existing deterministic model to test Workflow delivery. The other four components continue to use their real matrix providers. No real-provider test is replaced with a mock.

Confirmatory Acceptance

After diagnostic work is complete, freeze a final source SHA and publish the run IDs before examining confirmatory outcomes:

1. Compare 100 complete baseline-suite trials at PR base a24b68c5774bffeb0e4e762c3aab3ec7ab64a179 with 300 complete final-candidate-suite trials.
2. One suite trial contains all five components above. Join component trials by their preassigned global ordinal, (shard - 1) * repetitions + repetition, never by outcome. Use ten shards of ten per baseline component and ten shards of thirty per candidate component.
3. Primary failure is the union of terminal failure, a failed hard or soft assertion, a filtered/error model step, a failed model step, or an uncompleted compaction request in any component. Thus eventual success after a refused/retried call cannot qualify as clean.
4. Primary comparison: one-sided Fisher exact test for a lower candidate suite-failure rate, alpha 0.05. Report component counts separately; this is a suite-level significance claim, not a claim that each component's individual improvement is significant.
5. Require zero primary failures in all 300 candidate suite trials, plus the significant primary comparison and green final-head full CI. The exact one-sided 95% binomial upper bound for 0/300 is approximately 0.994%, conditional on independent, stationary suite trials. That is not proof of a literally zero failure rate.
6. All components and shards must finish with the intended source, model, eval coverage, and matching harness hashes. Missing, cancelled, malformed, or incomplete evidence cannot pass. Do not pool pilots into the holdout, extend a failed holdout, or omit failures.

This scopes the statistical claim to the observed compaction, reviewer-delegation/cache, and background-report failure surfaces. It does not establish that every possible future CI test is flake-free. Shared-provider correlations and time variation remain limitations and will be reported.

[chadhietala]

Compaction Pilot Result And Next Candidate

The f404c4521 full compaction pilot completed all 20 trials per model:

• OpenAI: 0/20 terminal or strict-reliability failures.
• Anthropic: 3/20 terminal and strict-reliability failures, all in redundant-tool-calls. All three exhausted four workflow attempts on the second compaction. Each retained providerStopCategory: reasoning_extraction, summaryAttempt: 1, olderMessageCount: 2, and recentMessageCount: 3.
• stale-todo-work, task-survival, and content-output-file-stub passed all 20 Anthropic trials. No failed soft metrics were hidden by terminal success.
• The normal full CI run at f404c4521 is green, but the repeated failures disqualify that candidate for the confirmatory holdout.

Before dispatching the next pilot:

• Source: df18762722c07179232f37f0c3a6bfafddd41270.
• Only behavior change since f404c4521: the summarizer's source-limit sentence now states that the supplied records are the complete source and only explicitly recorded facts may be reported. It no longer mentions private reasoning or hidden instructions, even negatively. Renderer exclusions, transcript caps, summary content requirements, fixture inputs, models, retry behavior, and E2E assertions are unchanged.
• This is a wording hypothesis, not proof that a specific phrase caused the refusal.
• Run 20 fresh-process trials per model, four shards of five, for agent-compaction-regressions on anthropic-opus and openai-sol.
• Exact evals: content-output-file-stub redundant-tool-calls stale-todo-work task-survival.
• Retain every trial. This replaces no earlier evidence and contributes no observations to the confirmatory holdout.
• Focused local compaction tests: 71 passed; targeted formatting/lint passed.

The purchasing-review pilots at f404c4521 continue separately. Their evals forbid compaction, and this new candidate does not change their fixture or the ordinary model-call path. The final confirmatory suite will still use one frozen SHA for every component.

[chadhietala]

Frozen Confirmatory Holdout

Starting the requested holdout on September 10, 2026, without another pilot or code change. This follows the previously published five-component confirmatory design. The candidate has known failing fixture-tasks checks in current CI; those failures remain evidence, and final-head green CI remains a separate acceptance requirement.

• Baseline source: a24b68c5774bffeb0e4e762c3aab3ec7ab64a179.
• Candidate source and repetition harness: aa463d6d1dc57f7179fde8c4417a28ba38f81f71.
• Baseline: 100 fresh suite trials, ten shards of ten per component.
• Candidate: 300 fresh suite trials, ten shards of thirty per component.
• Dispatch each arm with an explicit source revision. Audit the workflow/harness revision and matching parser/runner hashes in every artifact.

Component	Matrix model	Exact eval IDs
agent-compaction-regressions	anthropic-opus	content-output-file-stub redundant-tool-calls stale-todo-work task-survival
agent-compaction-regressions	openai-sol	content-output-file-stub redundant-tool-calls stale-todo-work task-survival
agent-prompt-cache	anthropic-opus	parallel-review/0000 parallel-review/0001
agent-prompt-cache	openai-sol	parallel-review/0000 parallel-review/0001
agent-background-tools	openai-sol	background-export.update-and-complete empty-delivery-marker scheduled-export.quiet-launch

Background-tools keeps its existing deterministic Workflow-delivery model. The other four components use their real matrix providers. This is 2,000 component trials, representing 100 baseline and 300 candidate suite trials, not 2,000 independent suite trials.

Join all five components by the preassigned ordinal (shard - 1) * repetitions + repetition. A suite fails if any component has a terminal/gate/soft assertion failure, filtered/error model step, failed step, or uncompleted compaction request. Retain failed, missing, cancelled, malformed, and incomplete evidence. No selective reruns, early stopping, sample extension, or pooling with pilots.

Acceptance remains zero candidate failures, one-sided suite-level Fisher exact p < 0.05, the exact one-sided 95% failure-rate upper bound below 1%, and green final-head full CI. Report all component counts and limitations from shared-provider correlation, temporal drift, and the tested surfaces.

Workflow run IDs will be recorded after dispatch and before examining holdout outcomes. No claim of success is being made.

[chadhietala]

Confirmatory Holdout Dispatched

These run IDs are recorded before examining holdout outcomes. All runs use the frozen aa463d6d1dc57f7179fde8c4417a28ba38f81f71 repetition harness. Baseline source is a24b68c5774bffeb0e4e762c3aab3ec7ab64a179; candidate source is aa463d6d1dc57f7179fde8c4417a28ba38f81f71.

Component	Baseline: 10 x 10	Candidate: 10 x 30
Compaction / Anthropic	34495917165	34495917045
Compaction / OpenAI	34495917176	34495917082
Prompt cache / Anthropic	34495917305	34495917168
Prompt cache / OpenAI	34495917318	34495917506
Background reports	34495917301	34495917379

Expected evidence: 100 shards and 2,000 component trials, joined into 100 baseline and 300 candidate suite trials by the preassigned ordinal. Every failure and incomplete shard remains in the outcome.

Frozen runner SHA-256: 8fc28586cba8533c92e380c9d80316f84648000fa4e440b47bcfe439b5e798b2.

Frozen parser SHA-256: a302346b645b7814e4452e7799484740afec5cf4a4dd680d32ad73739f11e1a8.

Frozen statistical collector SHA-256: 743b2cd00351f144e7f87f33512a1dfac79ff6aec25d66217c5c9aa4d75b7ac8.

The 20 repetition-parser self-tests and the SciPy 1.13.1 statistical self-test passed locally before dispatch. No code changes or reruns are planned during the holdout.

[chadhietala]

Holdout Complete: Acceptance Failed

The preregistered holdout completed on September 10, 2026. The candidate improved the measured suite-failure rate significantly, but it did not eliminate failures.

Primary suite outcome	Baseline	Candidate
Frozen source	a24b68c5774bffeb0e4e762c3aab3ec7ab64a179	aa463d6d1dc57f7179fde8c4417a28ba38f81f71
Complete suite trials	100	300
Failed suite trials	52 (52%)	21 (7%)

• One-sided Fisher exact p: 5.533752737638454e-21.
• Candidate exact one-sided 95% failure-rate upper bound: 9.923926099%.
• The zero-candidate-failure and below-1% upper-bound acceptance criteria failed.
• Final-head full CI is also not green: fixture-tasks failed in the local-world, Postgres, and Vercel workflows. No normal-CI checks are still pending.

Component Failure Counts

Component	Baseline failures / 100	Candidate failures / 300
Compaction / Anthropic	43	11
Compaction / OpenAI	0	0
Prompt cache / Anthropic	13	9
Prompt cache / OpenAI	1	2
Background reports	1	0

The primary result is the union by preregistered suite ordinal, not the sum of component failures: one suite trial can fail in multiple components. All terminal failures and strict reliability failures had matching trial counts. The candidate compaction failures contained 44 uncompleted compaction requests; the Anthropic prompt-cache cohort contained three filtered model steps. These are retained, not excused by eventual success.

Evidence Audit

• All 100 shards and 2,000 component trials are present, forming 100 baseline and 300 candidate suite trials.
• Every shard completed its registered count with the expected source SHA, run ID, model, exact eval selection, parser hash, and runner hash.
• Every workflow ran once (githubRunAttempt=1). No failed trials were excluded, no failed jobs were rerun, and pilots were not pooled into this holdout.
• The frozen statistical collector hash matched; its self-test and the 20 repetition-parser tests passed. Analysis used SciPy 1.13.1.
• No code changed during the holdout. The unrelated local lockfile edit was left untouched.
• Registration and all ten workflow run IDs were published before outcomes were examined.

The significance calculation and binomial bound assume independent, stationary suite trials. Shared-provider correlation, time variation, and the explicitly tested surfaces limit the inference. This is a suite-level improvement result, not proof that each component improved significantly or that all future CI runs are flake-free.

Conclusion: statistically significant improvement, but the holdout failed and the PR remains draft. Flakes remain.

[chadhietala]

Task-Fixture Correction; PR Still Incomplete

Pushed signed commit 0dfc9d4afbc6ea341c96f594843eb57f469eca17, limited to the task-test wait helper and regression tests.

The failed repeated-cancellation eval already had a successful task_cancel result showing the task was cancelled. The client correctly correlated the repeated send to its own delivery and advanced past the earlier cancellation notification. My new helper incorrectly required that notification again. It now recognizes a recorded successful terminal task view, then performs the same verification turn and assertions. Tests still wait for the notification when they have no terminal evidence; incorrect task identities, working states, and failed actions do not qualify.

Validation before push in the frozen-dependency checkout:

• 8,491 unit tests passed; one existing skip.
• Cancellation notification integration test passed.
• Package and task-fixture typechecks passed.
• Formatting, lint, task-transition validation, and invariant guard passed.
• New terminal-evidence tests failed before the fix; a real-client regression covers the skipped-notification cursor behavior.
• GitHub verified the signed commit and its DCO trailer is present.

The rejected formatting-only compaction diagnostic also remains recorded: five paired replays of a failed holdout input refused under both the current format and the alternative format, each reporting reasoning_extraction. No formatting candidate was added to the PR.

The holdout's remaining failures are unchanged: eleven compaction refusals, three reviewer refusals, seven cache-reuse failures, and one split reviewer launch, overlapping into 21 failed suite trials. This commit does not claim to solve them. New CI is running; the failed holdout remains the statistical result.