Skip to content

fix(eve): deflake workflow and fixture tests - #3165

Merged
chadhietala merged 8 commits into
mainfrom
chad/deflake-actions
Sep 9, 2026
Merged

chadhietala merged 8 commits into
mainfrom
chad/deflake-actions

Conversation

@chadhietala

@chadhietala chadhietala commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Intermittent Actions failures came from shared test state, buffered events, fixture dependency resolution, nondeterministic evals, and scaffold installs racing npm publication. This PR fixes those conditions, adds bounded Windows workflow-read retries, preserves pending tasks through compaction, and retains scripted mock responders. The scaffold test now installs the local eve tarball while preserving the 48-hour registry release-age policy; production scaffolding, authorization and ordering assertions, and the default E2E matrix are unchanged. Scope remains limited to fixes and regression coverage, with general repetition tools, manual CI controls, and unproven dev-server diagnostics excluded.

Workflow Release

The temporary Workflow patch remains necessary until a release containing vercel/workflow#4051 is available and adopted. This PR keeps the patch in place.

Validation

Latest scaffold publication-race fix (a6b5c5c77):

  • The focused release-age scenario passed in the ordinary environment.
  • pnpm --filter eve exec vitest run --config vitest.scenario.config.ts test/scenarios/eve-init.scenario.test.ts --retry=0, with GITHUB_HEAD_REF=changeset-release/main and a local pnpm store: all 10 tests passed, none skipped.
  • Regression coverage exercises an intentionally unpublished eve dependency range, verifies the local-tarball override, and checks that the generated project retains strict release-age policy without added exclusions. The obsolete release-branch skip is removed.
  • pnpm --filter eve exec tsc -p tsconfig.json --noEmit, targeted formatting/lint, pnpm guard:invariants, and the whitespace check passed.
  • The full scenario suite was not rerun for this single-file fix.

CI and the Local, Vercel, and Postgres E2E workflows passed on 5f9547b2a, before the scope cleanup and latest scaffold fix.

After scope cleanup, before the latest scaffold fix:

  • pnpm --filter agent-channels test:unit: 3 deployment-revision regression tests passed.
  • pnpm --filter eve exec vitest run --config vitest.unit.config.ts --maxWorkers=4 --silent: 8,415 passed; 1 existing skip.
  • pnpm --filter eve exec tsc -p tsconfig.json --noEmit: passed.
  • pnpm guard:invariants: passed. Targeted formatting and lint passed.
  • The restored workflow and discovery script match main byte-for-byte. Default discovery output is unchanged: 37 model-suite jobs and 29 fixture jobs per world.
  • No references to the removed repetition tooling remain. The task-reporting experiment is already on main and is not part of this PR's diff.

Earlier fix validation included 44 successful workspace typecheck tasks, 765 integration tests, and local shuffled repetition samples. The recorded samples are evidence for their selected tests, not proof that every possible CI flake is eliminated. Their runners and raw evidence are not part of this PR.

Full E2E suites run only in CI. Earlier green runs do not establish current-head CI status; required checks must pass on the latest head before merge. No existing E2E assertion or default matrix leg was removed.

Checklist

  • This change was requested or approved by a maintainer
  • I ran the relevant checks from CONTRIBUTING.md
  • I added tests and documentation where relevant
  • I added a changeset if this touches the published eve package
  • DCO sign-off passes for every pushed commit (git commit --signoff)

@vercel

vercel Bot commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
eve-docs Ready Ready Preview, v0 Sep 9, 2026 3:16pm UTC
eve-docs-4759 Ready Ready Preview, v0 Sep 9, 2026 3:16pm UTC
1 Skipped Deployment
Project Deployment Actions Updated
eve-pkg Skipped Skipped v0 Sep 9, 2026 3:16pm UTC

@github-actions

github-actions Bot commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Bundle + Package Summary: apps/fixtures/weather-agent

Key takeaways

  • No notable deltas vs main (6435811).

Delta vs main (6435811)

Area Metric Baseline Current Delta
Package Packed tarball 8.70 MB 8.70 MB +20 B ⚠️
Package Unpacked publish size 32.77 MB 32.77 MB +2.9 kB ⚠️
Package Installed footprint 77.90 MB 77.91 MB +2.9 kB ⚠️
Package Published files 3909 3911 +2
Package Installed files 7812 7814 +2
Package Installed package instances 33 33 0
Package Distinct installed package names 32 32 0
Package Installed dependency edges 51 51 0
Package Installed optional peer edges 8 8 0
Runtime Unique function payloads 2 2 0
Runtime Total function bytes 20.78 MB 20.78 MB +666 B ⚠️
Runtime Public routes 18 18 0
Changed function payloads vs main (6435811) (2)
Function Status Baseline Current Delta Route changes
functions/__server.func changed 10.39 MB 10.39 MB +333 B ⚠️ none
functions/.well-known/workflow/v1/flow.func changed 10.39 MB 10.39 MB +333 B ⚠️ none

eve init install

Metric Baseline Current Delta
Installed footprint 116.34 MB 116.34 MB +2.9 kB ⚠️
Installed packages 113 113 0
dependencies 4 4 0
devDependencies 2 2 0
Dependency package bytes 48.94 MB 48.94 MB +2.9 kB ⚠️
devDependency package bytes 5.04 MB 5.04 MB 0 B ➖
eve init dependency changes vs main (6435811)

dependencies

  • Changed: eve@file:eve-0.52.3.tgz -> eve@file:eve-0.52.4.tgz
Build Metadata
  • Preset: vercel
  • Nitro: nitro@3.0.260903-beta
  • Output directory: apps/fixtures/weather-agent/.vercel/output
  • Build metadata timestamp: 2026-09-09T15:17:17.159Z
  • Route aliases: 18 public, 1 internal (19 total aliases)
  • Vercel routes in config: 21
  • Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small
Package Drill-Down

Package Details

  • Package: eve@0.52.4
  • Package directory: packages/eve
  • Tarball: 8.70 MB (eve-0.52.4.tgz)
  • Unpacked payload: 32.77 MB across 3911 published files
  • Installed footprint: 77.91 MB across 7814 installed files
  • Installed root package: 31.34 MB
  • Installed dependencies: 46.56 MB
  • Installed package instances: 33
  • Distinct installed package names: 32
  • Installed dependency edges: 51
  • Installed optional peer edges: 8
  • Runtime dependencies: 2
  • Peer dependencies: 6 (5 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.
Graph metrics read only package.json files in package directories directly beneath a node_modules boundary, including nested boundaries. Each directory is one package instance; distinct names come from those manifests. Dependency edges count each unique name in dependencies or optionalDependencies per instance; optional peer edges count peerDependencies marked optional.

Heavy installed dependencies

  • eve: 31.34 MB (40.2%)
  • @rolldown/binding-linux-x64-gnu: 19.31 MB (24.8%)
  • ai: 7.06 MB (9.1%)
  • zod: 6.41 MB (8.2%)
  • undici: 3.51 MB (4.5%)
Publish payload breakdown
Published file size
🔴 dist/src/compiled/shadcn-registry/index.js       [#############...........] 9.76 MB 29.8%
🟠 dist/src/compiled/@photon-ai/chat-adapter-ime... [###.....................] 2.27 MB 6.9%
🟠 dist/src/compiled/@ai-sdk/code-mode/index.js     [#.......................] 1.03 MB 3.1%
🟡 dist/src/compiled/@vercel/blob/index.js          [#.......................] 901.4 kB 2.8%
🟡 dist/src/compiled/_chunks/workflow/signal-exi... [#.......................] 514.6 kB 1.6%
🔴 Other published files                            [########################] 18.30 MB 55.9%
Installed footprint breakdown
Installed package size
🔴 eve                             [########################] 31.34 MB 40.2%
🔴 @rolldown/binding-linux-x64-gnu [###############.........] 19.31 MB 24.8%
🔴 ai                              [#####...................] 7.06 MB 9.1%
🔴 zod                             [#####...................] 6.41 MB 8.2%
🟠 undici                          [###.....................] 3.51 MB 4.5%
🟠 nitro                           [#.......................] 1.89 MB 2.4%
🔴 Other installed packages        [######..................] 8.37 MB 10.7%
Runtime dependencies (2)
Package Range Notes
nitro 3.0.260903-beta
undici 8.9.0
Peer dependencies (6)
Package Range Notes
@opentelemetry/api ^1.0.0 optional peer
ai catalog:
braintrust ^3.0.0 optional peer
dd-trace ^6.13.0 optional peer
just-bash ^3.1.0 optional peer
microsandbox ^0.5.0 optional peer
eve init install drill-down

eve init install details

  • Command: eve init my-agent
  • Package manager: npm
  • Installed footprint: 116.34 MB across 9710 installed files
  • Installed packages: 113 total (107 transitive-only)
  • dependencies: 4 direct packages totaling 48.94 MB
  • devDependencies: 2 direct packages totaling 5.04 MB
  • Other transitive package files: 62.36 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

  • eve: 31.34 MB (26.9%)
  • @typescript/typescript-linux-x64: 27.95 MB (24.0%)
  • @rolldown/binding-linux-x64-gnu: 19.31 MB (16.6%)
  • zod: 10.37 MB (8.9%)
  • ai: 7.06 MB (6.1%)
Installed footprint breakdown
Installed package size
🔴 eve                              [########################] 31.34 MB 26.9%
🔴 @typescript/typescript-linux-x64 [#####################...] 27.95 MB 24.0%
🔴 @rolldown/binding-linux-x64-gnu  [###############.........] 19.31 MB 16.6%
🔴 zod                              [########................] 10.37 MB 8.9%
🔴 ai                               [#####...................] 7.06 MB 6.1%
🟠 undici                           [###.....................] 3.51 MB 3.0%
🔴 Other installed packages         [#############...........] 16.80 MB 14.4%
dependencies (4)
Package Range Installed size Share
@vercel/connect 1.0.0 167.9 kB 0.1%
ai ^7.0.82 7.06 MB 6.1%
eve file:eve-0.52.4.tgz 31.34 MB 26.9%
zod 4.5.4 10.37 MB 8.9%
devDependencies (2)
Package Range Installed size Share
@types/node 24.x 2.54 MB 2.2%
typescript 7.0.2 2.50 MB 2.1%
Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 10.39 MB 50.0%
🔴 functions/__server.func                         [########################] 10.39 MB 50.0%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 10.39 MB
Metric Value
Public routes /.well-known/workflow/v1/flow
Runtime nodejs24.x
Handler index.mjs
Payload 10.39 MB
Function files 10.39 MB across 113 files
Traced dependencies 0 B
Signal 🟠 Bundled file index.mjs is 2.34 MB (22.5%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                        [###########.............] 2.34 MB 22.5%
🟡 _libs/undici.mjs                 [#####...................] 980.8 kB 9.4%
🟡 _chunks/esm-Fqlolk7e.mjs         [###.....................] 723.3 kB 7.0%
🟡 _chunks/chatgpt-model.mjs        [###.....................] 698.9 kB 6.7%
🟡 _chunks/signal-exit-Dsy-TT0V.mjs [###.....................] 616.2 kB 5.9%
🔴 Other bundled files              [########################] 5.03 MB 48.4%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ],
  "environment": {
    "WORKFLOW_PRECONDITION_GUARD": "1"
  }
}

🟠 functions/__server.func • 17 public routes, 1 internal alias • 10.39 MB
Metric Value
Public routes /
/.well-known/workflow/v1/webhook/[token]
/eve/v1/activity/[token]
/eve/v1/callback/[token]
/eve/v1/connections/[name]/callback/[attemptId]/[token]
/eve/v1/connections/[name]/callback/[token]
/eve/v1/health
/eve/v1/info
/eve/v1/session
/eve/v1/session/[parentSessionId]/subagents/[callId]/[childSessionId]/stream
/eve/v1/session/[sessionId]
/eve/v1/session/[sessionId]/cancel
/eve/v1/session/[sessionId]/clear
/eve/v1/session/[sessionId]/compact
/eve/v1/session/[sessionId]/reset
/eve/v1/session/[sessionId]/stream
/eve/v1/task-input/[token]
Internal aliases /__server
Runtime nodejs24.x
Handler index.mjs
Payload 10.39 MB
Function files 10.39 MB across 113 files
Traced dependencies 0 B
Signal 🟠 Bundled file index.mjs is 2.34 MB (22.5%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                        [###########.............] 2.34 MB 22.5%
🟡 _libs/undici.mjs                 [#####...................] 980.8 kB 9.4%
🟡 _chunks/esm-Fqlolk7e.mjs         [###.....................] 723.3 kB 7.0%
🟡 _chunks/chatgpt-model.mjs        [###.....................] 698.9 kB 6.7%
🟡 _chunks/signal-exit-Dsy-TT0V.mjs [###.....................] 616.2 kB 5.9%
🔴 Other bundled files              [########################] 5.03 MB 48.4%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}

Build Timing: e2e/fixtures/agent-tools-sandbox

This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.

Build mode: deployable Vercel build with sandbox template prewarm included.

  • Build pipeline: 5.86 s -> 5.95 s (+88.6 ms) vs main (6435811).
  • Timing is informational: shared GitHub runners are too variable for a hard timing budget.
Detailed phase timings vs `main (6435811)`
Phase Baseline Current Delta
extension.check 18.5 ms 22.3 ms +3.8 ms
project.resolve 3.1 ms 6.1 ms +3.0 ms
workspace.create 2.9 ms 5.9 ms +3.0 ms
host.prepare 425.2 ms 455.2 ms +30.0 ms
vercel.service-prefix.resolve 2.0 ms 2.0 ms 0.0 ms
nitro.create 582.3 ms 585.2 ms +2.9 ms
sandbox.prewarm 379.2 ms 318.8 ms -60.4 ms
nitro.cache.prepare 0.2 ms 0.3 ms +0.1 ms
nitro.prepare 0.8 ms 1.0 ms +0.2 ms
nitro.public-assets 0.8 ms 0.8 ms 0.0 ms
nitro.prerender 0.4 ms 0.5 ms +0.1 ms
nitro.bundle 4.39 s 4.49 s +101.3 ms
nitro.cache.write 0.4 ms 0.4 ms 0.0 ms
vercel.workflow-function.materialize 48.2 ms 51.7 ms +3.5 ms
agent-summary.emit 0.6 ms 0.8 ms +0.2 ms
nitro.close 0.1 ms 0.2 ms +0.1 ms
output.publish 3.3 ms 3.8 ms +0.5 ms
workspace.remove 2.1 ms 2.3 ms +0.2 ms

Signed-off-by: Chad Hietala <chad.hietala@vercel.com>
Signed-off-by: Chad Hietala <chad.hietala@vercel.com>
Signed-off-by: Chad Hietala <chad.hietala@vercel.com>
Reset test-owned mock and renderer state, isolate fixture dependencies, and refresh npm metadata during scenario installs. Preserve task instructions through compaction and make principal-forwarding fixture dispatch deterministic while retaining authorization checks.

Validated unit, integration, and scenario suites locally, plus repeated targeted selections with retries disabled. Remote E2E validation remains pending on this commit.

Signed-off-by: Chad Hietala <chad.hietala@vercel.com>
Restore the test hook import required by main's new batching tests. Keep explicitly authored mockModel responders when runtime model mocking is enabled, while retaining generic replacement for real providers and the subagent fixture's default mock model ID.

Validated 44 workspace typecheck tasks, 8415 unit tests, 765 integration tests, and 30 shuffled focused runs. Vercel continuation timeouts remain under investigation.

Signed-off-by: Chad Hietala <chad.hietala@vercel.com>
Signed-off-by: Chad Hietala <chad.hietala@vercel.com>
Remove repetition runners and manual CI controls used for investigation, and drop unproven dev-server diagnostic edits. Preserve deployment-revision regression coverage in the fixture's normal unit suite without changing the default E2E matrix or existing eval assertions.

Signed-off-by: Chad Hietala <chad.hietala@vercel.com>
@chadhietala chadhietala changed the title fix(eve): deflake workflow tests and add repeat validation fix(eve): deflake workflow and fixture tests Sep 9, 2026
@chadhietala
chadhietala marked this pull request as ready for review September 9, 2026 14:56
Resolve the package under test from the existing scenario tarball through a temporary pnpm hook while retaining registry dependency release-age checks. Remove the release-branch skip and exercise an intentionally unpublished eve range.

Signed-off-by: Chad Hietala <chad.hietala@vercel.com>
@chadhietala
chadhietala merged commit d630b21 into main Sep 9, 2026
123 checks passed
@chadhietala
chadhietala deleted the chad/deflake-actions branch September 9, 2026 15:35

This branch was successfully deployed

2 active and 1 inactive deployments
Preview – eve-docs — a6b5c5c7 Deployed Sep 9, 2026 by vercel[bot]
Preview – eve-docs-4759 — a6b5c5c7 Deployed Sep 9, 2026 by vercel[bot]
Preview – eve-pkg — a6b5c5c7 Deployed Sep 9, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants