Skip to content

test: stop unit tests waiting out real 5 s fallbacks (~285 s per run) and fix two flaky tests - #37362

Merged
nwparker merged 1 commit into
mainfrom
test/prune-expensive-low-value-tests
Nov 2, 2026
Merged

nwparker merged 1 commit into
mainfrom
test/prune-expensive-low-value-tests

Conversation

@nwparker

@nwparker nwparker commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 4 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​23 $\color{#cf222e}{\Huge{\mathbf{−}}}$​5 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​18
Prod 1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​4 $\color{#cf222e}{\Huge{\mathbf{−}}}$​1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​3

Cuts about 285 s of test-execution time from every full unit run by removing real 5 s waits that protected nothing. Also fixes the two tests that recently failed intermittently on main. No test declarations are deleted and no workflow gating changes.

Where unit test time goes

Measured from the per-file phase artifacts of green main run 41392747240 (12 shards, 13,535 files):

Phase (summed worker time) Seconds
Total 16,624
Collect (imports/transform) 11,111
Test execution 4,840

Imports take about two thirds of worker time and grow with the number of files, not the number of declarations. So removing individual cheap tests saves little. Most of the execution time that can be recovered sits in tests that wait out a real production timeout. I ranked the 90 slowest files, ran them locally with --reporter=json to get per-test durations, and traced every timer of 1.5 s or longer in the suspicious files.

The largest finding was a test-harness bug. createConptyPreludeSender sent the mock ConPTY DA1 query on a microtask, so only listeners subscribed in that same microtask received it. #36422 added an earlier onData subscriber (the physical owner's initial-output observer), and the startup watch subscribes after an await. The watch never saw the prelude, so every Windows-path spawn in these suites waited the full CONPTY_STARTUP_DECISION_LIMIT_MS (5 s), timed out to undecided, and passed anyway. Sending the prelude on a macrotask (captured before tests fake timers) matches real node-pty output, which reaches every listener subscribed during spawn. Because of the bug, these tests had been exercising the fallback path instead of the input-reader path they describe.

Changes

File What changed Why (anti-pattern) CI s before (run 41392747240) Local s after
src/main/daemon/pty-subprocess-test-harness.ts (affects 9 pty-subprocess-* suites) Send the mock ConPTY prelude on a real macrotask instead of a microtask Waits on real time; the mock contradicted the real producer (output reaches all listeners), so every spawn waited out a 5 s fallback 261.1 0.7
structured-agent-session-outgrown-child.test.ts The fixture's runSetup answers right away for an account that is already prepared. Unprepared accounts still wait on the test-controlled setup promise The fixture contradicted the real producer: each launch of the ready account waited the 5 s refresh bound in prepareClaudeProfileForLaunch 16.3 1.1
worktree-creation-chat-handoff.test.tsx The mocked chat pane registers as actionable through registerNativeChatPane, as the real pane does Same: the handoff waited its 5 s HANDOFF_READY_TIMEOUT_MS fallback instead of covering the ready path 10.5 ~0.5
structured-agent-shell-environment.test.ts (flaky) Dropped the counter-file read Root cause: once /profile/B is served, a background run 4 truncates the counter (echo $n > file) while the test reads it, so it read 0. The assertion was redundant anyway, because only run 3 and later print /profile/B, which the next line asserts 0.6 1.3
claude-status-sign-in.test.tsx (flaky) getByText became findByText for the sign-in marker. In the remote-server case the marker is now awaited before asserting there is no Sign in button Root cause: the marker comes from the async account snapshot, after the settings-backed rows that openAccounts waits for. The old remote-server negative check could also pass before the marker loaded (a negative case that never reached its rejection path) 0.7 0.9

Total for the affected files: 293.4 s → 8.4 s (about 285 s, roughly 5.9% of unit test-execution time per full run). All 291 tests in the affected files pass locally; the counts are unchanged.

Deliberately kept despite cost

File CI s Why kept
codex-interrupt-order, codex-goal-stop-settings, codex-stopped-send-order, codex-turn-end-settlement ~45 The tests deliberately answer the interrupt before turn/completed, so Stop waits out CODEX_STOP_GRACE_MS (5 s). Faking timers across sqlite and lockfile timers is risky, and an injectable grace would be a test-only production seam
remote-runtime-request-connection 10 Asserts that the real 5 s cancellation timer elapses (marked with ast-grep-ignore)
account-add-process, remote-node-runtime-host-download, server-claude-helper-*, wsl-distro-holder-script, codex app-server teardown/supervision ~150 Real-process regression tests tied to fix commits (#36292, #35869, #35930) covering real deadlines and process groups, including WSL and remote hosts
Scaling suites (input-order-projection N=100, journal-*-scaling, acp-history-scaling) ~90 They count work rather than time, as complexity guards. Shrinking N (for example 100 to 30 in input-order-projection, about 14 s) is up to the owners
use-ai.operation-recovery "cold saved recovery (false)" 10 Waits twice on the 5 s chat-discovery bound in acquireAgentLaunchChoices, because the runtime fixture lacks host capabilities. A fix needs a runtime catalog fixture; left for a follow-up
journal-checkpoint-scaling 18 Each case pays about 2 s inside the checkpoint worker's own schedule; it can't be reduced without a production seam

Checks

  • pnpm test <affected files> --maxWorkers=2: 291 passed, 0 failed. The two flaky files were also run 3 times.
  • pnpm exec oxlint and pnpm exec oxfmt --check on the changed files: clean.
  • pnpm check:flaky-test-pattern-ratchet: OK. No baseline references changed, and no files were deleted, so the shard timings need no edit.
  • pnpm tc:node: the only error is the pre-existing agent-model-catalog-scaling.test.ts import that is failing on main right now.

The unit shard balancer still uses the October 25 timing baseline, so it will overweight the pty-subprocess-* files until its next refresh.

@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 9 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d735e6e0-7d88-4ee6-b226-d94c92712040
Commits

Reviewing files that changed from the base of the PR and between d712485 and 2264809.

Files selected for processing: 5
  • src/main/daemon/pty-subprocess-test-harness.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-outgrown-child.test.ts
  • src/main/runtime/structured-agent-shell-environment.test.ts
  • src/renderer/src/app-shell/worktree-creation-chat-handoff.test.tsx
  • src/renderer/src/components/status-bar/claude-status-sign-in.test.tsx
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…aky tests

The ConPTY prelude mock delivered its DA1 query on a microtask, which reached only
the physical owner's early onData listener (added in #36422); the startup watch
subscribes after an await, never saw it, and every Windows-path spawn waited out
CONPTY_STARTUP_DECISION_LIMIT_MS (5 s). Deliver it on a macrotask like real output.

Also give two fixtures the readiness their production peers provide, and fix the
counter-file race in structured-agent-shell-environment and the async sign-in
marker race in claude-status-sign-in.
@nwparker
nwparker force-pushed the test/prune-expensive-low-value-tests branch from 061cbdb to 2264809 Compare November 2, 2026 06:34
@nwparker
nwparker merged commit 0fad93e into main Nov 2, 2026
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant