Skip to content

Fix send-integration e2e flake: teardown budget asymmetry (10s afterAll vs 120s beforeAll) - #1302

Merged
waleedkadous merged 1 commit into
mainfrom
fix/send-e2e-teardown-timeout
Jul 30, 2026
Merged

Fix send-integration e2e flake: teardown budget asymmetry (10s afterAll vs 120s beforeAll)#1302
waleedkadous merged 1 commit into
mainfrom
fix/send-e2e-teardown-timeout

Conversation

@waleedkadous

Copy link
Copy Markdown
Contributor

Problem

send-integration.e2e.test.ts fails intermittently in the Tower Integration Tests CI job with Error: Hook timed out in 10000msafter all 5 tests pass. It has hit three PRs in two days (#1283, #1290, #1301) plus multiple earlier branches, each time requiring a manual rerun.

Root cause

(Credit: the air-1288 builder's diagnosis on PR #1301.) The suite's afterAll declares an explicit 10_000ms budget, overriding the config's 300s hookTimeout, while beforeAll in the same suite gets 120_000 — an unintentional asymmetry. The teardown performs two unbounded deactivate fetches, stopServer's 2s SIGTERM wait, and three rmSync calls; on a loaded ubuntu runner that exceeds 10s.

Fix

  • Raise the afterAll budget to 120_000, matching beforeAll.
  • Bound the two deactivate fetches with AbortSignal.timeout(10_000) so a hung Tower can't consume the enlarged budget either — the teardown still completes and the existing .catch(() => {}) handles the abort.

Testing

Suite passes locally under the e2e config: 5/5 in 1.8s. No production code touched — one test file, +6/−3.

🤖 Generated with Claude Code

…ch beforeAll

The suite's afterAll declared an explicit 10s budget, overriding the config's
300s hookTimeout, while beforeAll gets 120s — an unintentional asymmetry. The
teardown does two deactivate fetches + stopServer's SIGTERM wait + 3 rmSync,
which exceeds 10s on a loaded CI runner, failing the whole suite after all 5
tests pass. Hit PRs #1283, #1290, and #1301 across two days.

Raise the budget to 120s and bound the two deactivate fetches with
AbortSignal.timeout(10s) so a hung Tower can't eat the enlarged budget either.
@waleedkadous

Copy link
Copy Markdown
Contributor Author

Additional evidence from PR #1301's branch (air-1288 builder's tally): the send-integration teardown failed 3 of 4 first-attempt CI runs on that branch alone, passing on both re-runs. That hit rate is well past 'occasional flake' on loaded runners and supports the full fix (budget parity with beforeAll + bounded fetches) over a smaller bump.

@waleedkadous
waleedkadous merged commit 160a4e8 into main Jul 30, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant