Skip to content

test: widen timing margins on two Windows-flaky CI tests#359

Merged
gnanam1990 merged 1 commit into
mainfrom
fix-windows-ci-flaky-timing-tests
Jul 1, 2026
Merged

test: widen timing margins on two Windows-flaky CI tests#359
gnanam1990 merged 1 commit into
mainfrom
fix-windows-ci-flaky-timing-tests

Conversation

@gnanam1990

@gnanam1990 gnanam1990 commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Investigated two windows-latest smoke-test failures from today's merges. Both are pre-existing test-timing issues, unrelated to the content of the PRs whose merge-to-main runs they failed on (#349, #356 — neither touches internal/specialist or internal/agenteval).

  • TestOutputToolBlocksUntilTaskCompletes: a 100ms poll timeout against a 5ms background write left only 20x margin — not enough headroom against Windows' coarser default timer granularity and heavier CI scheduling contention than Linux/macOS. Widened the timeout to 2000ms. This is a ceiling, not a target: PollInterval keeps the test fast in the success case (it still returns as soon as output lands), it just tolerates more jitter before failing.
  • TestHarnessTimeoutCancelsBlockedAgent: confirmed there's no actual data race first (harness.Agent.Run is a direct synchronous call on the test's own goroutine, not a spawned goroutine racing the assertions). The failure was workspace materialization alone consuming the full 10s budget on a contended Windows runner, so the agent was never reached before the deadline fired. Widened to 30s. Unlike the other fix, this test's own design blocks for the full timeout to observe cancellation, so this directly costs ~20s more wall-clock time — inherent to what it's testing, not avoidable without restructuring the harness contract.

Test plan

  • go build ./...
  • go test ./internal/specialist/ -count=1 -run TestOutputToolBlocksUntilTaskCompletes -v (0.01s)
  • go test ./internal/agenteval/ -count=1 -run TestHarnessTimeoutCancelsBlockedAgent -v (30.03s, as expected)
  • go test ./internal/specialist/ ./internal/agenteval/ -count=1 (full packages)
  • go test ./... -race -count=1 (full repo)

Summary by CodeRabbit

  • Tests
    • Adjusted automated test timing to better tolerate slower or contended CI environments.
    • Improved timeout handling in long-running test scenarios to reduce flaky failures and keep test runs more reliable.

Both failed on windows-latest today, on merges of otherwise-unrelated
PRs (#349, #356) - neither PR touched internal/specialist or
internal/agenteval. Windows GitHub Actions runners have coarser default
timer granularity and more scheduling contention than Linux/macOS, so
tight timing margins that hold on the other two platforms aren't
reliable there.

- TestOutputToolBlocksUntilTaskCompletes: a 100ms poll timeout against a
  5ms background write left only 20x margin. Widened the timeout to
  2000ms (a ceiling, not a target - PollInterval keeps the test fast in
  the success case; it only changes how much CI jitter it tolerates
  before failing).
- TestHarnessTimeoutCancelsBlockedAgent: confirmed there's no actual data
  race (harness.Agent.Run is a direct synchronous call, not a spawned
  goroutine racing the assertions) - the failure was workspace
  materialization alone consuming the full 10s budget on a contended
  Windows runner, so the agent was never reached. Widened to 30s. This
  test's own design blocks for the full timeout to observe cancellation,
  so unlike the other fix this directly costs ~20s more wall-clock time,
  but that's inherent to what it's testing.
@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 9890559b-2ebd-4816-b9de-c58391d8984e

📥 Commits

Reviewing files that changed from the base of the PR and between c0254e3 and ade89a9.

📒 Files selected for processing (2)
  • internal/agenteval/benchmark_test.go
  • internal/specialist/output_tool_test.go

Walkthrough

Two test files had their timeout values increased to reduce flakiness under CI contention: a benchmark harness timeout raised from 10s to 30s, and an output tool test timeout raised from 100 to 2000, both with added explanatory comments.

Changes

Test Timeout Adjustments

Layer / File(s) Summary
Benchmark harness timeout
internal/agenteval/benchmark_test.go
TestHarnessTimeoutCancelsBlockedAgent's BenchmarkInput.Timeout increased from 10s to 30s, with comment explaining insufficient headroom on contended Windows CI runners.
Output tool blocking timeout
internal/specialist/output_tool_test.go
TestOutputToolBlocksUntilTaskCompletes's timeout value increased from 100 to 2000, with comments clarifying the timeout acts as a ceiling and returns promptly once output is written.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related PRs

  • Gitlawb/zero#115: Targets the same output-tool blocking/polling behavior whose test timeout is adjusted here.
  • Gitlawb/zero#180: Relates to the same benchmark harness timeout/cancellation behavior adjusted here.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: widening timeouts on two Windows-flaky CI tests.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Zero automated PR review

Verdict: No blockers found

Blockers

  • None found.

Validation

  • [pass] Diff hygiene: git diff --check
  • [pass] Tests: go test ./...
  • [pass] Build: go run ./cmd/zero-release build
  • [pass] Smoke build: go run ./cmd/zero-release smoke

Scope

Head: ade89a97dd37
Changed files (2): internal/agenteval/benchmark_test.go, internal/specialist/output_tool_test.go

This deterministic review checks validation status and basic diff hygiene. A human reviewer still owns product judgment and design quality.

@gnanam1990
gnanam1990 merged commit 0914ced into main Jul 1, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant