Skip to content

Fix Windows file-retry deadlocks under mocked timers - #801

Merged
Fei2411 merged 1 commit into
devfrom
fix/windows-ci-hang
Nov 1, 2026
Merged

Fei2411 merged 1 commit into
devfrom
fix/windows-ci-hang

Conversation

@Fei2411

@Fei2411 Fei2411 commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Issue

Fix the intermittent Windows unit-test hang seen in runs 41283258046, 41303237378 and 41307553093. There is no separate issue for this CI failure.

What changed / 改了什么

Before: a real Windows filesystem retry could wait forever on a frozen test clock.

  • The group/recovery tests mock global setTimeout. replaceFile() also used global setTimeout for its 40ms filesystem retry after EPERM, EACCES or EBUSY.
  • If Windows briefly refused to replace state.json, the save awaited a fake timer that nobody advanced. Loading the fixture or disposing its store then awaited that save. This is a clock deadlock, not a leaked shell, watcher or socket.
  • It is Windows-specific because only Windows enters that busy-error retry branch. Ordinary runs do not encounter contention, explaining why rerunning the same commit passes.

Now: import the promise-based timer before the tests freeze application timers and use it for the real filesystem backoff. Retry counts, delays, lock checks, application scheduling and disposal behavior are unchanged. Add a regression that injects one Windows EPERM under frozen timers, checks that replacement completes without a clock tick, and releases the old implementation on failure so the regression reports an assertion instead of hanging.

Log evidence

The cancelled logs did not finish every test: they contain neither the remaining files nor the final test totals. The spec reporter presents results in file order.

Job Evidence in the REST job log
124779322868, attempt 1 Lines 577–582, 07:10:03 UTC: follow-through.test.js passes “a delivery that fails is not recorded as delivered…” and logs the expected delivery error from the next test. That next test, “a reply that fails to reach the asker…”, never reports a result. Line 583 is cancellation at 08:37:16.
124798143194, passing attempt 2, same head SHA Lines 577 and 583 contain both of those successful results. Line 639 contains the first group-work.test.js result. Lines 2247–2254 report 1351 passed, 1 skipped and completion in 28.2 seconds.
124838348252, attempt 1 Line 641, 11:47:22 UTC: the last group-status.test.js test passes. No result from the next file, group-work.test.js, appears; line 642 is cancellation at 12:30:46.
124850950522, attempt 1 Same boundary at line 641, 12:41:59 UTC; line 642 is cancellation at 13:01:09.

Reproduction

  • On macOS, force the Windows branch and inject one EPERM at the actual state-file rename while the first group-work.test.js test has frozen setTimeout. Before the fix it times out at store.save -> replaceFile; after the fix the same test passes in about 75ms.
  • The blocked process reports only its two test communication PipeWrap resources. The unresolved wait is the mocked 40ms timeout, not an OS timer handle. Inspecting active native handles alone would miss it.
  • The permanent regression fails on the old implementation and passes in about 42ms with the fix.
  • With a temporary local injector, all three implicated files pass together: 105/105, including 37 injected Windows rename failures under mocked time. The injector is not included in this PR.
  • The historical jobs used npm test without a per-test timeout (see log line 160). Local validation used the requested --test-timeout=120000. No force-exit, timeout, retry or workflow changes are included.

Changelog

none — prevent Windows unit tests from deadlocking when a real file retry meets a frozen test clock.

How I checked it / 怎么验证的

  • Full suite on macOS, Node 24.22.2, DSH 0.2.1-rc.4: node --import ./test/harness-resolve.mjs --test --test-timeout=120000 "test/*.test.js" — 1431 passed, 2 skipped, 0 failed (run twice).
  • node scripts/build-client.mjs --check
  • npm run lint — 0 errors; 39 existing warnings.
  • Smoke test on macOS with an isolated mock-model bench — 52/52, both owned processes stopped and test data removed.
  • Leak scan of the repository snapshot — 0 must-fix findings.
  • Native Windows CI, job 124865149276: 1432 passed, 1 skipped, 0 failed, process exited normally in 38.2s; --check and lint also passed. The frozen-clock regression passes at log line 555 in 45.3ms; final totals are at lines 2339–2346.
  • All four unit-test CI matrix jobs passed, including Linux with Node 22 and the older supported DSH.
  • Screenshot attached — no interface changes.

Not verified: manually inducing an actual Windows file-sharing violation; the cancelled historical processes and their original busy error codes are no longer available. The busy-error path was exercised deterministically on Mac and native Windows CI. A single green native run alone cannot rule out every intermittent failure; the before/after deterministic regression is the evidence for this deadlock fix. Desktop installers were not manually checked (there are no interface or packaging changes).

Checklist / 检查

  • One issue, one subject.
  • No client source or generated bundle changes.
  • No keys, tokens or personal data.
  • No AI tool as author or Co-authored-by.

The group and recovery tests mock setTimeout while exercising real file I/O. A transient Windows rename error then waits on a fake retry timer, so state loading or store disposal never completes.\n\nUse the pre-imported promise timer for filesystem backoff without changing retries, delays or lock checks. Cover a synthetic Windows sharing violation under frozen application timers; release the old wait before asserting so a regression fails rather than hanging.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-01T13:40:55.049543Z 5ef3ff4 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@Fei2411
Fei2411 merged commit 6dfbdf2 into dev Nov 1, 2026
14 checks passed
@Fei2411
Fei2411 deleted the fix/windows-ci-hang branch November 1, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant