Repository navigation
test: stop the AOT [EngineTest=Pass] run hitting its 5 minute budget - #7125
Conversation
The ExpectedStateTests.Pass AOT run on Windows CI failed with what looked like a NotInParallelLock deadlock: every remaining global [NotInParallel] test logged a TaskCanceledException from EnterExclusiveAsync. It was not a deadlock. The AOT invocation passes --timeout 5m, a whole-session budget, and healthy runs take 3.5-5 minutes on Windows CI. When the budget ran out, the sequential global NotInParallel phase (which runs last) hit the cancelled token at the writer gate for each remaining test. The JIT run in the same job took 5m15s and passed because it has no session timeout. - Raise the AOT session budget to 10 minutes. - DedicatedThreadExecutorCleanUpOrderTests (#6885) took about 16 seconds per test: the executor also runs the project's ~50 global test hooks, each with the 300 ms CleanUp sleep, all on the serial NotInParallel path. Only the test body's CleanUp now sleeps, so the class takes about 4 seconds instead of 52 and the full Pass run drops from about 2.5-3 minutes to under 2 locally. This also makes the regression sharper, since the hooks no longer put seconds between consecutive test bodies. Co-Authored-By: Claude <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
ReviewTest-only change, and it looks good. The diagnosis in the description is convincing: the AOT
Suggestions (non-blocking):
Approving from a code standpoint. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8a798693c6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
[Low impact] The PR appears safe to merge. SummaryRaises the AOT test-session budget and removes unnecessary delays from the cleanup-order regression tests.
Reviews (1) · Last reviewed commit: "test: stop the AOT [EngineTest=Pass] run..." · Reviewed by Greptile |
What happened
CI run 39128089083 (PR #7086) failed
ExpectedStateTests.Passin AOT mode on Windows. The output looked like a deadlock: every remaining global[NotInParallel]test (NotInParallelTests.*,TestBuildContextOutputCaptureTests.*) logged aTaskCanceledExceptionfromNotInParallelLock.EnterExclusiveAsync.It is not a deadlock. The run hit its time limit.
InvokableTestBasepasses--timeout 5m. That limit covers the whole session, and healthy[EngineTest=Pass]AOT runs on Windows CI take 3m35s to 4m56s. The diagnostics artifacts of four recent green runs show this.--hangdump-timeout).[NotInParallel]bucket runs last, one test at a time. When the token is cancelled, each remaining test fails atawait _writerGate.WaitAsync(token)(NotInParallelLock.cs line 89). That is the stack in the CI log. Nothing was holding the lock.failed: 0. Across about a dozen uncontended local runs (source-gen and reflection, with default parallelism and withDOTNET_PROCESSOR_COUNT=4), nothing hung.mainon 2026-09-29 (run 36630664644, at db75285). So it predates fix(engine): apply max parallelism cap to keyed NotInParallel tests #7084 and fix(engine): run synchronous keyed NotInParallel tests concurrently #7087, and neither caused it.Changes
DedicatedThreadExecutorCleanUpOrderTests([Bug]: DedicatedThreadExecutor completes the test before CleanUp() runs, so the next test starts while CleanUp() is still running #6885) took about 16 seconds per test, on the serial[NotInParallel]path. The executor also runs every Before/After(Test) hook, and the test project has about 50 global[BeforeEvery]/[AfterEvery]test hooks. Each hook ran on its own thread with the fixture's 300 msCleanUp()sleep. Now only the test body'sCleanUp()sleeps: the body sets a thread-static flag, and the body runs on the executor's dedicated thread. The class drops from 52s to about 4s, and the full Pass run drops from 2.5-3 minutes to 1m51s locally. The regression check also gets sharper, because the hooks no longer put seconds between consecutive test bodies.No engine or scheduling code changes, so this does not interact with #7115.
Follow-up worth considering (not in this PR)
When the session is cancelled, the sequential phase still walks every remaining global
[NotInParallel]test and logs each one as "Unhandled exception in test ...". That output is what made this look like a lock bug. Reporting them as cancelled, or stopping the loop, would give a clearer diagnosis. This is a scheduler change, so it would need to be coordinated with #7115.Tests
DedicatedThreadExecutorCleanUpOrderTestsin source-gen and reflection modes: 3/3 passed in each (4.7s and 3.8s, previously 52s).Issue6885Testsengine test (reflection mode; AOT is CI-only): passed./**[EngineTest=Pass]source-gen run: 6001 total, 0 failed, 1m51s.Summary by CodeRabbit