Skip to content

test: stop the AOT [EngineTest=Pass] run hitting its 5 minute budget - #7125

Merged
thomhurst merged 1 commit into
mainfrom
fix/aot-expectedstate-timeout
Oct 18, 2026
Merged

thomhurst merged 1 commit into
mainfrom
fix/aot-expectedstate-timeout

Conversation

@thomhurst

@thomhurst thomhurst commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

What happened

CI run 39128089083 (PR #7086) failed ExpectedStateTests.Pass in AOT mode on Windows. The output looked like a deadlock: every remaining global [NotInParallel] test (NotInParallelTests.*, TestBuildContextOutputCaptureTests.*) logged a TaskCanceledException from NotInParallelLock.EnterExclusiveAsync.

It is not a deadlock. The run hit its time limit.

  • The AOT invocation in InvokableTestBase passes --timeout 5m. That limit covers the whole session, and healthy [EngineTest=Pass] AOT runs on Windows CI take 3m35s to 4m56s. The diagnostics artifacts of four recent green runs show this.
  • The JIT run of the same test in the failing job took 5m15s and passed, because the non-AOT path has no session timeout (only --hangdump-timeout).
  • The global [NotInParallel] bucket runs last, one test at a time. When the token is cancelled, each remaining test fails at await _writerGate.WaitAsync(token) (NotInParallelLock.cs line 89). That is the stack in the CI log. Nothing was holding the lock.
  • I reproduced the exact signature locally by running the Pass filter under CPU contention with a 6-minute timeout. The cancellation hit the same sequential tail, at a different class, with failed: 0. Across about a dozen uncontended local runs (source-gen and reflection, with default parallelism and with DOTNET_PROCESSOR_COUNT=4), nothing hung.
  • The same failure appeared on main on 2026-09-29 (run 36630664644, at db75285). So it predates fix(engine): apply max parallelism cap to keyed NotInParallel tests #7084 and fix(engine): run synchronous keyed NotInParallel tests concurrently #7087, and neither caused it.

Changes

  • Raise the AOT session timeout to 10 minutes. It is a whole-session budget, not a hang detector.
  • DedicatedThreadExecutorCleanUpOrderTests ([Bug]: DedicatedThreadExecutor completes the test before CleanUp() runs, so the next test starts while CleanUp() is still running #6885) took about 16 seconds per test, on the serial [NotInParallel] path. The executor also runs every Before/After(Test) hook, and the test project has about 50 global [BeforeEvery]/[AfterEvery] test hooks. Each hook ran on its own thread with the fixture's 300 ms CleanUp() sleep. Now only the test body's CleanUp() sleeps: the body sets a thread-static flag, and the body runs on the executor's dedicated thread. The class drops from 52s to about 4s, and the full Pass run drops from 2.5-3 minutes to 1m51s locally. The regression check also gets sharper, because the hooks no longer put seconds between consecutive test bodies.

No engine or scheduling code changes, so this does not interact with #7115.

Follow-up worth considering (not in this PR)

When the session is cancelled, the sequential phase still walks every remaining global [NotInParallel] test and logs each one as "Unhandled exception in test ...". That output is what made this look like a lock bug. Reporting them as cancelled, or stopping the loop, would give a clearer diagnosis. This is a scheduler change, so it would need to be coordinated with #7115.

Tests

  • DedicatedThreadExecutorCleanUpOrderTests in source-gen and reflection modes: 3/3 passed in each (4.7s and 3.8s, previously 52s).
  • Issue6885Tests engine test (reflection mode; AOT is CI-only): passed.
  • Full /**[EngineTest=Pass] source-gen run: 6001 total, 0 failed, 1m51s.

Summary by CodeRabbit

  • Tests
    • Adjusted automated test timing to accommodate longer sequential test runs without prematurely cancelling healthy sessions.
    • Improved cleanup handling in tests involving execution hooks, preventing hook activity from being treated as test-body work while preserving cleanup delays for test bodies. These changes make test results more reliable in scenarios with slow cleanup and longer AOT runs.

The ExpectedStateTests.Pass AOT run on Windows CI failed with what looked
like a NotInParallelLock deadlock: every remaining global [NotInParallel]
test logged a TaskCanceledException from EnterExclusiveAsync. It was not a
deadlock. The AOT invocation passes --timeout 5m, a whole-session budget,
and healthy runs take 3.5-5 minutes on Windows CI. When the budget ran out,
the sequential global NotInParallel phase (which runs last) hit the
cancelled token at the writer gate for each remaining test. The JIT run in
the same job took 5m15s and passed because it has no session timeout.

- Raise the AOT session budget to 10 minutes.
- DedicatedThreadExecutorCleanUpOrderTests (#6885) took about 16 seconds
  per test: the executor also runs the project's ~50 global test hooks,
  each with the 300 ms CleanUp sleep, all on the serial NotInParallel path.
  Only the test body's CleanUp now sleeps, so the class takes about 4
  seconds instead of 52 and the full Pass run drops from about 2.5-3
  minutes to under 2 locally. This also makes the regression sharper,
  since the hooks no longer put seconds between consecutive test bodies.

Co-Authored-By: Claude <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-18T14:35:36.955828Z 8a79869 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2a470220-b1d0-424d-8037-65bfbe75fc83
📥 Commits

Reviewing files that changed from the base of the PR and between 540152e and 8a79869.

📒 Files selected for processing: 2
  • tests/TUnit.Engine.Tests/InvokableTestBase.cs
  • tests/TUnit.TestProject/Bugs/6885/Tests.cs

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 5 remain after this review.

📝 Walkthrough

The AOT test command timeout increases from five to ten minutes. The Bug 6885 test executor now tracks test-body execution per thread and skips cleanup delays on executor threads that did not run a test body.

Changes: AOT test command timeout

Layer / File(s) Summary
AOT session time budget
tests/TUnit.Engine.Tests/InvokableTestBase.cs
The AOT test command timeout increases from five to ten minutes. Comments describe the timeout as a whole-session budget and note the sequential global [NotInParallel] tail.

Bug 6885 cleanup behavior

Layer / File(s) Summary
Track test-body execution and gate cleanup
tests/TUnit.TestProject/Bugs/6885/Tests.cs
The executor marks the current thread when it runs a test body. CleanUp skips its delay and state reset on threads without that marker. The remarks describe hook execution and the test time budget.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 8a798

The AOT session has a longer time budget, and the cleanup adjustment has no demonstrated effect on later tests. No concrete merge-blocking risk was found.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly identifies the AOT test run’s five-minute timeout, which is the primary issue addressed by the pull request.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit checks the timer’s face,
Ten minutes give the tests more space.
A thread that ran the body waits,
While hook threads skip the cleanup gates.
The burrow’s tests proceed apace.

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Review

Test-only change, and it looks good. The diagnosis in the description is convincing: the AOT --timeout is a whole-session budget, and healthy Windows runs take 3.5–5 minutes against a 5m limit.

InvokableTestBase: Raising the timeout from 5m to 10m is reasonable, and the new comment explains the intent.

Bugs/6885/Tests.cs: The change is sound as far as I can tell.

  • The [ThreadStatic] flag is safe as long as each DedicatedThreadExecutor invocation gets a fresh thread. If the executor ever pooled or reused threads, a hook could inherit t_ranTestBody = true from an earlier test body, and the slow CleanUp would run again. That would only slow things down. It would not cause false passes.
  • The regression check still covers [Bug]: DedicatedThreadExecutor completes the test before CleanUp() runs, so the next test starts while CleanUp() is still running #6885. It needs the test body's CleanUp to be slow, so a late CleanUp resets the state mid-test, and that is preserved.
  • Hook CleanUps no longer set s_state = "reset". That is harmless, because the next Initialize sets "configured" again.

Suggestions (non-blocking):

  • MarkTestBodyRunning() has to be called manually from the test body. If someone adds a fourth test and forgets the call, the slow CleanUp is silently skipped. Because a skipped CleanUp can no longer overlap with the next test, the regression guard would silently stop catching [Bug]: DedicatedThreadExecutor completes the test before CleanUp() runs, so the next test starts while CleanUp() is still running #6885. Keeping the call in the shared AssertNoOverlapAsync helper, as this PR does, mitigates that. A short comment on MarkTestBodyRunning saying every test must call it would make this explicit.
  • Your follow-up note is the right long-term fix. Cancelled sequential tests should be reported as cancelled, not logged as unhandled exceptions, so the next timeout is not misread as a lock bug.

Approving from a code standpoint.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8a798693c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/TUnit.TestProject/Bugs/6885/Tests.cs
@greptile-apps

greptile-apps Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge.

Summary

Raises the AOT test-session budget and removes unnecessary delays from the cleanup-order regression tests.

  • AOT test sessions get ten minutes to finish.
  • Cleanup-order tests wait only after test bodies.

Reviews (1) · Last reviewed commit: "test: stop the AOT [EngineTest=Pass] run..." · Reviewed by Greptile

@thomhurst
thomhurst enabled auto-merge (squash) October 18, 2026 14:50
@thomhurst
thomhurst merged commit c2f4b9a into main Oct 18, 2026
25 of 26 checks passed
@thomhurst
thomhurst deleted the fix/aot-expectedstate-timeout branch October 18, 2026 15:07
This was referenced Oct 22, 2026

This branch had an error being deployed

1 failed deployment
Pull Requests — 8a798693 Deployed Oct 18, 2026 by thomhurst via modularpipeline (windows-latest) #20137
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant