Skip to content

test: deflake stale-reader grace test (release on dedicated thread, not ThreadPool timer) - #578

Merged
tylerkron merged 3 commits into
mainfrom
fix/deflake-stale-reader-grace-timer
Aug 23, 2026
Merged

tylerkron merged 3 commits into
mainfrom
fix/deflake-stale-reader-grace-timer

Conversation

@tylerkron

Copy link
Copy Markdown
Contributor

Flaky-test-fixer routine: CI history (gh run list) showed StreamMessageConsumerStallingReaderTests.Start_WhenStoppedReaderExitsWithinGrace_WaitsAndRestartsSameInstance fail on net9.0 with ConsumerThreadNotExitedException, then pass on the next run with only an unrelated commit (a different test file) in between — a genuine flake, not a real regression.

What was wrong: the test scheduled the stream release via a System.Threading.Timer callback due 200ms after Start() begins its 1000ms grace-period wait. Timer callbacks run on the shared .NET ThreadPool, and under CI load — many test classes executing in parallel — that queue can occasionally be starved well past the 200ms due time, blowing the 1000ms grace window and failing the test even though the production Start()/grace-join logic was working correctly. This was a test-timing issue, not a bug in StreamMessageConsumer.

How it was fixed: the release now runs on its own dedicated background Thread with Thread.Sleep(200) instead of a ThreadPool-queued Timer, so it isn't subject to pool contention from other running tests.

This is a test-only change — no production code touched, so it's safe by construction (same assertions, same timing budget, just a more reliable trigger for the release).

Verified: the affected test run 5x back-to-back (all green), full dotnet test suite green on net9.0 and net10.0 (3729 total, 0 failed, 2 skipped, both TFMs). No bench validation needed (test-only change).

Not merging — for review.

StreamMessageConsumerStallingReaderTests.Start_WhenStoppedReaderExitsWithinGrace_WaitsAndRestartsSameInstance
scheduled the stream release via a System.Threading.Timer callback, which
runs on the shared ThreadPool. Under CI load that queue can be starved past
its 200ms due time, occasionally blowing the consumer's 1000ms grace join
and failing the test with ConsumerThreadNotExitedException even though
production behavior was correct. Releasing from a dedicated background
thread instead avoids ThreadPool queuing delay.

Confirmed by inspecting a prior CI failure (run 32580100693) that reproduced
exactly this symptom with no intervening code change before the next run
passed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@tylerkron
tylerkron requested a review from a team as a code owner August 22, 2026 22:43
@tylerkron

Copy link
Copy Markdown
Contributor Author

/agentic_review

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Deflake stale-reader grace test by releasing on a dedicated thread

🧪 Tests 🕐 10-20 Minutes

Grey Divider

AI Description

• Replace ThreadPool Timer-based stream release with a dedicated background thread.
• Prevent CI ThreadPool starvation from delaying release past the 1s grace window.
• Keep production behavior and assertions unchanged; only stabilize test timing.
Diagram

graph TD
T["StallingReader grace test"] --> C["consumer.Start() grace join"] --> A["Assert running + StopSafely"]
T --> R["Background releaser thread"] --> S["Sleep(200) + stream.Release()"] --> C
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Task.Factory.StartNew(LongRunning) releaser
  • ➕ Similar semantics to a dedicated thread while keeping Task-based test code
  • ➕ Avoids shared ThreadPool contention when LongRunning creates a dedicated thread
  • ➖ More subtle: scheduler/thread creation is implementation-dependent and less explicit than Thread
  • ➖ Slightly more complex to reason about in tests than a plain Thread
2. Increase grace/timing budget in the test
  • ➕ Minimal code change and lowest conceptual overhead
  • ➖ Paper-over fix: still dependent on ThreadPool scheduling and can remain flaky under worse load
  • ➖ Longer tests and slower CI; does not address root cause

Recommendation: The dedicated background Thread approach is the most explicit and reliable for deflaking a timing-sensitive test, because it fully removes ThreadPool scheduling variability from the release trigger while preserving the existing assertions and timing budget. A LongRunning Task is a reasonable alternative but is less direct/obvious than a Thread in a test like this.

Files changed (1) +21 / -1

Tests (1) +21 / -1
StreamMessageConsumerStallingReaderTests.csReplace Timer-based release with dedicated thread to avoid CI flakiness +21/-1

Replace Timer-based release with dedicated thread to avoid CI flakiness

• Replaces a System.Threading.Timer callback (ThreadPool-queued) with a dedicated background Thread that sleeps 200ms and calls stream.Release(). Adds a try/finally to join the releaser thread to keep test cleanup deterministic and reduce timing races.

src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs

@qodo-code-review

qodo-code-review Bot commented Aug 22, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Releaser not joined on throw ✓ Resolved 🐞 Bug ☼ Reliability ⭐ New
Description
If consumer.Start() throws, the test exits before joining the releaser thread, so using
disposal can run while the background thread is still pending. The releaser can then call
stream.Release() after StallingStream.Dispose() has disposed its ManualResetEventSlim,
throwing ObjectDisposedException on a background thread and potentially terminating the test
process.
Code

src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[R48-51]

+        consumer.Start();
+
+        // Join before the test's `using` disposals run: a still-alive releaser calling
+        // stream.Release() after the stream is disposed would throw ObjectDisposedException on a
Relevance

●●● Strong

Accepted history favors hardening cross-thread test synchronization/cleanup to avoid
flaky/process-terminating failures.

PR-#350
PR-#384

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The test disposes stream/consumer via using var. If consumer.Start() throws, execution never
reaches the join, but disposal still runs; after disposal, StallingStream.Release() will attempt
_release.Set() on a disposed ManualResetEventSlim, which throws ObjectDisposedException.

src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[25-53]
src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[409-441]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`releaser.Join(...)` is only executed after `consumer.Start()` succeeds. If `consumer.Start()` throws, the method exits early and the `using` disposals run while `releaser` may still be sleeping, allowing it to call `stream.Release()` after the stream’s internals have been disposed.

### Issue Context
`StallingStream.Release()` calls `_release.Set()`, and `Dispose()` disposes `_release`. A late `Release()` after disposal will throw `ObjectDisposedException` from the background thread.

### Fix
Ensure the releaser thread is always joined before leaving the test body, even when `consumer.Start()` throws. For example:
- Start `releaser`
- `try { consumer.Start(); } finally { Assert.True(releaser.Join(...)); }`

(Alternatively or additionally, wrap `stream.Release()` in the releaser thread with a `try/catch (ObjectDisposedException)` to avoid process-terminating unhandled exceptions.)

### Fix Focus Areas
- src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[48-53]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Ignored releaser Join result ✓ Resolved 🐞 Bug ☼ Reliability
Description
The test calls releaser.Join(...) but ignores the boolean result, so it can proceed and dispose the
stream while the background thread may still call stream.Release(), potentially throwing
ObjectDisposedException on a background thread (process-terminating). This can convert rare
scheduling delays into a hard crash or cross-test interference instead of a clean test failure.
Code

src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[R52-55]

+        finally
+        {
+            releaser.Join(TimeSpan.FromSeconds(2));
+        }
Relevance

●●● Strong

Recent same-file precedents accept deterministic thread-lifecycle assertions and cleanup hardening
in flaky consumer tests.

PR-#384
PR-#350

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The PR adds a background thread that calls stream.Release(), but the test does not verify that the
thread finished before the stream is disposed. The stream’s Dispose explicitly disposes the
ManualResetEventSlim used by Release/Read; a late Release after disposal can throw
ObjectDisposedException on the background thread.

src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[52-55]
src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[433-440]
src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[401-412]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The test starts a dedicated background `Thread` that calls `stream.Release()` later, but in `finally` it uses `releaser.Join(...)` and ignores the returned `bool`. If the join times out, the test can continue and the `using var stream` disposal can run while the background thread is still pending; when it eventually calls `stream.Release()`, it may touch disposed synchronization primitives and throw on a background thread.

## Issue Context
`StallingStream.Dispose(...)` disposes `_release`/`_readEntered`. A late `Release()` after disposal can throw `ObjectDisposedException`. Unhandled exceptions on background threads can terminate the test process.

## Fix Focus Areas
- src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[52-55]

## Suggested fix
- Capture the return of `Join(...)` and `Assert.True(joined, "...")`.
- Optionally, guard the thread body with try/catch and surface any exception back to the test thread (e.g., store to a shared `Exception?` and assert-null after join).
- As a fallback if join fails, do an immediate `stream.Release()` on the test thread before disposing to minimize the chance of a late call racing disposal.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Tip of the day
💡 Did you know, you can turn these tips off under Display preferences

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Previous reviews

Review updated until commit d3bdc82

Results up to commit b7972ae ⚖️ Balanced


🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)


Remediation recommended
1. Ignored releaser Join result ✓ Resolved 🐞 Bug ☼ Reliability
Description
The test calls releaser.Join(...) but ignores the boolean result, so it can proceed and dispose the
stream while the background thread may still call stream.Release(), potentially throwing
ObjectDisposedException on a background thread (process-terminating). This can convert rare
scheduling delays into a hard crash or cross-test interference instead of a clean test failure.
Code

src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[R52-55]

+        finally
+        {
+            releaser.Join(TimeSpan.FromSeconds(2));
+        }
Relevance

●●● Strong

Recent same-file precedents accept deterministic thread-lifecycle assertions and cleanup hardening
in flaky consumer tests.

PR-#384
PR-#350

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The PR adds a background thread that calls stream.Release(), but the test does not verify that the
thread finished before the stream is disposed. The stream’s Dispose explicitly disposes the
ManualResetEventSlim used by Release/Read; a late Release after disposal can throw
ObjectDisposedException on the background thread.

src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[52-55]
src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[433-440]
src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[401-412]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The test starts a dedicated background `Thread` that calls `stream.Release()` later, but in `finally` it uses `releaser.Join(...)` and ignores the returned `bool`. If the join times out, the test can continue and the `using var stream` disposal can run while the background thread is still pending; when it eventually calls `stream.Release()`, it may touch disposed synchronization primitives and throw on a background thread.

## Issue Context
`StallingStream.Dispose(...)` disposes `_release`/`_readEntered`. A late `Release()` after disposal can throw `ObjectDisposedException`. Unhandled exceptions on background threads can terminate the test process.

## Fix Focus Areas
- src/Daqifi.Core.Tests/Communication/Consumers/StreamMessageConsumerStallingReaderTests.cs[52-55]

## Suggested fix
- Capture the return of `Join(...)` and `Assert.True(joined, "...")`.
- Optionally, guard the thread body with try/catch and surface any exception back to the test thread (e.g., store to a shared `Exception?` and assert-null after join).
- As a fallback if join fails, do an immediate `stream.Release()` on the test thread before disposing to minimize the chance of a late call racing disposal.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

Qodo round 1 on #578: the prior fix ignored releaser.Join()'s result, so if
the background thread were ever still alive when the test's `using var
stream` disposes, a late Release() call could throw ObjectDisposedException
on that background thread — process-terminating rather than a clean test
failure. Assert the join succeeds instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@tylerkron

Copy link
Copy Markdown
Contributor Author

/agentic_review

@qodo-code-review

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit de94fe0

Qodo round 2 on #578: removing the try/finally in the previous fix
reintroduced the exact problem it was meant to guard against — if
consumer.Start() throws, the releaser join is skipped entirely and the
using-block disposals can run while the background thread is still
pending, risking an ObjectDisposedException on that thread. Restore the
join in a finally block, but check its result afterward rather than
asserting inside finally, so a genuine exception from Start() is never
masked by the join assertion.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@tylerkron

Copy link
Copy Markdown
Contributor Author

/agentic_review

@qodo-code-review

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit d3bdc82

@tylerkron

Copy link
Copy Markdown
Contributor Author

Qodo-clean, CI green — ready for review.

@tylerkron
tylerkron added this pull request to the merge queue Aug 23, 2026
Merged via the queue into main with commit 3db0637 Aug 23, 2026
1 check passed
@tylerkron
tylerkron deleted the fix/deflake-stale-reader-grace-timer branch August 23, 2026 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant