Repository navigation
MeshWeaver.Threading.Test wedged with exit=124 and is still unexplained — two static ManualResetEventSlim signals the #2013 sweep reported as absent #2341
Description
The unexplained half of #2013
#2013 is closed by #2335 (0ba9ba8d8), which correctly fixed the MeshWeaver.Hosting.Monolith.Test / MeshWeaver.Persistence.Test residue — including the wedge that hit main and blocked CD.
But the second wedge of 2026-08-26 is still unexplained, and closing #2013 absorbs it:
[CI] MeshWeaver.Threading.Test exit=124 TIMEOUT (8m wall-clock cap hit — likely fixture/init hang)
PR #2318, shard 5, run 32943789488, ~08:00Z — identical signature to #2013 (host dies, no failing test named), different assembly.
Correcting the sweep's negative
#2335's report concluded that MeshWeaver.Threading.Test contains none of the blocking markers — "swept for all three ratcheted markers plus other blocking primitives (ManualResetEventSlim, SemaphoreSlim, lock, Task.Result). None exist in that assembly."
That is not correct. On origin/main:
test/MeshWeaver.Threading.Test/AiPoolDrainJoinsRoundTest.cs:181
internal static readonly ManualResetEventSlim Entered = new(false);
test/MeshWeaver.Threading.Test/AiPoolDrainJoinsRoundTest.cs:107
ParkingChatClient.Entered.Wait(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken)
test/MeshWeaver.Threading.Test/DelegationDrainJoinsParkedToolCallTest.cs:151
internal static readonly ManualResetEventSlim DelegationEntered = new(false);
test/MeshWeaver.Threading.Test/DelegationDrainJoinsParkedToolCallTest.cs:105
.Wait(TimeSpan.FromSeconds(60), TestContext.Current.CancellationToken)
The negative finding was load-bearing for closing #2013 — it is why the Threading wedge was left without an explanation or an owner.
What is PROVEN vs what is HYPOTHESIS
Proven: both signals are static readonly ManualResetEventSlim, which AGENTS.md forbids outright — "A static field that is a collection or cache is FORBIDDEN … [ThreadStatic] … Process-wide static state survives mesh disposal, so it bleeds across tests" — and ManualResetEventSlim is named explicitly in the hand-woven-async ban. Neither is reset between tests; Entered is only ever Set.
Hypothesis, not yet measured: a static signal that is never reset makes its own guard vacuous on every run after the first. Entered.Wait(30s) returns true immediately because a previous test set it, so the assertion "the streaming stub must be executing before the drain proves anything" passes without the precondition holding. The test then drives DrainAll() against a round that is not actually parked — and the drain is the part with the real blocking behaviour. That would produce an unbounded hang rather than a clean 30 s timeout, which matches exit=124 with no test named.
I have not proven that chain, which is exactly why this is a filed issue rather than a patch.
Suggested first step
Make both signals per-instance rather than static (the fixture already scopes per test), then re-run MeshWeaver.Threading.Test repeatedly under DOTNET_PROCESSOR_COUNT=4. If the vacuous-guard theory holds, the pre-fix version should wedge more readily when the class runs after another that sets the same signal — i.e. only under the FULL assembly, never in isolation (the #2031/#2185 family shape).
🚨 Do not raise the 8 m cap. Over budget means stuck.
Activity
rbuergi commentedon Aug 26, 2026
Confirming the negative on
origin/mainpost-#2335 — unchanged, still an open gap.#2335(merged, closes #2013) does not touchtest/MeshWeaver.Threading.Testat all (git show --statfor its merge:IStorageAdapterTestExtensions.cs,BlockingBridgeSites.allow,BlockingBridgeInTestRatchetGuard.cs/BlockingBridgeSelfDeadlockMechanismTest.cs, threeHosting.Monolith.Test/Persistence.Testfiles — nothing inThreading.Test).Both signals this issue names are still present, unchanged:
And the ratchet guard's marker set (
.ToEnumerable(),.Wait(),.GetAwaiter().GetResult()) isstring-literal —
Entered.Wait(TimeSpan.FromSeconds(30), ct)does not match the bare.Wait()marker, so this shape is structurally invisible to the #2013 ratchet regardless of which assembly
it runs over. The ratchet closing #2013 does not and cannot cover this issue's claim.
No fix attempted here (this is a triage pass, not a repair) — the vacuous-guard hypothesis in the
issue body is still unproven and is exactly the right next step for whoever picks this up. Leaving
open.
rbuergi commentedon Aug 26, 2026
Root-caused and fixed in #2390. Two corrections to the record, both measured rather than argued.
The static-
ManualResetEventSlimhypothesis is disproved.ParkingChatClient.EnteredandParkedDelegationFactory.DelegationEnteredare each referenced only from inside their own single-[Fact]class, so neither can beSet()by one test andWait()ed by another — each is set and waited exactly once per process, by the same test. Both waits are bounded (Wait(30 s/60 s, ct)), and in every wedged run's phase trace the two drain tests they belong to complete in 0.7–2 s. They are still wrong per AGENTS.md's no-static-state rule; they are not this defect, and #2390 leaves them alone.The actual cause is a busy-wait in
SubThreadHangRepro. The failing shard's own artifact says so:collected-logs/_meshweaver-test-trace.logshows 112INIT_DONEagainst 111DISPOSE_DONE— exactly one test method entered and never returned — and in all three failing runs whose artifacts survive it is the same class. Reproduced locally 2/2 attempts (six concurrent hosts,DOTNET_PROCESSOR_COUNT=4; 12 sequential single-host runs were green — it needs contention), and the wedged host's managed stack names it:Observable.Defer(...).Catch(_ => Observable.Empty<T>().Delay(200.Milliseconds())).Repeat()—Catchintercepts OnError only, and the point read of the not-yet-created sub-thread node completes without emitting. Instrumented on the real repro: 460 000 re-subscribes in 8.45 s (~54 kHz), zeroOnError— the 200 ms backoff had never executed once. Because Rx subscribes synchronously, the loop runs inside the assertion's own.ToTask()call and theawaitis never reached, which puts it beyond.Within(...),[Fact(Timeout)],methodTimeoutandMonolithMeshTestBase's hard-deadline watchdog. It also starves the hub that has to create the node it waits for, so on a 4-vCPU runner it can never terminate.Scope was wider than one site:
SubThreadColdStartReproandSubThreadStallReprocarried the same construct twice more. All three now use theGetQuery-existence-then-stream-content composition AGENTS.md prescribes, and a new ratchet (UnthrottledResubscribeInTestRatchetGuard, empty allow file, mutation-tested) refuses an argument-less.Repeat()/.Retry()undertest/.Verification: 12 rounds × 6 concurrent hosts = 72 full-assembly runs after the fix, 0 wedges, 139/139 in every one.
Merge pull request #2390 from Systemorph/fix/2341-unthrottled-resubsc…
The two new seed tests await the stream instead of blocking on it