fix: a pooled-I/O residual NAMES the leaf that would not unwind by rbuergi · Pull Request #2770 · Systemorph/MeshWeaver

rbuergi · 2026-08-30T13:04:40Z

Why

A dirty teardown reports teardown DIRTY — 1 pooled I/O leaf(s) still running and, since #2480, which pool. It does not report which leaf — so the failure is unactionable.

Measured 2026-08-30, MeshWeaver.AI.Test (1,630 tests) under load (DOTNET_PROCESSOR_COUNT=4 + CPU burners), 20 runs:

run	test	residual	drain
7	`MeshNodeAgentFileStoreTest`	`AgentStore=1`	30 081 ms
12	`PluginCatalogTest`	`Query=1`	30 004 ms
16	`MeshPluginTest`	`FileSystem=1`	30 043 ms

Three pools, three tests, always the full budget — and nothing to act on. #2480 added the pool name for this exact reason and stopped one level short.

How

IoPool keeps each in-flight leaf's work by reference — no stack capture, no string built on the hot path — and formats only when a drain reports a residual. A lambda's compiler-generated name carries its enclosing method, so the residual reads:

AgentStore=1 [MeshWeaver.AI.Stores.MeshNodeAgentFileStore+<>c__DisplayClass…<WriteCore>b__0]

All four entry points instrumented (Invoke, InvokeStream, InvokeBlocking, SubscribeThroughPool — the last has no delegate, so it reports the source observable's type). PoolResidual.Sites flows into both the drain warning and the pools=[…] phase trace MonolithMeshTestBase already prints, so a CI test-evidence-* artifact carries it with no further change.

Verification

IoPoolTest 25/25, including two new pins: a leaf that ignores its token must be nameable, and a clean pool reports no sites (the diagnostic must not become noise).

Note: on current main this class did not reproduce in 24 further loaded runs (P ≈ 0.02 under the measured 15 % rate), so something in the intervening commits — most plausibly the continuing ToTask sweep — has largely fixed it. The diagnostic is worth landing regardless: the next occurrence names itself instead of restarting this investigation.

A dirty teardown reported an anonymous 'AgentStore=1' — enough to know a pool did not drain, never enough to fix it. Measured 2026-08-30 on MeshWeaver.AI.Test under load: 3 of 20 runs ended in a dirty teardown naming AgentStore, Query and FileSystem on three different runs, each burning the full 30 s drain budget, and not one of them could be chased any further. #2480 added the POOL NAME for exactly this reason and stopped one level short. IoPool now keeps each in-flight leaf's work BY REFERENCE (no stack capture, no string built on the hot path) and formats it only when a drain reports a residual. A lambda's compiler-generated name carries its ENCLOSING method, so the residual becomes AgentStore=1 [MeshWeaver.AI.Stores.MeshNodeAgentFileStore+<>c__DisplayClass..<WriteCore>b__0] All four entry points are instrumented (Invoke, InvokeStream, InvokeBlocking, SubscribeThroughPool — the last has no delegate, so it reports the source observable's type). PoolResidual carries the sites; both the drain warning and the pools=[...] phase trace the test base prints now include them. Pinned by IoPoolTest: a leaf that ignores its token MUST be nameable, and a pool that drains clean reports no sites (so the diagnostic is not noise). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HWc8YSndL2gpW2i9wRUQWt

Copilot

Pull request overview

This PR improves pooled I/O teardown diagnostics by capturing and surfacing the “site” (delegate/observable identity) of in-flight IoPool leaves so residual drain reports point to the specific operation that failed to unwind.

Changes:

Track in-flight IoPool leaves (by reference) and expose PendingLeafSites for residual reporting.
Extend IoPoolRegistry.DrainAll to include residual leaf sites in both returned residuals and warning logs.
Add tests ensuring (a) a residual is nameable and (b) a clean pool reports no sites.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
test/MeshWeaver.Hosting.Test/IoPoolTest.cs	Adds coverage to pin residual leaf site naming and “no-noise on clean drain”.
src/MeshWeaver.Mesh.Contract/Threading/IoPoolRegistry.cs	Propagates and logs per-pool residual sites alongside residual counts.
src/MeshWeaver.Mesh.Contract/Threading/IoPool.cs	Tracks in-flight leaves and formats diagnostic “site” strings for residual reporting.

Suppressed comments (2)

src/MeshWeaver.Mesh.Contract/Threading/IoPool.cs:315

EnterLeaf(source) is outside the try/finally that releases the gate permit; if EnterLeaf throws, the leaf will leak _inFlight and the permit, potentially wedging Drain()/Dispose(). Move ticket acquisition inside the guarded region (initialize to 0) so ReleaseGate always runs.

                await _gate.WaitAsync(ct).ConfigureAwait(false);
                Interlocked.Increment(ref _inFlight);
                var leaf = EnterLeaf(source);

src/MeshWeaver.Mesh.Contract/Threading/IoPool.cs:490

EnterLeaf(source) is evaluated before the try/finally that releases the gate permit; if it throws, the permit and _inFlight increment are leaked and the pooled subscribe setup can wedge teardown. Acquire the ticket inside the guarded region (initialize to 0) so ReleaseGate always runs.

                            await _gate.WaitAsync(ct).ConfigureAwait(false);
                            Interlocked.Increment(ref _inFlight);
                            var subLeaf = EnterLeaf(source);

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

                await _gate.WaitAsync(ct).ConfigureAwait(false);
                Interlocked.Increment(ref _inFlight);
+                var leaf = EnterLeaf(io);


@@ -174,7 +178,10 @@ public int DrainAll(out IReadOnlyList<PoolResidual> byPool)
                _logger?.LogWarning(
                    "IoPoolDrain: pool '{PoolName}' did not finish {Residual} leaf(es) within "
                    + "the drain budget — a leaf ignored its cancellation token; fix the leaf, do not "
-                    + "widen the budget.", kvp.Key, residual);
+                    + "widen the budget. Still running: {Sites}", kvp.Key, residual,
+                    kvp.Value.PendingLeafSites.Count == 0
+                        ? "(no site captured)"
+                        : string.Join(" | ", kvp.Value.PendingLeafSites));


+            // Read the sites BEFORE the drain returns nothing to name: Drain() joins, so by the
+            // time it hands back a residual the surviving leaves are exactly the ones still
+            // registered — but a leaf that unwinds during the join must not be reported, so this
+            // is read after Drain and reflects what is genuinely still running.


                        Interlocked.Increment(ref _inFlight);
+                        var blockingLeaf = EnterLeaf(work);
                        if (Interlocked.Increment(ref _blockingInFlight) == 1)
                            _blockingIdle.Reset();


github-actions · 2026-08-30T13:11:44Z