Teardown lets accepted work finish and names what it had to stop — no forced teardown by rbuergi · Pull Request #3291 · Systemorph/MeshWeaver

rbuergi · 2026-09-04T09:49:39Z

Why

The red runs in the last 130 (MeshWeaver Build and Test) split into two families. Five were the AGENTS.md shared rule blocks gate — a satellite added the rest-not-graphql block before core's manifest declared it — cured by #3287. The rest are teardown-shaped flakes: NackReachesTheWaiterDuringTeardownTest (run 33847949620, docs-only PR #3286), PodHubTransportTest.CrossSiloNack… (merge-queue run for the docs-only #3285), and their earlier sibling LateNackReenqueueTest.

The mechanism, from the passing trace beside the failing one: the owner's merge turn is parked, the sync hub cannot process its ShutdownRequest, and after 8 s the disposal watchdog force-tore the sync hub down from its timer thread; only then could the owner reach ShutDown and mint the verdict the waiter was armed for — 3 ms later locally, but the assertion window was 10 s, i.e. "the watchdog plus two seconds", and a loaded CI shard lost those two seconds.

Production shows the same instrument doing worse: Loki on memex-cloud (2026-08-29 → 09-03) has DISPOSAL DEADLOCK DETECTED: Hub sync/… RunLevel=ShutDown → [FORCE-TEARDOWN] … out-of-band teardown complete after 22706ms dozens of times per shutdown, on sync hubs and the node hubs above them, and once on a pod that was not shutting down. A hub that reports itself disposed while its work is still running is exactly "disposal seems to be keeping something".

Maintainer directives this implements (2026-09-04): let accepted work — activities, pooled I/O, handler turns — finish its job; never cancel on entry; no forced teardown, not in prod and not in tests; detect a wedge as no progress for a budget, cancel it once cooperatively, and log every such kill or discard at Error with enough detail to reproduce, so the red-log triage files it as an issue (prod only; tests just log).

What changes

Hub (MessageHub / MessageService)

Dispose() no longer calls CancelExecution(); the ShutdownRequest queues FIFO behind accepted work and the pump drains it.
The watchdog is now a stall detector (OnDisposalStall): re-armed on every subtree RunLevel transition (unchanged), fires every 8 s of no progress, and reaches one of five verdicts — busy (turns completing, Information), wedged turn (cancel once, Error 7311 with the turn, its age, queue depth and the recursive snapshot), ignores cancellation (Error 7312 every budget, disposal stays pending), ShutDown phase blocked (Error 7314 — a registrant blocking inside DisposeImpl; the exact shape Loki showed), stalled below (Error 7313). ForceTeardownAfterWatchdog is deleted.
Shutdown-time discards are Errors with event ids: a delivery still deferred behind an init gate (7301) and turns still queued when the pump stops (7302), with message type/id/sender/gates/run level.

I/O pools (IoPool / IoPoolRegistry / IoPoolOptions)

Drain() gives a grace before it cancels: waits on the admission count with DrainGrace (8 s) restarted on every completion, so leaves that finish are never cancelled and a burst drains in as many completions; only what outlives a whole grace is named (CancelledLeafSites) and cancelled, then joined under the drain budget as before. New DrainAll(out residual, out cancelledAfterGrace); the old overloads stay.

Activities (ActivityTracker / ActivityRunner)

TrackRun(label, cancel) + ActivityRunHandle.Progress() (called from every ctx.Log line). Quiesce(stallBudget) waits for progressing runs, cancels a run with no progress for one budget (Error 7321), abandons one that ignores it a budget later (Error 7322). Track() is unchanged.

Orchestration

MeshTeardownExtensions.TeardownAsync uses Quiesce; TeardownReport gains CancelledActivities, AbandonedActivities, CancelledIoLeaves, CancelledIoByPool, WorkWasKilled (additive; Clean unchanged). The 30 s hang bound in TeardownAsync and MeshTeardownHostedService now carries the recursive disposal snapshot, at Error.
MonolithMeshTestBase runs the same phase 0, traces DISPOSE_WEDGED_WORK, and does not fail the class on killed work (maintainer: in tests, log).

Tests — the force-dependent tests are rewritten to the new contract: a cooperative wedged turn is cancelled at the stall budget and the phases finish; a non-cooperative one keeps disposal pending, is reported every budget, and completes when released with nothing forced; a backlog is busy, not wedged, and all 800 accepted turns run; the pool grace lets a finishing leaf finish and restarts per completion; the activity quiesce's four verdicts. NackReachesTheWaiterDuringTeardownTest and LateNackReenqueueTest park a turn that observes the owner's shutdown and assert the verdict inside 6 s with no stall verdict logged. DisposalRaceNackTest door one now asserts the accepted request is served before the hub goes down.

Docs — new Doc/Architecture/TeardownLayers (policy, verdict table, the Loki findings incl. the 28-minute preStop/live-circuit cause of the blocked rolls, which is a scheduling matter, not disposal); HubDisposalModel, MeshLifecycle, ControlledIoPooling, DebuggingDisposalAndLeaks updated; What's New entry (Fix).

Verification

Built every touched project with -c Release -warnaserror. Ran locally in Release: the teardown classes in MeshWeaver.Messaging.Hub.Test, MeshWeaver.Hosting.Test, MeshWeaver.Graph.Test; then the full projects Messaging.Hub.Test, Hosting.Test, Graph.Test, Data.Test, Layout.Test, Documentation.Test (link integrity) and Hosting.Orleans.Test under DOTNET_PROCESSOR_COUNT=4 — results in the PR conversation below. NackReachesTheWaiterDuringTeardownTest now completes in well under the 8 s stall budget and logs no stall verdict.

Pairs-with: none — no public type or member is removed; every surface change is additive. The two Plugins twins of the parked-turn tests (MeshWeaver.Hosting.Monolith.Test/{NackReachesTheWaiterDuringTeardownTest,LateNackReenqueueTest}.cs) are updated in a Plugins PR (behaviour change behind an unchanged signature, shape 7); the change there is backward compatible with the current pin. OwnerDisposalNackTest there needs no change — its NACK is minted when the owner's stream ends at the shutdown signal, never by the force.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HiucTsjSiMxEUxYM9ou688

… forced teardown The 8 s disposal watchdog force-tore a hub down from its timer thread while the wedged turn was still running and signalled Dead: a completion that was not true. CI: NackReachesTheWaiterDuringTeardownTest's verdict could only be minted AFTER the force, so its 10 s window was "watchdog + 2 s" and a loaded shard lost it (run 33847949620). Prod (Loki, memex-cloud 08-29 → 09-03): DISPOSAL DEADLOCK DETECTED … [FORCE-TEARDOWN] … 22706ms dozens of times per shutdown, on sync/* hubs and their node hubs, once on a pod still serving. Maintainer directive 2026-09-04: let activities, pooled I/O and handler turns sit out their job; never cancel on entry; no forced teardown in prod or in tests; detect a wedge as no progress for a budget, cancel once cooperatively, and log every kill or discard at Error with what a reproduction needs so the red-log triage files it (prod); in tests, log. - MessageHub: Dispose() no longer cancels the in-flight turn (except a hub still Starting — its bring-up is not accepted work); the watchdog is a stall detector with five report-only verdicts (busy / wedged turn cancelled once / ignores cancellation / ShutDown phase blocked / stalled below), each Error with an EventId and the recursive snapshot; ForceTeardownAfterWatchdog deleted. HandleInitialize races the buildup against its token. - MessageService: TurnsCompleted; shutdown-time discards logged at Error. - IoPool.Drain: grace (admission-count join, restarted per completion) → name wedged leaves → cancel → join; DrainAll reports cancelled-after-grace. - ActivityTracker: TrackRun + Progress + Quiesce(stall) (cancel, abandon); ActivityRunner reports progress per log line and logs a teardown kill. - TeardownAsync/TeardownReport: CancelledActivities, AbandonedActivities, CancelledIoLeaves/ByPool, WorkWasKilled; hang bounds carry the snapshot. - MonolithMeshTestBase: phase-0 quiesce; DISPOSE_WEDGED_WORK traced, not failed. - Tests rewritten to the new contract; Doc/Architecture/TeardownLayers added; HubDisposalModel, MeshLifecycle, ControlledIoPooling, DebuggingDisposalAndLeaks updated; What's New (Fix). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HiucTsjSiMxEUxYM9ou688

rbuergi · 2026-09-04T09:52:51Z

Local verification (Release, `-warnaserror` builds; full projects under `DOTNET_PROCESSOR_COUNT=4`)

project	result
MeshWeaver.Messaging.Hub.Test (full)	367 passed, 0 failed, 2 m 7 s
MeshWeaver.Hosting.Test (full)	350 passed, 0 failed, 15 s
MeshWeaver.Graph.Test (full)	1324 passed, 0 failed, 1 m 27 s
MeshWeaver.Data.Test (full)	474 passed, 0 failed, 45 s
MeshWeaver.Layout.Test (full)	445 passed, 0 failed, 25 s
MeshWeaver.Documentation.Test (full, incl. link integrity and both ratchet guards)	257 passed, 0 failed, 11 s
MeshWeaver.Hosting.Orleans.Test (full, incl. `PodHubTransportTest.CrossSiloNack…`)	251 passed, 0 failed, 30 s
Plugins `MeshWeaver.Hosting.Monolith.Test` teardown twins, built against this branch	22 passed, 0 failed, 10 s

What the bulk runs found and what this branch fixed in response, in order:

IoPoolTest grace test: the first-issued leaf is not the first admitted — the test now signals on whichever leaf runs first.
DisposalStallWatchdogTest backlog case: the stall verdict had no baseline on its first look, so 512 completed turns read as a wedge — the detector now baselines the pump's turn count at Dispose() and treats a turn younger than the budget as movement.
DisposalRaceNackTest door one: with the shutdown queued behind accepted work the request is served, not NACKed — the test asserts the new contract.
DataContextInitTimeoutTest / DataContextShutdownGateAnswersDeferredTest: a sync hub whose InitializeHubRequest waited on a source that never emits held its owner for the whole 120 s startup bound — the bring-up now races its cancellation token (which Dispose() cancels for a hub that never reached Started; the one place that still cancels on entry, and it is not accepted work).

NackReachesTheWaiterDuringTeardownTest completes inside the 6 s bound with no stall verdict logged; PodHubTransportTest.CrossSiloNack… (the other flake in the census) passed in the full Orleans run.

Plugins half: Systemorph/MeshWeaver.Plugins — the two twin tests that parked a turn blind to the shutdown, updated in step (backward compatible with the current pin).

Copilot

🟡 Changes recommended

There are concrete concurrency/teardown-correctness issues in the new ActivityTracker delta emission and IoPool drain terminality semantics that should be fixed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR reworks Mesh teardown semantics to eliminate out-of-band “forced teardown,” letting already-accepted work drain while a stall detector reports (and cooperatively cancels) genuinely wedged turns, and extends the same “progress vs stall” idea to pooled I/O drain and activity teardown.

Changes:

Replaces hub disposal “force teardown after watchdog” with a stall detector (OnDisposalStall) that distinguishes busy vs wedged and logs event-id’d verdicts with diagnostics.
Updates IoPool.Drain() to add a completion-based grace period before cancellation, and enhances teardown reporting (per-pool residuals + cancellations after grace).
Adds activity quiescing with progress tracking/cancel/abandon verdicts, threads these outcomes into teardown reports, tests, and documentation.

File summaries

File	Description
test/MeshWeaver.Messaging.Hub.Test/MessageHubTest.cs	Updates disposal regression coverage to the new “drain backlog / cooperative cancel” contract.
test/MeshWeaver.Messaging.Hub.Test/HostedHubDisposalJoinTest.cs	Aligns hosted-hub disposal join expectations with stall-based teardown (no force teardown).
test/MeshWeaver.Messaging.Hub.Test/DisposalStallWatchdogTest.cs	Rewrites watchdog tests to validate the new stall-detector verdicts and behaviors.
test/MeshWeaver.Messaging.Hub.Test/DisposalRaceNackTest.cs	Updates the race contract so accepted work is served (door one) and NACK behavior remains correct (door two).
test/MeshWeaver.Messaging.Hub.Test/DisposalDeadlockDiagnosticsTest.cs	Pins the new diagnostics-rich stall verdict output.
test/MeshWeaver.Hosting.Test/IoPoolTest.cs	Adds/updates tests for drain grace, restart-on-completion behavior, and cancellation reporting.
test/MeshWeaver.Hosting.Test/IoPoolResidualNamesItsPoolTest.cs	Extends drain option coverage to include `DrainGrace` default/contract checks.
test/MeshWeaver.Hosting.Test/ActivityTrackerQuiesceTest.cs	New tests validating activity quiesce verdicts (wait / cancel / abandon) and legacy surfaces.
test/MeshWeaver.Hosting.Monolith.TestBase/MonolithMeshTestBase.cs	Integrates activity quiesce + extended teardown reporting into the monolith test teardown trace/reporting.
test/MeshWeaver.Graph.Test/NackReachesTheWaiterDuringTeardownTest.cs	Updates the parked-turn arrangement to observe shutdown (no reliance on forced teardown) and asserts no stall verdicts.
test/MeshWeaver.Graph.Test/LateNackReenqueueTest.cs	Aligns late-verdict behavior with “accepted work drains” teardown and shutdown-observing parked turns.
src/MeshWeaver.Messaging.Hub/MessageService.cs	Adds turns-completed counter and error-level discard logging with event ids for shutdown-time discards.
src/MeshWeaver.Messaging.Hub/MessageHub.cs	Implements stall detector logic, removes forced teardown path, and adds event ids/verdict handling.
src/MeshWeaver.Messaging.Hub/HostedHubsCollection.cs	Updates rationale/comments around joining children without out-running their terminal state (now stall-detected).
src/MeshWeaver.Mesh.Contract/Threading/MeshTeardownSignal.cs	Extends `TeardownReport` with cancelled/abandoned activities and cancelled I/O leaves reporting.
src/MeshWeaver.Mesh.Contract/Threading/IoPoolRegistry.cs	Passes `DrainGrace` into pools and extends `DrainAll` to return cancelled-after-grace details.
src/MeshWeaver.Mesh.Contract/Threading/IoPoolOptions.cs	Adds configurable `DrainGrace` with production-default semantics.
src/MeshWeaver.Mesh.Contract/Threading/IoPool.cs	Adds drain grace behavior and cancellation-after-grace reporting surfaces.
src/MeshWeaver.Mesh.Contract/MeshTeardownExtensions.cs	Switches phase-0 from “wait for idle” to progress-based activity quiesce and logs event-id’d findings.
src/MeshWeaver.Mesh.Contract/MeshNodeStreamExtensions.cs	Updates commentary around recycle/shutdown probing to match the new stall-detector behavior.
src/MeshWeaver.Mesh.Contract/ActivityTracker.cs	Adds run tracking with progress/cancel/abandon quiesce and new public handle/report types.
src/MeshWeaver.Hosting/MeshTeardownHostedService.cs	Upgrades teardown-failure logging to Error and attaches disposal diagnostics.
src/MeshWeaver.GitSync/ActivityRunner.cs	Registers activity runs with label/cancel and reports progress on each appended log line; distinguishes “cancelled by teardown.”
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-04-shutdown-lets-work-finish-and-names-what-it-had-to-stop.md	What’s New entry describing the new shutdown/teardown contract.
src/MeshWeaver.Documentation/Data/Architecture/TeardownLayers.md	New architecture doc formalizing “application drains first, infra after; no forced teardown,” with verdict table.
src/MeshWeaver.Documentation/Data/Architecture/MeshLifecycle.md	Updates lifecycle teardown description to incorporate progress-based activity quiesce and I/O drain grace.
src/MeshWeaver.Documentation/Data/Architecture/HubDisposalModel.md	Updates hub disposal model to describe the stall detector and non-forcing behavior.
src/MeshWeaver.Documentation/Data/Architecture/DebuggingDisposalAndLeaks.md	Updates debugging guidance to the new “stall is reported, not forced” model.
src/MeshWeaver.Documentation/Data/Architecture/ControlledIoPooling.md	Updates I/O drain documentation to “grace then cancel/join” and how cancellations are reported.
src/MeshWeaver.Data/Workspace.cs	Adjusts disposal/join commentary to match “accepted work drains; stall is reported” model.
src/MeshWeaver.Data/Serialization/JsonSynchronizationStream.cs	Updates disposal/join commentary to match the new teardown semantics.
src/MeshWeaver.Data/DataExtensions.cs	Updates rationale around DI resolution during shutdown after removal of out-of-band force teardown.

Review details

Suppressed comments (2)

src/MeshWeaver.Mesh.Contract/ActivityTracker.cs:130

Complete calls deltas.OnNext(-1) directly from the caller thread, but Subject<T> is not thread-safe under concurrent OnNext calls. This can race with TrackRun and corrupt or fault the count stream. Emit the delta via the EventLoopScheduler (or otherwise serialize emissions) to keep counts consistent.

    internal void Complete(ActivityRunHandle handle)
    {
        if (runs.TryRemove(handle.Ticket, out _))
            deltas.OnNext(-1);
    }

src/MeshWeaver.Messaging.Hub/MessageHub.cs:1958

The inline comment says "three verdicts" but OnDisposalStall now produces five distinct outcomes; the comment should match the current contract to avoid misleading future debugging/triage.

        // Pending: the outer bounds — the test base's dispose deadline, the host's teardown
        // budget — report a hang with these diagnostics attached instead of a lie about
        // completion. See OnDisposalStall for the three verdicts.

Files reviewed: 32/32 changed files
Comments generated: 4
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

rbuergi · 2026-09-04T10:00:06Z

        if (!TryEnterGateRegion()) return 0;
        try
        {
+            // 🚨 GRACE FIRST — let the work finish its job. Nothing is cancelled until the pool has


Agreed and fixed: Drain() now sets _draining on its first line, and every entry point refuses a leaf issued after that (Cancelled<T>(), the same answer as after disposal). Already-admitted work keeps its grace; a producer issuing one leaf per completion can no longer keep the admission count moving. IoPoolTest (grace tests included) and the full MeshWeaver.Hosting.Test pass.

rbuergi · 2026-09-04T10:00:07Z

+    public ActivityRunHandle TrackRun(string label, Action? cancel)
    {
+        var handle = new ActivityRunHandle(this, Interlocked.Increment(ref ticketSeq), label, cancel);
+        runs[handle.Ticket] = handle;
        deltas.OnNext(1);


Fixed: TrackRun/Complete now go through Emit(delta), which schedules the OnNext onto the tracker's own EventLoopScheduler — the subject is only ever called from that one thread, as the surrounding comment claims. (Track() had the same shape before this PR; it is covered by the same change.) ActivityTrackerQuiesceTest passes.

rbuergi · 2026-09-04T10:00:08Z

+                quiesce = await activities.Quiesce(ActivityStallBudget).ObserveCompletion(
                    ex => logger?.LogError(ex,


Deliberate and pre-existing, not introduced here: TeardownAsync is the ONE sanctioned async edge — the mesh-teardown boundary — per Doc/Architecture/MeshLifecycle ("the only sanctioned await is that single three-phase drain at the boundary") and HubDisposalModel. It runs on the host's shutdown thread / xUnit's teardown thread, never on a hub turn, and every wait inside it is a subscription bridged through ObserveCompletion (RunContinuationsAsynchronously), which is what keeps hub threads from carrying it. The hub-side machinery this PR changes stays purely reactive.

rbuergi · 2026-09-04T10:00:10Z

+    /// <summary>
+    /// The STALL budget of the disposal watchdog: how long the subtree may make no
+    /// <see cref="RunLevel"/> progress before the watchdog looks at what is holding the
+    /// teardown. It is not a duration cap and it forces nothing — see
+    /// <see cref="OnDisposalStall"/> for the three verdicts it can reach.


Fixed: both comments now say five verdicts, matching OnDisposalStall and Doc/Architecture/TeardownLayers.

github-actions · 2026-09-04T09:56:49Z