Waiting for disposal must be 100% reactive — five sites still poll, bridge to Task, or race a timeout (CompilationCacheService unloads anyway after 5s)

The rule

must wait 100% reactively for disposal. observable.Subscribe(_ => /*dispose finish*/)

Any wait for disposal that is not a subscription is a defect: no polling loop, no SpinWait, no Task/.Wait()/.Result/.ToTask() gate, no timeout race, no fixed delay. Disposal completion is an IObservable<T>; the waiter subscribes and the continuation lives in Subscribe(_ => …).

The reference implementation already exists in this repo

MessageHubGrain.OnDeactivateAsync is exactly right, and its own comment records that the anti-pattern was removed from it:

hub.DisposalCompleted
    .Take(1)
    .Catch<Unit, Exception>(ex => { logger.LogError(ex, "… KEEPING its load context"); return Observable.Empty<Unit>(); })
    .Subscribe(_ => UnloadContextIfSafe(reason, grainId));

🚨 The unload is gated on a POSITIVE drain report and never on a timer. It used to run unconditionally: wait up to 5 s for DisposalCompleted, log "moving on", unload anyway — i.e. the one case where we KNEW a user was still live was also a case where we unloaded.

Note also that its fault handling is fluent (.Catch), not try/catch — a fault travelling through the stream arrives later, on another thread, and sails straight past a try around the Subscribe call.

Sites that still violate it

1. 🚨 `CompilationCacheService.cs:664-667` — the exact pattern `MessageHubGrain` removed

var drainDeadline = Environment.TickCount64 + 5_000;
var spin = new SpinWait();
while (Volatile.Read(ref _pins) > 0 && Environment.TickCount64 < drainDeadline)
    spin.SpinOnce();
…
Unload();

A spin loop, a 5 s ceiling, and — by its own comment — "after it we unload anyway (last-resort, matching the prior always-unload behaviour)". This is the same "unload while a user may be live" that the grain path was fixed to stop doing, and the consequence the code itself names is assembly corruption mid-GetTypes/attribute resolution (TypeLoadException '…format is invalid'), i.e. the #613 use-after-unload family.

Should be: a "pins reached zero" IObservable<Unit> that Unload() subscribes to. If pins never reach zero, that is a leak to fix — not a 5 s budget to spend.

2. `MeshTeardownExtensions.cs:73-76` — Task gate + timeout, then dispose regardless

await activities.WhenIdle.Timeout(timeout)
    .Catch<Unit, Exception>(_ => Observable.Return(Unit.Default))
    .ToTask();
mesh.Dispose();

.ToTask() is a Task-as-a-gate, .Timeout(...) is a race against the thing being waited for, and .Catch(→ Return(Unit)) swallows the timeout and disposes anyway — the writes this quiesce exists to protect ("those Append/Finish writes go back through hubs that are still alive") are exactly what proceeds unprotected.

3. `MonolithMeshTestBase.cs:1534-1545` — bridges the observable to a Task, then polls it

var disposal = Mesh.DisposalCompleted.Catch(…).FirstOrDefaultAsync().ToTask();
while (!disposal.IsCompleted) { … await disposal.WaitAsync(progressCts.Token); … }

The signal is reactive and is immediately converted into a Task and polled. Test-side, but this is the canonical "wait for disposal" every test runs, so it shapes what teardown failures look like across the suite.

4–5. Reactive core, timeout escape — weaker, listed for completeness

IoPoolSiloTeardown.cs:134 — subscribes registry.Disposed (correct) but .Timeout(JoinBudget).Catch(→ -1).ToTask(); a leaf that never unwinds is reported as a residual and shutdown continues.
ActivityWriteTracker.cs:252 — states.Where(count == 0).Take(1) (correct) but .Timeout(DrainTimeout).Catch(…).

Both at least observe a real completion signal and log loudly, so they are a different class from 1–3 — but each still races the thing it is waiting for.

Why this matters beyond tidiness

A non-reactive waiter fails in two different-looking ways, which is why the current Orleans cluster reads as several unrelated bugs:

it gives up too early → a timeout (OrleansGrainTeardownStragglerTest Flake: OrleansGrainTeardownStragglerTest — the activation never leaves the silo catalog (4 hits on 2026-08-25, all unrelated diffs) #2301, HubWorksAfterDisposal Flake: OrleansMeshTests.HubWorksAfterDisposal — shard-0 Orleans suite is red across unrelated PRs #2346);
it observes a state that has been reached but whose completion was never published to it → content-at-terminal-state (OrleansAutoExecuteTest: a cell at Completed still holding the placeholder, failing with time to spare — [took 20s] against three 30 s waits).

"Reached but not settled" is precisely what polling a state gives you instead of subscribing to its completion. So for the teardown-adjacent flakes the question is not "what is slow or contended" but "who is waiting for disposal, and how?"

Scope note

grep for blocking bridges inside dispose paths (.Wait(), .Result, .GetAwaiter().GetResult() near dispose/teardown/shutdown/deactivate) returns nothing in src/ and memex/ — the crude blocking form is already gone. What remains is the subtler set above: poll loops, Task bridges, and timeout escapes.

The same rule applied to READINESS finds the suspected Orleans deadlock — and the correct signal already exists

Extending the sweep past disposal, because "wait 100% reactively" is a statement about waiting, not about one lifecycle stage. AGENTS.md already bans the general form: "Task-as-a-gate is the same sin. TaskCompletionSource used to make callers 'await a signal'."

The grain already has the right shape

// MessageHubGrain.cs:76,90
private readonly ReplaySubject<IMessageHub> _hubReadyRaw = new(bufferSize: 1);
private IObservable<IMessageHub> HubReady => _hubReadyRaw.Synchronize();

A replaying single-value subject — fires once, replays to late subscribers. Exactly right.

…and then converts it back into a Task gate

MessageHubGrain.DeliverMessage:701 wraps a TaskCompletionSource<IMessageDelivery> completed only when _hubReadyRaw terminates. For the "hub never becomes ready" case, the only terminal is produced inside OnDeactivateAsync. Orleans counts that Task as an in-flight request and drains in-flight requests before calling OnDeactivateAsync — so the completion depends on the callback it is blocking. The only release is GrainCollectionOptions.DeactivationTimeout.

A circular wait on the grain's hot path, released by a timer. It also produces a fixed-length stall rather than a variable one, which is the signature to look for — and #2301's failure is a fixed 30 s.

🚨 Unconfirmed: nobody has yet proven Orleans 10.2.2 blocks deactivation on that pending TCS — only that the shape is self-referential. Recording it here because it is the same defect class as the five disposal sites, found by applying the same rule.

Note the fix constraint: Orleans' grain contract requires a Task at the boundary, so the bridge itself cannot simply be deleted. The defect is not the bridge — it is that the Task's completion depends on OnDeactivateAsync. The fix is to guarantee the readiness observable has a terminal for the never-ready case that fires outside the deactivation callback.

Two more Task-shaped readiness gates

MessageHub.cs:927 exposes the hub's own readiness as a Task:

private readonly TaskCompletionSource hasStarted = new();
public Task Started => hasStarted.Task;

Real consumers of the Task (as opposed to the enum RunLevel > MessageHubRunLevel.Started, which is a synchronous state read and fine):

WorkspaceOperations.cs:146 — if (!stream.Hub.Started.IsCompleted) — polling a Task for readiness;
GenericUnpartitionedDataSource.cs:319 — Task.WhenAll(Streams.Values.Select(s => s.Hub.Started)) — a Task gate fanned across every stream's hub.

MessageService.cs:149 (startupCompletionSource) is the same shape one layer down.

Since Started is public, replacing it with an IObservable<Unit> is an API change and wants its own decision — but the two consumers above are exactly the "await a signal" pattern the rule forbids, and .IsCompleted polling is the readiness twin of the disposal poll loops in the issue body.

Checked and clean

SemaphoreSlim — the AGENTS.md ban is fully honoured. Filtering comments out, the only real usage in src/ and memex/ is IoPool.cs:16,93, the one sanctioned site. Seven other files mention it solely in comments explaining why they do not use one. Worth stating so nobody re-runs that grep and files seven false positives.

A related site, and it needs stating carefully — two documented rationales that do not quite meet

Raised by the session that root-caused two members of #2346: VersionPlugin's four tools use a bare .SubscribeOn(TaskPoolScheduler.Default), which is verbatim what IPooledSubscribeScheduler describes itself as replacing:

Historically that hop used SubscribeOn(TaskPoolScheduler.Default) — a bare ThreadPool schedule that mesh teardown cannot see. During teardown that render subscribe keeps executing on a ThreadPool thread AFTER the hub's Autofac LifetimeScope is disposed (→ ObjectDisposedException) and, for a node whose render touches types compiled into a collectible AssemblyLoadContext, AFTER that ALC is unloaded (→ native use-after-unload SIGSEGV — the endemic teardown crash reproduced as exit=139).

🚨 But this is not a flat rule violation, and I checked before amplifying it. VersionPlugin carries its own deliberate rationale (VersionPlugin.cs:18-31), and it is a different argument from the one the interface sanctions:

The interface's exemption is narrow: the bare fallback is fine only for "bare messaging-only hubs [that] compile no collectible ALCs, so the drain has nothing to join."
VersionPlugin's argument is not that one. It is that ToolTask.Bridge settles every terminal and observes the round's cancellation token, "so a parked read cannot hold an IoPoolNames.Ai gate permit through teardown (Five agent tools still bridge an observable to a Task that cannot be cancelled — the #1863 shape, audited per site #1956)."

The precise question, which nobody has answered

Those two protect different things. Releasing an IoPoolNames.Ai gate permit is not the same as the subscribe leaf being drainable. A bridge that observes cancellation can perfectly well release the permit and hand a terminal back to the caller while the underlying ThreadPool subscribe keeps running — which is exactly the invisible-to-DrainAll() state the interface doc warns about.

So the question to settle is:

Does ToolTask.Bridge observing cancellation actually stop the SubscribeOn(TaskPoolScheduler.Default) work, or does it only stop waiting for it?

If the latter, VersionPlugin's rationale is sound about the gate permit and silent about the leaf, and these four tools are exactly the drain-invisible shape — on an AI hub, which is not the ALC-free case the exemption covers.

The session that raised it also reported that this contradicts the premise of the test covering those tools, which is the observable consequence: a test that believes the drain reaches this work is asserting over something the drain cannot see.

Recording rather than changing: the two rationales are each internally coherent, and deciding between them needs someone who owns ToolTask.Bridge's contract. It belongs with this issue because it is the same underlying rule — work you cannot observe the completion of is work you cannot wait for reactively.

For completeness, the other bare sites found in the same sweep (InstanceAutoRegistrationService.cs:307, ModuleDiscoveryService.cs:181, RegistryUpdateReconciler.cs:124, ActivityRunner.cs:272, MeshNodeStreamCache.cs:2260) were not assessed against their own rationales and should not be assumed to be defects on the strength of the pattern alone.

Root cause of the `HOST_CRASHED` teardown failures: `.ToTask()` cannot observe a LATE fault — and the interface documents it as the sanctioned shape

Maintainer, today: "this is coming when we don't await properly. need to be 100% reactive: disposedObservable.Subscribe(...)". That is exactly right, and here is the mechanism with the evidence.

The shape, as `IMessageHub.cs:264-273` currently prescribes it

/// Observable completion of disposal — fires Unit once, then completes, when the hub has
/// finished disposing (or OnError on a disposal fault). … disposal completion is OBSERVED,
/// never awaited on a hub thread. … At a genuine async edge (test teardown, grain
/// deactivation) bridge once with `DisposalCompleted.FirstOrDefaultAsync()` / `.ToTask()`.

The first half is right. The last sentence is the defect — and "test teardown, grain deactivation" is precisely where OrleansGrainTeardownStragglerTest (#2301) crashes.

Why `.ToTask()` is not equivalent to `Subscribe`

A Task settles exactly once and then has nowhere to put anything. After it completes — normally, by Timeout, or by cancellation — a fault still travelling the observable has no observer. It becomes an unobserved exception, surfaces on the finalizer as UnobservedTaskException, and xUnit v3 escalates that to a Catastrophic failure.

That is not inference. OrleansGrainTeardownStragglerTest's own docstring describes the escalation verbatim:

…the throw escapes RAW into the activation source's Rx chain … faults a ThreadPool task nobody observes, and xUnit v3 escalates the UnobservedTaskException to a Catastrophic failure that poisons the NEXT test class (OrleansSubThreadRoutingTest died as collateral in the incident).

Subscribe(onNext, onError) always has an error arm, including for a fault that arrives after the interesting value did. .ToTask() structurally cannot.

The two waits in the crashing test

await hub.DisposalCompleted.Catch(…).FirstOrDefaultAsync().ToTask(ct);        // ① bridge

await Observable.Interval(TimeSpan.FromMilliseconds(100))                      // ② POLL
    .SelectMany(_ => mgmt.GetDetailedGrainStatistics().ToObservable())
    .Where(stats => stats.All(s => !string.Equals(s.GrainId.ToString(), grainId, …)))
    .FirstAsync().Timeout(TimeSpan.FromSeconds(30)).ToTask(ct);

① is the documented bridge. ② is worse: a 100 ms poll of the silo catalog with a 30 s Timeout — and when that Timeout fires, the task settles while GetDetailedGrainStatistics() calls are still in flight against a silo mid-teardown. Every one of those can fault into nothing.

This also explains the 30 s that everyone has been reading as "a slow teardown". A previous investigation measured the activation leaving the catalog in 0.10 s when healthy — so a 30 s wait is never "nearly finished". It is the Timeout firing, settling the task, and orphaning whatever faults next.

The fix, and the reason it needs a real signal

② polls because there is no deactivation-completed observable to subscribe to. So "make it reactive" is not a rewrite of the test — it is exposing the signal the test is currently simulating:

a deactivation/catalog-departure completion as IObservable<Unit> (AsyncSubject-shaped: fires once, replays to late subscribers), and
every waiter Subscribe-ing it, with an error arm, rather than polling or bridging.

🚨 And IMessageHub.cs:273 must stop recommending .ToTask(). An interface doc that prescribes the crashing shape is why this recurred four times: each fix targeted the timeout, because the documentation said the bridge was correct.

Scope

This is the same rule as the five sites in this issue's body, arriving from the other direction: those are waits that poll or race a timeout; this is a wait that cannot observe a late fault at all. MeshTeardownExtensions.cs:76,117 and IoPoolSiloTeardown.cs:142 use the same .ToTask() bridge and should be assessed together.

Related: #2301 (four recurrences, now on main with HOST_CRASHED), #2505 (which made the crash visible at all).

The rule is right; the REASON is deadlock, and that reorders the site list

PR #2527 lands sites 2–5. One correction to my own earlier comment on this issue, which framed
.ToTask() as "settles once and then cannot observe anything":

That is the second-order effect. The primary failure is a DEADLOCK. Maintainer, today:
"totask? this is deadlock" — and the /async skill's Rule 1 says it directly: the blocking family
on a hub action block, grain turn or Blazor circuit "parks the single-threaded scheduler — the
message you are waiting for can never be processed."

Why it bites even where nobody wrote a block

Rx's ToTask() completes its TaskCompletionSource without
RunContinuationsAsynchronously, so the awaiter resumes inline on the signalling thread.
Everything after the await then runs on the hub's disposal thread or the grain's turn scheduler —
and await captures TaskScheduler.Current when there is no SynchronizationContext, so one
inline resumption routes every later await in that method onto the same scheduler.

That is #2301 exactly: the test resumed onto the deactivating grain's turn scheduler and then held
it for 30 s waiting for that grain's deactivation. Every occurrence stalled at precisely the
Timeout budget, never a distribution around it, while a healthy activation leaves the catalog in
0.10 s. A number that is always the budget is a deadlock; contention gives a spread.

The question to ask at each site, and the answers

whose thread am I blocking, and does the signal I am waiting for need it?

site	whose thread	verdict
2. `MeshTeardownExtensions:76,117`	the mesh hub's own disposal thread — and everything after the await ran there, including `ioPools.DrainAll()`, a synchronous JOIN of every pooled leaf	converted. Worse than "a swallowed fault": a hub thread parked on a join of work that may need the hub.
3. `MonolithMeshTestBase:1534`	same, and it is the wait every test in the suite runs	converted — the Task-poll loop is gone; progress is an `Observable.Interval` stopping on the signal's own terminal.
4. `IoPoolSiloTeardown:134`	nobody's. It composes and RETURNS — Orleans awaits it, this method blocks nothing	kept, verified. `ILifecycleObserver` genuinely declares `Task OnStop(CancellationToken)`, and the bridge sits at the outermost edge, which is the rule (`/async` Rule 1a). Its comment previously justified itself by citing `IMessageHub.DisposalCompleted`'s remarks — the sentence that turned out to be the bug — and now records the measurement instead.
5. `ActivityWriteTracker:252`	not assessed here	still open
`SharedOrleansFixture.CleanupClientAsync` (not in the original list)	the client hub's disposal thread, inside a bare `catch { }`	converted

Site 1 — CompilationCacheService:664, the SpinWait + 5 s "unload anyway" — is untouched and
still the worst one. It is a different mechanism (an ALC-unload safety decision, not a Task
bridge) and needs the "pins reached zero" observable this issue asks for, plus a decision about what
happens when pins never reach zero. Deliberately not folded in.

Two swallows became data rather than silence

TeardownReport now carries DisposalFault and ActivitiesQuiesced, both logged at Error. They do
not flip Clean — whether a faulted disposal should fail a test class is an escalation
decision of its own and would redden the trunk for a reason unrelated to this fix. Named so it is
not mistaken for done.

A site checked and ruled OUT, with evidence

DataContext.cs:449 — Task.WhenAny(allInit, Task.Delay(InitializationTimeout, …)).ContinueWith(…).
It is a genuine instance of the banned timer-race shape, but it is not the #2301 root and not
a deadlock:

it blocks no thread — it hangs a ContinueWith on TaskScheduler.Default and returns, so it
cannot park a grain turn or a hub action block;
its budget is 120 s, while every Flake: OrleansGrainTeardownStragglerTest — the activation never leaves the silo catalog (4 hits on 2026-08-25, all unrelated diffs) #2301 occurrence stalled at exactly 30 s, which is the
test's own Timeout;
the failing traces stall between "Hub disposal completed" and catalog departure, with no
initialization gate in them.

It still deserves a reactive rewrite, but one that preserves the Hub.IsShuttingDown → FailGate
branch (#1270), the disarm-on-dispose (#1122) and the time-box itself (the 2026-06-26
path-resolution storm). Its own PR, with its own tests.

The documentation that made this a rule nobody could follow

IMessageHub.DisposalCompleted prescribed the crashing bridge in its own remarks. Replaced, leading
with the deadlock; AsynchronousCalls.md gains a matching section; both properties are pinned by
DisposalWaitBridgeTest with controls asserting the old shape's behaviour, so the claim is
measured rather than asserted.

🤖 Generated with Claude Code

The readiness twin: `IMessageHub.Started` is the same defect at the other end of the lifecycle

Found while sweeping Task out of the mesh core for #2489 (#2531 / #2535). Reporting rather than doing, because it lands squarely in the files this issue is open in — and because it is one coherent change that should not be smuggled into a hygiene PR.

MessageHub.cs:927 declares private readonly TaskCompletionSource hasStarted and :931 exposes public Task Started. That is a Task-shaped readiness gate with exactly the properties this issue objects to in the disposal gate: it is polled with .IsCompleted, fanned with Task.WhenAll, and its failure semantics are hand-tuned around a Task-only hazard.

Consumer list (complete, measured)

src/ — two:

MeshWeaver.Data/WorkspaceOperations.cs:146 — if (!stream.Hub.Started.IsCompleted) throw new DataException(...). A non-blocking poll, but note it is subtly not RunLevel >= Started: FailStartup completes the task (cancelled/faulted) without moving RunLevel, so a failed-startup hub passes this check today.
MeshWeaver.Data/GenericUnpartitionedDataSource.cs:319 — public Task Initialized => Task.WhenAll(Streams.Values.Select(s => s.Hub.Started)), i.e. IDataSource.Initialized (:106).

test/ — two: FailStartupTeardownClassificationTest (5 assertions on IsCanceled/IsFaulted/Exception) and SubscribeDuringRecycleTest:80 (await area.Started.WaitAsync(...)).

Why it is more than a signature change

IDataSource.Initialized feeds DataContext, which is where the real machinery is — 4 more Task sites (:58 IsInitializing, :73 Initialization, :351, :427) and a ~120-line OpenInitializationGate built on Task.WhenAll + Task.WhenAny + Task.Delay + ContinueWith + a watchdogCancellation CTS. That is exactly the timer-race shape the /async skill's Rule 1a calls out: "what if the wait expired?" — and here the answer is a hand-written FailGate branch per outcome.

Converting to IObservable<Unit> (an AsyncSubject<Unit>) deletes that wholesale — Merge(...).Timeout(InitializationTimeout).TakeUntil(disposal) is the whole gate — and it retires a mechanism that only exists because of Task: FailStartup's TrySetCanceled-instead-of-TrySetException classification exists solely to avoid arming a TaskScheduler.UnobservedTaskException on a hub with no awaiter (the #228 "Catastrophic failure" capture, documented at MessageHub.cs:208-224). An observable has no unobserved-exception hazard, so that entire branch — and the test pinning it — becomes unnecessary rather than needing a port.

Recommendation

Right fix, own PR, and it belongs with this issue's work rather than beside it: IMessageHub.Started and MessageHub.FailStartup are in the same file as the teardown classification #2301/#2488 is changing, so doing them separately guarantees a conflict. It is a public-interface break (IMessageHub, IDataSource), so it also wants to ride the same release note.

Sequencing suggestion: land the disposal half first, then Started, so MessageHub is only rewritten once.

For the record, the rest of the #2489 sweep found no other core violation worth converting: GenericUnpartitionedDataSource.GetInitialValueAsync and OrleansRoutingService.EnqueueStreamTeardown are already the sanctioned one-line boundary bridges (StreamConfiguration.WithInitialization and AsyncDisposeQueue both take Func<…, Task>), with fully reactive bodies and comments saying so.

Answering the scoped question for `src/MeshWeaver.AI/` — PR #2537

establish whether ToolTask.Bridge actually stops the work or only stops waiting for it

It stops waiting, and requests the teardown. It cannot promise the work has stopped — and the file claimed it could.

Bridge disposes before it settles, which is right and stays. But Rx disposal is synchronous only for a chain that is synchronous to dispose, and every one of these tool pipelines hops a scheduler — and Rx runs a sequence's unsubscribe on the scheduler it subscribed on. So pending.Dispose() returns once the unsubscribe is scheduled; the source is torn down some time later, possibly after the caller's task has already thrown.

The repo already knew half of this. AgentToolCancellationTest waits for the disposal rather than reading it, and its comment says exactly why. ToolTask's own <summary> asserted the stronger claim ("the work the tool started has provably stopped before anyone can act on the call having ended"). Corrected in #2537, in both the summary and at Settle.

What can promise it is the pool, so the mechanism now matches the claim. VersionPlugin's bare .SubscribeOn(TaskPoolScheduler.Default) is the sibling of the sites listed in this issue — not a poll or a timeout race, but the third shape: work moved somewhere the drain cannot see. IIoPool.SubscribeThroughPool already documents itself as "the tracked, drainable replacement for a bare .SubscribeOn(TaskPoolScheduler.Default), which the drain cannot reach", so the fix was to use it. Seven hops converted (VersionPlugin ×4, CollaborationPlugin ×2, DelegationTool ×1) via a new ToolTask.Pooled, on the AgentStore pool — never Ai, since the call already sits inside a round holding an Ai permit. Both forms leave the caller's scheduler, which is all those comments ever asked for; only one is visible to Drain().

A sibling of site 1, found in the same folder

DelegationTool.Settle still read if (set()) pending.Dispose(); — the exact shape ToolTask's comment names as the defect it had removed, i.e. the caller is told the call ended and the work is released afterwards. With RunContinuationsAsynchronously those are not interchangeable: on cancellation the round could observe the cancel, release its Ai permit and let teardown through while a live subscription to the sub-thread's node stream was still open. Fixed to dispose-then-set.

Suggested addition to this issue's list

6. ToolTask.Bridge + a scheduler-hopping source — not a poll, not a timeout race, but the same family: the waiter believes disposal is synchronous when it is scheduled. Generalised: a dispose-before-settle ordering buys the caller-visible half only; joining the work at teardown is the pool's job, and only pooled work is joinable. Any site that hops TaskPoolScheduler.Default outside IIoPool is invisible to DrainAll() by construction, which is the same failure mode as sites 1–3 seen from the other side.

Scope note: #2537 is src/MeshWeaver.AI/ only. Sites 1–5 in the body (CompilationCacheService, MeshTeardownExtensions, MonolithMeshTestBase, IoPoolSiloTeardown, ActivityWriteTracker) are untouched and still open.

Sites 2–5 merged: #2527 (9fba37ced) and its review follow-ups #2538 (bea984da8), both CI-green.

Two things this issue should carry forward.

1. The rule's REASON, corrected. My earlier comment framed .ToTask() as "settles once and
then cannot observe anything". That is the second-order effect; the primary failure is a
deadlock, and the distinction is what made four fixes to #2301 aim at the timeout. Rx completes
a ToTask() TaskCompletionSource without RunContinuationsAsynchronously, so the awaiter
resumes inline on the signalling thread — a hub's disposal thread, or a grain's turn scheduler —
and then does the rest of its work there. It is also sticky: await captures
TaskScheduler.Current absent a SynchronizationContext, so one inline resumption routes every
later await in the method onto that scheduler.

So there are two questions to ask at each site, not one:

whose thread does the completion need, and am I holding it? (the deadlock, from the waiting side);
is anything left to observe a fault after the wait settles? (the lost fault).

And a third that #2538 added: can the caller carry a hub scheduler INTO this method? —
ConfigureAwait(false) is the answer, and it is why the teardown awaits now carry it.

2. What remains open here.

Site 1, CompilationCacheService.cs:664 — the SpinWait + 5 s "unload anyway". Untouched,
and still the most serious: it is the one site whose failure mode is a use-after-unload SIGSEGV
rather than a hang. It needs the "pins reached zero" observable this issue asks for, plus a
decision about what happens when pins never reach zero (the answer must not be "unload anyway").
Site 5, ActivityWriteTracker.cs:252 — not assessed.
Ruled OUT with evidence and split to its own issue: DataContext's init watchdog is a Task-shaped timer race (Task.WhenAny + Task.Delay + ContinueWith) — make the wait reactive without deleting the time-box #2528, DataContext.cs:449's
Task.WhenAny + Task.Delay + ContinueWith init watchdog. It is the banned timer-race SHAPE but
not the banned BEHAVIOUR (its expiry fails fast rather than proceeding anyway), it blocks no
thread, and its 120 s budget does not match Flake: OrleansGrainTeardownStragglerTest — the activation never leaves the silo catalog (4 hits on 2026-08-25, all unrelated diffs) #2301's invariant 30 s. Its fix must preserve three
incident-driven behaviours (Per-node hub DataContext initialization times out at 120s, taking every hub under a user partition dark (root of #3517, #2886, #1313) #1122, A gate that can never reopen, and an unbounded nested create, can strand a request forever #1270, the 2026-06-26 storm), so it is its own change.

🤖 Generated with Claude Code

Two pre-existing ALC tests asserted the MECHANISM, not the property

Merge pull request #2549 from Systemorph/fix/2488-reactive-disposal-w…

Closing — all five sites are now reactive, verified at HEAD.

Site	Fix
1. `CompilationCacheService` SpinWait + 5 s "unload anyway"	#2549 — the drain is a subscription; and the open decision from the last comment is answered in code: a faulted drain signal now KEEPS the load context instead of unloading anyway (`"Drain signal for AssemblyLoadContext {ContextName} faulted — KEEPING its load"`). No `SpinWait` remains in the file.
2–4	#2527 (`9fba37ced`) + review follow-ups #2538 (`bea984da8`), both CI-green.
5. `ActivityWriteTracker`	Verified at HEAD: `Drain()` is `Observable.Defer` + `Where(Count == 0)` + bounded `Timeout(DrainTimeout)`, completing immediately when idle and logging loudly — naming the abandoned paths — on expiry.

Deliberately split out and staying open: #2528 — DataContext's init watchdog, ruled out of this issue's scope in the comment above (banned shape, not banned behaviour; its fix must preserve three incident-driven behaviours).

The corrected rationale from the last comment (inline ToTask() resumption on the signalling thread = deadlock, plus sticky scheduler capture) is recorded in the thread and in HubDisposalModel.md.

Re-measured 2026-08-28: site 1 — the one this issue flags 🚨 — is DONE. Sites 2–5 remain.

#	Site	State
1	`CompilationCacheService.cs` — "the exact pattern `MessageHubGrain` removed"	✅ clean — no `ToTask`, `.Wait(`, `.Result`, `Task.Delay` or `.Timeout(` anywhere in the file (1,411 lines)
2	`MeshTeardownExtensions.cs` — Task gate + timeout, then dispose regardless	❌ 6 × `ToTask`, 1 × `.Timeout(`
3	`MonolithMeshTestBase.cs` — bridges the observable to a Task, then polls	❌ 6 × `ToTask`, 2 × `.Wait(`/`.Result`, 1 × `.Timeout(`
4	`IoPoolSiloTeardown.cs` — `.Timeout(JoinBudget).Catch(→ -1).ToTask()`	❌ 5 × `ToTask`, 1 × `.Timeout(`
5	`ActivityWriteTracker.cs` — `.Timeout(DrainTimeout).Catch(…)`	❌ 1 × `.Timeout(`

Counts are file-wide rather than at the originally cited line numbers, which have moved; they establish presence, not that every occurrence is a disposal wait.

Related work that landed in between: #2527 ("Waiting for disposal is a Subscribe, never a .ToTask") and #2531 ("Take Task out of the mesh core"). Between them they cleared site 1, which is why the issue's own 🚨 marker no longer points at live code — but the issue as written is not satisfied, so it stays open.

🚨 A note on how I checked, because the obvious method lies here. An empty grep is also what a missing file returns, and several files have moved repositories this week (the AI engine left core today). I verified CompilationCacheService.cs exists and has 1,411 lines before reading its empty match set as "clean" — without that step this comment would have reported a deleted file as a fixed one.

Waiting for disposal must be 100% reactive — five sites still poll, bridge to Task, or race a timeout (CompilationCacheService unloads anyway after 5s) #2488

Description

The rule

The reference implementation already exists in this repo

Sites that still violate it

1. 🚨 CompilationCacheService.cs:664-667 — the exact pattern MessageHubGrain removed

2. MeshTeardownExtensions.cs:73-76 — Task gate + timeout, then dispose regardless

3. MonolithMeshTestBase.cs:1534-1545 — bridges the observable to a Task, then polls it

4–5. Reactive core, timeout escape — weaker, listed for completeness

Why this matters beyond tidiness

Scope note

Activity

rbuergi commented on Aug 27, 2026

The same rule applied to READINESS finds the suspected Orleans deadlock — and the correct signal already exists

The grain already has the right shape

…and then converts it back into a Task gate

Two more Task-shaped readiness gates

Checked and clean

rbuergi commented on Aug 27, 2026

A related site, and it needs stating carefully — two documented rationales that do not quite meet

The precise question, which nobody has answered

rbuergi commented on Aug 27, 2026

Root cause of the HOST_CRASHED teardown failures: .ToTask() cannot observe a LATE fault — and the interface documents it as the sanctioned shape

The shape, as IMessageHub.cs:264-273 currently prescribes it

Why .ToTask() is not equivalent to Subscribe

The two waits in the crashing test

The fix, and the reason it needs a real signal

Scope

rbuergi commented on Aug 27, 2026

The rule is right; the REASON is deadlock, and that reorders the site list

Why it bites even where nobody wrote a block

The question to ask at each site, and the answers

Two swallows became data rather than silence

A site checked and ruled OUT, with evidence

The documentation that made this a rule nobody could follow

rbuergi commented on Aug 27, 2026

The readiness twin: IMessageHub.Started is the same defect at the other end of the lifecycle

Consumer list (complete, measured)

Why it is more than a signature change

Recommendation

rbuergi commented on Aug 27, 2026

Answering the scoped question for src/MeshWeaver.AI/ — PR #2537

A sibling of site 1, found in the same folder

Suggested addition to this issue's list

rbuergi commented on Aug 27, 2026

rbuergi commented on Aug 28, 2026

rbuergi commented on Aug 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. 🚨 `CompilationCacheService.cs:664-667` — the exact pattern `MessageHubGrain` removed

2. `MeshTeardownExtensions.cs:73-76` — Task gate + timeout, then dispose regardless

3. `MonolithMeshTestBase.cs:1534-1545` — bridges the observable to a Task, then polls it

Root cause of the `HOST_CRASHED` teardown failures: `.ToTask()` cannot observe a LATE fault — and the interface documents it as the sanctioned shape

The shape, as `IMessageHub.cs:264-273` currently prescribes it

Why `.ToTask()` is not equivalent to `Subscribe`

The readiness twin: `IMessageHub.Started` is the same defect at the other end of the lifecycle

Answering the scoped question for `src/MeshWeaver.AI/` — PR #2537