One stuck delivery no longer wedges a whole silo's routing

[rbuergi]

Fixes #1028.

What the evidence actually says

I pulled the original wedge out of Loki (client-i, pod memex-portal-deployment-76855f7cdd-gm69g, 2026-08-07 15:14 UTC onward). Three facts fall out, and the third contradicts the issue's premise:

1. The wedge starts on a fresh activation, ~35 s into pod startup, during the startup burst (ShippedReleaseSeed firing ~20 AgenticPension/* release triggers, DynamicTypePreWarmer building 4 NodeTypes, StaticRepoImport syncing 4 sources) — all funnelled through the one routing worker. IdlenessTimeSpan=00:00:34, Total Enqueued=3; Total processed=2.

2. RouteMessage never RETURNED — it was blocked in its own synchronous body. Orleans' diagnostic reports Executing Task Id=1 Status=Running for 00:00:29.463 with QueuedWorkItems=0 and Total processed frozen at 2. Orleans' request work item completes as soon as the grain method returns its Task; a turn awaiting something would have shown processed=3 and the activation merely waiting. It didn't. The turn was synchronously stuck.

3. It was not an awaited grain call. Grain calls are bounded by Orleans' 30 s ResponseTimeout, and those timeouts absolutely do get logged here — over the three days I queried there are 2068 IRoutingGrain.RouteMessage and 179 IMessageHubGrain.DeliverMessage "Response did not arrive on time" lines, and zero for IMemoryStreamQueueGrain.Enqueue.

That third point answers question 1 in the issue, in the negative. IAsyncStream.OnNextAsync on an Orleans memory stream has exactly one await — queueGrain.Enqueue(...) (MemoryAdapterFactory.QueueMessageBatchAsync, verified against Orleans v10.2.2 source) — and that award is bounded and would have been logged. So "a memory-stream OnNextAsync that never completes" is not what happened. What actually has no bound is the synchronous work RouteMessage performed on the turn, on both branches:

• stream branch: GetStream(...) plus OnNextAsync's synchronous prologue (Orleans serialization of the delivery) — and then it awaited the post, so the turn's completion was the post's completion;
• grain branch: the entire synchronous prologue of pathResolver.ResolvePath(...).Subscribe(...) — a cache hit emits synchronously, so the whole handler (including the grain-call marshalling and the failure post) ran inline on the turn.

I could not identify the exact blocking primitive from the surviving evidence — the pod is gone and there is no dump — and I am not going to guess one. What is determinable is stronger and sufficient: the turn executes an unbounded amount of foreign code, and RoutingGrain is [StatelessWorker(1)] + non-reentrant, so that is one shared, unbounded, silo-wide critical section by construction.

The fix (question 2)

A timeout on the turn is the wrong bound and could not have worked anyway: you cannot abandon a synchronously blocked thread — a .Timeout(...) wrapped around it never gets a chance to fire, because the calling thread never returns to observe it. The cure has to be structural.

RouteMessage now captures its two activation-bound handles (GetStreamProvider, GrainFactory — both O(1) lookups) and returns. The composed route — path resolution, the memory-stream post, the per-node grain hand-off, the DeliveryFailure NACK — is subscribed on a new mesh-scoped Routing IIoPool via SubscribeThroughPool, which runs the subscribe (the synchronous prologue that wedged prod) on a ThreadPool thread, gated and drainable. A leg that never terminates now costs one pool slot and nothing else.

No caller-visible contract moves. RouteMessage already returned Forwarded unconditionally — the old stream branch's ContinueWith returned Forwarded whether the post succeeded or faulted. A delivery's real outcome has always come back through the response / DeliveryFailure path on the sender's hub, and still does.

Bounds that are about reporting, not suppression. Each leg terminates and tells the sender:

• the stream post carries a 60 s .Timeout — it can only fire after Orleans' own 30 s bound on its single grain call has already been exceeded, i.e. when the post is provably never going to land. It converts a silent never-completing leg into a LogError + a DeliveryFailure at the sender, and releases the pool slot. It is not a retry and not a queue bound.
• a synchronous throw during route composition (the classic one: GetStream NRE'ing out of PersistentStreamProvider.IsRewindable while the stream provider is still starting — that NRE is in these very logs) now NACKs the sender explicitly. Pre-fix it faulted RouteMessage itself and reached the caller's error path; moving off the turn would otherwise have turned it into a log line and a parked sender. That gap is closed here.

Detection. The grain counts in-flight route dispatches and reports at LogCritical when 64 are outstanding and not terminating, with a recovery line when it drains. Event-driven at the dispatch site — no timer, no watchdog, no poller. This replaces the signal that was previously invisible: NonReentrancyQueueSize growing to 541 inside an Orleans warning that nobody was watching.

Tests, and the negative control run explicitly

RoutingGrainTurnIsolationTest (real OrleansTestBase cluster) decorates the silo's real IPathResolver with one whose Subscribe blocks the calling thread for a magic partition — the prod failure mode, reproduced deterministically. It fires one delivery into that partition, waits (reading the silo-side decorator directly, so there is no ordering race and no sleep) until the routing grain is provably inside the stall, then pings an ordinary partition root and asserts the stall is still running when the ping is answered.

I verified the negative control by reverting RoutingGrain.cs to the pre-fix version and re-running:

	result
pre-fix RoutingGrain	FAIL — TaskCanceledException at the 10 s probe budget; the probe cannot be served at all
post-fix	PASS — probe answered in 203 ms, with 1 route leg still stalled

RoutingGrainStreamPostBoundTest pins the terminal contract on a TestScheduler: a post that never completes still terminates and NACKs the sender; a faulting post NACKs; a delivered post does not. It also carries an explicit NegativeControl_AwaitingThePostInsideTheTurn_NeverReturns documenting why the old shape had no bound.

Local: 29/29 routing-focused Orleans tests green, DocumentationLinkIntegrityTest green, dotnet build -c Release -p:CIRun=true -warnaserror clean.

Trade-off worth a reviewer's eye

Stream posts used to be serialized by the single in-turn await, so posts to a given target were de-facto FIFO. They are now concurrent. I judged that acceptable because Orleans gives no ordering guarantee for the grain calls into the routing grain in the first place (so nothing may depend on it end-to-end), and because the cross-hub write contract is explicitly eventual-consistency-safe — RFC 7396 merge patches applied against the owner's current state, documented in AGENTS.md as safe against concurrent writers. If you would rather keep strict FIFO, the alternative is a Subject + Concat serial queue off the turn; say the word and I will add it.

Related, deliberately out of scope

MessageHubGrain.DeliverMessage has the same in-turn shape (HubReady.Take(1).Subscribe(...) running inline), but it is a per-node grain, so a wedge there is bounded to that one node's hub rather than the whole silo — consistent with the 179 DeliverMessage timeouts in the logs being spread across nodes. Worth a separate look, not a change here.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR addresses issue #1028 by ensuring a single stuck/blocked RoutingGrain.RouteMessage execution can no longer wedge silo-wide message routing. It does this by moving the potentially-blocking routing work (path resolution, stream post, grain hand-off, and NACK surfacing) off the non-reentrant routing-grain turn and onto a dedicated mesh-scoped Routing IIoPool, while adding bounded failure surfacing and back-pressure visibility.

Changes:

• Refactors RoutingGrain.RouteMessage to do O(1) work on-turn and dispatch the composed route via the new IoPoolNames.Routing pool, including saturation logging and bounded stream-post failure surfacing.
• Adds regression tests that (1) reproduce a synchronous “blocked subscribe” wedge against a real Orleans cluster and (2) pin the stream-post leg’s terminal timeout/NACK contract deterministically via TestScheduler.
• Documents the new Routing pool and the user-visible “What’s New” fix entry.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/MeshWeaver.Hosting.Orleans.Test/RoutingGrainTurnIsolationTest.cs	New Orleans-cluster regression test proving a synchronously-blocked route leg no longer blocks unrelated routing.
test/MeshWeaver.Hosting.Orleans.Test/RoutingGrainStreamPostBoundTest.cs	New deterministic unit test for stream-post timeout/fault behavior and sender NACK surfacing.
test/MeshWeaver.Hosting.Orleans.Test/MeshWeaver.Hosting.Orleans.Test.csproj	Adds Microsoft.Reactive.Testing to support deterministic scheduler-based timeout testing.
src/MeshWeaver.Mesh.Contract/Threading/IoPoolOptions.cs	Introduces IoPoolNames.Routing and an IoPoolOptions.Routing cap (default 256), wired into MaxConcurrencyFor.
src/MeshWeaver.Hosting/AssemblyInfo.cs	Grants InternalsVisibleTo for MeshWeaver.Hosting.Orleans.Test to support the new routing regression test wiring.
src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs	Core fix: dispatch routing work via routingPool.SubscribeThroughPool, add in-flight/back-pressure logging, and implement bounded PostToStream behavior.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-09-one-stuck-message-no-longer-stops-the-portal.md	Adds a “What’s New” entry describing the user-visible impact of the routing isolation fix.
src/MeshWeaver.Documentation/Data/Architecture/ControlledIoPooling.md	Documents the new Routing pool and its purpose as an isolation boundary for routing subscribes.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results (shard 5)

1 725 tests   1 533 ✅  4m 57s ⏱️
   11 suites    192 💤
   11 files        0 ❌

Results for commit dcd97b1.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 0)

690 tests   690 ✅  12m 30s ⏱️
 10 suites    0 💤
 10 files      0 ❌

Results for commit dcd97b1.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 4)

1 284 tests   1 279 ✅  7m 1s ⏱️
   11 suites      5 💤
   11 files        0 ❌

Results for commit dcd97b1.

[github-actions]

Test Results (shard 1)

1 199 tests   1 199 ✅  5m 52s ⏱️
   11 suites      0 💤
   11 files        0 ❌

Results for commit dcd97b1.

[github-actions]

Test Results (shard 3)

   11 files     11 suites   5m 47s ⏱️
1 976 tests 1 868 ✅ 108 💤 0 ❌
2 117 runs  2 009 ✅ 108 💤 0 ❌

Results for commit dcd97b1.

[github-actions]

Test Results (shard 2)

1 912 tests   1 910 ✅  6m 28s ⏱️
   11 suites      2 💤
   11 files        0 ❌

Results for commit dcd97b1.

[github-actions]

Test Results

   65 files     65 suites   42m 38s ⏱️
8 786 tests 8 479 ✅ 307 💤 0 ❌
8 927 runs  8 620 ✅ 307 💤 0 ❌

Results for commit dcd97b1.