One stuck delivery no longer wedges a whole silo's routing

src/MeshWeaver.Documentation/Data/Architecture/ControlledIoPooling.md

@@ -176,8 +176,20 @@ Pools are keyed by resource class and resolved lazily from `IoPoolRegistry` (a m
 | `Http` | 16 | 2 |
 | `Compile` | `Environment.ProcessorCount` | 3 |
 | `Process` | 4 | 3 |
+| `Routing` | 256 | — |
 | `pg:{adapter}` (per Postgres adapter) | **1** (mirrors the adapter's single Npgsql connection) | — |
 
+**`Routing` is an ISOLATION boundary, not a throttle.** `RoutingGrain` is `[StatelessWorker(1)]` and
+non-reentrant, so a silo has exactly ONE routing turn — and Orleans' request timeout applies to
+callers *waiting on* a grain, never to the turn itself. Anything the turn does inline is therefore
+unbounded by construction and blocks every other message the silo needs to route. Prod (atioz,
+2026-08-07) had one `RouteMessage` turn executing for `06:00:22` with `NonReentrancyQueueSize=541`;
+Orleans' own diagnostics showed the work item still `Running` with `Total processed` frozen — i.e.
+`RouteMessage` had never returned, it was blocked in its own synchronous body. The cure is structural
+(you cannot time out a synchronously blocked thread): `RouteMessage` captures its activation-bound
+handles and hands the composed route to this pool via `SubscribeThroughPool`, so a leg that never
+terminates costs one slot and nothing else. See issue #1028.
+
 ---
 
 ## Hidden inside the interfaces

src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-09-one-stuck-message-no-longer-stops-the-portal.md

@@ -0,0 +1,25 @@
+---
+Name: One stuck message no longer stops the whole portal from talking to itself
+Category: Fix
+Description: A single delivery that never finished could hold the portal's message router and silently queue up everything else behind it.
+Icon: Sparkle
+Order: -20260809
+---
+
+# One stuck message no longer stops the whole portal from talking to itself
+
+Everything in a portal — opening a page, saving an edit, running a chat round — is a message handed
+to one router that decides where it goes. That router handled one message at a time, and it did the
+delivery work itself. So a single delivery that got stuck was enough to hold the router, and every
+other message in the portal simply waited. Nothing reported it: the portal answered pages, looked
+healthy, and quietly stopped doing anything that needed a message delivered.
+
+In production this happened once and lasted 37 hours. A single delivery sat unfinished for six
+hours with more than five hundred messages queued behind it, and it only came to light later as
+"this portal is running an old version" — the update it needed was one of the messages in that
+queue. Restarting the portal cleared it, which is part of why it stayed hidden for so long.
+
+The router now hands each delivery off and moves straight on to the next one, so a delivery that
+gets stuck costs only itself. A delivery that cannot be completed now says so — the sender is told
+it failed instead of waiting forever — and a portal whose deliveries stop finishing reports it
+loudly in the log rather than leaving it to be discovered days later.

src/MeshWeaver.Hosting/AssemblyInfo.cs

@@ -9,3 +9,6 @@
 [assembly: InternalsVisibleTo("MeshWeaver.Hosting.Monolith.Test")]
 [assembly: InternalsVisibleTo("MeshWeaver.Hosting.PostgreSql.Test")]
 [assembly: InternalsVisibleTo("MeshWeaver.Hosting.Sqlite.Test")]
+// RoutingGrainTurnIsolationTest (issue #1028) decorates the real PathResolutionService with one
+// whose Subscribe BLOCKS, to prove a stalled route leg can no longer wedge the silo's routing.
+[assembly: InternalsVisibleTo("MeshWeaver.Hosting.Orleans.Test")]

src/MeshWeaver.Mesh.Contract/Threading/IoPoolOptions.cs

@@ -59,6 +59,26 @@ public static class IoPoolNames
     /// </summary>
     public const string AgentStore = "AgentStore";
 
+    /// <summary>
+    /// Silo-side message routing (<c>RoutingGrain.RouteMessage</c>), via
+    /// <see cref="IIoPool.SubscribeThroughPool{T}"/>. Exists so the routing SUBSCRIBE — path
+    /// resolution, the memory-stream post, the per-node grain hand-off and the DeliveryFailure
+    /// NACK — runs OFF the routing grain's activation thread.
+    ///
+    /// <para>🚨 This is the fix for issue #1028. <c>RoutingGrain</c> is <c>[StatelessWorker(1)]</c>
+    /// and NON-reentrant, so the silo has exactly ONE routing turn: any work the turn performs
+    /// inline is work the whole silo's routing waits on, and Orleans' request timeout does NOT
+    /// apply inside a turn. Prod (atioz, 2026-08-07) had a single <c>RouteMessage</c> turn
+    /// executing for <c>06:00:22</c> with <c>NonReentrancyQueueSize=541</c> — every other message
+    /// the silo needed to route was stuck behind it for 37 h. Routing work therefore never runs on
+    /// the turn; it runs here, where one stuck delivery costs one pool slot and nothing else.</para>
+    ///
+    /// <para>Generous cap: a drain hook + isolation boundary, not a throttle — the slot is held
+    /// only for the bounded subscribe window (see <see cref="IIoPool.SubscribeThroughPool{T}"/>),
+    /// and routing is the hottest path in the mesh.</para>
+    /// </summary>
+    public const string Routing = "Routing";
+
     /// <summary>CPU-bound compilation (Roslyn compile/script). Wave 3.</summary>
     public const string Compile = "Compile";
 
@@ -161,6 +181,15 @@ public sealed record IoPoolOptions
     /// </summary>
     public int AgentStore { get; init; } = 128;
 
+    /// <summary>
+    /// Concurrent silo-side routing subscribes (the <c>Routing</c> pool). A drain hook and an
+    /// isolation boundary, not a throttle — the slot is held only for the bounded subscribe
+    /// window — so the cap is generous (256): routing is the hottest path in the mesh and must
+    /// never queue behind itself. See <see cref="IoPoolNames.Routing"/> for why routing work is
+    /// not allowed on the routing grain's turn at all (issue #1028).
+    /// </summary>
+    public int Routing { get; init; } = 256;
+
     /// <summary>Concurrent compilations. CPU-bound; defaults to the processor count.</summary>
     public int Compile { get; init; } = Environment.ProcessorCount;
 
@@ -210,6 +239,7 @@ public int MaxConcurrencyFor(string name) =>
             IoPoolNames.Query => Query,
             IoPoolNames.Layout => Layout,
             IoPoolNames.AgentStore => AgentStore,
+            IoPoolNames.Routing => Routing,
             IoPoolNames.Compile => Compile,
             IoPoolNames.Process => Process,
             _ => Default,

test/MeshWeaver.Hosting.Orleans.Test/MeshWeaver.Hosting.Orleans.Test.csproj

@@ -37,6 +37,9 @@
     <ItemGroup>
       <PackageReference Include="Microsoft.Orleans.TestingHost" />
       <PackageReference Include="Microsoft.Orleans.Sdk" />
+      <!-- TestScheduler: RoutingGrainStreamPostBoundTest drives the stream-post timeout
+           deterministically (issue #1028) instead of waiting on the wall clock. -->
+      <PackageReference Include="Microsoft.Reactive.Testing" />
     </ItemGroup>
 
 

test/MeshWeaver.Hosting.Orleans.Test/RoutingGrainStreamPostBoundTest.cs

@@ -0,0 +1,144 @@
+using System;
+using System.Collections.Generic;
+using System.Reactive;
+using System.Reactive.Linq;
+using System.Reactive.Threading.Tasks;
+using System.Threading;
+using System.Threading.Tasks;
+using MeshWeaver.Messaging;
+using Microsoft.Extensions.Logging.Abstractions;
+using Microsoft.Reactive.Testing;
+using Xunit;
+
+namespace MeshWeaver.Hosting.Orleans.Test;
+
+/// <summary>
+/// Pins the terminal contract of the memory-stream leg of a route (issue #1028).
+///
+/// <para><b>Why this leg needs a terminal bound.</b> A dropped memory-stream post has NO downstream
+/// response / <see cref="DeliveryFailure"/> path of its own — the stream-routed hub simply never
+/// sees the message. Before #1028 the post was AWAITED inside the routing grain's turn, so a post
+/// that never completed wedged the whole silo's routing AND told nobody. Now the post runs off the
+/// turn, which fixes the wedge; this test pins the other half: it must always TERMINATE (so the
+/// pool slot and the in-flight count are released) and the sender must always be told (so its
+/// <c>Observe</c> fires OnError instead of parking forever).</para>
+///
+/// <para>Deterministic — <see cref="TestScheduler"/> for the timeout, no cluster, no wall clock.</para>
+/// </summary>
+public class RoutingGrainStreamPostBoundTest
+{
+    private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(60);
+    private static readonly Address Sender = new("client", "sender-1");
+
+    [Fact]
+    public void PostThatNeverCompletes_TerminatesOnTimeout_AndNacksSender()
+    {
+        // The prod shape: the post is issued and its Task simply never completes.
+        var never = new TaskCompletionSource();
+        var nacks = new List<(string Message, ErrorType Type)>();
+        var scheduler = new TestScheduler();
+        var terminated = false;
+
+        RoutingGrain.PostToStream(
+                post: () => never.Task,
+                addressPath: "portal/user-1",
+                deliveryId: "d1",
+                sender: Sender,
+                postFailureToSender: (m, t) => nacks.Add((m, t)),
+                logger: NullLogger.Instance,
+                timeout: Timeout,
+                scheduler: scheduler)
+            .Subscribe(_ => { }, () => terminated = true);
+
+        // Before the bound elapses: still in flight, nobody has been told anything. Correct —
+        // a slow post is not a failed post.
+        scheduler.AdvanceBy(Timeout.Ticks - 1);
+        nacks.Should().BeEmpty("a post that has not yet exceeded its bound is not a failure");
+        terminated.Should().BeFalse();
+
+        scheduler.AdvanceBy(2);
+
+        // 🚨 The two things that must ALWAYS happen, and did not before #1028:
+        terminated.Should().BeTrue(
+            "the leg must complete so the routing pool slot and the in-flight route count are "
+            + "released — a never-terminating leg is how a bounded pool silently fills up");
+        var nack = nacks.Should().ContainSingle(
+            "a delivery that neither lands nor reports is exactly the silent hang #1028 was made of "
+            + "— the sender must get a DeliveryFailure so its Observe fires OnError").Subject;
+        nack.Type.Should().Be(ErrorType.Failed);
+        nack.Message.Should().Contain("portal/user-1");
+    }
+
+    [Fact]
+    public async Task PostThatFaults_NacksSender_AndStillCompletes()
+    {
+        var nacks = new List<(string Message, ErrorType Type)>();
+
+        await RoutingGrain.PostToStream(
+                post: () => Task.FromException(new InvalidOperationException("stream provider down")),
+                addressPath: "portal/user-2",
+                deliveryId: "d2",
+                sender: Sender,
+                postFailureToSender: (m, t) => nacks.Add((m, t)),
+                logger: NullLogger.Instance,
+                timeout: Timeout)
+            .ToTask();
+
+        var nack = nacks.Should().ContainSingle().Subject;
+        nack.Type.Should().Be(ErrorType.Failed);
+        nack.Message.Should().Contain("stream provider down");
+    }
+
+    [Fact]
+    public async Task PostThatSucceeds_CompletesWithoutNack()
+    {
+        var nacks = new List<(string Message, ErrorType Type)>();
+
+        var result = await RoutingGrain.PostToStream(
+                post: () => Task.CompletedTask,
+                addressPath: "portal/user-3",
+                deliveryId: "d3",
+                sender: Sender,
+                postFailureToSender: (m, t) => nacks.Add((m, t)),
+                logger: NullLogger.Instance,
+                timeout: Timeout)
+            .ToTask();
+
+        result.Should().Be(Unit.Default);
+        nacks.Should().BeEmpty("a delivered post must never NACK the sender");
+    }
+
+    /// <summary>
+    /// NEGATIVE CONTROL for the structural half of the fix. This is the SHAPE the routing grain's
+    /// turn used to have — <c>return s.OnNextAsync(delivery).ContinueWith(...)</c> — and it shows
+    /// why nothing bounded it: the turn's completion IS the post's completion, so a post that never
+    /// completes is a turn that never returns. <c>RoutingGrain</c> is <c>[StatelessWorker(1)]</c>
+    /// and non-reentrant, so that one turn is the silo's ONLY routing turn; Orleans' request
+    /// timeout applies to messages WAITING on a grain, never to the turn itself. Prod:
+    /// <c>06:00:22</c> executing, <c>NonReentrancyQueueSize=541</c>.
+    ///
+    /// <para>The cure is that <c>RouteMessage</c> no longer returns anything derived from the
+    /// delivery work — see <see cref="RoutingGrainTurnIsolationTest"/>, which proves it against a
+    /// real cluster.</para>
+    /// </summary>
+    [Fact]
+    public async Task NegativeControl_AwaitingThePostInsideTheTurn_NeverReturns()
+    {
+        var never = new TaskCompletionSource();
+
+        // The pre-#1028 turn body, verbatim in shape.
+        Task<IMessageDelivery> LegacyTurn() =>
+            never.Task.ContinueWith(_ => (IMessageDelivery)new MessageDelivery<string>(), TaskScheduler.Default);
+
+        var turn = LegacyTurn();
+
+        // A "wait to confirm nothing happened" negative test — there is no positive signal to
+        // filter for, because the whole point is that the turn produces none.
+        var settled = await Task.WhenAny(turn, Task.Delay(TimeSpan.FromMilliseconds(250)));
+
+        settled.Should().NotBe(turn,
+            "this is the defect: the turn's completion is the post's completion, so a post that "
+            + "never completes holds the silo's single non-reentrant routing turn forever, with no "
+            + "timeout anywhere on that path (issue #1028)");
+    }
+}