One non-reentrant RouteMessage can wedge a whole silo's routing — no timeout on that path #1028

Found while root-causing #1020 (client-i self-update stalled ~37 h). Not currently live — a pod restart cleared it — but it is unbounded by construction and will recur.

What happened

client-i's Admin/UpdatePolicy hub was unreachable for 37 hours. The cause was not the node or the mesh: the silo's Orleans routing was wedged.

RoutingGrain is [StatelessWorker(1)] and non-reentrant. One RouteMessage sat executing for 06:00:22 with NonReentrancyQueueSize=541. A single RouteMessage whose memory-stream OnNextAsync never completes therefore blocks every other message the silo needs to route, and there is no timeout on that path.

Scope of the evidence: those lines came from exactly one pod (~8.4 M on 2026-08-08, ~6.0 M on 2026-08-09). The current pod has emitted zero in five hours, so the restart cleared it.

Why this is worth fixing rather than watching

• It is silent. The symptom surfaced 37 hours later as "the deployment is stale", not as a routing fault. Nothing reported that a route had been executing for six hours.
• It is unbounded. [StatelessWorker(1)] + non-reentrant + no timeout means a single stuck continuation is sufficient. There is no back-pressure signal short of the queue growing to 541.
• A restart masks it, which is what made client-i self-update stalled ~37h — a rollout restart masks it, so it will drift again #1020 look like a self-update defect for 37 hours.

Not a watchdog

Per AGENTS.md, the fix is not a timer that restarts a wedged grain. The questions are: why can OnNextAsync on a memory stream never complete, and should that path carry a bound at all. A timeout that abandons the turn may be legitimate here — Orleans's own request timeout does not apply inside a grain turn — but that is a design decision, not a mitigation to bolt on.

Mitigating factor already landed

PR #1025 (fixes #1020) decouples the roll-forward from the Admin/UpdatePolicy status write. A useful consequence: when this recurs, the install still rolls forward on the next tick, and rolling forward restarts the pod, so the wedge now self-clears. That reduces the blast radius; it does not fix this defect.

Where to look

• RoutingGrain — the [StatelessWorker(1)] declaration and the RouteMessage turn.
• The memory-stream OnNextAsync path it awaits.
• Whether NonReentrancyQueueSize growth is observable anywhere today (it should be, and loudly).