Grain activation no longer times out waiting on itself by rbuergi · Pull Request #1062 · Systemorph/MeshWeaver

rbuergi · 2026-08-10T05:15:37Z

The defect

On memex.systemorph.com (AKS ns memex) Orleans grain activations faulted and retried forever:

fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
      [ACTIVATE] Grain Edu: activation faulted for Edu
System.TimeoutException: No response received in hub cache/tYwK_RNWZUSCQGvEPNvvzg within 00:01:00
for request SubscribeRequest (id=...) → target Edu.

The error names the activating grain as an unreachable target — because the request was the grain's own.

Root cause

MessageHubGrain.OnActivateAsync merged streamCache.GetStream(addressPath) — the grain's OWN
address — into the activation source. The cache opens that upstream with
GetMeshNodeStreamBypassCache(path), i.e. a SubscribeRequest addressed at path, which the
RoutingGrain delivers straight back to this grain, where DeliverMessage parks it on
_hubReadyRaw until CompleteActivation runs.

So on a cold entry the branch is a strict self-loop: it cannot answer until the very activation
it feeds has finished. Its only possible terminal is "the grain did not answer itself" — no
information about the node. Observable.Merge forwarded that terminal anyway, so the cache hub's
60 s RequestTimeout became a hard ceiling on TOTAL activation.

That ceiling pre-empted both deliberate slow-path designs downstream, 100% of the time:

NodeTypeEnrichmentHelpers.WaitForCompileSettled disarms its wall clock while a compile is in
flight — a cold pod compiling many dynamic types is meant to run past the budget.
BuildEnrichmentChain catches a stuck type into a visible compilation-error overlay so the hub
activates anyway.

Neither could be reached. Orleans retried; the cache's transient-streak breaker then replayed the
cached TimeoutException into each fresh activation, so the retries faulted in milliseconds
(measured burst at 03:57Z: +0 s/+1 s/+2 s/+3 s). The Edu partition was unreadable to users and to
MCP with the identical exception.

Why only this portal (same image on all three)

Measured over 30 h of Loki (ns memex): ~36 activation faults an hour, and every faulting grain
was a nodeType:Store/Plugin node — Edu (231), Feedback, Skill, Essentials, Publish,
Collaboration, DataModelling, Agent, Instances. Exactly the nodeType:Store/Plugin set.

memex.systemorph.com: Store/Plugin is installed as a dynamic in-mesh NodeType sitting at
CompilationStatus = Error — CS0117: 'AiSettingsNodeType' does not contain a definition for 'AddSkillSource' (that method was deleted in AiSettingsNodeType.AddSkillSource is fire-and-forget — a fast uninstall can be undone by a late install write #683; the in-mesh plugin source still calls it, and
no build or test can see it). That is precisely the case where enrichment legitimately runs long.
memex.meshweaver.cloud (control): Store/Plugin does not exist at all, so enrichment's 3 s
existence probe answers immediately and Take(1) disposes the self-loop long before 60 s.

Data-dependent, not version-dependent — consistent with all three portals running
3.0.0-rc1.ci.2509. The Monolith routing path has no equivalent self-loop, which is why no existing
test caught it.

The fix

The cache branch may still supply a VALUE (a warm entry lets a reactivation skip the storage
read) but never a TERMINAL. ComposeActivationSource collapses a faulted self-referential branch
to Empty and logs the exception in full — it is not swallowed.

Activation keeps a complete set of terminal outcomes without it: the path resolver's node drives
enrichment; nothing within FirstNodeResolutionTimeout throws the precise "no MeshNode emitted / no
query provider claims its partition" error; both branches finishing with no node completes the
source (Empty, not Never, is what keeps the missing-node answer prompt); enrichment faults
surface as the overlay.

ComposeActivationSource / BuildActivationChain are extracted as internal static so the whole
chain runs in virtual time — same seam as WaitForCompileSettled, same precedent as
LongRunningOperationCapExceeded.

Verification

MessageHubGrainActivationSourceTest — 7 tests, virtual time, 40 ms. Reverting only the composition
to plain Observable.Merge fails 3 of them with the prod signature verbatim:

Activation faulted, but no branch of it had anything to fault ON.
TimeoutException: No response received in hub cache/tYwK_RNWZUSCQGvEPNvvzg within 00:01:00
for request SubscribeRequest (id=...) → target Edu.

dotnet build -c Release -warnaserror — MeshWeaver.Hosting.Orleans + its test project: 0 warnings, 0 errors.
20 grain tests green under Release/-warnaserror, including the 9 Orleans cluster tests that
guard the semantics touched here — GrainActivationCompletesFastTest.NonExistentPath_ActivationFailsFast_NotThirtySecondTimeout,
OrleansBrokenNodeTypeAccessTest (×2), OrleansIdleReactivationNoWedgeTest (×2),
OrleansUnresolvableNodeNoWedgeTest (×3), PartitionRootActivationTest.
WhatsNewEntryIntegrityTest + DocumentationLinkIntegrityTest green.

Follow-up (separate, not in this PR)

The aggravator is a data defect in the plugins repo: Store/Plugin's in-mesh source calls
AiSettingsNodeType.AddSkillSource, deleted from the framework in #683. It needs porting to
AiSourcesInstallHook. This PR makes the platform survive it (the type shows its compilation-error
overlay instead of wedging every plugin partition), but the type still needs to compile.

🤖 Generated with Claude Code

MessageHubGrain.OnActivateAsync merged `streamCache.GetStream(addressPath)` — the grain's OWN address — into the activation source. The cache opens that upstream with GetMeshNodeStreamBypassCache(path), i.e. a SubscribeRequest addressed at `path`, which the RoutingGrain delivers straight back to THIS grain, where DeliverMessage parks it on _hubReadyRaw until CompleteActivation runs. On a cold entry the branch is a strict self-loop: it cannot answer until the very activation it feeds has finished, so its only possible terminal is "the grain did not answer itself". Observable.Merge forwarded that terminal, so the cache hub's 60s RequestTimeout became a hard ceiling on TOTAL activation and surfaced as [ACTIVATE] Grain Edu: activation faulted for Edu TimeoutException: No response received in hub cache/<id> within 00:01:00 for request SubscribeRequest (id=...) -> target Edu. naming the activating grain as an unreachable target. The ceiling pre-empted both deliberate slow-path designs downstream: WaitForCompileSettled DISARMS its wall clock while a compile is in flight, and BuildEnrichmentChain catches a stuck type into a visible compilation-error overlay so the hub activates anyway. Neither could be reached. Orleans retried, the cache's transient-streak breaker replayed the cached timeout into each fresh activation, and the retries faulted in milliseconds. Measured on memex.systemorph.com (ns memex) 2026-08-09/10: ~36 activation faults an hour, and every faulting grain was a nodeType:Store/Plugin node (Edu, Agent, Collaboration, DataModelling, Essentials, Feedback, Instances, Publish, Skill). Same image on all three portals; only this one has Store/Plugin installed as a dynamic NodeType sitting at CompilationStatus=Error (CS0117 AiSettingsNodeType.AddSkillSource, deleted in #683), which is exactly the case where enrichment legitimately runs long. Data-dependent, not version- dependent. The Edu partition was unreadable to users and to MCP with the same exception. The cache branch may still supply a VALUE (a warm entry lets a reactivation skip the storage read) but no longer a TERMINAL: it is collapsed to Empty and the exception is logged in full. Activation keeps a complete set of terminal outcomes without it — the path resolver's node, the 30s first-emission TimeoutException, the "no usable node" completion (prompt, which is why Empty and not Never), and enrichment's overlay. ComposeActivationSource / BuildActivationChain are extracted as internal statics so MessageHubGrainActivationSourceTest drives the whole chain in virtual time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Fixes an Orleans grain activation deadlock/time-budget failure where a self-referential mesh-node-cache subscribe could time out during activation and incorrectly fault the activation (causing infinite retries and unrecoverable plugin partitions).

Changes:

Neutralize terminal faults from the self-referential “own address” cache branch during MessageHubGrain activation while still allowing it to contribute a warm value.
Extract activation-source composition and activation-chain building into testable internal static helpers, including virtual-time control of the first-node budget.
Add a focused virtual-time test suite to pin the regression and document the production signature; add a WhatsNew entry describing the user-visible impact/fix.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
test/MeshWeaver.Hosting.Orleans.Test/MessageHubGrainActivationSourceTest.cs	Adds virtual-time tests that reproduce and guard the activation self-loop timeout behavior.
src/MeshWeaver.Hosting.Orleans/MessageHubGrain.cs	Changes activation source composition to ignore self-cache terminal faults and factors activation chain into reusable helpers.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-10-plugin-pages-stop-timing-out.md	Adds a user-facing “What’s New” entry for the fix.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    /// <para><b>The branches.</b> <paramref name="pathResolverStream"/> is authoritative: it reads
+    /// the node straight out of storage, in-process, with no hub round-trip.
+    /// <paramref name="ownAddressCacheStream"/> is an ACCELERATOR: when the process-wide
+    /// <c>IMeshNodeStreamCache</c> already holds a hydrated entry for this path (a reader kept it
+    /// warm across an idle-collection), it replays the node instantly and activation skips the
+    /// storage read.</para>


+    /// compilation-error OVERLAY so the hub activates anyway. Neither could ever be reached: the
+    /// self-loop faulted the activation first, 100% of the time. Orleans then retried, the cache's
+    /// transient-streak breaker replayed the cached timeout into each fresh activation, and the
+    /// retries faulted in milliseconds — the node stayed unreadable to users and to MCP.</para></para>


github-actions · 2026-08-10T05:43:15Z