Repository navigation
MessageHubGrain treats expected host-disposal race as a fail-level hub-construction error #3243
Description
What is failing
When a MessageHubGrain activates while its host (or an ancestor scope) is already disposing, HostedHubsCollection.CreateHub resolves the hub's services from a disposed Autofac LifetimeScope. The resulting ObjectDisposedException is swallowed into a Failed to create hosted hub entry, and the grain then logs [ACTIVATE] Hub construction returned no hub at fail level — even though this is the designed freeze-during-disposal path, which the message text itself anticipates.
Probable cause — high confidence
A companion burst from the same instant on the same pod (MeshWeaver.Messaging.HostedHubsCollection: Failed to create hosted hub for address Admin/PlatformVersion, System.ObjectDisposedException at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()) proves the lifetime scope was already disposed — so this is the "host is disposing" branch, not a hub-configuration throw. The activation raced pod/silo teardown and lost. The sibling incident Admin/_LogIncident/8bd7c9c44c12e40b shows the same ODE-at-hub-creation signature recurring across four different portal deployments since mid-August, consistent with pod churn rather than a one-off fault. What remains a genuine (minor) defect: the grain does not detect host disposal and instead emits a fail-level error for an expected condition, which is what opened this incident.
Impact
Minimal functional impact — one occurrence, one pod, one instant, at teardown; when the host is not disposing, hub construction succeeds. The real cost is alerting noise: every shutdown race of this shape is fingerprinted and ticketed.
Where to look
MeshWeaver.Hosting.Orleans.MessageHubGrain— grain activation: check the host/ancestor disposal state before requesting hub construction (or whenCreateHubreturns no hub) and downgrade/skip the log instead of failing.HostedHubsCollection.CreateHub(src/MeshWeaver.Messaging.Hub/HostedHubsCollection.cs:349) andMessageHubConfiguration.Build(src/MeshWeaver.Messaging.Hub/MessageHubConfiguration.cs:463) — the resolution sites that throwObjectDisposedExceptiononce the scope is disposed.
Related (same event, sibling log site): Admin/_LogIncident/8bd7c9c44c12e40b — its triage also failed to produce a draft; both stem from the same hub-creation-during-disposal race.
Evidence
| Fingerprint | e2028eb86d6a85a6 |
| Category | MeshWeaver.Hosting.Orleans.MessageHubGrain |
| Severity | Error |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-7fd44b7695-xvwnf |
| Occurrences | 1 |
| First seen | 2026-09-03 17:21:23Z |
| Last seen | 2026-09-03 17:21:23Z |
Recent log lines
2026-09-03 17:21:23Z memex-portal-deployment-7fd44b7695-xvwnf fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
[ACTIVATE] Grain Admin/PlatformVersion: Hub construction returned no hub for Admin/PlatformVersion (NodeType: Markdown). Either the hub configuration threw — see the 'Failed to create hosted hub' entry logged immediately before this one, which carries the real exception — or hosted-hub creation is frozen because this host (or an ancestor) is disposing.
Opened automatically from Admin/_LogIncident/e2028eb86d6a85a6. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 4, 2026
Landed on
mainvia #3271 (97635cc90— "a hosted-hub creation that races pod teardown is not a fault"). Closing manually: the merge queue does not fireCloseslinks in this repo — four issues behaved this way today (#3228, #3156, #3216, #3238), each withclosingIssuesReferencesresolving correctly and the issue staying open.What landed matches what this issue actually asked for. The ask was not "log this at a lower level" — that would be dialling verbosity, which the house rules forbid. It was that the grain could not distinguish two genuinely different conditions and reported both as a fail-level hub-construction error:
Both
src/MeshWeaver.Hosting.Orleans/MessageHubGrain.csandsrc/MeshWeaver.Messaging.Hub/HostedHubsCollection.csnow carry that distinction, so the caller reads why construction returned no hub rather than re-deriving it from a swallowedObjectDisposedException.For the record on the original evidence: fingerprint
e2028eb86d6a85a6, one occurrence, one pod (memex-portal-deployment-7fd44b7695-xvwnf, nsmemex-cloud, 2026-09-03 17:21:23Z), with the companionFailed to create hosted hub for address Admin/PlatformVersion+Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()burst from the same instant proving the scope was already disposed. The sibling incidentAdmin/_LogIncident/8bd7c9c44c12e40brecorded the same signature recurring across four deployments since mid-August — consistent with pod churn, which is exactly why the alerting noise was worth removing rather than the race worth "fixing".The real cost was always alerting noise: every shutdown race of this shape was fingerprinted and ticketed. That stops now, without suppressing the case that matters.
Address the review on #4622: three real findings, two with their own …
systemorph-com commentedon Sep 18, 2026
Reopened — this fault is happening again.
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-7fd44b7695-xvwnf,memex-portal-deployment-bbf878dfb-ztq58Recent log lines
systemorph-com commentedon Sep 21, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-75b545bbd-nssvp,memex-portal-deployment-7fd44b7695-xvwnf,memex-portal-deployment-bbf878dfb-ztq58Recent log lines
Re-addressed by the current identity function: this incident inherited
e2028eb86d6a85a6(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 21, 2026
Still occurring: 1 more since the last update (4 total).
memex-portal-deployment-75b545bbd-nssvp,memex-portal-deployment-7fd44b7695-xvwnf,memex-portal-deployment-bbf878dfb-ztq58Recent log lines
Re-addressed by the current identity function: this incident inherited
e2028eb86d6a85a6(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 21, 2026
Cluster triage: this symptom has five distinct roots
Reviewed together with #1540 #1547 #1548 #3243 #3593 #2833 #2480 #5064 #4859 #5136 #4847 #4883 #4888 #4961 — every one of them reports
ObjectDisposedExceptionwithAutofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()on top, or a teardown that did not finish. They are not one bug. The grouping, the discriminators and the evidence are committed asDoc/Architecture/DisposedScopeAndDyingHubs(added in #5159).Already fixed on
main— awaiting a roll to verify in production.The classification fix this issue asked for is in
mainas commit8427013697("fix(orleans): the SECOND reporter of a teardown race is not a fault either (#3243)"), confirmed an
ancestor of
origin/main.src/MeshWeaver.Hosting.Orleans/MessageHubGrain.cs:HostedHubOutcomenow carries the condition out ofHostedHubsCollection, where it is known, andHubConstructionFailureReasonreports each of the three cases as what it is.Unclassified— anunknown, not a shutdown — deliberately stays at
Error.🚨 This is a legitimate classification fix rather than a swallow, and the reason is that the condition
is measured:
IsHostContainerDisposed()asks the container directly whether it can still resolve, so"the host is going down" is an observation. If the probe itself throws, the CLR treats the exception
filter as false and the loud branch runs — the conservative direction, by construction. A
classification that merely assumed the race was benign would be a swallow wearing a log level.
Not closing from here: the value of the fix is that a rollout stops fingerprinting this, and that is
only observable on a rolled portal. It should close on the first rollout after which this fingerprint
does not reappear.
Wider grouping:
Doc/Architecture/DisposedScopeAndDyingHubs.rbuergi commentedon Sep 23, 2026
Closing: fixed and deployed. The three reopens after the fix are a different defect, #1186.
The fix. The classification this issue asked for landed in two parts:
97635cc90): the missing-hub line.e58a86e14e, merged 2026-09-19 19:37Z): the activation-fault arm.MessageHubGrain.ActivationFaultLevellogs a teardown race at Debug and uses the sentence "activation of {path} was abandoned because the host (or its DI scope) is tearing down". Anything else stays at Error asactivation faulted for {path}.git merge-base --is-ancestor e58a86e14e <image>is true for three images:39f49494eb, which memex.meshweaver.cloud runs today (/api/version, read 2026-09-23 04:3xZ);9017d62c(ci.9056) and026442ff(ci.9076), the images rolled on 09-20/21.Why the reopens are not this defect. All three lines the bot folded here after the fix read
activation faulted for …, which is the non-teardown sentence. The classifier therefore judged each of them not a teardown race. They match #1186's own rows, and #1186's rows carry the exception:MessageHubGrain[0]bbf878dfb-ztq58Admin/_Notification/j7gX…Admin/_Notification/…→System.TimeoutException: No MeshNode emitted for … within 30s75b545bbd-nssvpStore/Catalog/_Activity/compile-stateStore/Plugin/_Activity/compile-state→ same TimeoutException75b545bbd-nssvpMyAi/_Activity/77db8c28Store/_Activity/e7558488,Publish/_Activity/…,Edu/_Activity/…→ same TimeoutExceptionThat is the node-resolution stall. #1186 and #5315 track it, and #5042 and #5048 carry its fixes. Logging it at Error is correct, because it is a real fault.
Recurrence since then. The bot has posted nothing on this issue after 2026-09-21 15:22Z. It is demonstrably alive: it posted "Still occurring" on other issues through 2026-09-23 02:30Z.
What I did NOT establish:
75b545bbd-nssvpran.If a teardown-race
ObjectDisposedExceptionis ever logged at Error again, the bot reopens this.