Repository navigation
fix: a hosted-hub creation that races pod teardown is not a fault (#3243) - #3271
Merged
meshweaver-cloud[bot] merged 1 commit intoSep 4, 2026
Conversation
) `MessageHubGrain` logged `[ACTIVATE] … Hub construction returned no hub` at fail level for BOTH of the conditions that produce a null hub, and the message text itself listed the two and committed to neither: Either the hub configuration threw — see the 'Failed to create hosted hub' entry logged immediately before this one … — or hosted-hub creation is frozen because this host (or an ancestor) is disposing. That vagueness was honest: `GetHostedHub` answers `null` for both, and a null carries no reason, so the grain genuinely could not tell them apart. But the LEVEL is a ticketing decision, not a verbosity knob — every red line a portal writes becomes an incident and a GitHub issue (Doc/Architecture/LogWatchTriage). So the shutdown case, which happens on every pod rollout, opened a ticket for a race in which nothing failed and nothing was written: incident e2028eb86d6a85a6 and its sibling 8bd7c9c44c12e40b, recurring across four deployments since mid-August. This is a real cost/value distinction between two conditions, not verbosity tuning. Both remain fully reported; only the one that is NOT a fault stops being red. The reason now travels with the answer -------------------------------------- `HostedHubsCollection.GetHubWithOutcome` returns a `HostedHubResult` — the hub plus a `HostedHubOutcome` (Available / Absent / HostShuttingDown / ConstructionFaulted) and the exception behind it. `IMessageHub.TryGetHostedHub` carries it to the caller (default-implemented, so any other implementation keeps compiling and reports `Unclassified`). The classification is made in the one place that can see the container, rather than re-derived two layers away by a caller that cannot. The shutdown verdict is MEASURED, not assumed --------------------------------------------- The production shape is not the creation freeze: the creation passed `IsDisposing` and the Autofac scope then died UNDER the build (`LifetimeScope.ThrowDisposedException`). So an `ObjectDisposedException` out of hub construction is classified by ASKING the container whether it can still resolve anything. A live scope that threw that type for its own reasons still reads as a fault, and if the probe itself cannot answer the CLR treats the exception filter as false and the outcome stays LOUD — the conservative direction, by construction. Downgrading a level is not licence to swallow an outcome: both branches still record the activation failure for the caller, still fail parked deliveries via `_hubReadyRaw.OnError`, and still `DeactivateOnIdle` for retry-on-next-access. `Unclassified` stays at fail level too — only a KNOWN shutdown is quiet. Tests ----- `HubCreationDuringTeardownIsNotAFaultTest` runs all three conditions against a real Autofac container built by the framework's own `SetupModules`; the disposed-container case reproduces the incident's exact frame. Verified as a negative control: with the classification disabled it fails with `Expected … HostShuttingDown … but found ConstructionFaulted`, over the verbatim production stack. `HubConstructionOutcomeReportingTest` pins the level and the wording per outcome, including that an unclassified answer is never quiet. Docs: LogWatchTriage gains the third "not a fault" row and the two rules this case adds — the condition must travel, and the benign verdict must be measured. Closes #3243 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HhT5AXByEKo2fRQtWH2wCW
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The outcome classification is implemented at the correct layer, propagated via a compatible interface extension, and is backed by targeted regression tests and documentation updates.
Pull request overview
This PR reduces incident noise during expected pod rollouts by carrying a classified “hosted hub creation outcome” (e.g., shutting down vs. construction fault) from HostedHubsCollection to callers like MessageHubGrain, allowing accurate log level and message selection without guessing from a null hub result.
Changes:
- Introduces
HostedHubResult/HostedHubOutcomeand addsIMessageHub.TryGetHostedHub(...)to return both the hub (if any) and the classified outcome. - Updates
HostedHubsCollectionto classify and report “host shutting down” vs. “construction faulted”, including a measured container-disposed probe forObjectDisposedException. - Updates
MessageHubGrainreporting to log shutdown races atDebug(with evidence) while keeping real construction faults atError, and adds regression tests + documentation/What’s New updates.
File summaries
| File | Description |
|---|---|
| test/MeshWeaver.Messaging.Hub.Test/HubCreationDuringTeardownIsNotAFaultTest.cs | New regression test covering fault vs. disposal-freeze vs. disposed-container-under-build cases with a real container. |
| test/MeshWeaver.Hosting.Orleans.Test/HubConstructionOutcomeReportingTest.cs | Pins log level and wording in MessageHubGrain for each HostedHubOutcome. |
| test/MeshWeaver.Hosting.Orleans.Test/HubConstructionFailureReasonTest.cs | Updates existing test to the new outcome-aware failure-reason signature. |
| src/MeshWeaver.Messaging.Hub/MessageHub.cs | Adds TryGetHostedHub(...) and routes legacy GetHostedHub(...) through it. |
| src/MeshWeaver.Messaging.Hub/IMessageHub.cs | Adds default-implemented TryGetHostedHub(...) to propagate outcome without breaking older implementations. |
| src/MeshWeaver.Messaging.Hub/HostedHubsCollection.cs | Returns HostedHubResult, classifies teardown races vs. faults, and preserves retry semantics by clearing per-address single-flight entries post-construction. |
| src/MeshWeaver.Messaging.Hub/HostedHubResult.cs | Defines HostedHubOutcome and HostedHubResult (including IsShutdownRace and optional error propagation). |
| src/MeshWeaver.Hosting.Orleans/MessageHubGrain.cs | Switches activation hub acquisition to TryGetHostedHub(...) and logs at outcome-dependent level with outcome-specific reason text. |
| src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-04-a-pod-shutting-down-is-no-longer-an-error.md | Adds a What’s New entry describing the user-visible reduction in error-level noise. |
| src/MeshWeaver.Documentation/Data/Architecture/LogWatchTriage.md | Documents hosted-hub-teardown-race as a third “not a fault” outcome, plus the “condition must travel” and “benign must be measured” rules. |
Review details
- Files reviewed: 10/10 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Test Results (shard 3)368 tests 366 ✅ 37s ⏱️ Results for commit 97635cc. |
Contributor
Test Results (shard 0)251 tests 251 ✅ 2m 15s ⏱️ Results for commit 97635cc. |
Contributor
Test Results (shard 1)459 tests 459 ✅ 58s ⏱️ Results for commit 97635cc. |
Contributor
Test Results (shard 4)2 211 tests 2 211 ✅ 2m 48s ⏱️ Results for commit 97635cc. |
Contributor
Test Results (shard 5) 4 files 4 suites 1m 50s ⏱️ Results for commit 97635cc. |
Contributor
Test Results (shard 2)581 tests 389 ✅ 2m 38s ⏱️ Results for commit 97635cc. |
Contributor
Test Results 15 files 15 suites 11m 8s ⏱️ Results for commit 97635cc. |
This was referenced Sep 4, 2026
Closed
rbuergi
deleted the
fix/3243-disposal-race-is-not-a-hub-construction-error
branch
September 13, 2026 10:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3243
The defect
MessageHubGrainlogged[ACTIVATE] … Hub construction returned no hubat fail level for both conditions that produce a null hub, and the message text listed the two and committed to neither:The vagueness was honest.
GetHostedHubanswersnullfor both and a null carries no reason, so the grain genuinely could not tell them apart. But the level a call site chooses is a ticketing decision, not a verbosity knob — every red line a portal writes becomes an incident and a GitHub issue (Doc/Architecture/LogWatchTriage). So the condition that happens on every pod rollout opened a ticket for a race in which nothing failed and nothing was written: incidente2028eb86d6a85a6and its sibling8bd7c9c44c12e40b, recurring across four deployments since mid-August.This is not a log-level dial. It is a real cost/value distinction between two conditions that were previously indistinguishable, and both remain fully reported — only the one that is not a fault stops being red.
The reason now travels with the answer
HostedHubsCollection.GetHubWithOutcomereturns aHostedHubResult: the hub, aHostedHubOutcome(Available/Absent/HostShuttingDown/ConstructionFaulted), and the exception behind it.IMessageHub.TryGetHostedHubcarries it to the caller. Default-implemented, so any otherIMessageHubimplementation keeps compiling and reportsUnclassified.The shutdown verdict is MEASURED, not assumed
The production shape is not the creation freeze: the creation passed
IsDisposing, and the Autofac scope then died under the build (LifetimeScope.ThrowDisposedException). So anObjectDisposedExceptionout of hub construction is classified by asking the container whether it can still resolve anything. A live scope that threw that type for its own reasons still reads as a fault; and if the probe itself cannot answer, the CLR treats the exception filter as false and the outcome stays loud — the conservative direction, by construction.Downgrading a level is not licence to swallow an outcome. Both branches still record the activation failure for the caller, still fail parked deliveries through
_hubReadyRaw.OnError, and stillDeactivateOnIdlefor retry-on-next-access.Unclassifiedstays at fail level too — only a known shutdown is quiet.Tests, and the negative control
HubCreationDuringTeardownIsNotAFaultTest(new) runs all three conditions against a real Autofac container built by the framework's ownSetupModules— no mocks, no hand-rolled gates, no async:ConstructionFaulted,Erroris the thrown exceptionHostShuttingDown, no exceptionHostShuttingDown,Erroris theObjectDisposedExceptionVerified it fails without the fix. With the classification disabled the third case fails with
over the verbatim production stack —
Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()→MessageHubConfiguration..ctor→CreateMessageHub→HostedHubsCollection.CreateHub, preceded by the companion[Error] Failed to create hosted hub for address …line from the sibling incident.HubConstructionOutcomeReportingTest(new) pins the level and the wording per outcome, including that an unclassified answer is never quietly downgraded.HubConstructionFailureReasonTest(#1693) is carried forward onto the new signature.Verification
dotnet build -c Release -warnaserror, one project per invocation,0 Error(s)on:MeshWeaver.Messaging.Hub,MeshWeaver.Hosting.Orleans,MeshWeaver.Documentation,MeshWeaver.Mesh.Operations,MeshWeaver.PluginCatalog,MeshWeaver.Hosting.AspNetCore,MeshWeaver.Compiler.Pipeline, and the three touched test projects.MeshWeaver.Messaging.Hub.Test: full suite green (365 tests).MeshWeaver.Hosting.Orleans.Test~HubConstruction: 8/8 green.MeshWeaver.Documentation.Test~WhatsNew/~DocumentationLinkIntegrity: green.Docs
Doc/Architecture/LogWatchTriagegains the third "not a fault" row and the two rules this case adds: the condition must travel (a caller cannot classify what it cannot see), and the benign verdict must be measured. What's New entry: A pod shutting down is no longer reported as an error (Category: Fix).🤖 Generated with Claude Code
https://claude.ai/code/session_01HhT5AXByEKo2fRQtWH2wCW