Repository navigation
fix(hub): a hosted hub's initialization bound is strictly inside its host's (#1122) - #5205
Conversation
…host's (#1122) Initialization is a nested wait — a hub's DataContext time-box waits on its data sources, each waits on the sync/{clientId} sub-hub serving its stream, and that sub-hub is a hub with an initialization of its own. Every level took the same independently-written 120 s constant, and nothing said the levels were supposed to be ordered. Equal is not an ordering: when the enclosing bound wins it errors the inner hub's streams and that hub's init ends as a recognised shutdown which records nothing — the level that knew which action hung says nothing at all. Exactly one bound is now configured per hub (WithStartupTimeout, else HubInitializationBudget.Root); everything nested inside it is derived by a strictly contracting Nest(), the same shape MeshOperationOptions uses. No bound is widened or narrowed: the root value is unchanged, and a hub nobody hosts — including a per-node hub — still gets 120 s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
HubInitializationBudget.Nest currently accepts sub-millisecond positive budgets where tick truncation can collapse derived rungs to zero, so the ladder’s “strictly contracting” guarantee can fail under misconfiguration.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
This PR makes hub initialization timeouts attributable across nested hub initialization by introducing a strictly contracting “initialization budget ladder”, so a hosted (inner) hub times out before its host (outer) and can report the specific pending BuildupAction rather than being torn down silently.
Changes:
- Introduces
HubInitializationBudgetand derives per-hubInitializationBudget/NestedInitializationBudget, stamping the enclosing rung onto hosted hubs during creation. - Updates
MessageHubandDataContextto use the derived rung (Configuration.NestedInitializationBudget) instead of independently-written constants. - Adds tests and docs to pin both the ladder’s structural invariants and the behavioral “inner reports, outer keeps serving” outcome.
| File | Description |
|---|---|
| test/MeshWeaver.Messaging.Hub.Test/NestedInitializationBudgetTest.cs | New structural + behavioral tests for the contracting budget ladder across host/hosted hubs. |
| test/MeshWeaver.Messaging.Hub.Test/InitializationTimeoutNamesThePendingActionTest.cs | Updates rationale comment to reflect the separation between “what the line says” vs “which line fires”. |
| test/MeshWeaver.Graph.Test/ContentReadPaysTheOwningHubsStartupTest.cs | Updates doc comment to reference the new ladder rung terminology. |
| src/MeshWeaver.Messaging.Hub/MessageHubConfiguration.cs | Adds derived InitializationBudget and NestedInitializationBudget, and carries enclosing budget for hosted hubs. |
| src/MeshWeaver.Messaging.Hub/MessageHub.cs | Switches buildup timeout to the derived rung and stamps enclosing rung onto hosted hub configuration. |
| src/MeshWeaver.Messaging.Hub/HubInitializationBudget.cs | New ladder definition and Nest function used to derive nested budgets. |
| src/MeshWeaver.Data/DataContext.cs | Makes DataContext initialization timeout default to the owning hub’s derived rung (with optional override). |
| src/MeshWeaver.Documentation/Data/Architecture/InitializationBudgetLadder.md | New architecture page documenting the ladder and why equal budgets lose attribution. |
| src/MeshWeaver.Documentation/Data/Architecture/HubInitializationFailure.md | Updates documentation to reference NestedInitializationBudget as the rung-2 bound. |
| src/MeshWeaver.Documentation/Data/Architecture/DataContextInitializationTimeout.md | Updates explanation of the sub-hub vs DataContext timeout ordering, now guaranteed by construction. |
| src/MeshWeaver.Documentation/Data/Architecture/ContentRoute503.md | Updates bound table to reference the ladder rather than a deleted constant. |
| src/MeshWeaver.Documentation/Data/Architecture.md | Adds the new ladder doc page to the Architecture topic map. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| /// <param name="enclosing">The bound this wait runs inside.</param> | ||
| /// <returns>A bound strictly smaller than <paramref name="enclosing"/>.</returns> | ||
| /// <exception cref="ArgumentOutOfRangeException">The enclosing bound is not positive, which is | ||
| /// the one domain on which the ladder cannot be made strictly decreasing.</exception> | ||
| public static TimeSpan Nest(TimeSpan enclosing) => | ||
| enclosing > TimeSpan.Zero | ||
| ? TimeSpan.FromTicks(Math.Max( | ||
| (enclosing - NestingReserve).Ticks, | ||
| (long)(enclosing.Ticks * MinNestingFraction))) | ||
| : throw new ArgumentOutOfRangeException(nameof(enclosing), enclosing, | ||
| "An initialization budget must be positive: a nested bound that does not contract " | ||
| + "cannot fire before the bound that encloses it, and then the level that knows " | ||
| + "which wait starved never reports (Systemorph/MeshWeaver#1122)."); |
There was a problem hiding this comment.
Correct, and it is the exact hole MeshOperationOptions.Timeout's 1 ms guard exists to close — its own remarks say so: "the domain on which the nested-budget ladder is strictly decreasing", because below it "integer truncation could let two rungs collapse onto the same tick". I copied the formula and not the domain.
It is also reachable without anyone writing a silly number: the fraction floor halves per level, so a configured 2 s budget hits sub-millisecond at roughly the twelfth nesting level. That is a hub tree nobody has today, but "nobody has it today" is not a guarantee, and a rung that reads TimeSpan.Zero is precisely the equal-rung collision this type exists to make unrepresentable — an Observable.Timer(TimeSpan.Zero) fires immediately, so it would turn the ladder into an instant FAILED state rather than into a lost diagnostic.
Fixing it the way you name: Nest now refuses an enclosing bound below 1 ms with an ArgumentOutOfRangeException that says what the domain is for, and the guard is covered by an assertion in NestedInitializationBudgetTest alongside the existing contraction checks — one case just above the floor (still strictly contracting) and one below it (refused), so the boundary is pinned rather than assumed.
I am deliberately NOT also validating WithStartupTimeout: the refusal belongs where the invariant is, and adding a second validator would let the two drift. Pushing the change now.
…s formula (#1122) Review finding: below a millisecond the tick arithmetic stops being an ordering — halving one tick truncates to zero, two rungs land on the same instant, and a timer at TimeSpan.Zero fires at once, which is the equal-bounds collision this type exists to make unrepresentable, in its worst form. It is reachable by nesting alone: the fraction floor halves per level, so a configured 2 s budget gets there around the twelfth level. Nest now refuses a smaller enclosing bound, and the test pins both sides of the boundary. Also: a HostedHubCreation.Never probe takes the unwrapped config again. HierarchicalRouting probes per stream message per parent-chain level, and that path has twice been a measured CPU hot frame — the stamp is only ever invoked on construction, so wrapping it on a pure read buys nothing and costs a closure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nit contracts (#1122) Self-correction: I claimed a per-node hub is created directly rather than as a hosted hub, so the ladder would not touch it. Measured and false — both MessageHubGrain and MeshExtensions create it through TryGetHostedHub. Stamping every hosted hub would therefore have contracted the per-node hub's whole initialization for a host that is not waiting on it: the mesh root reached Started long before routing activates it. The stamp is now applied only while the host's RunLevel is below Started, which is exactly the window in which one of its own waits can be waiting on the new hub — a data source's stream is served by a sync/{clientId} sub-hub built inside StartDataSourcesAndOpenGate. A per-node hub keeps the root budget; only the sub-hubs born inside its initialization contract. The test now carries the control on the other side of that discriminator: a hub created after the host started must read HubInitializationBudget.Root exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
) The behaviour changed in the previous commit; this type's class doc still described the rule it replaced ("a hosted hub's rung 1", "a hub with no parent"). A doc comment on the type whose whole job is to STATE the ordering rule is not a cosmetic detail — a stale rationale argues for the regression. Comment-only; no behaviour, no test change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…1122) Three places still said the rule the previous commit replaced — HOSTED, where the code now says BORN INSIDE THIS INITIALIZATION. A partial sweep of a corrected rule is the shape that lets the old rule be re-derived from whichever copy was missed, so this finishes it: the doc page, the class doc, the property doc and the inline comment now all state the same predicate the code branches on. Comment/prose only; no behaviour, no test change. DocumentationLinkIntegrityTest + ArchitectureTopicMapTest: total 5, failed 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…1, 115 s rung 2 (#1122) "Not contracted at all" overstated it: a per-node hub keeps the full root budget for its WHOLE initialization, but its own time-box is one rung in and reads 115 s. That is exactly the error this change set already corrected once, so the index line must not re-plant it. Prose only. DocumentationLinkIntegrityTest + ArchitectureTopicMapTest: total 5, failed 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Addresses the part of #1122 / #1186 that is fixable without a live occurrence. No bound is widened and none is narrowed.
The defect: a nested pair of equal bounds, where which one reports is a coin flip
Initialization is a nested wait:
Every level took the same number,
120 s, written independently in three places —MessageHub's buildup bound,DataContext.InitializationTimeout, and the sub-hub's own copy of the first. Nothing in the code said the three were supposed to be ordered at all, and the two outcomes are not symmetric:Started);InitializationError, nothing recorded. The level that knew the answer is torn down before it can give it.Which one happens is decided by scheduling.
InitializeDataSourcescreates the sub-hub a few microseconds beforeOpenInitializationGatearms the box, but the sub-hub's bound is armed on the SUB-HUB's action block after a scheduling hop. In the event consolidated on #1122 the two lines were 5 ms apart. This is #1198's defect, one layer up —MeshOperationOptions' own remarks name it: "equal budgets, and the outer clock always starts first".🚨 Correction to #5193's stated rationale, measured on
main. That PR's body says the two 120 s bounds nest within one hub and the outer "unsubscribes the inner bound before it can printDescribePendingInitialization()". It does not:DataExtensions.StartDataSourcesAndOpenGatearms the time-box and returnsObservable.Return(Unit.Default), so the buildup action completes at once and the box's ownObservable.Amb(...).Subscribe(...)is not rooted in theConcat. On one hub they are siblings over disjoint subjects. The real nest is across the hub boundary, whichDoc/Architecture/DataContextInitializationTimeoutalready had right. #5193's change is good and stands; its reason was wrong, and this PR corrects the sentence in its test's class doc.The fix
Exactly ONE bound is configured per hub —
MessageHubConfiguration.InitializationBudget, either an explicitWithStartupTimeoutorHubInitializationBudget.Root. Everything nested inside it is DERIVED byHubInitializationBudget.Nest(max(t − 5 s, t × 0.5), the shapeMeshOperationOptions.Nestalready uses), which is strictly contracting:InitializationBudgetNestedInitializationBudget=Nest(rung 1)Concat, and theDataContexttime-box — siblings over disjoint subjects, so equal is correct thereNest(rung 2)MessageHub.TryGetHostedHubstamps rung 2 onto the hosted hub's configuration as it creates it. Stamped, never resolved on read:MessageHubConfiguration.ParentHubanswers out of the parent scope'sIMessageHubregistration, which for a hub built on a root provider resolves to the hub ITSELF — the first attempt derived the ladder from it and recursed until the stack died (Stack overflow, exit 134, whole host down, zero tests run).🚨 HOSTED is not ENCLOSED — and my first version of this PR got that wrong. I claimed a per-node hub is created directly rather than hosted. Measured and false: both
MessageHubGrainandMeshExtensionscreate it throughTryGetHostedHub. Stamping every hosted hub would therefore have contracted the per-node hub's whole initialization on behalf of a host that is not waiting on it — routing activates it on demand, long after the mesh root reachedStarted. The stamp is now applied only while the host'sRunLevelis belowStarted, which is exactly the window in which one of its own waits can be waiting on the new hub (a data source's stream is served by async/{clientId}sub-hub built insideStartDataSourcesAndOpenGate). The test carries the control on the other side of that discriminator: a hub created after the host started must readHubInitializationBudget.Rootexactly.What that leaves moving, stated plainly. A per-node hub's rung 1 is unchanged at 120 s. Its rung 2 — its BuildupAction
Concatand itsDataContexttime-box — reads 115 s instead of 120 s, so the production sentence changes from "did not complete within 120s" to "… within 115s". That is the price of having a rung above them at all, and the rung above them is the one a configuredWithStartupTimeoutarms. TheDataContextandMessageHubLogIncidentfingerprints are built from the log TEMPLATE, which is untouched (that fingerprint already survived an exception-message rewrite). 🚨 A fingerprint built from the exception MESSAGE —MessageHubGrain'sActivationFaultReason— will re-key on the changed number, the same cost the query-layer change named for itself.Verification — both sides measured
dotnet build src/MeshWeaver.Messaging.Hub -c Release -warnaserror0 Warning(s),0 Error(s)dotnet build src/MeshWeaver.Data -c Release -warnaserror0 Warning(s),0 Error(s)dotnet build src/MeshWeaver.Graph,src/MeshWeaver.Hosting.Orleans0 Warning(s),0 Error(s)eachdotnet build test/MeshWeaver.Messaging.Hub.Test,test/MeshWeaver.Graph.Test,test/MeshWeaver.Documentation.Test0 Warning(s),0 Error(s)eachNestedInitializationBudgetTest(both cases)total: 2, failed: 0, succeeded: 2MeshWeaver.Messaging.Hub.Test(~Initializ)total: 10, failed: 0, succeeded: 10MeshWeaver.Data.Testprojecttotal: 533, failed: 0, succeeded: 533MeshWeaver.Graph.Test~Test(essentially the whole project, incl. the per-node-hub startup path)total: 1942, failed: 0, succeeded: 1942MeshWeaver.Messaging.Hub.Testprojecttotal: 503, failed: 0, succeeded: 503— twice: 3 m 34 s before theTryGetHostedHubchange and 3 m 35 s after it, so the hot path it touches did not get slower eitherDocumentationLinkIntegrityTest+ArchitectureTopicMapTesttotal: 5, failed: 0, succeeded: 5🚨 Negative control — rung 3 reverted to the flat constant (
InitializationBudget => StartupTimeout ?? Root: the hosted hub ignores the stamp, which is exactlymain's behaviour), tests kept, rebuilt (dll mtime1790077039):total: 2, failed: 2, and the surviving line is the ENCLOSING hub's:— which names the host's own action and says nothing about the hub that actually hung. That is the attribution loss, reproduced. With the fix the hosted hub reports at its own rung (
1.75 sagainst the host's7 s) and the host answers requests normally.The behavioural test asserts BOTH directions, so it distinguishes "the right level reported" from "a level reported": the hosted hub's error names
HangsForever, and((MessageHub)host).InitializationErrormust be null.A second instance of the same pair, closed by the same change
MessageServicearms a startup timer atConfiguration.StartupTimeoutin the hub's constructor, covering ALL of that hub's gates,DataContextInitincluded.HandleInitialize's bound used to be that same value armed later, on the init turn — so on a hub that configures one, the constructor-armed timer won by construction and answered every deferred caller with its anonymous "failed to initialize in … — gates still closed: […]", while the line naming the pending BuildupAction (#5193) landed afterwards, in a log nobody's request carried. WithHandleInitializeatNest(rung 1)it fires strictly first, so the caller'sDeliveryFailurecarries the named reason — visible in the negative control's own output above, where the delivered failure IS the named one.(In production this pair is dormant:
StartupTimeoutdefaults to null, so that timer is never armed.)Docs
New page
Doc/Architecture/InitializationBudgetLadder, listed in the Architecture topic map.DataContextInitializationTimeoutandHubInitializationFailureupdated — the latter's "this outer bound always fires first … unsubscribes the inner bound" paragraph is the refuted one;ContentRoute503's bound table repointed off the deletedDefaultInitializationTimeout.No What's New file (policy
whatsnew-cadence).Review round 2 — the 1 ms domain
NestcopiedMeshOperationOptions' formula but not its 1 ms domain, so a sub-millisecond enclosing bound truncated a rung toTimeSpan.Zero— the equal-bounds collision in its worst form, since a timer at zero fires at once. Reachable by nesting alone (the fraction floor halves per level, so a 2 s budget gets there around the twelfth level).Nestnow refuses belowSmallestNestableBound, and the test pins BOTH sides of that boundary. Also, aHostedHubCreation.Neverprobe takes the unwrappedconfigagain —HierarchicalRoutingprobes per stream message per parent-chain level, twice a measured CPU hot frame, and the stamp is only ever invoked on construction.What this does NOT fix, and what stays open
#1122 and #1186 stay OPEN. This makes a hang ATTRIBUTABLE; it does not say why any particular wait starved. The residual causes named on those threads — a
StorageAdapterMeshQueryProviderthat neither emits nor terminates (#5048), a swallowed store fault read as an absent node (#5165), and the FAILED-latch/transient-retire asymmetry, which is an undecided policy question — are untouched here.Addresses to recycle: none.
src/framework code carried by the image; the ladder is read during a hub's initialization, so every activation created after the roll takes it.Mirror-sync: none — no catalog key touched.
Pairs-with: none — no public surface removed.
MessageHubConfigurationgains two public properties andHubInitializationBudgetis new;MessageHub.DefaultInitializationTimeoutwasprivate.Implementers: none — no interface member added.
🤖 Generated with Claude Code