Skip to content

fix(hub): a hosted hub's initialization bound is strictly inside its host's (#1122) - #5205

Merged
meshweaver-cloud[bot] merged 6 commits into
mainfrom
fix/1122-initialization-budget-ladder
Sep 22, 2026
Merged

meshweaver-cloud[bot] merged 6 commits into
mainfrom
fix/1122-initialization-budget-ladder

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Addresses the part of #1122 / #1186 that is fixable without a live occurrence. No bound is widened and none is narrowed.

The defect: a nested pair of equal bounds, where which one reports is a coin flip

Initialization is a nested wait:

a hub's DataContext time-box                     ← waits on its data sources
  └─ IDataSource.Initialized                      ← waits on each stream's first frame
       └─ the sync/{clientId} sub-hub              ← a HUB, with an initialization of its own
            └─ its BuildupAction Concat            ← bounded by the sub-hub's own time-box

Every level took the same number, 120 s, written independently in three places — MessageHub's buildup bound, DataContext.InitializationTimeout, and the sub-hub's own copy of the first. Nothing in the code said the three were supposed to be ordered at all, and the two outcomes are not symmetric:

  • inner first — the sub-hub reports, naming the BuildupAction that never signalled, and the enclosing wait then completes normally (a FAILED hub still opens its gate and reaches Started);
  • outer first — the enclosing box expires, errors every stream the data source holds, and the sub-hub's init ends as a recognised shutdown: no error log, no InitializationError, nothing recorded. The level that knew the answer is torn down before it can give it.

Which one happens is decided by scheduling. InitializeDataSources creates the sub-hub a few microseconds before OpenInitializationGate arms the box, but the sub-hub's bound is armed on the SUB-HUB's action block after a scheduling hop. In the event consolidated on #1122 the two lines were 5 ms apart. This is #1198's defect, one layer up — MeshOperationOptions' own remarks name it: "equal budgets, and the outer clock always starts first".

🚨 Correction to #5193's stated rationale, measured on main. That PR's body says the two 120 s bounds nest within one hub and the outer "unsubscribes the inner bound before it can print DescribePendingInitialization()". It does not: DataExtensions.StartDataSourcesAndOpenGate arms the time-box and returns Observable.Return(Unit.Default), so the buildup action completes at once and the box's own Observable.Amb(...).Subscribe(...) is not rooted in the Concat. On one hub they are siblings over disjoint subjects. The real nest is across the hub boundary, which Doc/Architecture/DataContextInitializationTimeout already had right. #5193's change is good and stands; its reason was wrong, and this PR corrects the sentence in its test's class doc.

The fix

Exactly ONE bound is configured per hub — MessageHubConfiguration.InitializationBudget, either an explicit WithStartupTimeout or HubInitializationBudget.Root. Everything nested inside it is DERIVED by HubInitializationBudget.Nest (max(t − 5 s, t × 0.5), the shape MeshOperationOptions.Nest already uses), which is strictly contracting:

rung value bounds
1 InitializationBudget this hub's WHOLE initialization, as its host bounds it
2 NestedInitializationBudget = Nest(rung 1) one wait inside it: the BuildupAction Concat, and the DataContext time-box — siblings over disjoint subjects, so equal is correct there
3 a hosted hub's rung 1 = Nest(rung 2) that hub's whole initialization

MessageHub.TryGetHostedHub stamps rung 2 onto the hosted hub's configuration as it creates it. Stamped, never resolved on read: MessageHubConfiguration.ParentHub answers out of the parent scope's IMessageHub registration, which for a hub built on a root provider resolves to the hub ITSELF — the first attempt derived the ladder from it and recursed until the stack died (Stack overflow, exit 134, whole host down, zero tests run).

🚨 HOSTED is not ENCLOSED — and my first version of this PR got that wrong. I claimed a per-node hub is created directly rather than hosted. Measured and false: both MessageHubGrain and MeshExtensions create it through TryGetHostedHub. Stamping every hosted hub would therefore have contracted the per-node hub's whole initialization on behalf of a host that is not waiting on it — routing activates it on demand, long after the mesh root reached Started. The stamp is now applied only while the host's RunLevel is below Started, which is exactly the window in which one of its own waits can be waiting on the new hub (a data source's stream is served by a sync/{clientId} sub-hub built inside StartDataSourcesAndOpenGate). The test carries the control on the other side of that discriminator: a hub created after the host started must read HubInitializationBudget.Root exactly.

What that leaves moving, stated plainly. A per-node hub's rung 1 is unchanged at 120 s. Its rung 2 — its BuildupAction Concat and its DataContext time-box — reads 115 s instead of 120 s, so the production sentence changes from "did not complete within 120s" to "… within 115s". That is the price of having a rung above them at all, and the rung above them is the one a configured WithStartupTimeout arms. The DataContext and MessageHub LogIncident fingerprints are built from the log TEMPLATE, which is untouched (that fingerprint already survived an exception-message rewrite). 🚨 A fingerprint built from the exception MESSAGE — MessageHubGrain's ActivationFaultReason — will re-key on the changed number, the same cost the query-layer change named for itself.

Verification — both sides measured

step signal
dotnet build src/MeshWeaver.Messaging.Hub -c Release -warnaserror 0 Warning(s), 0 Error(s)
dotnet build src/MeshWeaver.Data -c Release -warnaserror 0 Warning(s), 0 Error(s)
dotnet build src/MeshWeaver.Graph, src/MeshWeaver.Hosting.Orleans 0 Warning(s), 0 Error(s) each
dotnet build test/MeshWeaver.Messaging.Hub.Test, test/MeshWeaver.Graph.Test, test/MeshWeaver.Documentation.Test 0 Warning(s), 0 Error(s) each
NestedInitializationBudgetTest (both cases) total: 2, failed: 0, succeeded: 2
every init test in MeshWeaver.Messaging.Hub.Test (~Initializ) total: 10, failed: 0, succeeded: 10
the WHOLE MeshWeaver.Data.Test project total: 533, failed: 0, succeeded: 533
MeshWeaver.Graph.Test ~Test (essentially the whole project, incl. the per-node-hub startup path) total: 1942, failed: 0, succeeded: 1942
the WHOLE MeshWeaver.Messaging.Hub.Test project total: 503, failed: 0, succeeded: 503 — twice: 3 m 34 s before the TryGetHostedHub change and 3 m 35 s after it, so the hot path it touches did not get slower either
DocumentationLinkIntegrityTest + ArchitectureTopicMapTest total: 5, failed: 0, succeeded: 5

🚨 Negative control — rung 3 reverted to the flat constant (InitializationBudget => StartupTimeout ?? Root: the hosted hub ignores the stamp, which is exactly main's behaviour), tests kept, rebuilt (dll mtime 1790077039): total: 2, failed: 2, and the surviving line is the ENCLOSING hub's:

Hub 'host/1' initialization failed: BuildupAction 1 of 1
  (NestedInitializationBudgetTest.WaitsForTheHostedChild) did not complete within 7s

— which names the host's own action and says nothing about the hub that actually hung. That is the attribution loss, reproduced. With the fix the hosted hub reports at its own rung (1.75 s against the host's 7 s) and the host answers requests normally.

The behavioural test asserts BOTH directions, so it distinguishes "the right level reported" from "a level reported": the hosted hub's error names HangsForever, and ((MessageHub)host).InitializationError must be null.

A second instance of the same pair, closed by the same change

MessageService arms a startup timer at Configuration.StartupTimeout in the hub's constructor, covering ALL of that hub's gates, DataContextInit included. HandleInitialize's bound used to be that same value armed later, on the init turn — so on a hub that configures one, the constructor-armed timer won by construction and answered every deferred caller with its anonymous "failed to initialize in … — gates still closed: […]", while the line naming the pending BuildupAction (#5193) landed afterwards, in a log nobody's request carried. With HandleInitialize at Nest(rung 1) it fires strictly first, so the caller's DeliveryFailure carries the named reason — visible in the negative control's own output above, where the delivered failure IS the named one.

(In production this pair is dormant: StartupTimeout defaults to null, so that timer is never armed.)

Docs

New page Doc/Architecture/InitializationBudgetLadder, listed in the Architecture topic map. DataContextInitializationTimeout and HubInitializationFailure updated — the latter's "this outer bound always fires first … unsubscribes the inner bound" paragraph is the refuted one; ContentRoute503's bound table repointed off the deleted DefaultInitializationTimeout.

No What's New file (policy whatsnew-cadence).

Review round 2 — the 1 ms domain

Nest copied MeshOperationOptions' formula but not its 1 ms domain, so a sub-millisecond enclosing bound truncated a rung to TimeSpan.Zero — the equal-bounds collision in its worst form, since a timer at zero fires at once. Reachable by nesting alone (the fraction floor halves per level, so a 2 s budget gets there around the twelfth level). Nest now refuses below SmallestNestableBound, and the test pins BOTH sides of that boundary. Also, a HostedHubCreation.Never probe takes the unwrapped config again — HierarchicalRouting probes per stream message per parent-chain level, twice a measured CPU hot frame, and the stamp is only ever invoked on construction.

What this does NOT fix, and what stays open

#1122 and #1186 stay OPEN. This makes a hang ATTRIBUTABLE; it does not say why any particular wait starved. The residual causes named on those threads — a StorageAdapterMeshQueryProvider that neither emits nor terminates (#5048), a swallowed store fault read as an absent node (#5165), and the FAILED-latch/transient-retire asymmetry, which is an undecided policy question — are untouched here.

Addresses to recycle: none. src/ framework code carried by the image; the ladder is read during a hub's initialization, so every activation created after the roll takes it.

Mirror-sync: none — no catalog key touched.
Pairs-with: none — no public surface removed. MessageHubConfiguration gains two public properties and HubInitializationBudget is new; MessageHub.DefaultInitializationTimeout was private.
Implementers: none — no interface member added.

🤖 Generated with Claude Code

…host's (#1122)

Initialization is a nested wait — a hub's DataContext time-box waits on its data
sources, each waits on the sync/{clientId} sub-hub serving its stream, and that
sub-hub is a hub with an initialization of its own. Every level took the same
independently-written 120 s constant, and nothing said the levels were supposed
to be ordered. Equal is not an ordering: when the enclosing bound wins it errors
the inner hub's streams and that hub's init ends as a recognised shutdown which
records nothing — the level that knew which action hung says nothing at all.

Exactly one bound is now configured per hub (WithStartupTimeout, else
HubInitializationBudget.Root); everything nested inside it is derived by a
strictly contracting Nest(), the same shape MeshOperationOptions uses. No bound
is widened or narrowed: the root value is unchanged, and a hub nobody hosts —
including a per-node hub — still gets 120 s.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

HubInitializationBudget.Nest currently accepts sub-millisecond positive budgets where tick truncation can collapse derived rungs to zero, so the ladder’s “strictly contracting” guarantee can fail under misconfiguration.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity

Open (1)
What changed in this PR

This PR makes hub initialization timeouts attributable across nested hub initialization by introducing a strictly contracting “initialization budget ladder”, so a hosted (inner) hub times out before its host (outer) and can report the specific pending BuildupAction rather than being torn down silently.

Changes:

  • Introduces HubInitializationBudget and derives per-hub InitializationBudget/NestedInitializationBudget, stamping the enclosing rung onto hosted hubs during creation.
  • Updates MessageHub and DataContext to use the derived rung (Configuration.NestedInitializationBudget) instead of independently-written constants.
  • Adds tests and docs to pin both the ladder’s structural invariants and the behavioral “inner reports, outer keeps serving” outcome.
File Description
test/​MeshWeaver.Messaging.Hub.Test/​NestedInitializationBudgetTest.cs New structural + behavioral tests for the contracting budget ladder across host/hosted hubs.
test/​MeshWeaver.Messaging.Hub.Test/​InitializationTimeoutNamesThePendingActionTest.cs Updates rationale comment to reflect the separation between “what the line says” vs “which line fires”.
test/​MeshWeaver.Graph.Test/​ContentReadPaysTheOwningHubsStartupTest.cs Updates doc comment to reference the new ladder rung terminology.
src/​MeshWeaver.Messaging.Hub/​MessageHubConfiguration.cs Adds derived InitializationBudget and NestedInitializationBudget, and carries enclosing budget for hosted hubs.
src/​MeshWeaver.Messaging.Hub/​MessageHub.cs Switches buildup timeout to the derived rung and stamps enclosing rung onto hosted hub configuration.
src/​MeshWeaver.Messaging.Hub/​HubInitializationBudget.cs New ladder definition and Nest function used to derive nested budgets.
src/​MeshWeaver.Data/​DataContext.cs Makes DataContext initialization timeout default to the owning hub’s derived rung (with optional override).
src/​MeshWeaver.Documentation/​Data/​Architecture/​InitializationBudgetLadder.md New architecture page documenting the ladder and why equal budgets lose attribution.
src/​MeshWeaver.Documentation/​Data/​Architecture/​HubInitializationFailure.md Updates documentation to reference NestedInitializationBudget as the rung-2 bound.
src/​MeshWeaver.Documentation/​Data/​Architecture/​DataContextInitializationTimeout.md Updates explanation of the sub-hub vs DataContext timeout ordering, now guaranteed by construction.
src/​MeshWeaver.Documentation/​Data/​Architecture/​ContentRoute503.md Updates bound table to reference the ladder rather than a deleted constant.
src/​MeshWeaver.Documentation/​Data/​Architecture.md Adds the new ladder doc page to the Architecture topic map.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +80 to +92
/// <param name="enclosing">The bound this wait runs inside.</param>
/// <returns>A bound strictly smaller than <paramref name="enclosing"/>.</returns>
/// <exception cref="ArgumentOutOfRangeException">The enclosing bound is not positive, which is
/// the one domain on which the ladder cannot be made strictly decreasing.</exception>
public static TimeSpan Nest(TimeSpan enclosing) =>
enclosing > TimeSpan.Zero
? TimeSpan.FromTicks(Math.Max(
(enclosing - NestingReserve).Ticks,
(long)(enclosing.Ticks * MinNestingFraction)))
: throw new ArgumentOutOfRangeException(nameof(enclosing), enclosing,
"An initialization budget must be positive: a nested bound that does not contract "
+ "cannot fire before the bound that encloses it, and then the level that knows "
+ "which wait starved never reports (Systemorph/MeshWeaver#1122).");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and it is the exact hole MeshOperationOptions.Timeout's 1 ms guard exists to close — its own remarks say so: "the domain on which the nested-budget ladder is strictly decreasing", because below it "integer truncation could let two rungs collapse onto the same tick". I copied the formula and not the domain.

It is also reachable without anyone writing a silly number: the fraction floor halves per level, so a configured 2 s budget hits sub-millisecond at roughly the twelfth nesting level. That is a hub tree nobody has today, but "nobody has it today" is not a guarantee, and a rung that reads TimeSpan.Zero is precisely the equal-rung collision this type exists to make unrepresentable — an Observable.Timer(TimeSpan.Zero) fires immediately, so it would turn the ladder into an instant FAILED state rather than into a lost diagnostic.

Fixing it the way you name: Nest now refuses an enclosing bound below 1 ms with an ArgumentOutOfRangeException that says what the domain is for, and the guard is covered by an assertion in NestedInitializationBudgetTest alongside the existing contraction checks — one case just above the floor (still strictly contracting) and one below it (refused), so the boundary is pinned rather than assumed.

I am deliberately NOT also validating WithStartupTimeout: the refusal belongs where the invariant is, and adding a second validator would let the two drift. Pushing the change now.

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

  1 files  ±0    1 suites  ±0   2m 31s ⏱️ -11s
320 tests ±0  320 ✅ ±0  0 💤 ±0  0 ❌ ±0 
324 runs  ±0  324 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 5279aa9. ± Comparison against base commit cc889c1.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

429 tests   - 14   429 ✅  - 14   1m 3s ⏱️ +8s
  2 suites  -  1     0 💤 ± 0 
  2 files    -  1     0 ❌ ± 0 

Results for commit 5279aa9. ± Comparison against base commit cc889c1.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

533 tests   - 1 075   533 ✅  - 1 075   1m 2s ⏱️ - 2m 11s
  1 suites  -     1     0 💤 ±    0 
  1 files    -     1     0 ❌ ±    0 

Results for commit 5279aa9. ± Comparison against base commit cc889c1.

♻️ This comment has been updated with latest results.

rbuergi and others added 5 commits September 22, 2026 13:50
…s formula (#1122)

Review finding: below a millisecond the tick arithmetic stops being an
ordering — halving one tick truncates to zero, two rungs land on the same
instant, and a timer at TimeSpan.Zero fires at once, which is the equal-bounds
collision this type exists to make unrepresentable, in its worst form. It is
reachable by nesting alone: the fraction floor halves per level, so a configured
2 s budget gets there around the twelfth level. Nest now refuses a smaller
enclosing bound, and the test pins both sides of the boundary.

Also: a HostedHubCreation.Never probe takes the unwrapped config again.
HierarchicalRouting probes per stream message per parent-chain level, and that
path has twice been a measured CPU hot frame — the stamp is only ever invoked
on construction, so wrapping it on a pure read buys nothing and costs a closure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nit contracts (#1122)

Self-correction: I claimed a per-node hub is created directly rather than as a
hosted hub, so the ladder would not touch it. Measured and false — both
MessageHubGrain and MeshExtensions create it through TryGetHostedHub. Stamping
every hosted hub would therefore have contracted the per-node hub's whole
initialization for a host that is not waiting on it: the mesh root reached
Started long before routing activates it.

The stamp is now applied only while the host's RunLevel is below Started, which
is exactly the window in which one of its own waits can be waiting on the new
hub — a data source's stream is served by a sync/{clientId} sub-hub built inside
StartDataSourcesAndOpenGate. A per-node hub keeps the root budget; only the
sub-hubs born inside its initialization contract.

The test now carries the control on the other side of that discriminator: a hub
created after the host started must read HubInitializationBudget.Root exactly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
)

The behaviour changed in the previous commit; this type's class doc still
described the rule it replaced ("a hosted hub's rung 1", "a hub with no
parent"). A doc comment on the type whose whole job is to STATE the ordering
rule is not a cosmetic detail — a stale rationale argues for the regression.

Comment-only; no behaviour, no test change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…1122)

Three places still said the rule the previous commit replaced — HOSTED, where
the code now says BORN INSIDE THIS INITIALIZATION. A partial sweep of a
corrected rule is the shape that lets the old rule be re-derived from whichever
copy was missed, so this finishes it: the doc page, the class doc, the property
doc and the inline comment now all state the same predicate the code branches
on.

Comment/prose only; no behaviour, no test change. DocumentationLinkIntegrityTest
+ ArchitectureTopicMapTest: total 5, failed 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…1, 115 s rung 2 (#1122)

"Not contracted at all" overstated it: a per-node hub keeps the full root
budget for its WHOLE initialization, but its own time-box is one rung in and
reads 115 s. That is exactly the error this change set already corrected once,
so the index line must not re-plant it.

Prose only. DocumentationLinkIntegrityTest + ArchitectureTopicMapTest: total 5,
failed 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

    2 files   -   1      2 suites   - 1   6m 14s ⏱️ +32s
2 558 tests +594  2 558 ✅ +594  0 💤 ±0  0 ❌ ±0 
2 559 runs  +594  2 559 ✅ +594  0 💤 ±0  0 ❌ ±0 

Results for commit 5279aa9. ± Comparison against base commit cc889c1.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 578 tests  +850   1 578 ✅ +1 042   9m 19s ⏱️ + 2m 51s
    2 suites  -   1       0 💤  -   192 
    2 files    -   1       0 ❌ ±    0 

Results for commit 5279aa9. ± Comparison against base commit cc889c1.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    5 files  ±  0      5 suites  ±0   14m 26s ⏱️ - 1m 7s
3 336 tests  - 636  3 336 ✅  - 634  0 💤  - 2  0 ❌ ±0 
3 340 runs   - 636  3 340 ✅  - 634  0 💤  - 2  0 ❌ ±0 

Results for commit 5279aa9. ± Comparison against base commit cc889c1.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results

   13 files   -   4     13 suites   - 4   34m 36s ⏱️ -1s
8 754 tests  - 281  8 754 ✅  - 87  0 💤  - 194  0 ❌ ±0 
8 763 runs   - 281  8 763 ✅  - 87  0 💤  - 194  0 ❌ ±0 

Results for commit 5279aa9. ± Comparison against base commit cc889c1.

@meshweaver-cloud
meshweaver-cloud Bot added this pull request to the merge queue Sep 22, 2026
Merged via the queue into main with commit 141ef4a Sep 22, 2026
37 checks passed
@rbuergi
rbuergi deleted the fix/1122-initialization-budget-ladder branch October 10, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants