Repository navigation
fix: the NodeType overlay could never be delivered — two equal timeouts - #2992
Conversation
NodeTypeEnrichmentHelpers.SlowPathTimeout (60s) exactly equalled
MessageHubConfiguration.RequestTimeout (60s). An inner bound that cannot fire
first cannot deliver anything, so the overlay's correct and specific message —
"NodeType 'X' is not registered (referenced by instance '…')" — has never once
reached a caller. Every occurrence surfaced as the ceiling's generic
"No response received in hub …", naming neither the culprit nor the remedy.
This is the rule ReadBudget states for exactly this pair ("a bound nested inside
another bound must be able to fire FIRST, because it is the only one that knows
WHICH read starved"), with three cited incidents. The identical collision is
already recorded from the test side in OrleansDynamicCompilationTest, where it
cost "five sessions theorised from that silence" — that fix raised the test's
budget and left the collision underneath untouched.
Lowering it costs nothing: WaitForCompileSettled disarms this budget while a
compile is in flight, and in-flight visibility is InFlightOverlayGrace (5s to a
self-healing progress overlay). It governs only the dead-wait case.
- SlowPathTimeout 60s -> 30s, invariant stated at the constant
- MessageHubConfiguration.DefaultRequestTimeout made public: it was a bare
literal invisible outside its assembly, so nested bounds hand-copied the
number into comments — which is how it drifted into collision
- NodeTypeOverlayBudgetTest pins ordering, headroom and grace ordering;
verified to fail on reintroduction
- Doc drift fixed: the XML doc argued "30 s leaves budget" above a constant of
60, and StaticNodeProviders.md said 60 s
- NodeUpdatePipeline no longer reports its own 10s timeout as "Node not found".
That conflated a non-verdict with a verdict and invited delete-and-recreate on
live content — the same lie the read path was fixed for (#974). Genuine
not-found is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Test Results (shard 0)216 tests 216 ✅ 2m 32s ⏱️ Results for commit f9062a5. |
Test Results (shard 3)586 tests 586 ✅ 1m 2s ⏱️ Results for commit f9062a5. |
Test Results (shard 1)419 tests 419 ✅ 1m 27s ⏱️ Results for commit f9062a5. |
Test Results (shard 2)529 tests 527 ✅ 1m 31s ⏱️ Results for commit f9062a5. |
Test Results (shard 5) 3 files 3 suites 58s ⏱️ Results for commit f9062a5. |
There was a problem hiding this comment.
Pull request overview
This PR fixes a production diagnostic gap where the NodeType “slow path” overlay could never surface its specific error message because its timeout equaled the hub request timeout ceiling, causing the outer ceiling to preempt the inner diagnostic.
Changes:
- Reduced
NodeTypeEnrichmentHelpers.SlowPathTimeoutfrom 60s to 30s to restore headroom under the hub request ceiling. - Exposed
MessageHubConfiguration.DefaultRequestTimeoutpublicly to avoid hand-copied timeout literals drifting into collisions. - Added
NodeTypeOverlayBudgetTestand updated documentation to pin and explain the timeout ordering invariant.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/MeshWeaver.Graph.Test/NodeTypeOverlayBudgetTest.cs | Adds a regression test that enforces timeout ordering/headroom invariants for overlay delivery. |
| src/MeshWeaver.Messaging.Hub/MessageHubConfiguration.cs | Single-sources the default request timeout via a new public DefaultRequestTimeout. |
| src/MeshWeaver.Hosting/NodeUpdatePipeline.cs | Improves update-path error reporting by distinguishing timeout “unavailable” from definitive not-found. |
| src/MeshWeaver.Graph/Configuration/NodeTypeEnrichmentHelpers.cs | Lowers slow-path timeout and documents the nested-timeout invariant. |
| src/MeshWeaver.Documentation/Data/Architecture/StaticNodeProviders.md | Updates doc values/wording to match the new 30s slow-path timeout rationale. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| Observable.Throw<MeshNode>(ex switch | ||
| { | ||
| DeliveryFailureException => new InvalidOperationException( | ||
| $"Node not found: {node.Path}"), | ||
| TimeoutException => new InvalidOperationException( |
Test Results (shard 4)1 671 tests 1 671 ✅ 1m 49s ⏱️ Results for commit f9062a5. |
Test Results 14 files 14 suites 9m 22s ⏱️ Results for commit f9062a5. |
The diagnostic that names the culprit has never once been delivered
NodeTypeEnrichmentHelpersalready builds exactly the right message when an instance references aNodeType that resolves to nothing:
It is correct, it is specific, and nobody has ever seen it in production. Two bounds are equal:
NodeTypeEnrichmentHelpers.SlowPathTimeoutMessageHubConfiguration.RequestTimeoutAn inner bound that cannot fire first cannot deliver anything. So every one of these activations
surfaced as the ceiling's generic
"No response received in hub … for request SubscribeRequest",which names neither the culprit nor the remedy — and, through
NodeReadOutcome.FromReadFailure,reaches an operator as
"Unavailable: … this read reached no verdict".This is the rule the codebase already writes down for itself, in
ReadBudget:ReadBudget.Default(10 s) is documented as "comfortably inside the 60 s hubRequestTimeoutsothis bound is the one that fires".
SlowPathTimeoutnever got that margin, andReadBudget's ownremarks cite three production incidents of this exact shape (#1563, #1693, #1748).
It has already been paid for once, from the test side.
OrleansDynamicCompilationTestcarriesthis note about the identical collision:
That fix raised the test's budget. The collision underneath it was never addressed; this PR
addresses it.
Why lowering it costs nothing
SlowPathTimeoutnever bounded a legitimate compile.WaitForCompileSettleddisarms it(
Observable.Never) for as long as a compile is genuinely in flight, and in-flight visibility isInFlightOverlayGrace(5 s to a live progress overlay that self-heals onto the real hub). So theconstant governs exactly one case — nothing compiling and the type stream still unsettled, i.e. a
dangling or unreachable NodeType — where the whole remaining wait is dead time.
30 s restores the margin the surrounding design has always argued for and leaves a full 30 s for the
overlay to be built and delivered inside the caller's ceiling.
What's in this PR
SlowPathTimeout60 s → 30 s, with the invariant stated at the constant.MessageHubConfiguration.DefaultRequestTimeoutis now public. It was a bare literal invisibleoutside its assembly, so every nested bound that had to stay under it hand-copied the number into
a comment — which is precisely how it drifted into collision. Behaviour is unchanged; the
relationship is now single-sourced and referenceable.
NodeTypeOverlayBudgetTestpins the ordering invariant, the headroom, andInFlightOverlayGrace < SlowPathTimeout. Verified to fail on reintroduction:Expected 00:01:00 to be less than 00:01:00. A comment cannot fail; this can.constant of 60, and
StaticNodeProviders.mdsaid 60 s. A comment that contradicts its constant ishow the next reader concludes the margin exists. (
UnregisteredNodeTypeTestin the Plugins repoalso says 30 s — that becomes correct again with this change, no edit needed.)
NodeUpdatePipelinestops reporting a non-verdict as a verdict. It mapped both a realDeliveryFailureExceptionand its own 10 sTimeoutExceptionto"Node not found: {path}".The timeout case is exactly how a node whose NodeType no longer resolves fails — so the write path
told callers an existing node was absent, and the only repair that reads as sensible from
there is delete-and-recreate, on live content. This is the same lie the read path was fixed for
(Three more sites report an availability failure as a definitive negative (permission-fold fault → "Access denied", read timeout → "Not found", cold cache → "user unknown") #974). Genuine not-found is byte-identical; only the timeout branch changes.
How this was found
Auditing two production meshes for instances whose
nodeTyperesolves to nothing. One realinstance on
memex.systemorph.com(rbuergi/_Draft/ClientA_EslProposalQA,nodeType: EmailDraft)reads as
Unavailableand never reaches a verdict. Chasing why that produced no diagnosis is whatsurfaced this.
Three measurement traps worth recording, because each one produces a confident wrong answer:
search nodeType:NodeTypecannot enumerate the valid type set. Most built-ins leaveNodeTypenull and the
AddMeshNodesseed bucket is suppressed undercontext:search. The rule actuallyenforced is
FindStaticNode(nodeType) is not null || persistence.Exists(nodeType).nodeTypewildcards work in a positive filter but are silently ignored in negation —-nodeType:Edu/*does nothing.get_diagnosticsis not a registration oracle — it returns the same trivialOkfor anystatic type, healthy or not.
Deliberately NOT in this PR — two holes that need a decision, not an implementation
Both are real and both have a real counterparty, so each is a maintainer call rather than a patch:
A.
updateaccepts anodeTypethat does not exist. Create validates it(
MeshExtensions.cs:756-786,NodeCreationRejectionReason.InvalidNodeType); update does not —UpdateAccordingToSourceNodecopiesNodeTypethrough unvalidated, and no registeredINodeValidatorever producesInvalidNodeTypeforNodeOperation.Update. So update is a supportedroute to create the orphan condition.
Counterparty:
StaticRepoImporterdeliberately relies on update skipping this check as anordering escape hatch for cases
ImportWriteOrdercannot sequence. A naive validator would break theimporter, which is worse than the hole.
B. Pruning a NodeType strands its instances.
ComputePrunableNodes:412andPackageInstaller.PruneRemovedNodes:2915delete a NodeType with no instance check; a rename(
HandleMoveNodeRequest:4697) never rewrites instances.Counterparty:
WhatsNew/2026-08-28-retired-node-prune.mddocuments pruning retired NodeTypes asintended. So the answer is probably not refusal but a loud, actionable report — and the detector
already exists as a UI-only query (
MeshNodeLayoutAreas.cs:817,nodeType:{name} scope:subtree)that nobody wired into the delete path. The policy is also already written down, in
V53_RetypeBuiltinSlideDeckToPublish: "every install must retype first — the deletion lands only arelease AFTER it has run everywhere." That is exactly the rule prune violates.
Worth stating plainly while both are open: there is currently no safe supported way to repair a
mistyped node.
patchrefusesnodeTypeoutright (PatchableFields,MeshOperations.cs:4221),so the only route is
update— which is hole A.Fixing either before this PR would have left them invisible anyway. That is the argument for landing
the instrument first.
🤖 Generated with Claude Code