feat: the query fan-in's Initial stall is a TERMINAL, not a warning by rbuergi · Pull Request #5048 · Systemorph/MeshWeaver

rbuergi · 2026-09-21T06:31:25Z

The decision

The query fan-in's Initial gate is a BOUND, not a wait. MeshQuery.MergeProviderObservables
gates its merged Initial on EVERY registered IMeshQueryProvider emitting one. A provider that
neither emits, completes nor errors starved that gate for ever. It now terminates the merged
query with QueryProviderStalledException, naming the providers that did not answer.

Registered as policy query-fanin-stall-terminal in Doc/Architecture/PolicyNotProse, because
AccessControl.md had reserved it as a platform decision in its own right.

The measurement

Measured on memex over the 400 minutes to 2026-09-21T04:12Z: 95 No MeshNode emitted for
faults (~14/hour) plus 200+ InitialStallProbe warnings whose own text reads "the query is
silently stalled on its all-providers Initial gate and its consumer hangs with no error" — and
nothing acted on any of them, because a warning is not a terminal and the consumers that could have
acted were the ones parked. Earlier instances of the same shape, from the code that records them: CI
2026-07-21 (ExportImportAccessControlTest watchdog-killed at 60 s, flat heap, not one log line) and
#1446 (a CreateNodeRequest sat Executing 33 s until its caller's RequestTimeout ended it).

Why the second half turned out to be already built

AccessControl.md's convergence contract had already forced every consumer that turns a mesh read
into a decision to classify a fault correctly — because no permissive seed of the permission fold
is anything but a hole. The terminal is the input those classifiers were built for and never
received. Swept exhaustively (two parallel read-only sweeps of src/, then verified by hand):

consumer	on a fault today	more permissive than the hang?
`PermissionEvaluator.GetEffectivePermissions` + all four legs	propagates (rewraps only a probe-confirmed scope teardown)	no
`ObserveScopePolicies` / `ObserveEffectiveAssignments` / `ObserveAllMembershipNodes`	propagate; deliberately unseeded	no
`ObserveGatedNodes`	`StartWith(empty)` — monotone-additive, so seeding is stricter	no
`CheckPermissionOutcome` (both overloads)	`Undetermined` ⇒ `IsGranted == false` ⇒ `ErrorType.Unavailable`	no — fail closed
`AccessControlPipeline` (message gate)	classified refusal; `Unavailable`/`ShuttingDown`	no
`AnonymousGate.Evaluate` / `.AllowAnonymous`	`Undetermined` → `false`, warned naming the path	no
`RlsNodeValidator` (node-operation gate)	`NodeRejectionReason.Unavailable`; the write does not proceed	no — fail closed
`MeshNodeStreamCache.GetQueryRaw`	`Replay(1)` latches, then `EvictFaultedQuery` drops the chain (#1316)	no — self-healing
`SyncedQueryMeshNodes`	propagates; no `Catch` anywhere in `BuildReadStreamCore`	no
`workspace.GetQuery` per-user RLS wrap	propagates; built fresh per call, so not latched	no
`PartitionOwningTypes.OwnsPartition`	already `.Timeout(10s, → null).Catch(→ null)`; tri-state, callers fail closed	no — its own bound fires first
`MeshOperations` export permission filter	node omitted, warning names the omission	no
every layout area on the fan-in	`LayoutAreaHost` error frame instead of an eternal spinner	no — an improvement

The four genuinely fail-OPEN consumers, and why the terminal does not reach them

All four already reach their permissive value through their own bound, strictly shorter than rung
4 (15 s), so the terminal changes nothing for them. Stated because it is what makes "no new open
door" checkable rather than hopeful:

consumer	permissive answer	its own bound
`CreatableTypesCreationValidator`	`Valid()` — the create proceeds (a recorded fail-open decision; `Permission.Create` is the real gate)	10 s
`SpaceAdminInvariantValidator`	`Valid()` — last-admin removal not blocked	10 s
`PluginSurfaceProbe.Exists`	`false`	800 ms
`NotificationService.HasRoutingRules`	`false` — send now rather than defer	its `LookupTimeout`

Four consumers where a terminal is WORSE than a hang — fixed here

Not open doors; each turned a hang into something worse, so each is fixed in this change:

MeshDataSource.HandleRunTests — a one-arm .Subscribe(onNext) directly on the fan-in. Rx
hands an unhandled OnError to its default handler, which rethrows on the delivering pool
thread: an unhandled exception and a RunTestsRequest that never gets a response. Now answers
the caller with the failure.
UserIdentityCache — the portal's user directory, and the one latch on the fan-in with no
repair path at all. It opened its read once in the constructor over a Publish(), so one
terminal set its failure reason for the life of the process: every Lookup answered
Unavailable, every WhenDetermined waited on a snapshot that could never arrive, only a pod
restart cleared it. The chain is now re-openable and IndexChanged is a stable façade over an
instance Subject (so a terminal is not replayed to later waiters). No timer, no poller, no
backoff — the dead chain is dropped and the next caller re-opens it, which on a live portal is
the next request or circuit start, and that chain's first snapshot un-parks whoever was waiting.
The EvictFaultedQuery discipline, one class over.
EventSubscriptionRunner.WatchTriggerNodeType — its one-watch-per-type guard keyed on a
dictionary entry it never removed on a terminal, so a faulted watch was dead for the runner's life,
silently stranding the deferred invite/grant reconcile (a user onboards and gets no access). The
entry is now dropped pair-exact so the next emission rebuilds the watch.
MeshNodeBindingExtensions.Exists — its doc promised "stays subscribed so a late answer still
lands", which a terminal cannot deliver. The ReadBudget (10 s) still degrades to false first;
the later fault is deliberately not caught (a .Catch(→ false) would turn "nobody can answer"
into the value a real absence produces) and AreaErrorClassifier.IsStorageUnavailable now
recognises it, so the render shows the localized degrade frame rather than a framework sentence
naming provider class names at an end user. Doc comment corrected to say what now happens.

The budget is DERIVED, and that was not optional

The probe's delay was a hard-coded 20 s — exactly PermissionEstablishmentBudget at the
production default. Promoting that constant to a terminal as-is would have recreated #1198
precisely: equal is not an ordering, the outer clock starts first, so the fan-in could never fire
first and the attribution it alone can produce would have been lost to a coin flip.

MeshOperationOptions.QueryInitialBudget is therefore rung 4 —
Nest(PermissionEstablishmentBudget), 30 / 25 / 20 / 15 s at the default. RlsNodeValidator has
a fourth why arm naming the provider instead of printing its own budget.

🚨 And it reaches the unsecured surface only because it is passed explicitly. The IMeshQueryCore
registration builds MeshQuery with a null hub, so the fan-in's lazy hub.ServiceProvider lookup
cannot see a configured WithMeshOperationTimeout — and the security fold reads through exactly that
registration. PersistenceExtensions now hands it sp.GetService<MeshOperationOptions>(). Without
that, the innermost rung would have been the one value nobody could configure, and no test would have
noticed.

QueryProviderStalledException lives in MeshWeaver.Data.Contract for the reason StorageFaults
does (Mesh.Contract references Layout, so a type declared there is invisible to the classifier),
and is deliberately not a TimeoutException: several catch arms match that type to mean "my own
bound elapsed" and would re-attribute it to themselves — StreamPostGuardTimeoutException's lesson in
reverse — and it keeps the stall out of every retry class. A stalled provider must be fixed, never
retried behind the caller's back, so no production knob can turn the terminal back into a hang (the
short-budget seam is internal, for tests).

Controls, on both sides

QueryFanInStallIsTerminalTest (test/MeshWeaver.Hosting.Test, 8 cases, 2.9 s):
- stalled provider → faults, Providers contains stalled and not healthy, Budget equals
  the ladder's rung, message carries "never bump the consumer's timeout";
- control — same merge, same budget, both providers answer: no fault, held live past the
  budget (not a Take(1), which would unsubscribe before it and prove nothing);
- the single-provider branch both ways (it is a different code path);
- a provider that completes without an Initial is still counted empty and named on the
  frame, never faulted — the separation that keeps SilentProviders meaningful;
- the ladder's strict contraction at three configured values.
StalledPolicyReadFailsClosedTest (test/MeshWeaver.Graph.Test, 16.4 s) — one user, one role,
two partitions on a real monolith mesh, and a provider that stalls one partition's _Policy
read: the control partition still ALLOWS (IsGranted == true, not undetermined), and the stalled
one answers Undetermined / IsGranted == false with the provider named in the reason — for a
caller who holds Admin there. Entitled, and still not granted; unanswerable, and still not "denied".
Negative control, measured: with the terminal disarmed on this branch, the stalled partition's
wait does not complete at all — TimeoutException at StalledPolicyReadFailsClosedTest.cs:107
after 1 m 13 s. That is the hang, on this code, reproduced.
Existing contract tests re-run green: MeshQueryMergeContractTest, MeshQueryTransientRetryTest,
ColdEmptyInitialIsNotCachedTest, PathResolutionCachePoisonTest,
PathResolutionFloorIsNotAnAnswerTest — 39/39.
MeshWeaver.Documentation.Test 604/604 (link integrity, topic map, and the guard that caught my
own await <observable> in the first draft of the policy test).
Full solution: dotnet build -c Release -warnaserror → 0 Warning(s), 0 Error(s).

The dependent (break shape 7 — no gate sees this)

Swept MeshWeaver.Plugins@origin/main read-only. It carries NO core pin any more — every run
resolves the newest sealed main-cd set via resolve-platform.py, MW_PLATFORM_REF is unset, and
scripts/platform-floor.json is a floor only. So there is no pin to bump and this change reaches
Plugins automatically on the next seal. What that sweep found:

No custom IPermissionEvaluator, no ObserveScopePolicies call site, no RlsNodeValidator
override, no NodeRejectionReason.Unavailable use, and no Subscribe(onNext) without onError on
a mesh-query chain. The absences are load-bearing here.
No Plugins test asserts that a query must hang — every "hang" test asserts the opposite.
ContentGateUndeterminedTest already asserts a faulted and a silent evaluator both yield 503,
not 404, and PermissionSwallowRatchetGuard fails their build on
CheckPermission(...).Catch(→ verdict). Plugins lands on the compatible side.
Two Plugins tests are at risk and need the Plugins half:
Hosting.Monolith.Test/StarvedPermissionReadTest is written around the old behaviour (its class
doc says the leg "never errors … so it can only STALL"); its assertions (Unavailable, error
contains "could not be established") still hold — UnestablishedCheck produces that text on the
fault arm too — but its 20 s MeshOperationOptions.Timeout now contracts to a 5 s rung, so its
prose and elapsed expectations want updating. CompileSourceSnapshotWedgeTest becomes
timing-sensitive (its Release() may land after the terminal).
Two Plugins doc/behaviour sites go stale: Store/Core/Source/MeshQueries.cs:84-87 documents
relying on the hang ("waiting writes nothing, guessing rewrote everything") — the invariant it
protects survives (an error writes nothing either), but the paragraph is now wrong; and
Store/Catalog/Source/StoreCatalogLayoutAreas.cs:776-786,841 .Catch(→ empty/Anonymous) will
render a confident wrong entitlement ("Get" for a purchased plugin) where a stall used to show a
spinner.
LogIncidentControlPlane's unbounded incident watch will now retry once a minute on a permanently
stalled provider and escalate at 5 min — new, bounded, visible noise.

I did not change Plugins. Core lands first; the Plugins half is the two test updates, the two doc
paragraphs and the catalog .Catch arms, and it is specified above.

What I did NOT establish

Nothing here was exercised against PostgreSQL or a multi-silo Orleans mesh. Both new tests run
on an in-memory monolith, so what is measured is the fan-in, the fold and the classifiers — not a
real partitioned provider's timing. Plugins' PostgreSqlPartitionedMeshQuery has Matches() => true
and runs its cross-schema fan-out inside an IIoPool leaf, so its Initial includes pool queue wait;
whether 15 s is comfortable for a partitions:all union on a loaded pod is not measured here.
The 15 s default is calibrated against the healthy worst case recorded in the probe's own
comment ("single-digit seconds"), not against a fresh measurement of the deployed fleet.
I did not audit test/ outside core, nor the satellite repos' in-mesh C# (invisible to any
dotnet build). BlazorHostingExtensions.AllowContentRead (Plugins, A freshly uploaded content file 503s for minutes (10s timeout per request) before it starts serving — /api/content settle window #2901) still launders a
non-verdict into a 404 — fail-closed, wrong message, and out of this repo.
The MeshHubDisposalLeakTest TimerQueue shape was reasoned about, not re-measured: the arm is
released on the answer, so the rooting window is the window in which the query has no answer, which
is strictly shorter than the old log-only probe's. That probe SKIPs on macOS.
Addresses to recycle after the deploy: none. Nothing in this diff is served by a per-node hub's
bound configuration — it is framework src/ code that ships with the image.

🤖 Generated with Claude Code

MeshQuery.MergeProviderObservables gates its merged Initial on EVERY provider. A provider that neither emits, completes nor errors starved that gate for ever and the consumer hung with no error, no consumer-side log line and nothing to grep. The only trace was InitialStallProbe's warning, which deliberately only logged: measured on memex over the 400 minutes to 2026-09-21T04:12Z, 200+ of those warnings and 95 `No MeshNode emitted for` faults (~14/hour), and nothing acted on any of them. The merge now TERMINATES such a query with QueryProviderStalledException, naming the providers that did not answer. Policy `query-fanin-stall-terminal`. Why the consumers needed no change: the seeding rule in AccessControl.md had already forced them to classify a fault correctly, because no permissive seed of the permission fold is anything but a hole. CheckPermissionOutcome answers Undetermined (IsGranted=false, ErrorType.Unavailable), RlsNodeValidator answers NodeRejectionReason.Unavailable, AnonymousGate projects to false, and EvictFaultedQuery stops a Replay(1) latching one terminal for the process's life. The terminal is the input they were built for and never received. The budget is DERIVED, and it had to be. The probe's delay was a hard-coded 20s — exactly PermissionEstablishmentBudget at the default. Promoting that constant as-is would have recreated #1198 precisely: equal is not an ordering, so the fan-in could never fire first and the attribution it alone can produce would have been lost. QueryInitialBudget is rung 4, Nest(PermissionEstablishmentBudget), 30/25/20/15s at the default — and it is passed EXPLICITLY to the null-hub IMeshQueryCore registration, which is what the security fold reads through. Four consumers that a terminal turns from latent into routine, each fixed here: MeshDataSource.HandleRunTests had a one-arm Subscribe on the fan-in (Rx rethrows on the pool thread and the request is never answered); UserIdentityCache opened its directory read once over a Publish() so one terminal made the portal's user directory Unavailable for the life of the process with no repair path at all; EventSubscriptionRunner.WatchTriggerNodeType left its watch registry entry behind on a terminal so the watch was dead for good; and MeshNodeBindingExtensions.Exists promised a self-heal a terminal cannot deliver. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The Plugins-side findings of this change set are an investigation finding, so their durable home is the doc page — a PR body is a pointer, not a substitute. Names the two tests written around the old behaviour, the two doc/behaviour sites that go stale, and the one place (the store catalog's entitlement .Catch) where the terminal makes a user-visible statement worse rather than better. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved cache correctness, waiter termination, configuration propagation, and error-information exposure issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

Stale indexes allow lookups during query rehydration · New

What changed in this PR

This PR makes stalled query-provider fan-ins terminate with attributed failures instead of hanging indefinitely.

Changes:

Adds a derived query budget and QueryProviderStalledException.
Updates consumers, caches, and authorization handling.
Adds tests and architecture documentation.

File	Summary
`test/MeshWeaver.Hosting.Test/QueryFanInStallIsTerminalTest.cs`	Tests terminal fan-in behavior.
`test/MeshWeaver.Graph.Test/StalledPolicyReadFailsClosedTest.cs`	Tests fail-closed policy reads.
`src/MeshWeaver.Mesh.Contract/Services/MeshOperationOptions.cs`	Adds the derived query budget.
`src/MeshWeaver.Mesh.Contract/MeshNodeBindingExtensions.cs`	Documents terminal binding behavior.
`src/MeshWeaver.Layout/AreaErrorClassifier.cs`	Classifies provider stalls as unavailable.
`src/MeshWeaver.Hosting/Persistence/Query/MeshQuery.cs`	Terminates stalled provider fan-ins.
`src/MeshWeaver.Hosting/Persistence/PersistenceExtensions.cs`	Passes options to query registrations.
`src/MeshWeaver.Hosting.AspNetCore/Portal/UserIdentityCache.cs`	Reopens faulted directory queries.
`src/MeshWeaver.Graph/Security/RlsNodeValidator.cs`	Reports stalled providers as unavailable.
`src/MeshWeaver.Graph/MeshDataSource.cs`	Handles query failures in responses.
`src/MeshWeaver.Graph/EventSubscriptionRunner.cs`	Evicts faulted watches.
`src/MeshWeaver.Documentation/Data/Architecture/QueryFanInStallTerminal.md`	Documents the new policy.
`src/MeshWeaver.Documentation/Data/Architecture/PolicyNotProse.md`	Registers the policy.
`src/MeshWeaver.Documentation/Data/Architecture/AccessControl.md`	Updates access-control guidance.
`src/MeshWeaver.Documentation/Data/Architecture.md`	Adds the architecture documentation entry.
`src/MeshWeaver.Data.Contract/QueryProviderStalledException.cs`	Defines the terminal exception.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

rbuergi · 2026-09-21T06:37:59Z

+        Volatile.Write(ref _subscriptionFailure, null);
+        Volatile.Write(ref _hydrated, false);


Correct, and checking it surfaced a second, worse ordering bug that is mine rather than pre-existing. Both fixed in the next push.

Your finding. Classify answers Found(hit) on line 1, before it consults hydrated or subscriptionFailure — a hit outranks both by design. So resetting _hydrated on re-open signals "not authoritative" to every reader except the one path that matters here, and a deleted or renamed user is served as Found off the dead chain's last snapshot until the replacement Initial lands. Note this also means your parenthetical alternative — retaining the unavailable state — would not have closed it: a hit still wins over subscriptionFailure. Clearing the indexes is the fix, and it is the one Apply already performs for a Reset, which is the same event (an authoritative index being rebuilt).

The worse one, which you got me to look at. My re-open wrote _subscriptionFailure = null before _hydrated = false. A Lookup interleaving between those two reads failure == null, hydrated == true over the OLD index — so a MISS answers a definitive Unknown, i.e. "no such user", which is precisely the false actionable verdict #974/#637 exist to prevent and the input that drives onboarding and provisioning. That is strictly worse than the stale Found you named, and it was reachable on every re-open.

The fix adopts Apply's own "down first, rebuild, then up again LAST" discipline verbatim: _hydrated = false -> clear _byEmail/_byId -> _subscriptionFailure = null. In that order the worst a concurrent lookup can observe is Unavailable, never a false definitive answer in either direction, and UntilDetermined turns that into a pending question that the replacement chain's first snapshot resolves. EmailOf needs nothing extra — it already gates on _hydrated.

One thing worth recording, because it changes how alarming this reads: on main the stale hit is served for the life of the process (the chain is never re-opened, so _subscriptionFailure stays set and _byEmail stays frozen at the last snapshot forever). This change makes the window bounded by one query round-trip — so it is an improvement that was still not clean, which is what you caught.

github-actions · 2026-09-21T06:40:01Z

Test Results (shard 0)

1 files ±0 1 suites ±0 1m 52s ⏱️ -47s
300 tests ±0 300 ✅ ±0 0 💤 ±0 0 ❌ ±0
302 runs ±0 302 ✅ ±0 0 💤 ±0 0 ❌ ±0