Repository navigation
PostgreSqlPartitionedMeshQuery stalls past the 15s query fan-in Initial budget, faulting MessageHubGrain activations (Admin/Build) #5393
Description
What is failing
Four MessageHubGrain activations under Admin/Build/* (Radzen, ClientFReporting, Publish, Claims) faulted within a single second because PostgreSqlPartitionedMeshQuery never delivered an Initial frame to the query fan-in within the 15 s QueryInitialBudget. Each activation failed with QueryProviderStalledException, so those Admin/Build hubs were reported unavailable until the provider answered again.
Probable cause
Medium confidence, evidence points at the PostgreSQL partitioned query provider's cold/partitioned read path exceeding the 15 s rung-4 budget — not at the grain activation logic, which is only the consumer surface. The fan-in stall terminal was deliberately calibrated (see QueryFanInStallTerminal) against a recorded healthy worst case on an in-memory monolith, and that doc explicitly states the 15 s default was never measured against PostgreSQL or a multi-silo Orleans mesh. This burst is the first fleet evidence on that untested path. The same provider also feeds sibling incidents at other consumer sites (46836767a771863f ExceptionHandlerMiddleware, cd1aa68ed4d343f4 CompileWatcher, aecfa9f924fd5bf5 / dcd4ea38eb202663 other MessageHubGrain paths) and has a large separate UnanchoredQueryException history (d4c8f6f74ecfa422), so treat this as the provider stalling, surfaced by the new terminal. Whether the provider is genuinely stalling (DB latency, partition fan-out) or the budget is too tight for a cold PostgreSQL provider is the first question to settle.
Impact
Low so far: 4 occurrences, 1 ms window, one portal pod, and the failure is retryable (ErrorType.Unavailable, never a permission verdict). No data loss; affected Admin/Build hubs were unavailable for the stall window. Likely related incidents suggest a recurring provider-stall pattern worth one investigation covering all of them.
Where to look
MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery— itsInitialemission path for path-scoped queries (path:"Admin/Build/..."fan-outs).MeshQuery.MergeProviderObservables/InitialStallProbe— the terminal that raised it; policyquery-fanin-stall-terminal.MeshOperationOptions.QueryInitialBudget(15 s default) — validate against a real cold PostgreSQL provider before assuming the provider alone is at fault.- Possibly duplicate of / related to the other
QueryProviderStalledExceptionincidents above; a human may fold them into one provider-level ticket.
Do not bump the consumer timeout — per the policy, a stalled provider must be fixed, not the budget widened.
Evidence
| Fingerprint | 23624cd171bed2bf |
| Category | MeshWeaver.Hosting.Orleans.MessageHubGrain |
| Severity | Error |
| Exception | MeshWeaver.Mesh.QueryProviderStalledException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-857546649-2c25b |
| Occurrences | 4 |
| First seen | 2026-09-22 18:53:19Z |
| Last seen | 2026-09-22 18:53:19Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.MessageHubGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-22 18:53:19Z memex-portal-deployment-857546649-2c25b fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
[ACTIVATE] Grain Admin/Build/Radzen: activation faulted for Admin/Build/Radzen
MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:"Admin/Build/Radzen"|"Admin/Build"|"Admin"' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 18:53:19Z memex-portal-deployment-857546649-2c25b fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
[ACTIVATE] Grain Admin/Build/ClientFReporting: activation faulted for Admin/Build/ClientFReporting
MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:"Admin/Build/ClientFReporting"|"Admin/Build"|"Admin"' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 18:53:19Z memex-portal-deployment-857546649-2c25b fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
[ACTIVATE] Grain Admin/Build/Publish: activation faulted for Admin/Build/Publish
MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:"Admin/Build/Publish"|"Admin/Build"|"Admin"' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 18:53:19Z memex-portal-deployment-857546649-2c25b fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
[ACTIVATE] Grain Admin/Build/Claims: activation faulted for Admin/Build/Claims
MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:"Admin/Build/Claims"|"Admin/Build"|"Admin"' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
Opened automatically from Admin/_LogIncident/23624cd171bed2bf. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 23, 2026
Duplicate of #5315, and the same episode as #5390. Evidence: #5315 (comment)
This is the same pod (
857546649-2c25b), 3.5 minutes after the #5390 burst, inside the ThreadPool-starvation episode recorded by #5388/#5389/#5391. It is the same site (theMessageHubGrain[ACTIVATE]ancestor path-resolution query) with the same provider set (PostgreSqlPartitionedMeshQueryonly, becauseDefersToNativeProvideris true forAdmin/Build/*). The reads were trivial single-schemaINlists. Raising the budget is not the fix. The post-roll verification of #5327 is tracked on #5315.systemorph-com commentedon Sep 23, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 26 more since the last update (30 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhncRecent log lines
systemorph-com commentedon Sep 26, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 105 more since the last update (135 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhnc,memex-portal-deployment-76cf776847-2x9rqRecent log lines
systemorph-com commentedon Sep 26, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
5e34fe5b952f42d0(MeshWeaver.Hosting.Orleans.MessageHubGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.dea4d1b649c6a1c15e34fe5b952f42d0MeshWeaver.Hosting.Orleans.MessageHubGrainMeshWeaver.Mesh.QueryProviderStalledException[ACTIVATE] Grain Manufacturing: activation faulted for Manufacturingmemexmemex-portal-deployment-56fbdcd48f-sz5stAdmin/_LogIncident/dea4d1b649c6a1c1What triage wrote about this fingerprint
What is failing
The
ManufacturingMessageHub grain faulted during activation becauseMeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuerydid not emit its Initial snapshot within the query fan-in's 15 s bound. Because the merged Initial gates on every provider, the hub had no snapshot to serve and the query was reported as unavailable (retryable) — an availability failure, not a permission verdict.Probable cause
Most likely a transient PostgreSql slowdown or connection stall (medium confidence): the ops log store shows many bursts of
PostgreSqlPartitionedMeshQuerystall entries (e.g. 2026-09-21 04:08–04:10) and multiple related incidents — PostgreSQL health checks failing withTaskCanceledException/OperationCanceledExceptionon 2026-09-22/23 and LISTEN-connection reconnects — suggesting the database is intermittently slow or unreachable. No in-code defect is evident from a single occurrence; the consumer's timeout should not be bumped.Impact
One occurrence on one portal pod (
memex-portal-deployment-56fbdcd48f-sz5st), for theManufacturingspace as usersystem-security. The failure is retryable, so impact is a one-off transient unavailability of that space's data, not an outage.Where to look
MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery— its Initial-emission path / PostgreSQL connection and query latency.Admin/_LogIncident/23624cd171bed2bf(same log site, generic path — this is itsManufacturinginstance),Admin/_LogIncident/46836767a771863f,Admin/_LogIncident/64b60a7e471950fc, and the PostgreSQL health-check incidents.Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-56fbdcd48f-sz5stRecent log lines
systemorph-com commentedon Sep 26, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 359 more since the last update (494 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhnc,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-76cf776847-9p7z4Recent log lines
systemorph-com commentedon Sep 26, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 5 more since the last update (6 total).
memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-4dpcnRecent log lines
systemorph-com commentedon Sep 27, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 3 more since the last update (497 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhnc,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-555d9cd496-qntshRecent log lines
systemorph-com commentedon Sep 27, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 5 more since the last update (499 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhnc,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-555d9cd496-qntshRecent log lines
systemorph-com commentedon Sep 27, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 411 more since the last update (910 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhnc,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-555d9cd496-qntshRecent log lines
systemorph-com commentedon Sep 27, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
5e34fe5b952f42d0(MeshWeaver.Hosting.Orleans.MessageHubGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.b90f51c17e5d5bfd5e34fe5b952f42d0MeshWeaver.Hosting.Orleans.MessageHubGrainMeshWeaver.Mesh.QueryProviderStalledExceptionmemexmemex-portal-deployment-648bf84cf7-vs6fgAdmin/_LogIncident/b90f51c17e5d5bfdWhat triage wrote about this fingerprint
Likely duplicate. This is the same failure as #5531 (incident
Admin/_LogIncident/df1ef8b39a2d8601: both providers stall) and #5393 (incidentAdmin/_LogIncident/23624cd171bed2bf:PostgreSqlPartitionedMeshQuerystalls, 910 occurrences). It got a new fingerprint because this burst carried the exception message as its detail, and it came from thememexdeployment rather thanmemex-cloud. A human can probably close it against #5531.What is failing
A
MessageHubGrainactivation for a nested chat thread underPortfolio/_Thread/...faulted. The activation's ancestor-path query (sixpath:alternatives, running assystem-security) got noInitialfrom eitherStorageAdapterMeshQueryProviderorPostgreSqlPartitionedMeshQuerywithin the fan-in's 15s bound. The fan-in raisedQueryProviderStalledExceptionas designed, and the grain was reported unavailable (retryable).Probable cause
Medium confidence: a shared stall in the provider/PostgreSQL layer, not in the grain or the fan-in. Both providers stalled together, which is the #5531 pattern and points at a shared downstream dependency (database latency or pool pressure). The exception itself says the fix belongs in the stalled provider and that the budget should not be raised. This single sample has no provider-side stack, so it can't identify the query or lock.
Impact
Low for this incident: one occurrence on one pod. A thread reply (likely an agent's sub-thread) may have failed once and should succeed on retry. There is no data loss and no access decision was involved. The combined impact belongs on the parent issues, where the pattern keeps recurring across deployments. This incident adds evidence that the
memexdeployment is affected too.Where to look
MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQueryandMeshWeaver.Hosting.Persistence.Query.StorageAdapterMeshQueryProvider: how each emitsInitialfor multi-alternativepath:ancestor queries.memexportal pod at the time of the burst.Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-648bf84cf7-vs6fgRecent log lines
systemorph-com commentedon Sep 28, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (7 total).
memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-5c476fdbf8-55v9jRecent log lines
systemorph-com commentedon Sep 29, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-648bf84cf7-vs6fg,memex-portal-deployment-5dc5995596-4mvj6Recent log lines
systemorph-com commentedon Sep 29, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
5e34fe5b952f42d0(MeshWeaver.Hosting.Orleans.MessageHubGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.676074c5787c54de5e34fe5b952f42d0MeshWeaver.Hosting.Orleans.MessageHubGrainMeshWeaver.Mesh.QueryProviderStalledExceptionmemexmemex-portal-deployment-5dc5995596-4mvj6,memex-portal-deployment-5dc5995596-knmbkAdmin/_LogIncident/676074c5787c54de_Threadsatellite reads starve the query fan-in Initial gate and fault grain activation — recurrence of Feedback#81What triage wrote about this fingerprint
Duplicate note. This is a recurrence of the provider-side Initial-stall defect already filed as MeshWeaver.Feedback#81 (handed over to GitHub shortly before this burst fired). Systemorph/MeshWeaver.Plugins#2790 is open; a human can fold this recurrence into it in seconds. Filed anyway so the recurrence is on record.
What is failing. When
MessageHubGrainactivates for a deep thread-cell path, none of the registered query providers emit anInitialwithin the query fan-in's 15s bound — the exception names bothStorageAdapterMeshQueryProviderandPostgreSqlPartitionedMeshQuery. The mergedInitialgates on every provider, so path resolution has no snapshot, the grain's activation faults, and the affected node reads as unavailable (retryable) to users and MCP.Probable cause — high confidence this is the mechanism documented in Feedback#81, plus one new wrinkle.
StorageAdapterMeshQueryProviderdefers scoped satellite reads (a_-prefixed path segment, per the Systemorph/MeshWeaver.Feedback#20 defer-to-native gate) to the pedestrianWalkAdapter(basePath, Subtree)+ per-pathReadwalk, andCollectMatchedends in aConcat().ToList()— nothing is emitted until every read completes, so one incomplete read starves the all-providers Initial gate. Every failing query in this burst is exactly that shape (_Threadcells under a user thread, and under this incident's own triage thread), matching Systemorph/MeshWeaver.Plugins#2790's satellite-read class and confirming it is broader than its originalAccessAssignmentsample. Wrinkle: the stall now co-namesPostgreSqlPartitionedMeshQuery, which Systemorph/MeshWeaver.Plugins#2790's original probe did not — either the terminal 15s bound catches that provider mid-flight, or it has its own slow Initial on_Thread-scoped reads; worth checking beside the storage adapter. Why the walk never completes (load vs. structurally non-terminating read) remains unestablished, as in Systemorph/MeshWeaver.Plugins#2790.Impact. A small burst on the memex deployment (counts, pod and timestamps in the footer): each fault makes the affected node read as unavailable until a retry succeeds, and the same exception also faulted the creation of this incident's own triage thread, so ops/admin-plane reads are not immune. The same mechanism historically ran at ~95 activation faults per 400 minutes on this deployment (see #1186/#81's measurements), so the recurrence rate, not this burst's size, is the number to watch. Bounded to scoped satellite reads, fail-closed, retryable, no wrong data served — not release-blocking on current evidence.
Where to look. Code sites as named in Systemorph/MeshWeaver.Plugins#2790's handover (source not re-read in this triage):
MeshQuery.MergeProviderObservablesand the fan-in bound that throwsQueryProviderStalledException(src/MeshWeaver.Hosting/Persistence/Query/MeshQuery.cs);StorageAdapterMeshQueryProvider—DefersToNativeProvider/CollectMatched/FindMatchingNodes(src/MeshWeaver.Hosting/Persistence/Query/StorageAdapterMeshQueryProvider.cs); thePostgreSqlPartitionedMeshQueryInitial path on scoped reads; consumer context inPathResolutionService.ResolveSegmentsCore(src/MeshWeaver.Hosting/PathResolutionService.cs) andMessageHubGrain(FirstNodeResolutionTimeout). Related: core #1186 (symptom history; consumer side fixed by PR #5042, which is why this now surfaces as a 15sQueryProviderStalledExceptionrather than the old 30s Amb timeout), #4557 (SilentProviders, the handled sibling violation), Systemorph/MeshWeaver.Feedback#20 (the defer-to-native gate that keeps these reads on the pedestrian walk).Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 3 more since the last update (3 total).
memex-portal-deployment-5dc5995596-4mvj6,memex-portal-deployment-5dc5995596-knmbkRecent log lines
systemorph-com commentedon Oct 2, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
5e34fe5b952f42d0(MeshWeaver.Hosting.Orleans.MessageHubGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.3d690e41dc5e5dfc5e34fe5b952f42d0MeshWeaver.Hosting.Orleans.MessageHubGrainMeshWeaver.Mesh.QueryProviderStalledExceptionmemexmemex-portal-deployment-7fc8c5f599-q4d8dAdmin/_LogIncident/3d690e41dc5e5dfcWhat triage wrote about this fingerprint
Likely duplicate — the provider-stall family. Same log site and exception, and the same two stalled providers, as the umbrella #5393 (incident
Admin/_LogIncident/23624cd171bed2bf) and its recurrence on this samememexdeployment (#5531, incidentAdmin/_LogIncident/676074c5787c54de); #5315 is the open provider-side investigation. A human can probably close this against #5393/#5315 in seconds.What is failing
A
MessageHubGrainactivation on thememexdeployment faulted for a freshly created compile activity node (Hosting/TriageStatus/_Activity/compile-20261002183129…, whose id embeds a creation timestamp 26 s before the fault). The activation's ancestor-path query received noInitialfrom eitherMeshWeaver.Hosting.Persistence.Query.StorageAdapterMeshQueryProviderorMeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuerywithin the query fan-in's 15 s bound, so the fan-in raisedQueryProviderStalledExceptionas designed and the grain was reported unavailable (retryable).Probable cause
Medium confidence: the same shared provider/PostgreSQL stall tracked under #5315/#5393 — not a defect in the grain or the fan-in, which are only the consumer surface. The provider set in the message is set by the query shape, not by the fault: the path's
_-prefixed segment (_Activity) makesStorageAdapterMeshQueryProvider.DefersToNativeProvideranswer false, so both providers really read and both are named — the exact pattern QueryFanInStallTerminal documents for this shape. The trigger here fits the write-then-activate wave the same doc describes (a new node's activation issuing its Initial amid the pool contention behind a compile), but no provider-side stack exists in this single sample, so the precise queue this time is not establishable from this incident alone. Note thememexdeployment was still affected on 2026-09-29 (incident676074c5787c54de) after the live-requery coalescing and exact-probe batching fixes, so it may simply not yet carry them.Impact
Minimal for this incident: 1 occurrence, one pod, a single retryable availability fault on an automated TriageStatus activity node — no data loss, no permission verdict, no user-visible surface. The systemic weight belongs on the parent issues; what this occurrence adds is one more reading that the stall family persists on
memexa week after the last one.Where to look
MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQueryandMeshWeaver.Hosting.Persistence.Query.StorageAdapterMeshQueryProvider—Initialemission for multi-alternativepath:ancestor queries; the parent investigation lives in MessageHubGrain activations fault when query providers stall past the 15s fan-in bound (access/entitlements queries unavailable) #5315/PostgreSqlPartitionedMeshQuery stalls past the 15s query fan-in Initial budget, faulting MessageHubGrain activations (Admin/Build) #5393.pg-read:pool reading, and the fleet history behind this diagnosis.MeshWeaver.Hosting.[MessageHub].MessageHubGrainactivation — consumer surface only; the faulted activation is the designed outcome, and per policy the stalled provider must be fixed, never the consumer's timeout widened.memexdeployment carries the coalescing (fix(query): a live query re-reads once per burst of changes, not once per change #5615) and exact-probe batching fixes at all, pluspg-read:pool / PostgreSQL telemetry for the 18:31:55Z window on the pod named in the evidence footer.Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-7fc8c5f599-q4d8dRecent log lines
systemorph-com commentedon Oct 3, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (8 total).
memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-5c476fdbf8-55v9j,memex-portal-deployment-6bd968f77b-lvxctRecent log lines
systemorph-com commentedon Oct 4, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
5e34fe5b952f42d0(MeshWeaver.Hosting.Orleans.MessageHubGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.3ea4ac0c67c69d565e34fe5b952f42d0MeshWeaver.Hosting.Orleans.MessageHubGrainMeshWeaver.Mesh.QueryProviderStalledExceptionmemexmemex-portal-deployment-76dd69c7fc-xzsdlAdmin/_LogIncident/3ea4ac0c67c69d56What triage wrote about this fingerprint
What is failing.
MessageHubGrainactivation faults for every grain under the bug-triage threadHosting/Triage/_Thread/bug-systemorph-meshweaver-5935(the worker thread for issue #5935): the multi-path exact-path query into that thread subtree (path:"Hosting/Triage/_Thread/bug-systemorph-meshweaver-5935/{id}"|…, usersystem-security) receives noInitialfrom eitherStorageAdapterMeshQueryProviderorPostgreSqlPartitionedMeshQuerywithin the query fan-in's 15 s bound, so activation raisesQueryProviderStalledExceptionand the thread's cells are reported unavailable (retryable). The bug-triage automation that was re-driving issue #5935 at the time cannot read its own thread.Probable cause (medium confidence). This is a recurrence of the provider-stall family tracked under #5315 — same exception, same two stalled providers — in a shape not previously ticketed: explicit multi-path probes into a
_Threadsatellite subtree. The mechanism established by PR #6094 (in review, not yet deployed): on the partitioned Postgres backend the pedestrianStorageAdapterMeshQueryProviderserves scoped satellite reads itself, and itsListChildPaths+Readwalk cannot see satellite rows at all while flooding the one process-wide FIFOpg-read:pool (cap 16) with ~2×N admissions — every other read, including these exact-path probes, queues behind it and misses the 15 s bound. The burst lands minutes after the #5935 worker was re-driven (20:04Z) and while the #6094 review round was running, which fits the walk's head-of-line starvation; no provider-side stack is in the sample, so the precise queue this time is inferred from the mechanism, not measured.Impact. Confined but real: 17 grain-activation faults inside ~0.6 s on one pod (
memex-portal-deployment-76dd69c7fc-xzsdl), all in the #5935 triage thread's subtree — that thread is unavailable to the triage automation and to anyone opening it until providers answer again. No data loss, no permission verdict; the fan-in correctly fails as unavailable (retryable). The systemic weight belongs on the #5315 family; this ticket records that the family persists on memex and now blocks an active triage worker, not just interactive views.Likely duplicate of the #5315 family (see #6071, the sibling memex recurrence) — a human can fold this against the parent — but the shape (exact-path fan-in into a satellite
_Threadsubtree faulting at grain activation, stalling the bug-triage pipeline itself) is new evidence worth a line on the parent.Where to look.
src/MeshWeaver.Hosting/Persistence/Query/StorageAdapterMeshQueryProvider.cs(DefersToNativeProviderand the walk) andMeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery— both missed the Initial bound; the fix in review is PR #6094 (defer scoped satellites, bound the walk), which ships with the next image roll. QueryFanInStallTerminal has the budget ladder and the pg-read starvation analysis. The consumer,MeshWeaver.Hosting.Orleans.MessageHubGrainactivation, only surfaces the stall — per policy, fix the stalled provider, never the consumer's timeout.Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 17 more since the last update (17 total).
memex-portal-deployment-76dd69c7fc-xzsdlRecent log lines
systemorph-com commentedon Oct 4, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-648bf84cf7-vs6fg,memex-portal-deployment-5dc5995596-4mvj6,memex-portal-deployment-67c67b456d-l7dp7Recent log lines
systemorph-com commentedon Oct 12, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (911 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhnc,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-555d9cd496-qntsh,memex-portal-deployment-5b786c4466-4fs6cRecent log lines
systemorph-com commentedon Oct 15, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (9 total).
memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-5c476fdbf8-55v9j,memex-portal-deployment-6bd968f77b-lvxct,memex-portal-deployment-6d4df8d45f-ktwf2Recent log lines
systemorph-com commentedon Oct 15, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 4 more since the last update (915 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhnc,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-555d9cd496-qntsh,memex-portal-deployment-5b786c4466-4fs6c,memex-portal-deployment-6d4df8d45f-ktwf2Recent log lines
systemorph-com commentedon Oct 21, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 2 more since the last update (917 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhnc,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-555d9cd496-qntsh,memex-portal-deployment-5b786c4466-4fs6c,memex-portal-deployment-6d4df8d45f-ktwf2,memex-portal-deployment-649d598dc4-jlx5kRecent log lines
systemorph-com commentedon Oct 21, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
5e34fe5b952f42d0(MeshWeaver.Hosting.Orleans.MessageHubGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.9ede99d90750583e5e34fe5b952f42d0MeshWeaver.Hosting.Orleans.MessageHubGrainMeshWeaver.Mesh.QueryProviderStalledExceptionmemex-cloudmemex-portal-deployment-649d598dc4-jlx5kAdmin/_LogIncident/9ede99d90750583eWhat triage wrote about this fingerprint
Observed
A
MessageHubGrainactivation for the nodeHosting/PlatformBuilds/manufacturingfaulted withMeshWeaver.Mesh.QueryProviderStalledException: the providerMeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuerydid not emit anInitialwithin the query fan-in's 15 sQueryInitialBudget, for the activation's node-resolution querypath:"Hosting/PlatformBuilds/manufacturing"|"Hosting/PlatformBuilds"|"Hosting", run assystem-security. The query is the grain's ancestor-path read (MeshQuery.MergeProviderObservablesgates the mergedInitialon every provider; a faulted activation is the designed terminal, and the next access re-resolves).Note on the named provider: the path has no
_-prefixed segment, soStorageAdapterMeshQueryProvider.DefersToNativeProvideris true for it and that provider answers at once with an emptyInitial— only the leg that really reads Postgres is left to be named. PerDoc/Architecture/QueryFanInStallTerminal("The provider set in the message is set by the query shape, not by the fault"), the name reflects the query shape, not necessarily where the fault sits.Where and when
A single occurrence on a single pod of the memex-cloud portal deployment, one isolated line — no burst, no spread across pods. Nothing on the incident says what the pod was doing around it.
Impact
One hub activation was discarded; consumers of that read see a retryable
Unavailable, fail-closed, never a permission verdict. Judged from the numbers this is a transient availability blip on one node's hub, not an outage.Not established
Candidate causes and the reading that separates them
pg-read:Postgrespool (cap 16) behind a burst of other reads on the same pod. This is the mechanism established for the 2026-09-27→10-04 control-instance stalls of the same one-rowpath:Hosting/PlatformBuildsprobe shape (pedestrian satellite walk flooding the pool). Predicts: admission-heavy reads or routing back-pressure lines on the same pod in the minutes before the fault. Reading: a pod-scoped log query over ~02:30–02:50Z for routing-leg back-pressure,ThreadSupervisor.Scan/Admin-Thread sweep lines, and CompileWatcher faults..NET Thread Pool execution stalledor routing-leg starvation lines on the same pod in the same window. The same pod-scoped reading as (1) separates them.Known family
This is the same query shape and provider as #5315, #5390 and #5393 (grain activations timing out on ancestor-path
INreads) and the recurringpath:Hosting/PlatformBuildsprobe stalls recorded inDoc/Architecture/QueryFanInStallTerminal→ "Head-of-line starvation of pg-read by the pedestrian satellite walk"; the root investigation of why the underlying providers stall remains open under #5315. This ticket should be linked there — if the next reading confirms one of the candidates above on this pod, it is a recurrence of that family rather than a new defect.Where to look
Doc/Architecture/QueryFanInStallTerminal— esp. "The first fleet readings", "What the Initial queues behind", and "Head-of-line starvation of pg-read by the pedestrian satellite walk"MeshWeaver.Hosting.Persistence.Query.MeshQuery→MergeProviderObservables/InitialStallProbe(the terminal and the named-provider rule)MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery(the named provider) and the process-widepg-read:PostgrespoolLeft closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-649d598dc4-jlx5kRecent log lines
systemorph-com commentedon Oct 21, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 4 more since the last update (921 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhnc,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-555d9cd496-qntsh,memex-portal-deployment-5b786c4466-4fs6c,memex-portal-deployment-6d4df8d45f-ktwf2,memex-portal-deployment-649d598dc4-jlx5kRecent log lines
systemorph-com commentedon Oct 22, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (10 total).
memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-5c476fdbf8-55v9j,memex-portal-deployment-6bd968f77b-lvxct,memex-portal-deployment-6d4df8d45f-ktwf2,memex-portal-deployment-5d997855c9-scc52Recent log lines
systemorph-com commentedon Oct 22, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
5e34fe5b952f42d0(MeshWeaver.Hosting.Orleans.MessageHubGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.48b3a1fa45f178845e34fe5b952f42d0MeshWeaver.Hosting.Orleans.MessageHubGrainMeshWeaver.Mesh.QueryProviderStalledExceptionmemex-cloudmemex-portal-deployment-5d997855c9-p4t6qAdmin/_LogIncident/48b3a1fa45f17884What triage wrote about this fingerprint
Observed
MessageHubGrain(categoryMeshWeaver.Hosting.Orleans.MessageHubGrain) forHosting/PlatformBuildson the memex-cloud portal faulted during grain activation at 2026-10-22T21:23:33Z: the mesh querypath:"Hosting/PlatformBuilds"|"Hosting"(usersystem-security) terminated withMeshWeaver.Mesh.QueryProviderStalledExceptionbecause the providerMeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQueryemitted noInitialwithin the query fan-in's 15 s bound. Because the mergedInitialgates on every provider, the query had no snapshot and was terminated as unavailable (retryable) rather than answered — an availability failure, not a permission verdict and not a wrong answer.Where and when
This burst is one occurrence on one pod (
memex-portal-deployment-5d997855c9-p4t6q), one tenant namespace (Hosting), at a single instant. The fingerprint family is not new:Ops/Logsholds bursts of the sameQueryProviderStalledExceptionfrom the same provider on the memex instance on 2026-09-09, 09-10, 09-14, 09-21, 09-27 and 2026-10-07, each across multiple portal pods (e.g.-df7f9f,-fd958f,-6b6b44,-545bbd,-9cd496,-466887,-6674bc), and a 24 h census (Ops/Actions/verify-1186-memex-20261007-activation) read 432 matching lines in the window 2026-10-06T16:07 → 2026-10-07T16:07 UTC. Today's line is the first of this fingerprint since that census, 15 days later.Possibly already known
The 2026-10-07 census action is titled "#1186/#5800 — activation faults + fan-in stalls on memex, 24 h", which points at an existing issue (#1186) and PR (#5800) covering this symptom. Whether today's burst predates or postdates a fix from that work could not be read from this mesh — dedupe against #1186 before treating this as new; if it is the same defect recurring, this ticket adds the 2026-10-22 occurrence as evidence it was not resolved.
Impact
At the observed instant the
Hosting/PlatformBuildsmessage hub could not activate and its queries answered "unavailable"; the failure is retryable, so a consumer retrying would recover. Judged over the history inOps/Logs, this is a recurring, intermittent availability failure of the primary mesh-query path on the memex instance, not a one-off.Not established
PostgreSqlPartitionedMeshQuerywas doing while stalled (a slow DB round-trip, a pool wait, or never scheduled at all).Candidate causes and the reading that separates them
warn/faillines from categoryMeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQueryat the same instants (the 2026-09-09 and 09-14 bursts includefail:lines under that category) and DB-side wait evidence. Reading: the provider's own log lines in 2026-10-22T21:23:18–33Z and the full text of those earlierfail:lines — they name the concrete DB failure if this cause holds.path:"Hosting/PlatformBuilds"fails on demand while other queries on the same pod answer.Where to look
MeshWeaver.Hosting.Orleans.MessageHubGrain— the activation path that logs[ACTIVATE] Grain {path}: activation faulted.Initialgating inMeshWeaver.Meshthat raisesQueryProviderStalledExceptionafter the 15 s bound.MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery— the provider named in the exception.The framework source for these types was not readable from this mesh; the type names above come from the incident's own evidence.
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-5d997855c9-p4t6qRecent log lines
systemorph-com commentedon Oct 23, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1230 more since the last update (2151 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhnc,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-555d9cd496-qntsh,memex-portal-deployment-5b786c4466-4fs6c,memex-portal-deployment-6d4df8d45f-ktwf2,memex-portal-deployment-649d598dc4-jlx5k,memex-portal-deployment-8556887fd9-hwx6zRecent log lines
systemorph-com commentedon Oct 23, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 451 more since the last update (2602 total).
memex-portal-deployment-857546649-2c25b,memex-portal-deployment-5f95fb94dc-pkhnc,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-555d9cd496-qntsh,memex-portal-deployment-5b786c4466-4fs6c,memex-portal-deployment-6d4df8d45f-ktwf2,memex-portal-deployment-649d598dc4-jlx5k,memex-portal-deployment-8556887fd9-hwx6zRecent log lines