PostgreSqlPartitionedMeshQuery stalls past the 15s fan-in bound, faulting 108 MessageHubGrain activations in one 12s window

What is failing

MessageHubGrain activation faults with QueryProviderStalledException: the query fan-in received no Initial from PostgreSqlPartitionedMeshQuery within its 15s bound, so grain activations for unrelated workspaces (Pricing/Guideline, Reinsurance/Acceptance, Ifrs17/ReportingNode, Manufacturing/MorningBrief, Admin/Build/DoublePendulum, …) failed and their queries were answered as unavailable (retryable).

Probable cause

Backend stall, not a hub code defect: a single provider — the Postgres partitioned mesh query — stalled, and because the merged Initial gates on every provider, 108 distinct grain activations all failed within the same ~12-second window (18:49:44–18:49:56 UTC on 2026-09-22). A stall this broad and this synchronized points at the Postgres store (or the pod's connection to it) going unavailable/slow for that window, rather than at any of the affected nodes themselves. Confidence: medium — the exception names the stalled provider, but the underlying failure (DB latency, pool exhaustion, pod saturation) needs pod metrics and Postgres logs for that window to pin down.

Impact

A 12-second burst of 108 failures on one pod (memex-portal-deployment-857546649-2c25b), self-resolved. Retryable availability failures across many workspaces — user-visible while it lasts, but short-lived and on a single pod. Not release-blocking on its own.

Where to look

PostgreSqlPartitionedMeshQuery (PostgreSQL mesh query provider) and the query fan-in merge that gates on every provider's Initial. Per the exception's own guidance: fix the stalled provider; do not raise the consumer's 15s bound.

Duplicates

Same log site and exception as incident Admin/_LogIncident/aecfa9f924fd5bf5, filed earlier the same day as issue #5315 (that burst stalled both StorageAdapterMeshQueryProvider and PostgreSqlPartitionedMeshQuery; this one stalls only the Postgres provider — likely the same underlying infrastructure flakiness). This ticket is plausibly a duplicate of #5315 and can probably be closed as such once the Postgres connection health for 2026-09-22 18:49 UTC is reviewed. Related cross-references: thread-pool-stall incidents (1dbb05f503932ad3) and the UnanchoredQueryException fan-out incidents on the same Postgres provider (c02f1ff6724001de, d4c8f6f74ecfa422).

Evidence


Fingerprint	`dcd4ea38eb202663`
Category	`MeshWeaver.Hosting.Orleans.MessageHubGrain`
Severity	Error
Exception	`MeshWeaver.Mesh.QueryProviderStalledException`
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-857546649-2c25b`
Occurrences	108
First seen	2026-09-22 18:49:44Z
Last seen	2026-09-22 18:49:56Z
Routing	not determined — no configured route matches the category `MeshWeaver.Hosting.Orleans.MessageHubGrain`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-22 18:49:56Z memex-portal-deployment-857546649-2c25b fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
      [ACTIVATE] Grain Pricing/Guideline: activation faulted for Pricing/Guideline
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:"Pricing/Guideline"|"Pricing"' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 18:49:56Z memex-portal-deployment-857546649-2c25b fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
      [ACTIVATE] Grain Admin/Build/DoublePendulum: activation faulted for Admin/Build/DoublePendulum
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:"Admin/Build/DoublePendulum"|"Admin/Build"|"Admin"' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 18:49:56Z memex-portal-deployment-857546649-2c25b fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
      [ACTIVATE] Grain Reinsurance/Acceptance: activation faulted for Reinsurance/Acceptance
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:"Reinsurance/Acceptance"|"Reinsurance"' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 18:49:56Z memex-portal-deployment-857546649-2c25b fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
      [ACTIVATE] Grain Manufacturing/MorningBrief: activation faulted for Manufacturing/MorningBrief
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:"Manufacturing/MorningBrief"|"Manufacturing"' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 18:49:56Z memex-portal-deployment-857546649-2c25b fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
      [ACTIVATE] Grain Ifrs17/ReportingNode: activation faulted for Ifrs17/ReportingNode
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:"Ifrs17/ReportingNode"|"Ifrs17"' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

_{Opened automatically from Admin/_LogIncident/dcd4ea38eb202663. Recurrences are folded into this issue rather than opening new ones.}

Duplicate of #5315. Full evidence: #5315 (comment)

Same site. A MessageHubGrain [ACTIVATE] fault on the ancestor path-resolution query (path:"X"|"parent"|…, run as system-security), ended by the fan-in's stall terminal.

Same discriminator. A Postgres-backed read stalled while the pod was starved of ThreadPool threads. On this pod (857546649-2c25b), LocalSiloHealthMonitor logged Thread Pool execution stalled for 24.7s at 18:47:10Z (#5388), and routing legs waited for pool threads 18:47:40–18:49:57Z (#5389, #5391). This burst sits inside that window.

Why the provider list differs. Here only PostgreSqlPartitionedMeshQuery is named, while #5315 also names StorageAdapterMeshQueryProvider. That follows from StorageAdapterMeshQueryProvider.DefersToNativeProvider: these paths have no _ segment, so that provider answers an empty Initial at once. It is not a different fault.

The post-roll verification of #5327 is tracked on #5315.