PostgreSQL partitioned mesh query provider exceeds the 15 s Initial budget, faulting space compile watchers

What is failing

The newly-armed query fan-in stall terminal (QueryProviderStalledException, see QueryFanInStallTerminal) fired 88 times in a ~33-second window on the memex-cloud portal pod: MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery did not emit an Initial within the 15 s QueryInitialBudget for the CompileWatcher's nodeType:Code subtree queries across many spaces (AppleMaps/Maps, Hosting/Issue, AgenticBusiness/Menu, Edu/*, …). Each fault tears down that space's sources watcher, which then re-establishes — so the code watchers for affected spaces were unavailable for at least one 15 s budget window each.

Probable cause

The terminal itself is behaving as designed — it names the stalled provider and is retryable. What is stalling is the PostgreSQL partitioned provider. Two candidate explanations, in order of confidence:

The 15 s budget is miscalibrated for the real partitioned fleet. The design doc states explicitly that the budget was calibrated against the probe's recorded healthy worst case, never measured against PostgreSQL or a multi-silo Orleans mesh. A cold or load-spiked PostgreSqlPartitionedMeshQuery Initial that takes >15 s — connection-pool contention, partition fan-out, or a transient DB slowdown — is now surfaced as a hard error where it previously hung silently.
A genuine PostgreSQL-side stall (lock contention, slow partition query) that the doc's fleet measurement never caught because nothing surfaced it before.

Confidence in the location (the PostgreSQL provider, rung 4 of the timeout ladder): high — every sample names it as the sole non-answering provider. Confidence in which of the two mechanisms above: low — the burst's tight 33 s window on a single pod looks like one transient DB/pool event fanning out across all watchers, which favours (1)/(2) combined rather than a per-space defect.

Per policy query-fanin-stall-terminal, the fix belongs in the provider — do not raise the consumer's timeout to mask this.

Impact

88 faults across many spaces' compile watchers on one pod (memex-portal-deployment-5c444645f8-8pdzb) in ~33 s, all under user system-security. Self-correcting (each watcher re-establishes), so this is a bounded availability blip on the background compile pipeline, not an outage. Worth watching whether it recurs: a repeated pattern would mean the fleet's real PostgreSQL Initial latency sits at or above the budget, making the stall terminal fire routinely.

Where to look

MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery — the named stalled provider; start with its Initial emission path and its per-partition fan-out timing.
MeshQuery.MergeProviderObservables / MeshOperationOptions.QueryInitialBudget (rung 4, 15 s default) — QueryFanInStallTerminal, especially the "Not established" section: the budget was never measured against PostgreSQL or a multi-silo mesh. First action: measure real PostgreSqlPartitionedMeshQuery Initial latency on the deployed fleet and either fix what makes it slow or recalibrate the ladder.
MeshWeaver.Graph.CompileWatcher — the re-establishing consumer; its recovery worked as designed.

Not a duplicate: no existing incident or issue covers this component/symptom combination — this is the stall terminal's first production sighting against the PostgreSQL provider, and earlier PostgreSqlPartitionedMeshQuery log entries (Ops/Logs, 2026-09-09 → 09-14) are provider-side faults of a different shape, predating the terminal.

Evidence


Fingerprint	`cd1aa68ed4d343f4`
Category	`MeshWeaver.Graph.CompileWatcher`
Severity	Error
Exception	`MeshWeaver.Mesh.QueryProviderStalledException`
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-5c444645f8-8pdzb`
Occurrences	88
First seen	2026-09-22 14:43:02Z
Last seen	2026-09-22 14:43:35Z
Routing	not determined — no configured route matches the category `MeshWeaver.Graph.CompileWatcher`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-22 14:43:35Z memex-portal-deployment-5c444645f8-8pdzb fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on AppleMaps/Maps — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:AppleMaps/Maps/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 14:43:35Z memex-portal-deployment-5c444645f8-8pdzb fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Hosting/Issue — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Hosting/Issue/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 14:43:35Z memex-portal-deployment-5c444645f8-8pdzb fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on AgenticBusiness/Menu — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:AgenticBusiness/Menu/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 14:43:35Z memex-portal-deployment-5c444645f8-8pdzb fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Edu/LearningJourney — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Edu/LearningJourney/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 14:43:35Z memex-portal-deployment-5c444645f8-8pdzb fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Edu/Quiz — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Edu/Quiz/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

_{Opened automatically from Admin/_LogIncident/cd1aa68ed4d343f4. Recurrences are folded into this issue rather than opening new ones.}

Still occurring: 186 more since the last update (274 total).

Last seen: 2026-09-22 14:44:15Z
Pods: memex-portal-deployment-5c444645f8-8pdzb

Recent log lines

2026-09-22 14:44:15Z memex-portal-deployment-5c444645f8-8pdzb fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Ifrs17/CreditRiskRating — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Ifrs17/CreditRiskRating/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 14:44:15Z memex-portal-deployment-5c444645f8-8pdzb fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on ILS/Journal — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:ILS/Journal/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 14:44:15Z memex-portal-deployment-5c444645f8-8pdzb fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Ifrs17/ProjectionConfiguration — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Ifrs17/ProjectionConfiguration/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

Still occurring: 1715 more since the last update (1989 total).

Last seen: 2026-09-22 14:50:30Z
Pods: memex-portal-deployment-5c444645f8-8pdzb

Recent log lines

2026-09-22 14:50:30Z memex-portal-deployment-5c444645f8-8pdzb fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Hosting/Build — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Hosting/Build/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 14:50:30Z memex-portal-deployment-5c444645f8-8pdzb fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Claims/Claim — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Claims/Claim/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-22 14:50:30Z memex-portal-deployment-5c444645f8-8pdzb fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Ifrs17/Engine — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Ifrs17/Engine/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

Code lead against `main` (`39f49494eb` core, `e9d9283ab` Plugins): the 15 s clock includes the wait for a slot on a BOUNDED pool

The ticket gives two candidates: a miscalibrated budget, or a genuine PostgreSQL stall. Reading the code gives a third, and it fits this burst's shape better than either. 88 faults, one pod, ~33 s, every one a CompileWatcher nodeType:Code subtree query, and the pod …-5c444645f8-8pdzb had just been rolled in (the same pod as #5346's roll-coincident FleetWatch stall).

Where the clock starts. The single-provider path in MeshQuery (src/MeshWeaver.Hosting/Persistence/Query/MeshQuery.cs, ~L515–560) arms InitialStallProbe with OperationOptions.QueryInitialBudget (15 s at the production default) inside Observable.Create, i.e. at Subscribe.
Where the provider's work runs. PostgreSqlPartitionedMeshQuery (Plugins, ~L134 and ~L472) runs its fan-out as _ioPool.Invoke(async ct => …). _ioPool is the registry's IoPoolNames.FileSystem pool, which is bounded and shared with every other storage read on the pod.
So the probe measures queueing plus the query. When a freshly rolled pod starts one compile watcher per space and each subscribes its sources query at once, the leaves queue for pool slots. A leaf that waits more than 15 s for a slot reads as "provider [PostgreSqlPartitionedMeshQuery] did not emit an Initial" even if its SQL then runs in milliseconds.
The terminal can amplify it. Each fault tears down that space's sources watcher. The watcher re-establishes (per the ticket, "its recovery worked as designed") and re-subscribes, which re-queues the leaf behind the same wave.

That is consistent with one transient event fanning out across all watchers, which the ticket already suspects. It is also consistent with the policy it cites (query-fanin-stall-terminal: do not raise the consumer's timeout). Raising the budget would only move the threshold at which a boot wave reads as a stall.

What would decide it. The pool's queue depth and wait time on that pod between 14:28Z and 14:30Z, against the SQL time per leaf. If slots were exhausted and the leaves ran fast once they started, this is a boot-wave or attribution defect: the probe blames a provider for time spent waiting on the pool. It is not a PostgreSQL defect, and the fix is either to measure the provider from the moment its leaf starts, with pool starvation reported as its own named cause, or to stop the boot wave from subscribing every space's sources at once. Which of those is right is a design call on the stall terminal, so I am not opening a PR on a reading.

What I did NOT establish

Any measurement: pool depth, per-leaf SQL time, and whether PostgreSQL itself was slow at that moment. No Logs/Sample read was possible; the control instance's MCP answers 503 this session.
Whether the multi-provider path arms the same way. I read only the single-provider branch, which is what a one-provider PG portal takes.

Still occurring: 5 more since the last update (1994 total).

Last seen: 2026-09-23 13:26:53Z
Pods: memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-t9222

Recent log lines

2026-09-23 13:26:42Z memex-portal-deployment-5f95fb94dc-t9222 fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Underwriting/Guideline — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Underwriting/Guideline/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-23 13:26:53Z memex-portal-deployment-5f95fb94dc-t9222 fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Underwriting/CapacityCap — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Underwriting/CapacityCap/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-23 13:26:53Z memex-portal-deployment-5f95fb94dc-t9222 fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Hosting/DeploymentStatus — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Hosting/DeploymentStatus/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

Consolidated bot-filed duplicates of this defect (closed not_planned): #5530 — QueryProviderStalledException faulting the CompileWatcher sources watcher. The filer fan-out that minted them (no lookup of an existing issue for the same log site) is fixed in Systemorph/MeshWeaver.Plugins#2290.

Still occurring: 115 more since the last update (2109 total).

Last seen: 2026-09-23 14:45:13Z
Pods: memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-t9222, memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-777677694-hjkc8

Recent log lines

2026-09-23 13:34:49Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Hosting/Publication — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Hosting/Publication/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-23 13:34:49Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on TaskManagement/Task — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:TaskManagement/Task/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-23 14:45:13Z memex-portal-deployment-777677694-hjkc8 fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Store/Plugin — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Store/Plugin/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

Still occurring: 123 more since the last update (2232 total).

Last seen: 2026-09-23 18:52:36Z
Pods: memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-t9222, memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-777677694-hjkc8, memex-portal-deployment-6f497dc9b-r4nkg

Recent log lines

2026-09-23 18:52:36Z memex-portal-deployment-6f497dc9b-r4nkg fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Hosting/Publication — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Hosting/Publication/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-23 18:52:36Z memex-portal-deployment-6f497dc9b-r4nkg fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Reinsurance/AmountType — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Reinsurance/AmountType/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-23 18:52:36Z memex-portal-deployment-6f497dc9b-r4nkg fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on SocialMedia/Profile — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:SocialMedia/Profile/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

Why the Postgres Initial misses 15 s: code finding and fix, in review

The terminal still fires on images carrying #5327:

MessageHubGrain activations fault when query providers stall past the 15s fan-in bound (access/entitlements queries unavailable) #5315: 12 [ACTIVATE] faults, the latest at 2026-09-23 13:26:31Z on 5f95fb94dc-t9222.
PostgreSQL partitioned mesh query provider exceeds the 15 s Initial budget, faulting space compile watchers #5344: 2,109 CompileWatcher faults, including 13:26–13:34Z on 5f95fb94dc-* and 14:45Z on 777677694-hjkc8.

So moving compiles off the shared ThreadPool was not the whole cause.

Where the 15 s goes. MeshQuery arms the stall probe at Subscribe, so the budget also covers every queue the read waits in before its SQL runs. For an anchored query, PostgreSqlMeshQuery runs its leaf on the FileSystem pool (256 slots). Inside that slot, PostgreSqlStorageAdapter.QueryNodesAsync → ReadPooled waits for pg-read:Postgres. That pool is one per process, cap 16, and shared by every per-schema adapter.

The only reading of that pool's queue (ControlledIoPooling.md, memex.systemorph.com, 828 min uptime):

31.9 M admissions, about 640/s;
mean queue wait 342 ms;
48,122 waits over 1 s.

So the gate is saturated in steady state, before any boot wave. (This corrects the earlier claim that the leaves "bypass the pg-read: cap". They hold a FileSystem slot while they wait for one.)

One thing that fills it. Both per-schema providers re-ran a live query once per change notification, through an unbounded changeBuffer.Select(RunQuery).Concat() queue:

StorageAdapterMeshQueryProvider (core), which serves every _-segment path on this backend, so every path in MessageHubGrain activations fault when query providers stall past the 15s fan-in bound (access/entitlements queries unavailable) #5315;
PostgreSqlMeshQuery (Plugins), which serves every anchored query.

A burst of N writes therefore cost N back-to-back full reads, and the queue grows fastest exactly when the pool is contended. Plugins#2328 removed the same queue from the fan-out provider today (it had measured ~35 identical unions in 11 s).

Fix (the 15 s bound is unchanged). One re-query at a time; notifications that arrive while it runs fold into ONE follow-up, which reads every write they announced. There is no timer.

core: fix(query): a live query re-reads once per burst of changes, not once per change #5615 (CoalesceWhileRunning, StorageAdapterMeshQueryProvider);
Plugins: Systemorph/MeshWeaver.Plugins#2337 (PostgreSqlMeshQuery).

Each has a test that holds a re-query in flight, delivers 12 changes, and counts the reads: 2, versus 12 (core) and more than 2 within 3 s (PG), with the per-change queue restored. Full suites: core MeshWeaver.Hosting.Test 948/0 failed; MeshWeaver.Hosting.PostgreSql.Test 1216/0 failed. Write-up: Doc/Architecture/QueryFanInStallTerminal → What the Initial queues behind.

Not established

What share of the ~640 reads/s per-change re-queries account for. There is no per-query read census, and the control instance's Logs/Sample were unreachable this session.
Whether this alone keeps a pod start inside 15 s. The boot wave of Initials, and the CompileWatcher's 1 s re-establish after each fault, are untouched contributors.

Closes on (owner reading, after a roll carrying both halves): QueryProviderStalledException count within 15 min of pod start, at [ACTIVATE] and at Sources watcher faulted, compared with the 2026-09-23 counts above. That needs a Logs InstanceAction on the control instance.

Update: all three fixes are merged.

core fix(query): a live query re-reads once per burst of changes, not once per change #5615: e45b95d872
Systemorph/MeshWeaver.Plugins#2337: 2d0fe843c6
Systemorph/MeshWeaver.Plugins#2332: 3fa4bd36f5

They are not live yet. They reach memex and memex-cloud with the next release, seal and roll that carries them. The post-roll reading named above decides whether this closes.

Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet

What is running.

memex-cloud: /api/version returns 9a61fed964. Ops/Status/memex-cloud (06:42:00Z) shows 4/4 replicas on 3.0.0-ci.9263. CD run 9263 built core 9a61fed964 with Plugins 41bcfe968f.
memex: /api/version returns 39f49494eb. Ops/Status/memex shows the serving pods on 3.0.0-ci.9218, which is core 39f49494eb with Plugins e9d9283abe. A third pod, on ci.9260, is unready, so the roll is converged: false.

This issue's fix: core #5615 (e45b95d872) + Plugins#2337 (2d0fe843c6). It is not an ancestor of either running build (git merge-base --is-ancestor, exit 1).

Where it is. 3.0.0-ci.9287 (CD run 35956222018) is core bf5ac85526 with Plugins c4795a8d5b, and it contains this fix. The registry has sealed it. memex-cloud's self-updater logged [SelfUpdate] handing 3.0.0-ci.9287 to the control lane (https://memex.systemorph…) at 06:47Z. So the roll now waits on the control instance, which is an owner step and was not done here.

Still open. It closes on a post-roll reading once a ci.9287-or-later image serves the namespace this fingerprint fires in.

Closing: fix live, no recurrence for ≥ 6 h, positive control present

Image. memex-cloud (memex.meshweaver.cloud) serves 3.0.0-ci.9291. /api/version returns core 140c5fb7ae, and Ops/Status/memex-cloud (sampled 14:49Z) shows all 4 replicas on that tag, converged: true. The Plugins half is b3601fecbe: the portal-ai build job of CD run 35970157348 (job 107542528900) logs HEAD is now at b3601fe. The first pods of this ReplicaSet (memex-portal-deployment-589ff8f895) started at 08:41:05Z and 08:42:17Z. More pods of the same ReplicaSet, on the same image, joined later (gqsfn at 14:06Z, vw5jj at 14:10Z, and others). The control instance memex.systemorph.com still serves core 39f49494eb.

Window and method. I ran Logs InstanceActions on memex.meshweaver.cloud between 14:43Z and 14:54Z, with sinceMinutes 370/375, filtered to this ReplicaSet with | pod=~"memex-portal-deployment-589ff8f895-.+". That is ≥ 6 h of this image only, from 08:41Z to about 14:50Z. Coverage control: Ops/logs-memexcloud-20260924-close6h-c-bootcoverage (warm-up complete) lands 9 lines from 9 pods of this ReplicaSet. The earliest are ghgjt, vn49b and tp564 at about 08:51Z, so the window reaches back to the first boots.

Fix: core #5615 (e45b95d872) + Plugins#2337 (2d0fe843c6). It is an ancestor of the running build (git merge-base --is-ancestor, exit 0).

reading	query	lines
fingerprint	`\|~ "(?i)watcher faulted"` (`Ops/logs-memexcloud-20260924-close6h-f-srcwatcher`)	0
positive control (same logger category)	`\|~ "MeshWeaver.Graph.CompileWatcher"` (`Ops/logs-memexcloud-20260924-close6h-c-compilewatcher`)	≥ 10 (cut at limit 10)

Related, and not this fingerprint: 12 QueryProviderStalledException lines did land in the window (…-f-stalled). They are all on pod ghgjt between 14:17Z and 14:31Z, logged by ProvisionPlan, and they name StorageAdapterMeshQueryProvider on path:<space>/_Entitlements/<user> reads. None comes from CompileWatcher or names PostgreSqlPartitionedMeshQuery. I recorded them on #5315.

Closing as completed. If this shape recurs on an image that contains the fix, reopen and treat it as a new path. Readings were taken read-only through the memex MCP.

Left closed — the newest occurrence (2026-09-24 09:55:55Z) predates the close (2026-09-24 14:55:54Z), so this is a late update, not a regression.

Still occurring: 2 more since the last update (2234 total).

Last seen: 2026-09-24 09:55:55Z
Pods: memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-t9222, memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-777677694-hjkc8, memex-portal-deployment-6f497dc9b-r4nkg, memex-portal-deployment-56fbdcd48f-4dpcn

Recent log lines

2026-09-23 18:52:36Z memex-portal-deployment-6f497dc9b-r4nkg fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on SocialMedia/Profile — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:SocialMedia/Profile/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-24 09:55:55Z memex-portal-deployment-56fbdcd48f-4dpcn fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Store/Catalog — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Store/Catalog/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-24 09:55:55Z memex-portal-deployment-56fbdcd48f-4dpcn fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Store/Plugin — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Store/Plugin/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

Left closed — the newest occurrence (2026-09-24 10:04:11Z) predates the close (2026-09-24 14:55:54Z), so this is a late update, not a regression.

Still occurring: 1 more since the last update (2235 total).

Last seen: 2026-09-24 10:04:11Z
Pods: memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-t9222, memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-777677694-hjkc8, memex-portal-deployment-6f497dc9b-r4nkg, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st

Recent log lines

2026-09-24 09:55:55Z memex-portal-deployment-56fbdcd48f-4dpcn fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Store/Catalog — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Store/Catalog/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-24 09:55:55Z memex-portal-deployment-56fbdcd48f-4dpcn fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Store/Plugin — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Store/Plugin/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-24 10:04:11Z memex-portal-deployment-56fbdcd48f-sz5st fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Feedback/EvalCase — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Feedback/EvalCase/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

Folded in — a DIFFERENT fingerprint from the same log site.

This issue tracks the log site 0b63f818bef1705a (MeshWeaver.Graph.CompileWatcher), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.


Folded fingerprint	`c51d033118c2264a`
Log site	`0b63f818bef1705a`
Category	`MeshWeaver.Graph.CompileWatcher`
Severity	Error
Exception	`MeshWeaver.Mesh.QueryProviderStalledException`
Detail	Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in'{value}'path:{path} scope:children select:path'{value}'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
Namespace	`memex`
Pods	`memex-portal-deployment-7b99649c6f-khd2s`
Occurrences	1
First seen	2026-09-28 12:42:28Z
Last seen	2026-09-28 12:42:28Z
Already folded here	1 fingerprint(s)
Incident	`Admin/_LogIncident/c51d033118c2264a`
Triage called it	CompileWatcher late-release-adoption watcher faults when PostgreSQL partitioned query stalls past the 15 s Initial budget

What triage wrote about this fingerprint

What is failing

The CompileWatcher (MeshWeaver.Graph) has a [LateReleaseAdoption] watcher that lists a node type's releases (path:<Type>/Release scope:children select:path, run as system-security). On the memex deployment this query for Essentials/Mailbox got no answer: MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery did not emit an Initial within the 15 s query fan-in budget. The fan-in raised QueryProviderStalledException. The watcher faulted and then re-established itself. While the stall lasted, it could not see or adopt a new release for that type.

Probable cause

Likely duplicate: same root cause as #5344 and #5530. Those issues cover the same stalled provider faulting the CompileWatcher's sources watchers (namespace:<Type>/Source scope:subtree nodeType:Code). This incident is the same stall at a different watcher inside the same component. It is not a separate defect in the late-release-adoption path.

Confidence that the stall is in the PostgreSQL provider: high. The exception names it as the only provider that did not answer. Confidence in why the provider stalled: low. There is one sample and no provider-side log, so I can't tell a transient DB or connection-pool slowdown from a 15 s budget that is set too tight for PostgreSQL. QueryFanInStallTerminal says the budget was never measured against PostgreSQL. As the exception itself says, do not raise the consumer timeout.

This is also the first time the stall has been seen on the memex deployment. The earlier incidents were on memex-cloud. So the provider stall is not limited to one deployment.

Impact

Small. There was one occurrence on one pod, and the watcher retries on its own. At worst, adopting a new Essentials/Mailbox release was delayed by one budget window plus the time to re-establish. The only reason to keep this open is as more evidence for #5344/#5530. If those are fixed, this can be closed.

Where to look

MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery: the path that emits Initial for scope:children queries under a Release namespace. The fix is tracked in PostgreSQL partitioned mesh query provider exceeds the 15 s Initial budget, faulting space compile watchers #5344.
MeshQuery.MergeProviderObservables / MeshOperationOptions.QueryInitialBudget: see QueryFanInStallTerminal.
MeshWeaver.Graph.CompileWatcher: the [LateReleaseAdoption] watcher's fault/re-establish handling. It recovered here as designed.

Reopened — this fault is happening again.

Still occurring: 1 more since the last update (1 total).

Last seen: 2026-09-28 12:42:28Z
Pods: memex-portal-deployment-7b99649c6f-khd2s

Recent log lines

2026-09-28 12:42:28Z memex-portal-deployment-7b99649c6f-khd2s fail: MeshWeaver.Graph.CompileWatcher[0]
      [LateReleaseAdoption] Essentials/Mailbox faulted on Essentials/Mailbox — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Essentials/Mailbox/Release scope:children select:path' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

Still occurring: 1 more since the last update (2 total).

Last seen: 2026-09-28 12:51:55Z
Pods: memex-portal-deployment-7b99649c6f-khd2s

Recent log lines

2026-09-28 12:42:28Z memex-portal-deployment-7b99649c6f-khd2s fail: MeshWeaver.Graph.CompileWatcher[0]
      [LateReleaseAdoption] Essentials/Mailbox faulted on Essentials/Mailbox — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Essentials/Mailbox/Release scope:children select:path' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-28 12:51:55Z memex-portal-deployment-7b99649c6f-khd2s fail: MeshWeaver.Graph.CompileWatcher[0]
      [LateReleaseAdoption] Signature/DeepSignCredential faulted on Signature/DeepSignCredential — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Signature/DeepSignCredential/Release scope:children select:path' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

I read this issue's latest memex recurrences against the pod's LIVENESS heartbeats: #5315 (comment). At the stall minute, no heartbeat recorded ≥ 1,000 queued ThreadPool items. So this recurrence is not the pod-wide surge behind #5800/#5799. The 15 s Initial is being missed for another reason. The pg-read gate wait and Postgres lock waits at those minutes are the next readings, and I have not taken them. No bound was changed.

Still occurring: 2 more since the last update (4 total).

Last seen: 2026-10-04 20:14:58Z
Pods: memex-portal-deployment-7b99649c6f-khd2s, memex-portal-deployment-76dd69c7fc-xzsdl

Recent log lines

2026-09-28 12:51:55Z memex-portal-deployment-7b99649c6f-khd2s fail: MeshWeaver.Graph.CompileWatcher[0]
      [LateReleaseAdoption] Signature/DeepSignCredential faulted on Signature/DeepSignCredential — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Signature/DeepSignCredential/Release scope:children select:path' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-10-04 20:14:58Z memex-portal-deployment-76dd69c7fc-xzsdl fail: MeshWeaver.Graph.CompileWatcher[0]
      [LateReleaseAdoption] Store/Publishing faulted on Store/Publishing — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Store/Publishing/Release scope:children select:path' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-10-04 20:14:58Z memex-portal-deployment-76dd69c7fc-xzsdl fail: MeshWeaver.Graph.CompileWatcher[0]
      [LateReleaseAdoption] Store/InstallRequest faulted on Store/InstallRequest — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:Store/InstallRequest/Release scope:children select:path' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

Back off sources-watcher re-establish after a query provider stall (R…

docs(compiler): name the reads the stall backoff covers, at the wrap …

Merge pull request #6182 from Systemorph/bugfix/systemorph-meshweaver…

Every #5344 occurrence I could read is on a GC-thrash pod: 8pdzb 09-22, t9222/pkhnc/hjkc8/r4nkg 09-23 (78–81 % GC, heap 10.7–11.6 GiB), and 4dpcn 09-24 (16 % GC, 12.9 s gaps, heap 9.7 GiB). The 09-28 12:42 khd2s recurrence (memex) was not in my reading set. These close with the retention fix (#6261 + Plugins#3109). Table: #5315 (comment)

Consolidated into #5315, which tracks the provider-stall family and keeps its per-occurrence split table.

This issue's consumer, CompileWatcher, re-establishes on the stall terminal as designed, so the issue carries no consumer defect of its own. Its occurrences sort into two groups. Most were on GC-thrash pods (8pdzb, t9222, pkhnc, hjkc8, r4nkg, 4dpcn); those close with the retention fix tracked on #5800 (#6261 + Plugins#3109). xzsdl (10-04, heap 11.6 GiB, GC 2.4 %) is mixed. khd2s (09-28) was never read. Both remaining cases belong to #5315's healthy-pod half.

Readings on the current memex image. memex runs 3.0.0-ci.10310 (core 8b791643) on all 3 replicas, booted 11:15–11:44Z today (Ops/Status/memex, 14:32Z). That image carries the #6260 IoPool census, which #5315 needs to split Postgres time from slot wait. A governed Logs read (Ops/Actions/sevm-20261009-memex-stall) covers 11:12–14:37Z with did not emit an Initial|Failed to track activity|could not be claimed|UNANCHORED|TokenValidation. It returned 3 lines, none of them a stall: they are the InstanceSyncConfig GRACE line, once per replica, which is #5508 and fixed in Systemorph/MeshWeaver.Plugins#3230. The positive control is the 3 DynamicTypePreWarmer: warm-up complete lines in the same window (Ops/Actions/sevm-20261009-memex-prewarm), so the window is live. A 3.5 h zero does not by itself close a stall that recurs every few days. That is why this folds into #5315 instead of closing as fixed.

Closing as a duplicate of #5315. If a stall recurs at this log site, it folds there.

Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

Still occurring: 1 more since the last update (2236 total).

Last seen: 2026-10-13 00:08:51Z
Pods: memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-t9222, memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-777677694-hjkc8, memex-portal-deployment-6f497dc9b-r4nkg, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-54856649cb-m2jn9

Recent log lines

2026-09-24 09:55:55Z memex-portal-deployment-56fbdcd48f-4dpcn fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Store/Plugin — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Store/Plugin/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-09-24 10:04:11Z memex-portal-deployment-56fbdcd48f-sz5st fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Feedback/EvalCase — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Feedback/EvalCase/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
2026-10-13 00:08:51Z memex-portal-deployment-54856649cb-m2jn9 fail: MeshWeaver.Graph.CompileWatcher[0]
      Sources watcher faulted on Store/Catalog — re-establishing
      MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'namespace:Store/Catalog/Source scope:subtree nodeType:Code' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.

Navigation Menu

Repository navigation

PostgreSQL partitioned mesh query provider exceeds the 15 s Initial budget, faulting space compile watchers #5344

What is failing

Probable cause

Impact

Where to look

Evidence

Code lead against `main` (`39f49494eb` core, `e9d9283ab` Plugins): the 15 s clock includes the wait for a slot on a BOUNDED pool

Why the Postgres Initial misses 15 s: code finding and fix, in review

Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet

Closing: fix live, no recurrence for ≥ 6 h, positive control present

What is failing

Probable cause

Impact

Where to look

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PostgreSQL partitioned mesh query provider exceeds the 15 s Initial budget, faulting space compile watchers #5344

Description

What is failing

Probable cause

Impact

Where to look

Evidence

Activity

systemorph-com commented on Sep 22, 2026

systemorph-com commented on Sep 22, 2026

rbuergi commented on Sep 23, 2026

Code lead against main (39f49494eb core, e9d9283ab Plugins): the 15 s clock includes the wait for a slot on a BOUNDED pool

systemorph-com commented on Sep 23, 2026

rbuergi commented on Sep 23, 2026

systemorph-com commented on Sep 23, 2026

systemorph-com commented on Sep 23, 2026

rbuergi commented on Sep 23, 2026

Why the Postgres Initial misses 15 s: code finding and fix, in review

rbuergi commented on Sep 24, 2026

rbuergi commented on Sep 24, 2026

Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet

rbuergi commented on Sep 24, 2026

Closing: fix live, no recurrence for ≥ 6 h, positive control present

systemorph-com commented on Sep 26, 2026

systemorph-com commented on Sep 26, 2026

systemorph-com commented on Sep 28, 2026

What is failing

Probable cause

Impact

Where to look

systemorph-com commented on Sep 28, 2026

rbuergi commented on Sep 30, 2026

systemorph-com commented on Oct 4, 2026

rbuergi commented on Oct 8, 2026

rbuergi commented on Oct 9, 2026

systemorph-com commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Code lead against `main` (`39f49494eb` core, `e9d9283ab` Plugins): the 15 s clock includes the wait for a slot on a BOUNDED pool