FollowGo waits unboundedly for a GO that cannot cross processes — the PG change listener is registered but never started

Found while implementing the cross-cluster build claim (#1424 / PR #1437). The doc was corrected there; the underlying defect was not fixed and needs its own home.

The claim the doc made

BuildCoordination.md stated that the GO broadcast rides the Postgres LISTEN/NOTIFY feed and is therefore "cross-process by construction".

It is not. PostgreSqlChangeListener is registered but never started in either partitioned-PG overload — AddPartitionedPostgreSqlPersistence has its AddHostedService commented out, and the Aspire overload never had one. So there is no cross-process change feed at all: a mirror can be arbitrarily far behind a rival's write and will never be told.

The consequence

BuildProtocolDriver.FollowGo subscribes to ObserveBuildGo with no bound. A follower in a different cluster is therefore waiting for a notification that cannot reach it, with nothing to end the wait.

So the current honest description of a separate-cluster bake host is "publishes the GO earlier", not "spares its peers the bake" — the peers never learn it happened.

Note this is not merely a slow path: an unbounded wait for an event that is structurally undeliverable never completes. That is the same class as several defects fixed this week (a request handled but never answered; a subscription whose terminal signal was discarded), where the caller has no way to distinguish "still working" from "will never happen".

Two things to decide, and they are different

Should the change listener be started? If cross-process notification is intended, the commented-out AddHostedService is the gap — but starting it is a real behavioural change to every partitioned-PG deployment and wants its own justification and measurement.
FollowGo must not wait unboundedly regardless. Even with the feed running, an unbounded wait on a remote event is the wrong shape. The arbiter that fix(build): the build claim is exclusive across clusters, not just within one #1437 added deliberately reads the witness rather than waiting to be notified, precisely because the feed cannot be relied upon — FollowGo should do the same, or carry a bound with an explicit outcome.

🚨 Do not fix this by adding a poll timer as the primary mechanism. #1366 removed a 10-minute staleness clock for exactly that reason, and #1437's arbiter is level-triggered by design: a refused claim writes nothing and re-decides on the next real change.

Context worth carrying

#1437's cross-cluster claim does not depend on this being fixed — it reads the durable row rather than awaiting a notification, which is why it works today across clusters that share a database. This issue is about the follower half: a peer that could have skipped a bake but is never told the winner started one.

Reopening: closed as COMPLETED on 2026-08-13, but the defect is live and it caused a production outage tonight

This issue's own body says it plainly:

The doc was corrected there; the underlying defect was not fixed and needs its own home.

It was then closed as COMPLETED. The doc fix closed the ticket that existed to track the code fix. Verified still live on origin/main just now:

src/MeshWeaver.Hosting.PostgreSql/PostgreSqlExtensions.cs
367:        // services.AddSingleton(sp => new PostgreSqlChangeListener(baseDataSource, ..., ...));
368:        // services.AddHostedService<PostgreSqlChangeListenerHostedService>();

It is one of three broken legs behind #1814

Tonight every course cover on memex.meshweaver.cloud served NodeType 'Store/Plugin' could not be prepared to anonymous visitors for ~2 hours. Root cause chain on #1814; the part this issue owns is that there is no cross-process change feed at all, measured three ways:

leg	state on prod
the `mesh_node_notify` trigger fires	missing on 143 of 144 memexcloud schemas, including `store` (owner of `Store/Plugin`) — the one present is `public`, empty by design
something LISTENs	this issue — registration commented out for the partitioned setup, and memex-cloud is definitively partitioned
the notification carries the node	`PostgreSqlChangeListener.cs:130` publishes `Entity = null`, and `MeshDataSource`'s adoption path discards `notification.Entity is not MeshNode`

The consequence in #1814 is exactly the "a mirror can be arbitrarily far behind a rival's write and will never be told" sentence above, with a concrete victim: one pod's MeshNodeStreamCache could never learn that another pod had written latestAssemblyCollection/latestAssemblyPath, so its cross-hub patch base was stale by construction. That is why the write conflict was deterministic rather than a race — UpdateRemote re-enqueued twice against a "fresh activation" and failed identically both times, because the staleness was in the snapshot, not the activation.

Proven by experiment: at one replica (no rival mirror) plus a cleared latch, the page rendered in 2.6 s. At two replicas it had been failing for two hours.

Why fixing this leg alone is not enough

Restoring the trigger (V54, #1816) would make NOTIFY fire into a channel nobody listens on; restoring the listener without the entity would deliver notifications MeshDataSource discards. All three legs have to land together, which is a materially bigger change than any one of them and carries its own risks — notably a NOTIFY storm per bulk import, and a per-notification re-read to supply the missing entity.

Reopening rather than filing fresh, because this issue already states the defect correctly and the history matters: it was known on 2026-08-13, documented, and closed while still live.

docs: Durable Streams Are Mesh Nodes — the design that retires the Or…

Navigation Menu

Repository navigation

FollowGo waits unboundedly for a GO that cannot cross processes — the PG change listener is registered but never started #1440

The claim the doc made

The consequence

Two things to decide, and they are different

Context worth carrying

Reopening: closed as COMPLETED on 2026-08-13, but the defect is live and it caused a production outage tonight

It is one of three broken legs behind #1814

Why fixing this leg alone is not enough

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

FollowGo waits unboundedly for a GO that cannot cross processes — the PG change listener is registered but never started #1440

Description

The claim the doc made

The consequence

Two things to decide, and they are different

Context worth carrying

Activity

rbuergi commented on Aug 18, 2026

Reopening: closed as COMPLETED on 2026-08-13, but the defect is live and it caused a production outage tonight

It is one of three broken legs behind #1814

Why fixing this leg alone is not enough

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions