Repository navigation
Sync stream give-up leaves the reader on a frozen last value; one owner teardown is counted as several activations #6602
Description
What is failing
When a subscriber's sync stream gives up on its owner in JsonSynchronizationStream.CreateExternalClient (Stream …: giving up on owner <path> — N DISTINCT owner activations have refused this stream (budget 3), or the same-activation ride-out variant), the stream neither errors nor completes. The re-arm is disarmed and the stream stays alive on its last value: "degrades to the pre-existing orphaned-stream behaviour". Every reader of it then holds a frozen snapshot that looks like a live one. No error, no completion, and nothing for the reader's own recovery to react to.
Where it is seen
On memex-cloud on 2026-10-12 the give-up lines named Hosting/PlatformBuilds, Hosting/Modules, Ops/Status/memex and many other nodes, in bursts during rolls (~08:45Z, 09:14Z, 09:32Z, 09:58Z, 11:08Z; Ops/Actions/w5011-memexcloud-giveup-15h). This is a candidate cause for the reader-side shape of #5011: the fleet watch held a read of Ops/Status/build that stopped advancing while the store had fresh rows, and its Unread/Blind handling never fired because the read never ended. (Details: the newest comment on #5011.)
Why a roll exhausts the budget (code reading)
The budget counts DISTINCT refusing activations since the last answered re-ask. One owner teardown can be charged as two or three distinct activations:
StreamEndedEventcarries no activation identity, so every announcement (recycle announce, the owner's answer to a heartbeat for a stream it does not serve, an idle release) is charged as a new activation, and it also breaks the same-activation chain the NACKs build;MessageHubGrain.RefuseBecauseTheHostIsLeavingtags itsShuttingDownNACK with the GRAIN's identity, while the hub's own NACKs (MessageService,MessageHub) tag the HUB's. So one activation on a leaving silo has two different tags.
During a roll, two such teardowns are enough to exceed a budget of 3 with no answered re-ask in between.
Expected
- A give-up ends the stream with a typed, named terminal, so the reader's recovery runs. The cache evicts the faulted entry, and the next read opens a fresh subscription.
- One activation is counted once, whichever carrier reports it.
Refs #5011
Activity
rbuergi commentedon Oct 12, 2026
Fix: core #6606 (terminal give-up + activation accounting), Plugins Systemorph/MeshWeaver.Plugins#3503 (heartbeat re-reads ended reads on next write).
fix(data): the ride-out budget is charged per refusing activation (#6602
Merge origin/main into #6539
rbuergi commentedon Oct 13, 2026
Verified on memex-cloud, where this was seen. It runs core
ec7763883b(/api/version), which contains #6606 (5d675f73ec). Governed reads on control, 2026-10-13:Ops/Actions/v6602-memexcloud-giveup-since-0000-1013:{namespace="memex-cloud"} |~ "(?i)giving up on owner", 2026-10-12 23:56Z – 2026-10-13 15:11Z, cut at 40. These are two bursts, ~04:00:44Z and ~04:01:04Z, on pods…6649cb-m2jn9and…6649cb-54lxb. EveryStream <id>: giving up on owner <path>line is followed, on the same pod and for the same stream id, byMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream <id>…. Owners includeHosting/PlatformBuilds, so the reader now gets a typed terminal and no longer sits on a frozen value.Ops/Actions/v5011-memexcloud-stale-since-0000-1013, same window,past the deadline its own writer declared | was NOT written this pass: no FleetWatchHeartbeat past-deadline census at all. There are 3NOT written this passlines from one pass at 00:09Z: the writer missed one pass, and the next pass recovered before any deadline. On 2026-10-12 the same instance logged about two stale-census episodes overnight.Ops/Actions/v6602-memexcloud-giveup-6h-1013, 09:11Z – 15:11Z, a window with memex-cloud rolls at 10:50Z, 11:08Z and 14:28Z: 0 give-up lines.Not established: the accounting half ("one activation is counted once") cannot be read from a log line. The 04:00Z give-ups still say "4 DISTINCT owner activations", and I cannot tell from outside whether those were four real activations. The consumer half that re-opens an ended read is Systemorph/MeshWeaver.Plugins#3503. It is armed after I answered its review today and is not merged yet. #5011 keeps watching the census across the next rolls.
Closing this one: its defect, a give-up that freezes the reader, is gone on the running portal.