Repository navigation
Sync stream of a user thread node ended after 4 owner activations refused it — the phuong/_Thread address recycled in a loop #6727
Description
Observed
The sync stream UUGYFaNslEWJ4hHpidHm_A, owned by the thread node phuong/_Thread/where-do-i-add-materials-to-build-our-di-e79aaebbd6b6366796a0, ended itself with MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException after 4 DISTINCT owner activations of that address refused the stream against the recycle re-arm budget of 3 — the give-up path of JsonSynchronizationStream's ride-out in MeshWeaver.Data (the #6602 give-up contract, per Riding Out a ShuttingDown Address) firing as designed: the owner address was recycling in a loop, each successor activation dying before it could answer the re-ask. The exception names the shape, not the requester.
The MeshWeaver.Blazor.Components.ButtonView log site is only where the ended stream reached a reader: both samples carry the same stream id and owner path, loading undoAllStyle/undoAllLabel in area Header/Changes/UndoAll — the Undo All control of the changes/diff header over that thread. One stream's terminal error fanned out across every view reading it: sibling incidents in the same ~1.3-second window record the same stream id and owner path at DispatchView (691), NavLink (548), Label (275), BadgeView, ItemTemplate, ContainerView, LayoutStackView, MarkdownEditorView, NavLink and more — one read event, not thousands of failures.
Where and when
One pod of the memex-cloud portal, one owner address, one user namespace (phuong), a single burst around 2026-10-13T20:01:58Z. Nothing else — no other tenant, node or pod — appears in this fingerprint.
Impact
One user's live view of that thread's changes/diff header lost its feed for the burst window. Readers recover by design (the stream ends with the error so its readers can re-read), so impact is bounded to the recycle window unless the loop persisted — which this evidence does not show.
Not established
- What requested the recycles of that thread node — the exception names the recycle loop at the owner, not its requester.
- Whether the owner address settled after the burst, and whether the user's view recovered on re-read.
- The node's own state:
phuong/_Thread/where-do-i-add-materials-to-build-our-di-e79aaebbd6b6366796a0is not readable under the identity this triage ran with (a direct Get and a namespace search ofphuongboth return nothing), so its version history and activity at the burst could not be read. - Whether other streams owned by the same node behaved the same.
- Whether a roll, deploy or module live-update coincided with the burst.
Candidate causes and the reading that separates them
- Overlay / stale-assembly self-heal repeatedly recycling the thread's hub — each successor hits the same stale state and dies again. Predicts recycle-request lines naming this owner path with a stale-build/overlay reason just before the burst. Reading: the pod log around 2026-10-13T20:01:57Z for recycle-request lines for that path and their stated reason.
- A build or activation fault killing each successor during bring-up of the thread node's type (a live-update window would fit). Predicts a repeated build/init failure for that type in the same window. Reading: build/compile log lines for the type in the window, and
GetDiagnosticson the type on the portal instance. - A writer repeatedly triggering the recycles — e.g. an agent or the user writing to this active thread while the diff view was open. Predicts write/version activity on the node at the burst. Reading:
GetVersionson the node (under an identity that can readphuong) for entries in the window. - Reader-side mis-accounting — one settling teardown charged as four distinct activations (the Sync stream give-up leaves the reader on a frozen last value; one owner teardown is counted as several activations #6602 drift shape: a carrier without an activation tag). Predicts exactly ONE distinct activation token for this owner in the window. Reading: count the distinct activation tokens in the
ShuttingDownNACK /StreamEndedEventlines for this owner path — 4 real tokens rules this out; 1 rules it in.
Where to look
JsonSynchronizationStream's recycle re-arm latch inMeshWeaver.Data— the budget machinery that minted the give-up; the exception is by design here, not the defect.- Riding Out a ShuttingDown Address — the two-axis budget, the give-up contract (Sync stream give-up leaves the reader on a frozen last value; one owner teardown is counted as several activations #6602), and the two recycling shapes; per that page, the thing to examine is whatever is asking for the recycles.
- The owner node
phuong/_Thread/where-do-i-add-materials-to-build-our-di-e79aaebbd6b6366796a0on the memex-cloud portal — its version history and activity at the burst (needs an identity that can read thephuongnamespace).
Already known?
The same exception shape was filed today as #6712 (owner Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b, pod memex-portal-deployment-54856649cb-m2jn9, also 4 activations against budget 3, ~10 minutes earlier), and further bursts of the same fingerprint family appear in the incident log today around 17:20Z and 19:52–19:57Z on yet other owners and pods. This ticket is not a duplicate of #6712 — different owner address, different pod — but the recurrence across owners, namespaces and pods within hours makes a shared trigger (e.g. today's roll or live-update activity) a live question, and the readings in candidates 1–2 are exactly what would answer it. Sibling incidents for the same stream at other view sites are being triaged in parallel and will land as adjacent tickets covering the same event; a human may fold them.
Evidence
| Fingerprint | 67ac9f960fe8cf60 |
| Category | MeshWeaver.Blazor.Components.ButtonView |
| Severity | Error |
| Exception | MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-54856649cb-54lxb |
| Occurrences | 2 |
| First seen | 2026-10-13 20:01:57Z |
| Last seen | 2026-10-13 20:01:57Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Blazor.Components.ButtonView. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-10-13 20:01:57Z memex-portal-deployment-54856649cb-54lxb fail: MeshWeaver.Blazor.Components.ButtonView[0]
Loading 'undoAllStyle' in area 'Header/Changes/UndoAll' in Area Header/Changes/UndoAll
MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream UUGYFaNslEWJ4hHpidHm_A: giving up on owner phuong/_Thread/where-do-i-add-materials-to-build-our-di-e79aaebbd6b6366796a0 — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
2026-10-13 20:01:57Z memex-portal-deployment-54856649cb-54lxb fail: MeshWeaver.Blazor.Components.ButtonView[0]
Loading 'undoAllLabel' in area 'Header/Changes/UndoAll' in Area Header/Changes/UndoAll
MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream UUGYFaNslEWJ4hHpidHm_A: giving up on owner phuong/_Thread/where-do-i-add-materials-to-build-our-di-e79aaebbd6b6366796a0 — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
Opened automatically from Admin/_LogIncident/67ac9f960fe8cf60. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 68a441c9a9ca2c22: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
rbuergi commentedon Oct 14, 2026
This is a duplicate of #6729, which is the anchor for the whole
SyncStreamOwnerRefusedExceptionfamily. The fix is in #6756.The root cause was the subscriber's own host stopping, not an owner recycle loop. The pod logged
Application is shutting down...in the same second as this burst. Its router answered each re-ask with an untaggedShuttingDownrefusal (Host is shutting down, cannot route to …), and the sync stream counted each of those as a "distinct owner activation". The fix rides out such routing-layer refusals on the paced time axis, and their give-up names the route instead of an owner.Closing as a duplicate. Verification follows on #6729.
systemorph-com commentedon Oct 14, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
68a441c9a9ca2c22(MeshWeaver.Blazor.Components.ButtonView), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.3a320201ad1fbc5e68a441c9a9ca2c22MeshWeaver.Blazor.Components.ButtonViewMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionStream AC0OZ9PNY0{n}p8JZZl2hnjA: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.memexmemex-portal-deployment-567896cdc8-84nqzAdmin/_LogIncident/3a320201ad1fbc5eWhat triage wrote about this fingerprint
Observed — On the memex portal (pod
memex-portal-deployment-567896cdc8-84nqz), theButtonViewloading theHeader/Changes/UndoAllarea of a user's thread view failed at the same instant for two values (undoAllStyle,undoAllLabel): the sync streamAC0OZ9PNY06p8JZZl2hnjA, owned by the thread-node hub atmkleiner/_Thread/wo-ist-jetzt-der-businessplan-fd1de4a8cc8b305e8df8, ended withMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException— 4 DISTINCT owner activations refused the stream against a budget of 3. The framework's own message records that the owner address is recycling in a LOOP, each successor activation dying before it can answer, and that the stream ends with this error so its readers can re-read.Where and when — Both occurrences in the same millisecond (2026-10-14T10:25:48Z), one pod, one tenant namespace (
memex), one owner address, one stream id: a single burst, not a steady rate. The same message shape ("giving up on owner … 4 DISTINCT owner activations", budget 3) appears inOps/Logsagainst many different owners (Approvals, BinaryClickerV2, UWDeepfield/*, Hosting/Triage threads,Hosting/FleetWatchPass, Essentials/OperationRequest), historically clustered near restarts and rolls — this incident is that shape with a user thread hub as the owner. #5011's investigation logged the identical signature as its "item 4" pattern around FleetWatchPass owner churn during rolls; that issue is closed and covered the fleet-watch component, not this one.Impact — One user's thread view failed to render its Undo-All header control at that instant; the stream ends so readers can re-read. Two occurrences on one pod in one instant is low direct impact, but the underlying signal — an owner address recycling faster than an activation can answer — is the same pattern that produced lost fleet-watch rows in #5011.
Not established — What requested the recycles or what killed each of the four activations. Whether the address is still recycling now. Whether other log sites saw the same address. Whether this pod restarted or rolled around the burst. Whether a type-scoped recycle ran in the window. No cause is established here.
Candidate causes and the reading that separates them
recyclejob /ActivationRecycle(PR ActivationRecycle: recycle a NodeType's live activations on every process #6054) and the RecycleCascade dispose live activations when a definition is rewritten; NodeType compiles rewrite the definition on every activation: version churn (~322k versions, ~10k writes/day) with unchanged source fingerprint #6038 documents NodeType compiles rewriting the definition on every activation (version churn), which would re-arm a cascade per successor activation — a genuine loop. Predicts:[QUIESCE-START]lines formkleiner/_Thread/wo-ist-jetzt-…carrying an ActivationRecycle reason, and anAdmin/_Recycle/{id}request with per-process reports in the window naming the Thread NodeType. Reading those request/report children and the quiesce reasons settles this cause in or out.Where to look — The give-up site in
MeshWeaver.Data.Serialization(the exception's own message names its semantics); the per-node hub lifecycle for_Threadnodes; theAdmin/_Recycleaudit children around 2026-10-14T10:25Z;Doc/Architecture/StaleStateUntilRecycle; prior art in #5011 (item 4, same signature, closed), #6038 (definition rewrite churn), PR #6054 (ActivationRecycle).Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 2 more since the last update (2 total).
memex-portal-deployment-567896cdc8-84nqzRecent log lines