Thread hub recycles in a loop — its sync stream ends with SyncStreamOwnerRefusedException and that thread's Changes-header badges fail to load #6709

Activity

1. systemorph-com

   added

   bugSomething isn't workingSomething isn't working

   sev:MMedium - secondary path broken, or easy workaroundMedium - secondary path broken, or easy workaround

   on Oct 13, 2026
2. systemorph-com commented on Oct 13, 2026

   

   systemorph-comBot

   on Oct 13, 2026

   ContributorAuthor

   More actions

   Folded in — a DIFFERENT fingerprint from the same log site.

   This issue tracks the log site e96e96ad08bdc004 (MeshWeaver.Blazor.Components.BadgeView), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.

   Folded fingerprint	b2a48be977e4575f
   Log site	e96e96ad08bdc004
   Category	MeshWeaver.Blazor.Components.BadgeView
   Severity	Error
   Exception	MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException
   Detail	Stream UUGYFaNslEWJ4hHpidHm_A: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
   Namespace	memex-cloud
   Pods	memex-portal-deployment-54856649cb-54lxb
   Occurrences	548
   First seen	2026-10-13 20:01:57Z
   Last seen	2026-10-13 20:01:58Z
   Already folded here	1 fingerprint(s)
   Incident	Admin/_LogIncident/b2a48be977e4575f
   Triage called it	Thread-hub recycle loop ends a diff-view sync stream with SyncStreamOwnerRefusedException on memex-cloud

   What triage wrote about this fingerprint

   Observed

   On pod memex-portal-deployment-54856649cb-54lxb of memex-cloud (the public plugin-registry instance), the MeshWeaver.Blazor.Components.BadgeView slots that load a document's before/after columns — areas Header/Changes/Rows/View/Before and Header/Changes/Rows/View/After, loading before, beforeStyle, after, afterStyle — failed loading with MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException for sync stream UUGYFaNslEWJ4hHpidHm_A. The exception reports that 4 DISTINCT owner activations of the hub at phuong/_Thread/where-do-i-add-materials-to-build-our-di-e79aaebbd6b6366796a0 refused the stream against a budget of 3, that the owner address is recycling in a LOOP rather than settling — each successor activation dies before it can answer — and that the stream was ended with this error so its readers re-read. The log category (BadgeView) is the reader reporting a failed load; the failure it reports is in the stream-ownership negotiation in MeshWeaver.Data.Serialization.

   Where and when

   A single ~25 ms burst: 20:01:57.986Z → 20:01:58.011Z, 548 occurrences, all on one pod, all for one stream id and one owner address, one tenant (user phuong). The count is a client-side amplification of one underlying refusal set, not 548 independent failures: the stream ends with the error so its readers re-subscribe, and the four BadgeView slots of the before/after view each re-fail — which accounts for the shape of the burst. The watcher folded two detail shapes onto this incident (274 occurrences each), one flagged [case-only-path-collision]; variants is 1 and every visible sample carries the same owner address, so both shapes are the same defect counted twice — but the concrete per-instance paths that raised the collision flag were masked and cannot be read from this node. Whether the loop continued after the burst, whether other streams or owner addresses on memex-cloud showed the same, and whether other pods were involved are all not established: the window observed is this burst only.

   Impact

   One user's thread workspace view — the before/after changes view of where-do-i-add-materials-to-build-our-di-… — could not be served from its owning hub while that address was recycling. Scoped to one thread and one tenant; no evidence in this window of impact on other documents, other tenants, or the registry functions of memex-cloud.

   Not established

   The cause is not established, and the evidence here does not choose between the candidates below. Additionally not determined: whether the incident still recurs; whether the thread is currently openable on memex-cloud; and whether a duplicate issue exists — no other incident or issue nodes are readable from this partition, so no duplicate check was possible. The framework source (MeshWeaver.Blazor.Components.BadgeView, MeshWeaver.Data.Serialization) is not readable from this mesh either, so no file, type or method beyond those named in the log itself could be verified.

   Candidate causes and the reading that separates them

   1. The hub fails during activation — a deterministic defect in loading that thread's state (e.g. a node in its subtree that fails in that activation). Predicts: activation-time exceptions for that grain id in memex-cloud logs inside the burst window, and the thread staying unopenable until the defect is gone. Reading: the hub's own activation log lines for phuong/_Thread/where-do-i-add-materials-to-build-our-di-e79aaebbd6b6366796a0 around 20:01:57–58Z name the exception; opening the thread on memex-cloud reproduces it.
   2. Something keeps requesting Recycle of that address — the exception's own text points here ("the fix is at whatever is requesting the recycles, not here"). A server-side actor repeatedly quiescing the hub — e.g. the stale-build auto-recycle path, as configured on this instance class, or a stuck-detector — would kill each successor before it can answer without the hub itself being defective. Predicts: [QUIESCE-START] lines for that grain id carrying a reason naming the recycler, with no activation exception. Reading: those quiesce lines separate "defective grain" from "something repeatedly tearing a healthy grain down".
   3. Resource/deactivation churn — the grain evicted under memory or GC pressure before answering. Less consistent with 4 distinct activations refusing within milliseconds, but not excluded. Predicts: memory/GC or eviction lines on the pod at that instant and the same shape on other owner addresses under load. Reading: pod-level logs for memex-portal-deployment-54856649cb-54lxb around 20:01:57Z rule this in or out.

   Where to look

   The raise site of MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException — the stream-owner refusal and recycle-retry negotiation in MeshWeaver.Data.Serialization — and the activation/quiesce path of the thread hub at phuong/_Thread/where-do-i-add-materials-to-build-our-di-e79aaebbd6b6366796a0 on memex-cloud. The reader that re-subscribes is MeshWeaver.Blazor.Components.BadgeView's load of the Header/Changes/Rows/View/Before/After areas. No source file could be linked from this partition; those names come from the log itself.

   Still occurring: 548 more since the last update (548 total).

   • Last seen: 2026-10-13 20:01:58Z
   • Pods: memex-portal-deployment-54856649cb-54lxb

   🚨 This incident covers 2 distinct message shapes, so its own Last seen is the newest of ALL of them. Read the row for the shape this issue is about — a shape whose last sighting predates a fix is not recurring, whatever the counter says.

   Shape	Occurrences	First seen	Last seen
   2dfd6d35b91c4df5 — [case-only-path-collision] Stream UUGYFaNslEWJ4hHpidHm_A: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a L…	274	2026-10-13 20:01:57Z	2026-10-13 20:01:58Z
   b2a48be977e4575f — Stream UUGYFaNslEWJ4hHpidHm_A: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a L…	274	2026-10-13 20:01:57Z	2026-10-13 20:01:58Z

   Recent log lines

   2026-10-13 20:01:58Z memex-portal-deployment-54856649cb-54lxb fail: MeshWeaver.Blazor.Components.BadgeView[0]
         Loading 'before' in area 'Header/Changes/Rows/View/Before' in Area Header/Changes/Rows/View/Before
         MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream UUGYFaNslEWJ4hHpidHm_A: giving up on owner phuong/_Thread/where-do-i-add-materials-to-build-our-di-e79aaebbd6b6366796a0 — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
   2026-10-13 20:01:58Z memex-portal-deployment-54856649cb-54lxb fail: MeshWeaver.Blazor.Components.BadgeView[0]
         Loading 'afterStyle' in area 'Header/Changes/Rows/View/After' in Area Header/Changes/Rows/View/After
         MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream UUGYFaNslEWJ4hHpidHm_A: giving up on owner phuong/_Thread/where-do-i-add-materials-to-build-our-di-e79aaebbd6b6366796a0 — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
   2026-10-13 20:01:58Z memex-portal-deployment-54856649cb-54lxb fail: MeshWeaver.Blazor.Components.BadgeView[0]
         Loading 'after' in area 'Header/Changes/Rows/View/After' in Area Header/Changes/Rows/View/After
         MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream UUGYFaNslEWJ4hHpidHm_A: giving up on owner phuong/_Thread/where-do-i-add-materials-to-build-our-di-e79aaebbd6b6366796a0 — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
   
3. rbuergi commented on Oct 14, 2026

   

   rbuergi

   on Oct 14, 2026

   Contributor

   More actions

   This is a duplicate of #6729, which is the anchor for the whole SyncStreamOwnerRefusedException family. The fix is in #6756.

   The root cause was the subscriber's own host stopping, not an owner recycle loop. The pod logged Application is shutting down... in the same second as this burst. Its router answered each re-ask with an untagged ShuttingDown refusal (Host is shutting down, cannot route to …), and the sync stream counted each of those as a "distinct owner activation". The fix rides out such routing-layer refusals on the paced time axis, and their give-up names the route instead of an owner.

   Closing as a duplicate. Verification follows on #6729.
4. rbuergi

   closed this as not plannedon Oct 14, 2026
5. systemorph-com commented on Oct 14, 2026

   

   systemorph-comBot

   on Oct 14, 2026

   ContributorAuthor

   More actions

   Folded in — a DIFFERENT fingerprint from the same log site.

   This issue tracks the log site e96e96ad08bdc004 (MeshWeaver.Blazor.Components.BadgeView), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.

   Folded fingerprint	46ec664c09804d7c
   Log site	e96e96ad08bdc004
   Category	MeshWeaver.Blazor.Components.BadgeView
   Severity	Error
   Exception	MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException
   Detail	Stream AC0OZ9PNY0{n}p8JZZl2hnjA: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
   Namespace	memex
   Pods	memex-portal-deployment-567896cdc8-84nqz
   Occurrences	16
   First seen	2026-10-14 10:25:48Z
   Last seen	2026-10-14 10:25:48Z
   Already folded here	2 fingerprint(s)
   Incident	Admin/_LogIncident/46ec664c09804d7c
   Triage called it	Owner hub of an actively-written thread recycles in a loop; its sync stream exhausts the owner-activation budget and portal views give up

   What triage wrote about this fingerprint

   Observed

   A portal sync stream (AC0OZ9PNY06p8JZZl2hnjA) owned by the thread node mkleiner/_Thread/wo-ist-jetzt-der-businessplan-fd1de4a8cc8b305e8df8 was refused by 4 DISTINCT owner activations against a budget of 3, and ended with MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException. The exception's own text states the owning address is recycling in a loop — each successor activation dies before it can answer the stream. The failure surfaced in the memex portal while MeshWeaver.Blazor.Components.BadgeView loaded the areas Header/Changes/Rows/View/Before and Header/Changes/Rows/View/After (fields before, beforeStyle, after, afterStyle) — i.e. a changes view over that thread.

   Where and when

   • All occurrences fall inside one second (2026-10-14 10:25:48 UTC) on a single pod of the memex namespace — one burst, not a steady rate.
   • The owner thread is fresh (created 10:10:25 UTC) and was under an active agent session: an ExecutiveAssistant thread with composer Provider/OpenRouterEU/z-ai/glm-5.3, supervision episode dd411faed51744738e8fc2ffa3b2ed2d, last written at 10:24:23 UTC — roughly 85 s before the burst — after ~107 versions in 14 minutes. The recycle loop hit a thread being written to at high frequency.
   • This incident is one fan-out of a single event: at the same instant, sibling incidents were opened for the same stream and owner at other view components — ButtonView (Admin/_LogIncident/3a320201ad1fbc5e), ContainerView (72f86b7b1b2d5f61), DispatchView (bb44668bb70d7c07), Label (6dda29fd55c6413a), NavLink (7991243deee2a7b3), ItemTemplate (873a0b9719b198d2), LayoutStackView (034a9354b3a27ce5). Those are the same stream dying, reported at nine log sites — one defect, not nine; a human can close the siblings as duplicates of whichever ticket is picked up first.
   • Similar SyncStreamOwnerRefusedException bursts appear earlier the same morning (07:28–07:31 UTC) for other streams in the same namespace, so the shape is not unique to this thread — but this incident's evidence covers only this stream.
   • The incident carries two fingerprint shapes differing only by a [case-only-path-collision] marker on one key — the same log detail modulo letter case in a path; one event, not a second symptom.

   Impact

   One tenant, one thread, one pod, one burst: the affected view areas of that thread's page in the memex portal could not be served for the duration of the recycle loop (the stream ends with this error so readers re-read). No evidence of portal-wide impact or data loss. The owner thread itself is Active with 12 messages and a completed summary, so the thread content appears intact.

   Not established

   • What requested the recycles — no hub-lifecycle evidence is on this incident.
   • Whether each activation died from an initialization fault inside the hub or was quiesced externally; the incident shows only refusals.
   • Whether the thread's views render correctly now, and whether the earlier 07:28–07:31 bursts share a cause with this one.
   • The exact source of the log site: no BadgeView / MeshWeaver.Data.Serialization source nodes were findable in the readable mesh partitions, so no file or line can be cited.

   Candidate causes and the readings that separate them

   1. The owner hub is being recycled repeatedly by something external (e.g. the supervision episode or a convergence quiesce), and each successor dies before answering. Predicts repeated quiesce/dispose lines with reasons for the same address in the seconds before 10:25:48. Reading: hub lifecycle logs for mkleiner/_Thread/wo-ist-jetzt-der-businessplan-fd1de4a8cc8b305e8df8 on that pod, 10:25:40–10:25:48 — each entry carries the recycler's reason.
   2. Each activation crashes during hub initialization (e.g. deserializing the thread's content), never reaching the point of accepting stream ownership. Predicts an initialization exception in the same pod at the same timestamps, one per activation. Reading: pod logs around 10:25:4x for the first activation failure.
   3. The loop is ordinary write churn: the active session's rapid successive writes (version 107 in 14 minutes) each invalidate the hub, and four activations churned past a budget of 3. Predicts a 1:1 correlation between recycle timestamps and the thread's version timestamps, with no exceptions inside the activations. Reading: diff the thread's version history (GetVersions on the owner node) against the recycle times.

   Readings 1 and 2 rule each other in or out on the presence of an exception inside the activations; reading 3 is settled by the version-vs-recycle timeline.

   Where to look

   • Systemorph/MeshWeaver: MeshWeaver.Data.Serialization — the SyncStreamOwnerRefusedException type and the owner-activation budget logic that emits "giving up on owner … (budget …)"; and MeshWeaver.Blazor.Components.BadgeView — the Loading '{field}' in area '{area}' log site.
   • The owner node mkleiner/_Thread/wo-ist-jetzt-der-businessplan-fd1de4a8cc8b305e8df8 and its version history around 10:24–10:26 UTC.
   • The sibling incidents listed above (same stream, same instant) for the full fan-out picture.

   Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.

   Still occurring: 16 more since the last update (16 total).

   • Last seen: 2026-10-14 10:25:48Z
   • Pods: memex-portal-deployment-567896cdc8-84nqz

   🚨 This incident covers 2 distinct message shapes, so its own Last seen is the newest of ALL of them. Read the row for the shape this issue is about — a shape whose last sighting predates a fix is not recurring, whatever the counter says.

   Shape	Occurrences	First seen	Last seen
   c805f0798f2eebc4 — [case-only-path-collision] Stream AC0OZ9PNY0{n}p8JZZl2hnjA: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a…	8	2026-10-14 10:25:48Z	2026-10-14 10:25:48Z
   46ec664c09804d7c — Stream AC0OZ9PNY0{n}p8JZZl2hnjA: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a…	8	2026-10-14 10:25:48Z	2026-10-14 10:25:48Z

   Recent log lines

   2026-10-14 10:25:48Z memex-portal-deployment-567896cdc8-84nqz fail: MeshWeaver.Blazor.Components.BadgeView[0]
         Loading 'before' in area 'Header/Changes/Rows/View/Before' in Area Header/Changes/Rows/View/Before
         MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream AC0OZ9PNY06p8JZZl2hnjA: giving up on owner mkleiner/_Thread/wo-ist-jetzt-der-businessplan-fd1de4a8cc8b305e8df8 — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
   2026-10-14 10:25:48Z memex-portal-deployment-567896cdc8-84nqz fail: MeshWeaver.Blazor.Components.BadgeView[0]
         Loading 'afterStyle' in area 'Header/Changes/Rows/View/After' in Area Header/Changes/Rows/View/After
         MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream AC0OZ9PNY06p8JZZl2hnjA: giving up on owner mkleiner/_Thread/wo-ist-jetzt-der-businessplan-fd1de4a8cc8b305e8df8 — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
   2026-10-14 10:25:48Z memex-portal-deployment-567896cdc8-84nqz fail: MeshWeaver.Blazor.Components.BadgeView[0]
         Loading 'after' in area 'Header/Changes/Rows/View/After' in Area Header/Changes/Rows/View/After
         MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream AC0OZ9PNY06p8JZZl2hnjA: giving up on owner mkleiner/_Thread/wo-ist-jetzt-der-businessplan-fd1de4a8cc8b305e8df8 — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.