Repository navigation
Subscription/payment hub addresses recycle through four owner activations without settling — MarkdownView streams for Admin/Subscriptions/sglauser give up with SyncStreamOwnerRefusedException #6725
Description
Observed
The portal markdown view MarkdownView failed to load the three style/content bindings of the Overview/2 area of a subscription view (style, markdown, html — all three against Area Overview/2). Each load terminated with MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException on the view's synchronization stream uv075yPRUkaGh6TCDiRLhA: the stream gave up on its owner — the subscription hub at Admin/Subscriptions/sglauser — after 4 DISTINCT owner activations refused the stream against a budget of 3. Per the exception's own text, the owner address "is recycling in a LOOP, not settling: each successor dies before it can answer", and the stream deliberately ends with this error so its readers can re-read.
Three occurrences are the same give-up event surfacing at each of the view's three bindings; one stream, one owner address, one instant. The MarkdownView log site is the reporter only — it binds the stream, it does not cause the recycles.
The owner node itself (Admin/Subscriptions/sglauser, nodeType Store/Subscription, viewer sglauser, tier enterprise/staff-unmetered) is intact and Active; the failure is in the hub activation/handoff cycle serving that address, not in the persisted content.
Where and when
All three occurrences fall inside ~53 µs (19:52:18.851037–19:52:18.851090 UTC, 2026-10-13), on a single pod, one namespace (memex-cloud). This is the first occurrence of this fingerprint — the incident opened within ~35 s of the burst.
The same window produced two sibling incidents on this pod, same exception type, same "4 DISTINCT owner activations … budget 3" loop wording, different owners: Admin/_LogIncident/9c4c66c8fab3b978 and 1542d4b5163b29aa, both MarkdownView failing to load Area Overview/2 with owner Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b (two distinct stream ids on the same owner). A pod-local cause acting on hub activation generally is therefore consistent with what I can see; I could not read anything outside this pod to confirm or exclude a wider scope. Notably, the "Overview/2" area path is identical across all three incidents — the same portal page reading multiple subscription/payment hubs failed at the same instant — which is consistent with one page load hitting three addresses mid-recycle, and also with a single fault tearing all three down together. Which of the two is not established.
The same give-up shape has been filed before on other owners: #6709 (a thread hub, 2026-10-13 17:17) and #6713 (same mechanism, same day), plus fleet-wide bursts of the older wording on 2026-10-07 / 2026-10-10 per those tickets. Those tickets are the same failure mechanism at other owner addresses, not duplicates of this instance.
Impact
One user's view of the subscription/payment Overview failed to load at that instant, three times in the same instant, on one pod. The stream is designed to end so readers can re-read; whether the reload succeeded is not established. No data loss is evidenced — the owner node is intact. Judged sev:M from the numbers: a single user-visible view-load failure with designed end-and-re-read containment, one pod, one burst. Three owners giving up in the same window on the same pod is what would re-rule this upward.
Not established
- What requested each recycle of
Admin/Subscriptions/sglauser— the incident carries no stack frame and no QUIESCE reason. - Why each successor activation died before answering — no per-activation fault is on this incident.
- Whether the give-up repeated after 19:52:18, and whether the view recovered on re-read.
- Whether the two sibling incidents (
9c4c66c8fab3b978,1542d4b5163b29aa) share a cause with this one or merely a window — I could not read pod-level evidence tying them together. - Whether the same moment shows on other pods — I have no visibility outside this burst.
Candidate causes and the reading that separates them
- A recycle storm on this pod — some trigger recycled hubs across the
Admin/Subscriptions/Store/Paymentsaddresses at the same instant, and each address went through ≥4 activations within the stream's re-read window. Consistent with three different owners giving up within milliseconds on one pod. Predicts:[QUIESCE-START]lines for all three owner addresses at 19:52:18 on this pod, naming one requester/reason. Reading: pod logs for the three owner paths around 19:52:18 would settle it in one pass. - A per-owner poison activation — something specific to the subscription/payment hubs faults each successor during initialization (e.g. an inaccessible source on first load). Consistent with the loop wording ("each successor dies before it can answer") being reached on each owner independently. Predicts: identical activation-fault log lines per owner, repeated per activation, on the three addresses. Reading: activation/init fault logs for
Admin/Subscriptions/sglauserandStore/Payments/_Rejected/973c…in the seconds before 19:52:18. - A systemic recycler (deployment roll, build rollout, NodeType recycle wave —
Doc/WhatsNew/2026-08-25-auto-recycle-on-stale-builddescribes production auto-recycling onto new builds). Predicts: the same give-up on other pods/tenants at the same time and a deploy marker in the window. Reading: whether other pods emitted the same fingerprint at 19:52:18 — absent from this burst, but this burst covers only one pod.
Where to look
MeshWeaver.Data.Serialization—SyncStreamOwnerRefusedException, the owner-refusal budget (refused by 4 distinct activations against a budget of 3), and the handoff that counts DISTINCT owner activations.MeshWeaver.Blazor.Components.MarkdownView— the reporting log site only.- The owner addresses
Admin/Subscriptions/sglauser(nodeTypeStore/Subscription) andStore/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b, plus the sibling incidents 9c4c66c8fab3b978 and 1542d4b5163b29aa. - Prior same-mechanism tickets Thread hub recycles in a loop — its sync stream ends with SyncStreamOwnerRefusedException and that thread's Changes-header badges fail to load #6709 / ThreadChatView fails to load a chat thread whose hub recycled through four owner activations without settling #6713 (filed 2026-10-13, other owners) — the exception's own text in every instance points at whatever is requesting the recycles, not at the stream.
Evidence
| Fingerprint | c9b666c065db562c |
| Category | MeshWeaver.Blazor.Components.MarkdownView |
| Severity | Error |
| Exception | MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-54856649cb-m2jn9 |
| Occurrences | 3 |
| First seen | 2026-10-13 19:52:18Z |
| Last seen | 2026-10-13 19:52:18Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Blazor.Components.MarkdownView. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-10-13 19:52:18Z memex-portal-deployment-54856649cb-m2jn9 fail: MeshWeaver.Blazor.Components.MarkdownView[0]
Loading 'style' in area 'Overview/2' in Area Overview/2
MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream uv075yPRUkaGh6TCDiRLhA: giving up on owner Admin/Subscriptions/sglauser — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
2026-10-13 19:52:18Z memex-portal-deployment-54856649cb-m2jn9 fail: MeshWeaver.Blazor.Components.MarkdownView[0]
Loading 'markdown' in area 'Overview/2' in Area Overview/2
MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream uv075yPRUkaGh6TCDiRLhA: giving up on owner Admin/Subscriptions/sglauser — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
2026-10-13 19:52:18Z memex-portal-deployment-54856649cb-m2jn9 fail: MeshWeaver.Blazor.Components.MarkdownView[0]
Loading 'html' in area 'Overview/2' in Area Overview/2
MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: Stream uv075yPRUkaGh6TCDiRLhA: giving up on owner Admin/Subscriptions/sglauser — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.
Opened automatically from Admin/_LogIncident/c9b666c065db562c. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 797bcddf96b4b1b9: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
systemorph-com commentedon Oct 13, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
797bcddf96b4b1b9(MeshWeaver.Blazor.Components.MarkdownView), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.1542d4b5163b29aa797bcddf96b4b1b9MeshWeaver.Blazor.Components.MarkdownViewMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionStream _qeQIr5atUWnOqtIf2a7Rw: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.memex-cloudmemex-portal-deployment-54856649cb-m2jn9Admin/_LogIncident/1542d4b5163b29aaWhat triage wrote about this fingerprint
Observed
A
MarkdownViewon the memex-cloud portal (memex-portal-deployment-54856649cb-m2jn9), rendering areaOverview/2, tried to load its three document areas —style,markdownandhtml— and all three loads terminated withMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionon the SAME stream,_qeQIr5atUWnOqtIf2a7Rw, whose owner is the node pathStore/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b. The exception's own text states the mechanism: 4 DISTINCT owner activations refused the stream against a budget of 3, the owner address was "recycling in a LOOP, not settling", and each successor activation died before it could answer the subscription.Three separate log lines (one per document area) carrying the same stream id and the same exception at the same instant is one burst from one component:
MeshWeaver.Blazor.Components.MarkdownView— the log category — raising the exception surfaced by the platform's sync-stream layer (MeshWeaver.Data.Serialization).The owner path is a retained, signature-unverifiable webhook delivery: per
Store/Payments("When a delivery cannot be verified"), a delivery that names an order but fails HMAC verification is kept atStore/Payments/_Rejected/{id}withrejectedAt/rejectionReason/receivedSignaturestamped, and_Rejectedsits under the Public/Anonymous deny (CatalogGate.SensitiveSegments).Where and when
A single burst, within one fifth of a second, on a single pod of one deployment (memex-cloud). No steady rate, no recurrence across pods, no coincident event (roll, restart, maintenance job) was read on the incident. Note this is NOT the same shape as the fleet-wide "giving up on owner" storm recorded across many hubs on 2026-09-26 and 2026-10-07 (different streams, different owners, different deployments) — this incident names exactly one stream and one owner path.
Impact
One portal view (
Overview/2on memex-cloud) failed to render its style/markdown/html at that instant. Three occurrences, one pod, one burst: low impact as observed — but the failing owner is a payments-security container, and the same burst pattern appears in historical logs at many other owner paths.Not established
Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4bwas not read.MeshWeaver.Data.Serialization's owner-recycling code.GetofStore/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4bunder the triage identity returns "Not found" — but_Rejectedis under the Public/Anonymous deny, so that read is permission-shaped and says nothing about the node's existence. Its state, itsrejectionReason, and its created/modified timestamps were not read.Candidate causes and the reading that separates them
_Rejectednode's hub (per the exception's own diagnosis). Predicts: a recycle-requesting log site or job namingStore/Payments/_Rejectedor its hub, near 2026-10-13T19:52Z, on that pod. Reading: the pod's container logs around the burst, and theOps/Queues/maintenance-jobs registry for any job or watcher touching the Payments hub — thePaymentInboxWatcherarms onStore/Payments/_Inboxand the whole Payments tree is one hub, so a watcher-side recycle request would land here._Rejectednode type, such as itsreceivedSignature/rejectionReasoncontent schema). Predicts: a startup/activation exception in the pod log at the same instant, naming the hub and the failing initialization step. Reading: the pod log lines immediately BEFORE these three, and[QUIESCE-START]/activation lines for the owner address.Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b(or an ancestor) just before the burst; a Store admin's audit trail would show it. Reading:GetVersions/ the delete audit for that path, and whether the node reads under a Store-admin identity now. This would rhyme with the knownMarkdownView-on-deleted-node family (issues MarkdownView hard-fails rendering text whose backing node was deleted — stale view keeps reading a deleted OnePager comment #6130/MarkdownView hard-fails rendering text whose backing node was deleted — stale view keeps reading a deleted OnePager comment #6129/CollaborativeMarkdownView hard-fails when its target document node was deleted — read ends with the delete (stale Hardware/Workstations) #6080/CollaborativeMarkdownView hard-fails loading a document whose node was deleted (stale Hardware/SamuelGlauserMacBookAir) #6076) but with the sync-stream owner layer failing instead of the render layer — not established as the same defect.MarkdownViewin areaOverview/2subscribes to an owner path it is not authorized to read (the_Rejecteddeny), and each activation is refused by the authorization gate, with the recycle loop a secondary effect. Predicts: an authorization-refusal log line naming the triage identity and the path, one per activation attempt. Reading: the pod's auth/log lines for the owner address across the burst. This would make the view the defective component rather than the stream layer.None of these is established; each is separable by the named reading.
Where to look
MeshWeaver.Data.Serialization— the code that emitsSyncStreamOwnerRefusedExceptionand implements the "budget of N distinct owner activations" (no source node for it is present in the mesh I could read; it lives in the framework repo).MeshWeaver.Blazor.Components.MarkdownView— the log site; how it subscribes its style/markdown/html area loads and which node path it derives the owner address from.Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b(read under a Store-admin identity; the runbook for_Rejectedis at/Store/Payments, section "When a delivery cannot be verified").PaymentInboxWatcher.AddPaymentInboxWatcher, per/Store/Payments).Not a duplicate of any existing triage issue (searches over
Hosting/Triage/issueand the LogIncident partition found no prior ticket for this exception); the raw-log appearances at other owner paths (2026-09-26, 2026-10-07) were never triaged into issues and their owners differ from this one.Still occurring: 3 more since the last update (3 total).
memex-portal-deployment-54856649cb-m2jn9Recent log lines
systemorph-com commentedon Oct 13, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
797bcddf96b4b1b9(MeshWeaver.Blazor.Components.MarkdownView), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.9c4c66c8fab3b978797bcddf96b4b1b9MeshWeaver.Blazor.Components.MarkdownViewMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionStream HO0YVkEcjEmUYQMy_iqSZg: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.memex-cloudmemex-portal-deployment-54856649cb-m2jn9Admin/_LogIncident/9c4c66c8fab3b978What triage wrote about this fingerprint
Observed
A
MarkdownViewrender of the portal page areaOverview/2(bindingsstyle,markdown,html) terminated onMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionfor sync streamHO0YVkEcjEmUYQMy_iqSZg: the stream gave up on its owner — the hub atStore/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b— after four distinct owner activations refused it against a budget of three. The exception's own text states the address was "recycling in a LOOP, not settling: each successor dies before it can answer", and that the stream deliberately ends with this error so its readers can re-read.The owner path is a retained Stripe webhook delivery: per Store/Payments ("When a delivery cannot be verified"), a delivery whose signature does not verify but which names an order is kept verbatim at
_Rejected/{id}withrejectedAt/rejectionReason/receivedSignaturestamped, and_Rejectedsits under the Public/Anonymous deny (CatalogGate.SensitiveSegments) — the runbook's own reading of a non-empty_Rejectedis "someone may have paid and not been served".MarkdownViewis the reporting log site only; the view binds the stream, it does not cause the recycles.Where and when
A single burst inside one second, one pod, one namespace; first window of this fingerprint. The same second produced sibling incidents off the same page render: 1542d4b5163b29aa — the same owner address on a second stream (
_qeQIr5atUWnOqtIf2a7Rw), same four-activations-against-budget-three loop — and c9b666c065db562c — ownerAdmin/Subscriptions/sglauser, third stream, same wording. The same three stream ids surface across that window's other view-component incidents (Label, IconView, HtmlView, ContainerView, DispatchView, LayoutStackView). One page render, areaOverview/2, was reading at least three owner addresses, and every stream it held gave up at the same instant.Impact
One portal page failed to render at that instant on one pod. The stream is designed to end with this error so readers can re-read; whether the re-read recovered is not established. No data loss is evidenced. Judged
sev:M(the same-owner sibling was filedsev:L) because three distinct owner addresses — two subscription/payment hubs, one of them the payment-evidence container_Rejected— looped in the same window on one pod; a human may re-rule downward.Not established
Store/Payments/_Rejectedhub — the incident carries no stack frame and no[QUIESCE-START]reason. NoActivationRecyclerequest naming this path appears inAdmin/_Recyclefor this day (the requests readable there target other addresses, e.g.recycle-mcp-20261013182842381-1a9493→Personal/PersonalApp); that search is partition-limited, so it is absence of evidence, not evidence of absence.rejectionReasonsays: a direct read returns not-found, but_Rejectedis under the Public/Anonymous deny, so that read is permission-shaped and settles nothing.Admin/Subscriptions/sglauserloop shares a cause with this owner's loop or merely a window.Candidate causes and the reading that separates them
[QUIESCE-START]lines forStore/Payments/_Rejected/973c…in the seconds before the burst, each carrying a reason. Reading: the pod's container logs for that address around the burst settle it in one pass._Rejectedhub faults during initialization (for example on the retained-delivery content or its deny-gated access check) and never answers. Predicts an identical activation/init fault line repeated once per activation. Reading: the pod log lines immediately before the give-up._Rejected/{id}node (or an ancestor) just before the burst. Reading: the node's version history / delete audit under a Store-admin identity. Off-pattern if confirmed — the runbook retains rejected deliveries rather than deleting them, so a delete would be an unexplained write.Overview/2render subscribes several addresses and a single teardown wave (roll, type recycle) caught all of them. Predicts the same give-up on other pods at the same instant and a deploy marker in the window. Reading: whether any other pod emitted this fingerprint in the same window.Where to look
MeshWeaver.Data.Serialization—SyncStreamOwnerRefusedException, the distinct-owner-activation budget, and the handoff that counts them.MeshWeaver.Blazor.Components.MarkdownView— the reporting site only.Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b, readable under a Store admin; runbook: Store/Payments, "When a delivery cannot be verified".Already known / likely duplicate
This is the same event window as #6725, opened from this second's sibling incidents (
1542d4b5163b29aa— the same owner address on a second stream;c9b666c065db562c—Admin/Subscriptions/sglauser), and the same give-up mechanism was filed earlier the same day as #6709/#6713 on other owner addresses (per the record on those incident nodes). This ticket is a distinct fingerprint on the same owner and instant; it can likely be closed as a duplicate of #6725 in seconds.Still occurring: 3 more since the last update (3 total).
memex-portal-deployment-54856649cb-m2jn9Recent log lines
systemorph-com commentedon Oct 13, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
797bcddf96b4b1b9(MeshWeaver.Blazor.Components.MarkdownView), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.436616b05ce8abc4797bcddf96b4b1b9MeshWeaver.Blazor.Components.MarkdownViewMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionStream daxfo7{n}X4{n}yAQZdHkDPA0Q: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.memex-cloudmemex-portal-deployment-85f6665f88-mzv8mAdmin/_LogIncident/436616b05ce8abc4What triage wrote about this fingerprint
Observed
On pod
memex-portal-deployment-85f6665f88-mzv8m(namespacememex-cloud), aMarkdownView(MeshWeaver.Blazor.Components.MarkdownView) rendering areaOverview/2/1/2failed all three of its content loads ('style', 'markdown', 'html') withMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionon sync streamdaxfo70X40yAQZdHkDPA0Q: the owner hub atAdmin/Subscriptions/sglauserwas refused by 4 DISTINCT owner activations against a budget of 3. The exception's own text records that the owner address was recycling in a loop, not settling — each successor activation died before it could answer — and that the stream ends with this error so its readers can re-read.Where and when
A single burst: 3 log lines within ~65 ms at 2026-10-13T20:22:46Z, all on one pod, all one stream id and one owner. The three lines are the three sub-loads of one MarkdownView instance — one page-load failure, not three independent events. Nothing on the incident establishes what coincided with it (a roll, a recycle job, a sync); no coincidence is claimed.
Impact
One user-facing page area (
Overview/2/1/2) failed to render its markdown content on that load. The stream is designed to end with this error so readers re-read; whether the re-read recovered is not in the evidence.Not established
Admin/Subscriptions/sglauser— an activation that dies during startup, an external recycle trigger, or an operational action — is not fixed by this evidence.Approvals,UWDeepfield/Section,RFPProcessor,ThinkInStreams/_Policy— each "4 DISTINCT owner activations … budget 3"), but no existing GitHub issue covers the symptom; searches of Hosting/Triage found none. This ticket covers the concrete burst above.Candidate causes and the reading that separates them
Admin/Subscriptions/sglauser, each fresh activation dying during startup before it can serve the subscribe. Predicts recycle/teardown log lines at that address around 20:22:46 and refusals of other streams owned by it. Reading: the owner hub's own logs on the same pod at that timestamp — the recycle reason and how many recycles.Recyclejob touching that address) coincided with the page load; the loop is not a defect of this code path. Predicts one recycle window and no recurrence after it. Reading: Hosting/Queues job records or admin audit entries namingAdmin/Subscriptions/sglausernear 20:22:46, and whether later loads of the same area succeed.Where to look
MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionand the owner-subscription retry budget (budget 3) in MeshWeaver.Data's sync-stream serialization code — the architecture pages in this mesh locate the stream lifecycle insrc/MeshWeaver.Data/Serialization/JsonSynchronizationStream.cs(seeDoc/Architecture/AReleaseRefusedAtTheSource,Doc/Architecture/EvictedStreamRetention).MeshWeaver.Blazor.Components.MarkdownView(log site:Loading '{value}' in area '{value}' in Area {path}).Admin/Subscriptions/sglauserand its activation/recycle history around the burst.The exception's own message states the diagnosis the framework itself records: the stream ending is by design (so readers re-read); the defect, if any, is at whatever was requesting the recycles at
Admin/Subscriptions/sglauser— which is exactly what none of the current evidence identifies.Still occurring: 3 more since the last update (3 total).
memex-portal-deployment-85f6665f88-mzv8mRecent log lines
systemorph-com commentedon Oct 13, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
797bcddf96b4b1b9(MeshWeaver.Blazor.Components.MarkdownView), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.e16c16e07b296cd0797bcddf96b4b1b9MeshWeaver.Blazor.Components.MarkdownViewMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionStream {n}SFoN8p8{n}CmrYYhEJYi-g: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.memex-cloudmemex-portal-deployment-85f6665f88-mzv8mAdmin/_LogIncident/e16c16e07b296cd0What triage wrote about this fingerprint
Observed
Three
faillog lines fromMeshWeaver.Blazor.Components.MarkdownView, one per resource ('style', 'markdown', 'html') of a MarkdownView rendering areaOverview/2/1/2, all at 2026-10-13T20:22:46.340Z on podmemex-portal-deployment-85f6665f88-mzv8m(namespacememex-cloud), each carryingMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: sync stream0SFoN8p830CmrYYhEJYi-gended withgiving up on owner Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b — 4 DISTINCT owner activations have refused this stream (budget 3). The address is recycling in a LOOP, not settling: each successor dies before it can answer.So the portal view's own loads succeeded; the failure is upstream on the stream's owner — four successive activations of the node-type hub owning
Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4bdied before they could serve the stream, and the delivery layer stopped retrying after its budget of 3 owner-activation changes and terminated the stream with this exception so its readers re-read. The three lines are the same exception delivered to three reader components of one render, not three incidents.Where and when
All three occurrences are within 36 ms of each other at 20:22:46.340Z, on a single pod, against a single node path. This is one burst at one node, not a fleet-wide condition. The owner address lies under
Store/Payments/_Rejected/— a rejected-payment record — so the failing activation is plausibly the rejected-payments store's own hub, but that was not readable from this mesh (noStore/Paymentspartition is visible to this triage; the node itself could not be read from here).Impact
A portal page in the payments area renders without its markdown content (the view logged 'Loading ... in area Overview/2/1/2' and its stream ended in error). With three occurrences on one pod in one burst, the user-visible impact is one page render at one instant; the exception is by design terminal-but-recoverable ("the stream now ENDS with this error so its readers can re-read"). Judged from the numbers this is a narrow, single-node event — but the message itself asserts the owner activation is dying repeatedly (4 distinct successors in one window), which if it persists at that address is a real defect in whatever keeps recycling that hub.
Already-known shapes (checked, not duplicates)
Ops/Logscarries many earlierSyncStreamOwnerRefusedExceptionlines on both memex and memex-cloud pods (e.g.memex-...-94f9b5-ggf882026-09-26, ownersApprovals,Marketing,Testing,Crm/Client,Chat,UWDeepfield...;...-b7c796-jqrtw2026-10-07;...-64bddf-v5s6s2026-10-07) — always "4 DISTINCT owner activations ... (budget 3)", i.e. the same loop signature at many owner addresses. None of these appear inHosting/Triageas a filed issue, so this is the first ticket for the shape; earlier occurrences may attach to it later.Hosting/Triage/issue/systemorph-meshweaver-6034— "Second write racing a retype's hub recycle is refused as Unknown instead of retry-this" — is about a different exception class and different mechanism (a write refused during a retype recycle), not a stream owner refusing across four activations. Not a duplicate, but the same neighbourhood of hub-recycle behaviour.Feedback/_Submissions/fb-a25024edf541725c0ca4ac7c7ce44729(2026-10-04, cross-hub writer saw store refusal as a bare timeout forOps/Status/memex) documents the sibling failure mode — the requester sees silence while the owner dies — but for a different exception and path. Related context, not a duplicate.Not established
DisposeRequestloop, aStorepersistence fault). Cause not established.Ops/Logshistory, which suggests either (a) a systemic cause that keeps recycling hubs fleet-wide, or (b) the delivery layer's give-up budget (3) simply being exceeded routinely whenever any owner recycles more than 3 times. Not established from this evidence.MarkdownViewsite.Candidate causes and the reading that separates them
SyncStreamOwnerRefusedExceptionat the SAME owner path over time, and owner-side log lines (activation init failures,DisposeRequest, store faults) fromStore/Payments/_Rejectedon the same pod in the same window. Reading that separates it: owner-side logs forStore/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4baround 20:22:46Z onmemex-portal-deployment-85f6665f88-mzv8m— a repeated init exception or repeatedDisposeRequestthere confirms it; absence of owner-side faults weakens it.Recycleaction, see theActivationRecyclework inHosting/Triage: PRs ActivationRecycle: recycle a NodeType's live activations on every process #6054, fix(graph): defer recycle service resolution until request delivery #6072, EXPRESS: ActivationRecycle resolves nothing heavy on the mesh hub's init turn (Autofac lock-order deadlock reds Plugins main) #6084, issue Recycle jobs for the Hosting platform's own NodeTypes are enqueued without their declaration seal and refused #6217 — recycle jobs for the Hosting platform's own NodeTypes enqueued without their declarations). If a recycle storm repeatedly tore down the owner's activation, four successors within the delivery window would produce exactly this. Predicts a recycle/job log entry in the same window, and the same exception at OTHER owner addresses simultaneously. Reading that separates it: whether other nodes' streams failed with the same exception in the same seconds (would show inOps/Logsaround 20:22:46Z), and whether a recycle job was enqueued for that NodeType then.Ops/Logsentries above suggest it is not rare); if owner-side logs are clean at every occurrence, this candidate gains weight.Where to look
MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptioninMeshWeaver.Data.Serialization— the delivery layer that counts distinct owner activations and ends the stream at budget 3.Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b— its activation init and its disposal/recycle path on podmemex-portal-deployment-85f6665f88-mzv8m.Doc/Architecture/StreamLivenessAndTheHubReference(hub teardown vs. stream liveness; the "overlay self-heal recycle window" andHostedHubsCollectionbehaviour) andDoc/Architecture/HubDisposalModel(whyDispose()returning is not teardown having happened).Hosting/Triage/issue/systemorph-meshweaver-6034for the adjacent retype-recycle refusal shape.Severity reasoning: one burst, one pod, one node path, three lines in 36 ms — the affected page can re-read and recover. But the exception names a mechanism (an owner recycling without settling) that the logs show recurring at many addresses. Judged as a clear defect with a narrow blast radius: sev:M. A human with owner-side logs may re-rule.
Still occurring: 3 more since the last update (3 total).
memex-portal-deployment-85f6665f88-mzv8mRecent log lines
systemorph-com commentedon Oct 13, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
797bcddf96b4b1b9(MeshWeaver.Blazor.Components.MarkdownView), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.a0bda63fb05163b2797bcddf96b4b1b9MeshWeaver.Blazor.Components.MarkdownViewMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionStream ctBtZgYcpUKmU4{n}Z-{n}wZyQ: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.memex-cloudmemex-portal-deployment-85f6665f88-mzv8mAdmin/_LogIncident/a0bda63fb05163b2What triage wrote about this fingerprint
Observed
MeshWeaver.Blazor.Components.MarkdownViewfailed to load three bound properties (style,markdown,html) in areaOverview/2/1/2of the nodeStore/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4bwhen the sync streamctBtZgYcpUKmU47Z-1wZyQended withMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedException: 4 DISTINCT owner activations at that address refused the stream, exceeding the per-stream budget of 3. The exception's own text records that the address was recycling without settling — each successor activation died before it could answer — and that the stream was ended so its readers can re-read.MarkdownView is where the symptom surfaced, not where the failure lives: the same stream and owner produced sibling incidents at the same instant on the same pod at the DispatchView, Label, ContainerView, HtmlView, IconView and LayoutStackView log sites (e.g.
Admin/_LogIncident/797bbeaa68e47f7a), all one render tree reading one node. The failing component is the sync-stream owner at the addressStore/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b, whose activations repeatedly die.Where and when
One burst: 3 log lines within 22 ms at 2026-10-13T20:22:46Z, all on pod
memex-portal-deployment-85f6665f88-mzv8m, all the same stream, owner and view area — one user's view load. That pod belongs to deployment revision85f6665f88, which does not appear in the pod list of the roll-churn incident (Admin/_LogIncident/roll-churn-1e98b289db877f91, last seen 20:04:49Z), so the burst landed on a pod of a newer revision ~18 minutes after that churn window — consistent with the burst coinciding with a roll, but that is inference, not observation. Fleet-wide NodeType bake failures were active until 20:12 (Admin/_LogIncident/448cabb8229c98be).Impact
Judged from the numbers: the portal view of one rejected payment node failed to render once, on one pod, in one burst. The stream ends with the error so its readers can re-read. No other pod, tenant or node shows this fingerprint in the window.
Not established
storepartition (searches againstStore/Paymentsreturn empty under this identity), so the node's type, sources and recent versions could not be inspected.Candidate causes and the reading that separates them
GetDiagnosticson the node's NodeType.Duplicate check
No existing issue in
Hosting/Triagecovers this condition. The symptom class recurs:Ops/Logscarries "giving up on owner … 4 DISTINCT owner activations" bursts since 2026-09-26 against many different owners (UWDeepfield, Approvals, BinaryClickerV2, RFPProcessor, thread paths), always folded per log site; each burst's owner differs, so per-instance causes may differ too.Where to look
MeshWeaver.Data.Serialization—SyncStreamOwnerRefusedExceptionand the owner-activation/refusal-budget machinery that emitted the LOOP wording.Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4band its activation/recycle records.memex-portal-deployment-85f6665f88-mzv8m, log for 2026-10-13T20:22:40–50Z.Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4bin thestorepartition (readable by identities with access there).Still occurring: 3 more since the last update (3 total).
memex-portal-deployment-85f6665f88-mzv8mRecent log lines
rbuergi commentedon Oct 14, 2026
This is a duplicate of #6729, which is the anchor for the whole
SyncStreamOwnerRefusedExceptionfamily. The fix is in #6756.The root cause was the subscriber's own host stopping, not an owner recycle loop. The pod logged
Application is shutting down...in the same second as this burst. Its router answered each re-ask with an untaggedShuttingDownrefusal (Host is shutting down, cannot route to …), and the sync stream counted each of those as a "distinct owner activation". The fix rides out such routing-layer refusals on the paced time axis, and their give-up names the route instead of an owner.Closing as a duplicate. Verification follows on #6729.
systemorph-com commentedon Oct 14, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
797bcddf96b4b1b9(MeshWeaver.Blazor.Components.MarkdownView), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.d0725adb9e7a8d3d797bcddf96b4b1b9MeshWeaver.Blazor.Components.MarkdownViewMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionStream nZAlMSigYEeetBkkgrZ0rA: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.memex-cloudmemex-portal-deployment-6bd97645d4-fwns9Admin/_LogIncident/d0725adb9e7a8d3dWhat triage wrote about this fingerprint
Observed
Three
MeshWeaver.Blazor.Components.MarkdownViewloads — the fieldsstyle,markdownandhtmlin areaOverview/2/1/2— terminated withMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionfor streamnZAlMSigYEeetBkkgrZ0rA, whose owner isAdmin/Subscriptions/sglauser(node typeStore/Subscription, "sglauser — Enterprise (staff)", state Active, untouched since 2026-09-06). The stream layer reports 4 DISTINCT owner activations refused the stream against a budget of 3, and that the address is recycling in a LOOP — each successor activation dies before it can answer — after which the stream ends so its readers can re-read. MarkdownView is where the failure surfaced; the exception's own diagnostic locates the origin at whatever is requesting the recycles of that address, not at the component.At the same instant on the same pod, two sibling incidents of the same burst show the same refusal at a second, different owner address:
ContainerViewloads ofOverview/2/1/1…on the same stream/owner, and aThreadChatViewcomposer-context resolution on streamQlv5xA9kB0WbKiLVHT_zDAwhose owner is the user hubsglauseritself. So two addresses tied to one user — the subscription node and the user's own hub — were both recycling in loops at the same moment.Where and when
One pod of deployment generation
6bd97645d4, one instantaneous burst (all three loads within ~100 ms), namespacememex-cloud. This is the third deployment generation in two days showing this failure mode: bursts on generation679d7b6984(2026-10-13 ~17:20), on54856649cb(2026-10-13 19:52–20:27, dozens of component log sites and hundreds of bindings per burst, across unrelated owners on multiple pods), and now6bd97645d4. Unlike yesterday's bursts, which fanned across unrelated owners (agent threads, aStore/Payments/_Rejected/…node), this one touches only addresses belonging to a single user.Impact
At that instant, the subscription-overview view bound to
Admin/Subscriptions/sglauserfailed to render its MarkdownView cells on one pod, and the same user's chat composer failed to resolve its context. The stream ends by design so its readers re-read; whether the views actually recovered is not established. Yesterday's bursts show the same mechanism can terminate hundreds of bindings at once on arbitrary owners, so the family is broader than this instance of it. Judgedsev:Mfor this ticket: a secondary rendering path broken for one user at one instant, with the reader-re-read mechanism already in place; the family's spread is tracked under the anchor issues below.Likely duplicate — reference before closing
#6723 (filed 2026-10-13 19:52 from incident
Admin/_LogIncident/846736a784dac1ec, ownerStore/Payments/_Rejected/973c…) and #6732 (fromAdmin/_LogIncident/078d47784ea3865c, owner agent threadphuong/_Thread/…) cover the same exception and the same "owner recycling in a loop, 4 activations vs budget 3" mechanism at other owners, streams, pods and instants. If the recycle loops have one requester-side cause, this ticket folds into the anchor; this burst contributes the per-user multi-address shape (subscription node and user hub refusing simultaneously) as evidence of the family's spread, and a human can close it as a duplicate in seconds.Not established
Admin/Subscriptions/sglauserand of the user hubsglauserat that moment, or why each successor owner activation died before it could answer. The incident carries no stack beyond the component log sites and no recycle/quiesce lines.Candidate causes and the reading that separates them
Admin/Subscriptions/sglauserand the user hubsglauserin the seconds before the burst on this pod. Reading: hub activation/quiesce logs for this pod and window.6bd97645d4— hubs not yet settled when first served. Predicts a pod start time close to the burst, no further bursts from this pod once warm, and the same failure for any user first hitting a cold pod. Reading: deployment and pod-start events for the generation against the burst timestamp. Note the family spans three generations in two days, which weakens a single-rollout explanation for the family as a whole but not for this burst.Where to look
MeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionin the stream layer ofMeshWeaver.Data— the per-stream DISTINCT-activation refusal budget it reports (3 here).Store/Subscription) and the user hubsglauser— their hub activation/quiesce logs on the reporting pod around the burst instant.Admin/_LogIncident— the requester of the recycles is the common unknown.Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 3 more since the last update (3 total).
memex-portal-deployment-6bd97645d4-fwns9Recent log lines
systemorph-com commentedon Oct 14, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
797bcddf96b4b1b9(MeshWeaver.Blazor.Components.MarkdownView), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.412bc5ee9348a867797bcddf96b4b1b9MeshWeaver.Blazor.Components.MarkdownViewMeshWeaver.Data.Serialization.SyncStreamOwnerRefusedExceptionStream IGyFG7mKpU6xDUJA2KdWdg: giving up on owner {path} — {n} DISTINCT owner activations have refused this stream (budget {n}). The address is recycling in a LOOP, not settling: each successor dies before it can answer. The stream now ENDS with this error so its readers can re-read; the fix is at whatever is requesting the recycles, not here.memex-cloudmemex-portal-deployment-6bd97645d4-fwns9Admin/_LogIncident/412bc5ee9348a867What triage wrote about this fingerprint
Observed
A
JsonSynchronizationStream(MeshWeaver.Data.Serialization) gave up on its owner addressStore/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4bafter 4 distinct owner activations refused the stream against an activation budget of 3 — the "recycling in a LOOP, not settling" shape. Each successor activation of the owner died before it could answer the re-ask, so the stream ended withSyncStreamOwnerRefusedException(the #6602 give-up terminal) and theMarkdownViewreading it (areaOverview/2/1/2, loadingstyle/markdown/html) faulted instead of rendering.This is not an isolated event at one log site. In the same sub-second window, sibling incidents record the same 4-distinct-activations refusal on at least three other streams and two other owner addresses:
Admin/Subscriptions/sglauser(incidentAdmin/_LogIncident/d0725adb9e7a8d3d), the user addresssglauserfor aThreadChatViewcomposer context (incidentAdmin/_LogIncident/0cf50d9d7d625b31), and furtherSyncStreamOwnerRefusedExceptionbursts acrossHtmlView,LayoutStackView,ContainerView,DispatchView,IconViewandLabel(incidentsAdmin/_LogIncident/de37656edeebd428,cfbde2ffdf2d3b31,4e9b02f64b62821d,0719220b05ede97b,db058e1f769a9600,4858ce400f52ca71,35b45f73bdabd399,0750968ec7cf42a4,0f068e74f04f62bb,aebee61e6325fca7,c808344675bbdedf,0fe35bde1741db3e,2e3172f47480d12b). Multiple owner addresses recycling through several activations each, in the same instant, is the "many owners, many probes" shape — the cause is at whatever is requesting the recycles, not at the stream rider.Where and when
One pod of the
memex-cloudnamespace, one burst well under a second long. The wider incident board shows the same deployment in heavy churn in the minutes before:MeshNodeCompilationServiceCompilationExceptionfailures,nodetype_bakehealth checks reportingUnhealthy,MessageServiceACTION-BLOCK OVERLOAD, andSiloUnavailableExceptionroll-churn across the memory silos. Whether the recycle loop is part of that churn or independent of it is not established.Impact
Portal views reading the affected nodes failed to load for the duration of the loop — a bounded, user-visible availability blip. By design the stream ENDING lets its readers re-read, and
MeshNodeStreamCache/AreaErrorClassifiertreat the type as transient, so recovery is automatic once an owner activation survives long enough to answer. No data loss is indicated; nothing on the incident shows a write at risk. Judged from the numbers this is a secondary-path blip on one pod, but a genuine defect signal: four owner activations dying back-to-back is not normal teardown.Not established
StreamEndedEvent, grain-vs-hub tag divergence in Activity tracking fails to hydrate user activity node — inner UpdateNode aborts with no initial state within 30s #5256).Candidate causes and the reading that separates them
DisposeRequestpostings forStore/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b,Admin/Subscriptions/sglauserand the user hub in the seconds before 07:26:58, and genuinely distinct activation tags in the per-rejection Warning lines. Reading: the Loki warnings for streamIGyFG7mKpU6xDUJA2KdWdgand its siblings — each line names the owner and the activation tag; four distinct tags confirm the storm, and whoever posted the teardowns is the next hop.MeshNodeCompilationServiceincidents,nodetype_bakeUnhealthy) can fail activation, and a failed activation answers nothing — a succession of those is the loop. Predicts the owner paths' NodeTypes appear in the compile-failure incidents on the same window. Reading: cross-check the compile-failure fingerprints forStore/Payments/ subscription NodeTypes in the minutes around the burst.Where to look
MaxRecycleReArms = 3, the give-up terminal, and the "many owners, many probes → look at whatever is asking for the recycles" rule.JsonSynchronizationStream(MeshWeaver.Data.Serialization) — where the budget is charged andSyncStreamOwnerRefusedExceptionis minted; the per-rejection Warning lines it logs before giving up are the primary evidence for candidate 1 vs 2.ShutdownNack(MeshWeaver.Messaging.Contract) — activation-token minting/parsing; a tagging drift would live here or at aStreamEndedEventproducer.ASyncStreamThatGivesUpOnItsOwnerEndsTest(MeshWeaver.Data.Test) — pins the counting behaviour; a repro of the mis-count shape belongs next to it.Not a duplicate: no existing issue covers
SyncStreamOwnerRefusedException; the closest neighbours are #5344 (aQueryProviderStalledExceptionat the same portal, different terminal, Done) and the roll-window read/write riders #5256/#5011 (different failure mode — silent wait, not a budget give-up).Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 3 more since the last update (3 total).
memex-portal-deployment-6bd97645d4-fwns9Recent log lines