Bulk stream-teardown flood saturates the memex-cloud mesh hub turn loop, tripping the aggregate action-block breaker

What is failing. A single hub on memex-cloud (mesh/YxuIePH430S3wXXs-HRa8A) received a burst of sheddable infrastructure traffic — predominantly StreamEndedEvent, with some HeartBeatEvent — that drove its single-threaded action-block inbound queue past the 10,000-message aggregate watermark within seconds. The aggregate breaker shed the sheddable traffic as designed (user-facing and lifecycle messages are never shed), so the hub stayed alive, but an amplifying source is still flooding this block far past its drain rate.

Probable cause. The shape of the shed traffic points at a bulk stream teardown: hundreds of sync/area streams ended in rapid succession (e.g. a disconnecting client that held many subscriptions, a mass cancel, or a teardown that itself re-emits). This is the exact scenario Action-Block Wedge Prevention Invariant 3 was built to survive — the breaker worked; the source that amplified the flood is the defect to find. Confidence: medium — the log tells us what flooded (StreamEndedEvent at scale) but not which sender or client drove it; a delivery fate-trail (SHED_AGGREGATE) analysis on that hub will identify the amplifier.

Impact. 454 overload events in ~36 seconds on one pod and one hub. User-facing messages were protected by the shed policy, so the likely user effect is latency/stalls for whoever was active on that hub's partition during the window, not message loss. Not a systemic outage — one hub, one pod, one window.

Where to look. Start with MeshWeaver.Messaging.MessageService ingestion logging on pod memex-portal-deployment-5f95fb94dc-pkhnc, hub mesh/YxuIePH430S3wXXs-HRa8A, window 2026-09-23 13:36:18–13:36:54 UTC: pull the shed deliveries' SHED_AGGREGATE fate trails and MessageStormBreaker.AggregateSheds to identify the sender(s) emitting the StreamEndedEvent storm. Then inspect the stream-teardown / StreamEndedEvent emit sites (SynchronizationStream, area streams) for a teardown path that emits per-stream endings without coalescing, and any client that holds very many streams on this hub. Related background: #1200 (legitimate wide fan-out vs breaker keys) and #1485 (envelope-vs-payload exemption checks) — if the flood turns out to be legitimate bulk fan-out rather than a defect, this ticket becomes 'raise/override the hub watermark' via MessageHubConfiguration.WithAggregateWatermark(...) instead.

Evidence


Fingerprint	`f2efe2be834b6801`
Category	`MeshWeaver.Messaging.MessageService`
Severity	Error
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-5f95fb94dc-pkhnc`
Occurrences	454
First seen	2026-09-23 13:36:18Z
Last seen	2026-09-23 13:36:54Z
Routing	not determined — no configured route matches the category `MeshWeaver.Messaging.MessageService`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-23 13:36:53Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/YxuIePH430S3wXXs-HRa8A: inbound queue depth 10000 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. HeartBeatEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-09-23 13:36:54Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/YxuIePH430S3wXXs-HRa8A: inbound queue depth 10000 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-09-23 13:36:54Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/YxuIePH430S3wXXs-HRa8A: inbound queue depth 10000 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-09-23 13:36:54Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/YxuIePH430S3wXXs-HRa8A: inbound queue depth 10000 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-09-23 13:36:54Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/YxuIePH430S3wXXs-HRa8A: inbound queue depth 10000 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.

_{Re-addressed by the current identity function: this incident inherited 7a92a4f2b427a9ef (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/f2efe2be834b6801. Recurrences are folded into this issue rather than opening new ones.}

Found a sender from code that is pure waste by construction. Fixed in #5635 (merged, `bf5ac85526`), awaiting roll.

The amplifier. JsonSynchronizationStream posts StreamEndedEvent from every owner-side stream disposal (#2191), and that included the disposal the SUBSCRIBER asked for (UnsubscribeRequest → hub.Dispose()). By then the subscriber's sync/ hub had already disposed itself, so the announcement crossed the subscriber pod's mesh hub only to be dropped there ([CanBeIgnored]). Now take a subscriber that releases N mirrors at once, for example a cache hub going down: it sends N UnsubscribeRequests, and N owners each send one StreamEndedEvent back through the same router turn loop. That is one sender shape producing N distinct-key sheddable messages to ONE hub, which is what this ticket describes.

Fix (root, not batching): the owner's sync hub records that the subscriber ended it (SynchronizationStream.MarkSubscriberEnded, set in the UnsubscribeRequest handler before dispose), and the announcement skips that end. Owner-side ends (idle release, EvictClientSubscriptions) are still announced. SubscriberReleaseIsNotAnnouncedBackTest: a release gives 0 announcements, and the owner-side end (the control) gives 1. With the fix disabled it fails with "expected 0 … but found 1". Doc: LiveMirrorsAndTheChangeFeed (step 5).

Not established: which emitter produced the 2026-09-23 13:36Z burst on mesh/YxuIePH430S3wXXs-HRa8A. The other emitter, Workspace.AnnounceRecycleToClientSubscriptions, sends one event per LIVE subscriber of a recycled hub. Those events are needed, and they are not changed here. Telling the two apart needs a Logs read of that window (the SHED_AGGREGATE fate trails, and the sender address on the dropped StreamEndedEvents). The control instance is disconnected for this session.

Deployed: memex.meshweaver.cloud runs core 9a61fed964, which predates this. Verification after the roll: no aggregate-watermark trips shedding StreamEndedEvent on a replica carrying bf5ac85526. If they recur there, read the sender. A sender that is a carrier hub (nodeops) points at the recycle announcer.

Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet

What is running.

memex-cloud: /api/version returns 9a61fed964. Ops/Status/memex-cloud (06:42:00Z) shows 4/4 replicas on 3.0.0-ci.9263. CD run 9263 built core 9a61fed964 with Plugins 41bcfe968f.
memex: /api/version returns 39f49494eb. Ops/Status/memex shows the serving pods on 3.0.0-ci.9218, which is core 39f49494eb with Plugins e9d9283abe. A third pod, on ci.9260, is unready, so the roll is converged: false.

This issue's fix: #5635 (bf5ac85526). It is not an ancestor of either running build (git merge-base --is-ancestor, exit 1).

Where it is. 3.0.0-ci.9287 (CD run 35956222018) is core bf5ac85526 with Plugins c4795a8d5b, and it contains this fix. The registry has sealed it. memex-cloud's self-updater logged [SelfUpdate] handing 3.0.0-ci.9287 to the control lane (https://memex.systemorph…) at 06:47Z. So the roll now waits on the control instance, which is an owner step and was not done here.

Still open. It closes on a post-roll reading once a ci.9287-or-later image serves the namespace this fingerprint fires in.

Closing: fix live, no recurrence for ≥ 6 h, positive control present

Image. memex-cloud (memex.meshweaver.cloud) serves 3.0.0-ci.9291. /api/version returns core 140c5fb7ae, and Ops/Status/memex-cloud (sampled 14:49Z) shows all 4 replicas on that tag, converged: true. The Plugins half is b3601fecbe: the portal-ai build job of CD run 35970157348 (job 107542528900) logs HEAD is now at b3601fe. The first pods of this ReplicaSet (memex-portal-deployment-589ff8f895) started at 08:41:05Z and 08:42:17Z. More pods of the same ReplicaSet, on the same image, joined later (gqsfn at 14:06Z, vw5jj at 14:10Z, and others). The control instance memex.systemorph.com still serves core 39f49494eb.

Window and method. I ran Logs InstanceActions on memex.meshweaver.cloud between 14:43Z and 14:54Z, with sinceMinutes 370/375, filtered to this ReplicaSet with | pod=~"memex-portal-deployment-589ff8f895-.+". That is ≥ 6 h of this image only, from 08:41Z to about 14:50Z. Coverage control: Ops/logs-memexcloud-20260924-close6h-c-bootcoverage (warm-up complete) lands 9 lines from 9 pods of this ReplicaSet. The earliest are ghgjt, vn49b and tp564 at about 08:51Z, so the window reaches back to the first boots.

Fix: #5635 (bf5ac85526). It is an ancestor of the running build (git merge-base --is-ancestor, exit 0).

reading	query	lines
fingerprint	`\|~ "(?i)ACTION-BLOCK OVERLOAD"` (`Ops/logs-memexcloud-20260924-close6h-f-overload`)	0
positive control (same logger category)	`\|~ "MeshWeaver.Messaging.MessageService"` (`Ops/logs-memexcloud-20260924-close6h-c-messageservice`)	≥ 5 (cut at limit 5)

Before the fix, this fired as a single 454-line burst (2026-09-23 13:36Z). Across ≥ 6 h that includes pod joins and replacements, the fixed image logged no overload line in any hub.

Closing as completed. If this shape recurs on an image that contains the fix, reopen and treat it as a new path. Readings were taken read-only through the memex MCP.

Reopened — this fault is happening again.

Still occurring: 2 more since the last update (456 total).

Last seen: 2026-10-11 19:21:36Z
Pods: memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-b79b66b68-w488f

Recent log lines

2026-09-23 13:36:54Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/YxuIePH430S3wXXs-HRa8A: inbound queue depth 10000 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-11 19:21:35Z memex-portal-deployment-b79b66b68-w488f fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub portal/nodeops-ZjzXQ4RCv0K2l430tkdI5A: inbound queue depth 30907 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-11 19:21:36Z memex-portal-deployment-b79b66b68-w488f fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/ZjzXQ4RCv0K2l430tkdI5A: inbound queue depth 44118 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.

_{Re-addressed by the current identity function: this incident inherited 7a92a4f2b427a9ef (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Reopened — this fault is happening again.

Still occurring: 2 more since the last update (456 total).

Last seen: 2026-10-11 19:21:36Z
Pods: memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-b79b66b68-w488f

Recent log lines

2026-09-23 13:36:54Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/YxuIePH430S3wXXs-HRa8A: inbound queue depth 10000 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-11 19:21:35Z memex-portal-deployment-b79b66b68-w488f fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub portal/nodeops-ZjzXQ4RCv0K2l430tkdI5A: inbound queue depth 30907 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-11 19:21:36Z memex-portal-deployment-b79b66b68-w488f fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/ZjzXQ4RCv0K2l430tkdI5A: inbound queue depth 44118 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.

_{Re-addressed by the current identity function: this incident inherited 7a92a4f2b427a9ef (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

test(data): reproduce the per-stream recycle announcement fan-out

Still occurring: 1 more since the last update (457 total).

Last seen: 2026-10-12 01:26:43Z
Pods: memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-b79b66b68-w488f, memex-portal-deployment-6c95ffcd84-h4tr5

Recent log lines

2026-10-11 19:21:35Z memex-portal-deployment-b79b66b68-w488f fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub portal/nodeops-ZjzXQ4RCv0K2l430tkdI5A: inbound queue depth 30907 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-11 19:21:36Z memex-portal-deployment-b79b66b68-w488f fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/ZjzXQ4RCv0K2l430tkdI5A: inbound queue depth 44118 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-12 01:26:43Z memex-portal-deployment-6c95ffcd84-h4tr5 fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/xf4z3K4sdEOX0Jhh-nBNCA: inbound queue depth 16868 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.

_{Re-addressed by the current identity function: this incident inherited 7a92a4f2b427a9ef (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

fix(data): announce an owner recycle once per subscriber, not per stream

Status of the fix, PR #6539 (head 81cba5dc38), 2026-10-12 16:35Z:

MeshWeaver Build and Test: 23 of 24 jobs are green. The remaining one, Dependent suites (MeshWeaver.Plugins), is still running. No job has failed.
The only red is Automatic review answered: "the automatic review has not landed — no review by the automatic reviewer on this pull request". The reviewer opened 0 threads and nothing is unanswered. That needs the review to land, or a maintainer re-request or review-waived. An agent can do neither.
Auto-merge is now armed (squash).

Not checked: whether the 2026-10-12 01:26:43Z occurrence (mesh/xf4z3K4sdEOX0Jhh-nBN… on …-6c95ffcd84-h4tr5) has the same producer as the 10-11 19:21Z one that #6539 is built on.

Still occurring: 2 more since the last update (459 total).

Last seen: 2026-10-12 23:39:00Z
Pods: memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-b79b66b68-w488f, memex-portal-deployment-6c95ffcd84-h4tr5, memex-portal-deployment-5b786c4466-4fs6c

Recent log lines

2026-10-12 01:26:43Z memex-portal-deployment-6c95ffcd84-h4tr5 fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/xf4z3K4sdEOX0Jhh-nBNCA: inbound queue depth 16868 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-12 23:38:57Z memex-portal-deployment-5b786c4466-4fs6c fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/-dLslaS_80q6Xpd2KjJrKA: inbound queue depth 10305 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-12 23:39:00Z memex-portal-deployment-5b786c4466-4fs6c fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub portal/nodeops--dLslaS_80q6Xpd2KjJrKA: inbound queue depth 59431 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.

_{Re-addressed by the current identity function: this incident inherited 7a92a4f2b427a9ef (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 7 more since the last update (466 total).

Last seen: 2026-10-13 04:00:59Z
Pods: memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-b79b66b68-w488f, memex-portal-deployment-6c95ffcd84-h4tr5, memex-portal-deployment-5b786c4466-4fs6c, memex-portal-deployment-54856649cb-m2jn9, memex-portal-deployment-54856649cb-4hwrp

Recent log lines

2026-10-13 00:08:41Z memex-portal-deployment-54856649cb-m2jn9 fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub portal/nodeops-c9JVoVItPkah_rquk6wNJA: inbound queue depth 10591 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-13 00:08:50Z memex-portal-deployment-54856649cb-m2jn9 fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/c9JVoVItPkah_rquk6wNJA: inbound queue depth 10301 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-13 04:00:59Z memex-portal-deployment-54856649cb-4hwrp fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub portal/nodeops-5kH5XgIhYUOQctcGO_TXIg: inbound queue depth 18720 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.

_{Re-addressed by the current identity function: this incident inherited 7a92a4f2b427a9ef (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Merge origin/main into #6539

test(data): pin every mirror's re-ask and the per-stream fallback of …

#6539's review threads are now answered and the PR is armed (merge). On 2026-10-13 its only red was Automatic review answered: three Copilot threads from 10-12 18:02Z that no push had addressed. Commit 1658104456 fixes all three:

the end-to-end control now counts distinct re-asking mirrors as (owner, stream id), not a total. Local run: 60 from 60 distinct mirrors, carrier client/…=3;
CoalesceRecycleEnds is an internal seam over plain rows, and a unit test pins the rolling-deploy fallback: an undeclared subscriber still gets one plain end per stream. Negative control run: {"l1"} vs {"l1","l2","l3"};
the AcceptsStringSplice doc wording.

Each thread has a reply naming the commit. The 2026-10-13 00:08Z overloads in the last fold are the shape #6539 addresses: a recycle cascade's per-stream ends on portal/nodeops-…. Closes on: no ACTION-BLOCK OVERLOAD … StreamEndedEvent across a module live update after #6539 rolls.

Still occurring: 1 more since the last update (467 total).

Last seen: 2026-10-14 07:05:25Z
Pods: memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-b79b66b68-w488f, memex-portal-deployment-6c95ffcd84-h4tr5, memex-portal-deployment-5b786c4466-4fs6c, memex-portal-deployment-54856649cb-m2jn9, memex-portal-deployment-54856649cb-4hwrp, memex-portal-deployment-7d5c89fc56-8ljrw

Recent log lines

2026-10-13 00:08:50Z memex-portal-deployment-54856649cb-m2jn9 fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/c9JVoVItPkah_rquk6wNJA: inbound queue depth 10301 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-13 04:00:59Z memex-portal-deployment-54856649cb-4hwrp fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub portal/nodeops-5kH5XgIhYUOQctcGO_TXIg: inbound queue depth 18720 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-14 07:05:25Z memex-portal-deployment-7d5c89fc56-8ljrw fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub portal/nodeops-UyjBlHLYt0KuVdjpWqZ8xA: inbound queue depth 15992 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.

_{Re-addressed by the current identity function: this incident inherited 7a92a4f2b427a9ef (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 2 more since the last update (469 total).

Last seen: 2026-10-15 00:47:09Z
Pods: memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-b79b66b68-w488f, memex-portal-deployment-6c95ffcd84-h4tr5, memex-portal-deployment-5b786c4466-4fs6c, memex-portal-deployment-54856649cb-m2jn9, memex-portal-deployment-54856649cb-4hwrp, memex-portal-deployment-7d5c89fc56-8ljrw, memex-portal-deployment-6d4df8d45f-ktwf2

Recent log lines

2026-10-14 07:05:25Z memex-portal-deployment-7d5c89fc56-8ljrw fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub portal/nodeops-UyjBlHLYt0KuVdjpWqZ8xA: inbound queue depth 15992 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-15 00:47:05Z memex-portal-deployment-6d4df8d45f-ktwf2 fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub portal/nodeops-CQIqeIjW80aQ96e3VctZHg: inbound queue depth 10315 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-15 00:47:09Z memex-portal-deployment-6d4df8d45f-ktwf2 fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/CQIqeIjW80aQ96e3VctZHg: inbound queue depth 10066 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.

_{Re-addressed by the current identity function: this incident inherited 7a92a4f2b427a9ef (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 1 more since the last update (470 total).

Last seen: 2026-10-15 00:48:36Z
Pods: memex-portal-deployment-5f95fb94dc-pkhnc, memex-portal-deployment-b79b66b68-w488f, memex-portal-deployment-6c95ffcd84-h4tr5, memex-portal-deployment-5b786c4466-4fs6c, memex-portal-deployment-54856649cb-m2jn9, memex-portal-deployment-54856649cb-4hwrp, memex-portal-deployment-7d5c89fc56-8ljrw, memex-portal-deployment-6d4df8d45f-ktwf2

Recent log lines

2026-10-15 00:47:05Z memex-portal-deployment-6d4df8d45f-ktwf2 fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub portal/nodeops-CQIqeIjW80aQ96e3VctZHg: inbound queue depth 10315 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-15 00:47:09Z memex-portal-deployment-6d4df8d45f-ktwf2 fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/CQIqeIjW80aQ96e3VctZHg: inbound queue depth 10066 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.
2026-10-15 00:48:36Z memex-portal-deployment-6d4df8d45f-ktwf2 fail: MeshWeaver.Messaging.MessageService[0]
      ACTION-BLOCK OVERLOAD in hub mesh/CQIqeIjW80aQ96e3VctZHg: inbound queue depth 10000 crossed the aggregate watermark 10000. Shedding sheddable [CanBeIgnored] traffic (e.g. StreamEndedEvent) at ingestion to keep the single-threaded turn loop draining — user-facing and lifecycle messages are NEVER shed. An amplifying source (many distinct keys) is driving this block past its drain rate; find and fix it.

_{Re-addressed by the current identity function: this incident inherited 7a92a4f2b427a9ef (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

The 2026-10-15 00:47Z overloads came from the same module-swap wave as the #6520 and #5718 recurrences, on an image that already contains #6539.

Pod, namespace, image. The pod is memex-portal-deployment-6d4df8d45f-ktwf2 in namespace memex-cloud (Ops/Actions/rw-6520-memexcloud-cqiq-20261015 on the control instance). Its mesh hub is mesh/CQIqeIjW80aQ96e3VctZHg, and the same suffix is on portal/nodeops-CQIq…. The generation is the 3.1.10841 roll (completed 2026-10-14 23:44:40Z). Every main commit from 2026-10-14 05:41Z onward contains fix(data): announce an owner recycle once per subscriber, not once per stream #6539 (bd72dab2d5), so these overloads are not a stale image.
What ran on that pod at that moment. A live module swap (ModuleLiveUpdate: MeshWeaver.Graph.Views swapped live …; MeshWeaver.OgCard and MeshWeaver.Markdown.Export appear in the same minute). It recycled every live per-node hub. Every NodeType definition hub among them then started its own RecycleCascade, and each cascade posted one DisposeRequest per stored address of its network from the node-operation issuing hub (portal/nodeops-…). The cascade lines on this pod (Ops/Actions/rw-6520-memexcloud-wave-20261015, 289 lines across the namespace) give fan-outs of up to 409 addresses per cascade, with about 200 cascades. So tens of thousands of disposes went through that one issuing hub. My inference, which I did not measure: each disposed hub ends its streams, and that would produce the StreamEndedEvent traffic the breaker names.
Fix in flight: fix(modules): one landing is one recycle wave, and the wave does not cascade (#6729 recurrence) #6800 (open). The swap's wave carries CascadedFrom, so it no longer starts those cascades, and one landing pass is one wave instead of one per module. That removes the per-definition fan-out from portal/nodeops-….

Not established:

The exact composition of the 10,315 queued messages at 00:47:05Z. That is 41 s BEFORE the first INCOMPLETE cascade line (00:47:46Z). The swap's own wave (every live hub, posted from the same issuing hub) and the cascades that completed (Information, which did not reach Loki for this namespace) both fall in that window, and the shed fate trail was not read.
Whether fix(data): announce an owner recycle once per subscriber, not once per stream #6539's coalescing was exercised here at all. It announces an OWNER recycle once per subscriber. A swap-plus-cascade wave produces many distinct owners at once, which fix(data): announce an owner recycle once per subscriber, not once per stream #6539 does not coalesce across.

Closes on: no ACTION-BLOCK OVERLOAD on portal/nodeops-… during a module live update after #6800 rolls.

Closing: the producer named in the last comment is fixed, merged and running.

Producer: a live module swap recycled every per-node hub. Each NodeType hub then started its own RecycleCascade and posted one DisposeRequest per address through portal/nodeops-…, about 200 cascades with up to 409 addresses each.
Fix: fix(modules): one landing is one recycle wave, and the wave does not cascade (#6729 recurrence) #6800 (4a308d1dc0): one landing is one recycle wave, and that wave does not cascade. feat(recycle): a NodeType recycle cascade is ONE ActivationRecycle request (#6520) #6810 (64908a635e) makes a cascade a single ActivationRecycle request.
Running: both commits are ancestors of core 9a3809f308, which /api/version reports on all three portals (read 2026-10-16 ~05:40Z).
Log window: {namespace="memex-cloud"} |~ "(?i)ACTION-BLOCK OVERLOAD", 2026-10-16 00:28Z to 05:48Z, read 0 lines (Ops/Actions/iw-mc-overload-since-0028-20261016, control instance). No module swap ran in that window either (|~ "(?i)swapped live" read 0). So the zero shows the breaker is quiet, not the fix under a swap wave.

If the overload returns on an image that contains #6800, reopen this issue with the pod's producer.

Navigation Menu

Repository navigation

Bulk stream-teardown flood saturates the memex-cloud mesh hub turn loop, tripping the aggregate action-block breaker #5532

Evidence

Found a sender from code that is pure waste by construction. Fixed in #5635 (merged, `bf5ac85526`), awaiting roll.

Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet

Closing: fix live, no recurrence for ≥ 6 h, positive control present

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Bulk stream-teardown flood saturates the memex-cloud mesh hub turn loop, tripping the aggregate action-block breaker #5532

Description

Evidence

Activity

rbuergi commented on Sep 24, 2026

Found a sender from code that is pure waste by construction. Fixed in #5635 (merged, bf5ac85526), awaiting roll.

rbuergi commented on Sep 24, 2026

Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet

rbuergi commented on Sep 24, 2026

Closing: fix live, no recurrence for ≥ 6 h, positive control present

systemorph-com commented on Oct 11, 2026

systemorph-com commented on Oct 11, 2026

systemorph-com commented on Oct 12, 2026

rbuergi commented on Oct 12, 2026

systemorph-com commented on Oct 12, 2026

systemorph-com commented on Oct 13, 2026

rbuergi commented on Oct 13, 2026

systemorph-com commented on Oct 14, 2026

systemorph-com commented on Oct 15, 2026

systemorph-com commented on Oct 15, 2026

rbuergi commented on Oct 15, 2026

rbuergi commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Found a sender from code that is pure waste by construction. Fixed in #5635 (merged, `bf5ac85526`), awaiting roll.