Overlay self-heal staleness verdict recycles every Store/Plugin-bound hub at once, discarding 105 deferred requests mid-burst

What is failing

At 2026-09-19 08:31:56–57 UTC, every hub on pod memex-portal-deployment-85f5bcf79d-n6lj2 bound to the Store/Plugin overlay (Store, DataModelling, HomeAssistant, AppleWeather, …) simultaneously posted a self-addressed DisposeRequest — the "Overlay self-heal" recycle — and tore down with 105 accepted-but-deferred GetDataRequest/SubscribeRequest deliveries still parked behind their [DataContextInit, MeshNodeInit] gates. Those messages were discarded and their senders answered ShuttingDown.

Probable cause (medium confidence)

The hub-level defect is expected behaviour: a hub going down must not process deferred work, and the senders got a ShuttingDown NACK that read/subscribe callers can retry on. The real anomaly is the correlated, fleet-wide trigger: one overlay staleness verdict on NodeType 'Store/Plugin' fired across all bound hubs in the same ~300 ms window. That is either (a) a genuine Store/Plugin overlay build landing, in which case the mass rebind is by design and this ticket is about the discard noise of 105 messages per pod per rebind, or (b) the per-hub overlay watcher producing a false stale verdict (e.g. a shared timestamp/build-id compared against a race-updated build), causing an unnecessary recycle storm. The log text itself says "its own watcher found stale" — no evidence in the samples distinguishes (a) from (b); check whether a Store/Plugin overlay build was actually published at 08:31:56 UTC.

Impact

Single pod, single sub-second burst (firstSeen == lastSeen, 289 ms apart), no recurrence since. Senders were answered transiently; reads and subscriptions have their own recovery, so no sustained user-facing outage is evident. Treat as an incident-worth-investigating rebind storm, not an outage.

Where to look

The overlay staleness watcher and the "Overlay self-heal" recycle attribution in MeshWeaver.Messaging.MessageService (log event 7301, [DISPOSE-DISCARD]).
Why all hubs bound to the same overlay recycle in the same tick, and whether the stale verdict is durable-checked before the self-posted DisposeRequest.
Background: Debugging Disposal, Storms and Leaks, Hub Disposal Model, and Refused replies during teardown (why ShuttingDown — not Failed — is the right answer for the discarded senders).

Evidence


Fingerprint	`dec214a24ee7f642`
Category	`MeshWeaver.Messaging.MessageService`
Severity	Error
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-85f5bcf79d-n6lj2`
Occurrences	105
First seen	2026-09-19 08:31:56Z
Last seen	2026-09-19 08:31:57Z

Recent log lines

2026-09-19 08:31:57Z memex-portal-deployment-85f5bcf79d-n6lj2 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub DataModelling is disposing with GetDataRequest (id=CsSkEGuO4kagd_pgf7RPNg, from DataModelling/DataModelExplorer) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (DataModelling), i.e. a rebind or self-heal recycle; why: Overlay self-heal: the instance is bound to an overlay of NodeType 'Store/Plugin' that its own watcher found stale, so the hub is recycled to re-bind against the current build. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-19 08:31:57Z memex-portal-deployment-85f5bcf79d-n6lj2 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub HomeAssistant is disposing with SubscribeRequest (id=N5VPgO3OPE2i5snna3z5cA, from cache/IqwnCmEjA0CQeKBVtKRD_g) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (HomeAssistant), i.e. a rebind or self-heal recycle; why: Overlay self-heal: the instance is bound to an overlay of NodeType 'Store/Plugin' that its own watcher found stale, so the hub is recycled to re-bind against the current build. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-19 08:31:57Z memex-portal-deployment-85f5bcf79d-n6lj2 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub HomeAssistant is disposing with GetDataRequest (id=2Q0vnXRdPEqG0-Kq02N3jg, from HomeAssistant/Home) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (HomeAssistant), i.e. a rebind or self-heal recycle; why: Overlay self-heal: the instance is bound to an overlay of NodeType 'Store/Plugin' that its own watcher found stale, so the hub is recycled to re-bind against the current build. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-19 08:31:57Z memex-portal-deployment-85f5bcf79d-n6lj2 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub AppleWeather is disposing with GetDataRequest (id=YS-UeARG3UqgwoyE-2Dn-Q, from AppleWeather/Weather) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (AppleWeather), i.e. a rebind or self-heal recycle; why: Overlay self-heal: the instance is bound to an overlay of NodeType 'Store/Plugin' that its own watcher found stale, so the hub is recycled to re-bind against the current build. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-19 08:31:57Z memex-portal-deployment-85f5bcf79d-n6lj2 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub AppleWeather is disposing with SubscribeRequest (id=eSZRDSMf50W0O2LzE2h5fQ, from cache/IqwnCmEjA0CQeKBVtKRD_g) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (AppleWeather), i.e. a rebind or self-heal recycle; why: Overlay self-heal: the instance is bound to an overlay of NodeType 'Store/Plugin' that its own watcher found stale, so the hub is recycled to re-bind against the current build. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.

_{Re-addressed by the current identity function: this incident inherited d2249f800ffc2577 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/dec214a24ee7f642. Recurrences are folded into this issue rather than opening new ones.}

The remedy on this ticket is right and the blast radius is the wrong frame — plus what the recycle CANNOT fix

Grouped as part of the area:mesh timeout cluster. Two points, one correcting an obvious-looking reading of this ticket, one bounding what a fix can achieve.

1. It is a drain defect, not a blast-radius defect

The title frames it as "recycles every Store/Plugin-bound hub at once", which reads as "too many hubs". The 2026-09-21 consolidation comment (folding in #5053) already reframed it correctly and it is worth restating as the disposition: the defect is that the self-heal path disposes a hub MID-INITIALIZATION without draining the work it has already accepted.

The evidence is in the log line itself: initialization gates closed at deferral: [DataContextInit, MeshNodeInit]. Those messages were accepted and parked behind gates that had not opened yet. The teardown report's own contract — quoted in the line — says "Accepted work must be drained before a hub goes down". The self-posted DisposeRequest path does not honour it.

That distinction decides the fix. Narrowing the blast radius would reduce the count from 105 to a smaller number and leave every one of them a discard. The remedy #5053 names closes both: wait for open gates / drain deferred work before disposing, or re-post accepted-but-deferred work to the successor hub. One hub losing accepted work is the same defect as 105 doing it; 105 is the volume, not the bug.

2. What a recycle cannot fix, and why that bounds the (a)/(b) question

The ticket leaves open whether the correlated trigger is (a) a genuine Store/Plugin overlay build landing or (b) a false stale verdict. The platform's standing rule constrains what (b) would even mean:

A dispose makes the activation RE-READ; it does NOT decide what the re-read FINDS — so it is never itself a fix, and a second recycle proves nothing the first did not. A dispose-only recycle compiles nothing: the fresh activation re-resolves the store key (nodeTypePath, version), and a recompile of an already-Ok type does not rewrite its node, so it re-binds the same local copy.

So a self-heal recycle is only ever useful when the re-read would find something different. That is exactly why NodeTypeEnrichmentHelpers already carries a deferral ladder rather than a plain retry — "Overlay self-heal: instance '{InstancePath}' is stuck on NodeType '{NodeType}' AGAIN after {Heals} self-heal(s) — the type reports a usable build but the instance cannot bind it. Deferring the next recycle by {Delay:0}s." (:2592) — and a terminal that notifies platform admins (:2697). The machinery for "recycling is not working" exists.

What it does not bound is the FIRST recycle of a correlated wave: all bound instances see the same overlay advance in the same tick, each posts its own dispose, and each discards whatever it had accepted. So the question that decides (a) from (b) is narrower than "was a build published at 08:31:56Z": it is whether the watcher's stale verdict is taken against a durable reading before the dispose is posted. If it is, this is (a) and the discard is the whole ticket. If it is not, it is (b) and the deferral ladder is catching it a cycle late.

Disposition

Leave open with the #5053 remedy as the scope: drain before dispose, or hand the deferred work to the successor. sev:M stands — the senders got a ShuttingDown NACK that reads and subscribes can retry on, so nothing is lost, but accepted-and-discarded is a contract violation the report itself names.

Not a duplicate of #4888 — that is a direct Dispose() with no routed request and no reason, a different caller and a different fix, as this ticket already records.

What I did NOT establish

I did not check whether a Store/Plugin overlay build was published at 2026-09-19 08:31:56Z, which is the one reading that separates (a) from (b). Live portals were read-only for this session.
I did not read the watcher's verdict path to determine whether the stale comparison is durable-checked before the self-post. That is the narrowed question above and it is a source read I did not make.
I did not establish that draining is safe in every case — a hub whose dispose is a rebind can drain, but one being torn down because its configuration is unusable may not be able to run the deferred work at all, and then re-posting to the successor is the only correct arm of the two. Which arm applies per trigger is undecided here.
105 discards, one pod, one sub-second burst, no recurrence since. Nothing here establishes a rate.

Grouping method committed as Doc/Architecture/OneLogSiteManyTerminals (PR #5170).

Root cause found, and it is why the discarded work was LOST rather than re-asked — fixed in PR #5272

The thread's frame is right and this does not change it: the defect is that accepted work is
discarded, not that 105 hubs recycled at once. What was missing was why a discard costs anything
at all, given the sender is answered with the transient ErrorType.ShuttingDown that reads and
subscribes are supposed to retry on. It costs everything, because no reader could tell it was
transient.

The mechanism

ErrorType is not what decides a retry. Every classifier that chooses "re-probe the fresh
activation" over "take this answer as final" reads the message text:

classifier	assembly	matches
`MeshNodeStreamCache.IsTransientOwnerFailure`	`MeshWeaver.Hosting`	`"is shutting down"`, `"Rejecting now"`, `"invalid activation"`, …
`AreaErrorClassifier.IsTransientHubFailure` / `IsHubRecycling`	`MeshWeaver.Layout`	the same markers
`OrleansRoutingService.ClassifyRoutedFailure`	`MeshWeaver.Connection.Orleans`	the same markers
`ShutdownNack.IsAnsweredByOwner`	`MeshWeaver.Messaging.Contract`	`ShutdownNack.Banner` — "Hub X is shutting down"

ShutdownNack.IsAnsweredByOwner is deliberately not an ErrorType test — its own doc comment
says the routing layer mints that classification off the same text, so an ErrorType test answers the
question with the routing layer's echo of it. The banner is the only evidence that names the speaker.

The disposal drain — MessageService.Dispose → DrainDeferredDeliveries → NackThroughParent — was
the one ShuttingDown NACK this service minted without going through that seam. It hand-wrote its
sentence. Here is the literal text, captured from the real path by a test run against the unfixed
tree:

Hub recognised/1 was disposed while GatedRequest (id=REBT7dydZE6tlIxom0t9NQ) was still deferred;
initialization gates closed at deferral: [recognised-gate-never-opens] — the message was never
processed. The teardown was requested by host/1 (routed DisposeRequest); why: … The address may
reactivate (recycle / restart); retry to get the authoritative answer.

No banner. No "is shutting down". No "Rejecting now". No activation tag. So:

ShutdownNack.IsAnsweredByOwner(msg, address) → false — the sender concludes the ROUTING LAYER
refused it, not the owner;
IsTransientOwnerFailure(msg) → false; AreaErrorClassifier.IsHubRecycling(msg) → false;
ShutdownNack.ExtractActivationTag(msg) → null, so the two re-probe riders cannot tell one
wedged activation from a recycle storm (ThreadAgentIntegrationTest flakes on shard 4 again — now AddressRecyclingException, not the #1362 timeout #2025) even if they did retry.

The envelope said transient. Every reader said terminal. That is the difference between 105
retryable NACKs and 105 requests nobody ever answers — and it is why this line is an Error rather
than teardown noise. The sentence ends "retry to get the authoritative answer" addressed to readers
that could not tell it was retryable.

It is silent by construction: nothing fails to compile, the delivery is still refused, the log line is
still written. MessageService's own header already states the rule it broke — "Every ShuttingDown
NACK this service mints for a delivery the reader can re-probe must call this" (of ActivationTag())
— and ShutdownNack.RetryForTheAuthoritativeAnswer is documented for exactly this case: "work this
hub ACCEPTED and can no longer finish — a queued turn that came too late, machinery that can no longer
be created, a gate that can never open".

The fix, and the control on both sides

PR #5272 composes the drain's refusal through that seam, carrying every fact the old sentence had
(the gates recorded at deferral, who asked for the teardown, the reason they stated) inside its what
clause, plus the activation tag.

Control: DeferredDeliveryNackedOnDisposeTest.ADiscardedDeferredDelivery_IsRecognisedAsTheOwnersAnswer
discards a REAL deferred delivery and requires the refusal the sender actually receives to be
recognised as the owner's — by Address and by path — with an activation tag on it. On the unfixed
tree: failed, with the text above printed. With the fix: passed. Full class 6/6; full
MeshWeaver.Messaging.Hub.Test suite 504 passed, 0 failed. Negative control: a different address
must NOT be recognised. Positive controls: the gate name and the stated reason must survive the
rewording, so a rewrite cannot trade one fact for another.

And the guard that should have caught it. OwnerAnswerRecognitionGuard runs the real predicate
over each producer's real sentence — but the producer list is written by hand, which is #3017's
own trap one level up ("a guard over an enumeration can only assert the members that were already
written down": a fifth terminal was missing, a sixth was missing at the same time, and the class's
own guard passed throughout). This drain was the seventh, and it stayed unlisted through both. It
is now in the enumeration, and TheProducerSetCoversEverySeam's floor moved 7 → 8.

Why NOT "drain before dispose" or "re-post to the successor"

Those are the two arms #5053 named, and the third arm — the platform's own recovery contract — is the
one that was broken. A recycled address reactivates and the sender re-asks; that is the documented
design (Doc/Architecture/RidingOutAShuttingDownAddress: "ShuttingDown is the only delivery
failure in the mesh that is a promise"). The drain had silently opted out of it. Restoring it is the
root cause, and the other two arms carry hazards this ticket already anticipated:

Waiting for gates can wait forever — a hub serving a fallback card is exactly the state whose
gates may never open, and PROD memex: every course cover serves the 'type has not finished compilation' fallback — Store/Plugin build has not settled #1814 is the outage that produced the self-heal recycle in the first
place. A bounded wait is a guessed timeout, which is not a fix.
Re-posting from inside Dispose duplicates what the sender's own re-ask does and adds a
ping-pong hazard between an instance and its successor.

This ticket's own "What I did NOT establish" already flagged the first: "a hub being torn down
because its configuration is unusable may not be able to run the deferred work at all".

The (a)/(b) question, answered from source

The ticket asks whether the correlated trigger is (a) a genuine Store/Plugin overlay build
landing or (b) a false stale verdict, and the 2026-09-22 comment narrowed it to "whether the
watcher's stale verdict is taken against a durable reading before the dispose is posted". Reading
NodeTypeEnrichmentHelpers.ArmOverlaySelfHeal, the verdict has three rungs and none of them is a
cached-query read:

rung	source	durable?
`advanced` (version moved past the overlay)	`meshHub.GetWorkspace().GetMeshNodeStream(nodeType)`	the per-node stream — authoritative and live by the CQRS rule, not a query
`graced` (usable but version never moved)	the same stream, re-subscribed after `SelfHealGrace`	the code's own note says this replays the workspace's cached snapshot
`reEvaluated` (the #1814 ladder)	`AuthoritativeTypeRead`	resolved from the mesh's query providers — storage

So the verdict is durable on the fast rung and the ladder rung, and the grace rung's stale-positive
risk is bounded by OverlayHealBudget spacing across hub lifetimes. That reads as (a): all
instances bound to the overlay see the same build land on the same live stream in the same tick and
each posts its own dispose. The recycle is by design; the answer it gave to the work it displaced was
not.

One consequence worth keeping: the overlay watcher is armed in WithInitialization, and unlike its
sibling ArmStaleAssemblySelfHeal it has no settle-window Throttle on the fast rung — so an
instance activating against a type that has already reached a usable build can post its dispose
while its own init gates are still closed. That is precisely the [DataContextInit, MeshNodeInit]
shape in the samples. It is not a defect on its own (the recycle is correct and the sender now
re-asks), but it is why this ticket's discards land in the activation window rather than at idle.

What I did NOT establish

That every sender in the 105 actually re-asks. The fix restores the classification the riders
read; I verified the classifiers and the riders from source, not the end-to-end recovery of a
GetDataRequest from a layout area or a SubscribeRequest from a cache hub under this exact
teardown.
Whether a Store/Plugin overlay build was published at 2026-09-19 08:31:56Z. The (a) reading above
is from the watcher's code, not from that mesh's publish history.
Any rate. One pod, one sub-second burst, no recurrence since — unchanged from the filing.
This is a src/ change and CI compiles it; nothing here touches in-mesh source, so nothing in this
fix needs a portal recycle to take effect beyond the ordinary roll.

Disposition

Leave open until PR #5272 merges and the portals roll. sev:M stands.

Disposition note — this thread's own wording, corrected

My earlier comment ended "Leave open until PR #5272 merges and the portals roll." That is the
sev:B/sev:H rule (policy severity-closes-on-verification), and it does not apply here:
#4866 is sev:M, so it closes on the merge, which the pull request's own Fixes keyword will do.

The live confirmation is still worth taking afterwards, and it is cheap: after the portals roll, a
discarded deferred delivery's NACK must read as the owner's answer. The one-line check on a captured
DeliveryFailure.Message is that it opens with Hub <address> is shutting down — the
ShutdownNack.Banner all four classifiers key off. Before the fix it opened Hub <address> was disposed while … and matched none of them.

fix: drop the disposal-drain NACK half — MeshWeaver#5272 carries it

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-21)

Consolidation: #5053 closed into this ticket — the same self-heal recycle, caught mid-initialization

#5053 is this defect with a smaller blast radius and a sharper diagnosis: the overlay self-heal recycle does not drain accepted-but-deferred work before tearing the hub down. Its own search found no existing issue for the race; this ticket, filed two days earlier, is it. Migrating what #5053 carried:

The init-gate race, stated precisely

Two per-issue hubs — Ops/Issue/fleet-watch__Unobserved and Ops/Issue/client-a__Unobserved, instances of the Hosting/Issue NodeType — accepted SubscribeRequests from cache hub cache/Gs3WGiD6-ku2Nf6cjfXSAg and deferred them behind their still-closed gates [DataContextInit, MeshNodeInit]. Before those gates opened, each hub's own overlay watcher found its bound overlay stale and posted a self-DisposeRequest. At RunLevel=ShutDown the accepted work was discarded and the sender answered ShuttingDown.

So the hub was disposed mid-initialization — it had accepted work it had not yet been able to run. The teardown report's own contract says accepted work must be drained before a hub goes down; the self-heal DisposeRequest path does not honour it. The why: string naming the overlay watcher as requester is what makes this attributable rather than a mystery teardown.

The remedy #5053 names

The self-heal path must either wait for open gates / drain deferred work before disposing, or re-post accepted-but-deferred work to the successor hub. Either closes both this ticket's 105-discard stampede and #5053's 2-discard race — they are the same missing step.

Structural point worth keeping

"Any self-heal recycle during a build change can hit every Hosting/Issue instance at once and discard a whole batch of subscriptions." That is this ticket's stampede, predicted from the other end. Both samples land in windows of general fleet slowdown (2026-09-19 08:31Z here, 2026-09-20 22:22Z there), which widens the race rather than causing it.

Not the same as #4888

#4888 is a direct Dispose() with no routed DisposeRequest and no stated reason on a plugin-install hub — a different caller and a different fix (find the caller; make it route a request with a reason). It stays open separately.

Navigation Menu

Repository navigation

Overlay self-heal staleness verdict recycles every Store/Plugin-bound hub at once, discarding 105 deferred requests mid-burst #4866

Evidence

The remedy on this ticket is right and the blast radius is the wrong frame — plus what the recycle CANNOT fix

1. It is a drain defect, not a blast-radius defect

2. What a recycle cannot fix, and why that bounds the (a)/(b) question

Disposition

What I did NOT establish

Root cause found, and it is why the discarded work was LOST rather than re-asked — fixed in PR #5272

The mechanism

The fix, and the control on both sides

Why NOT "drain before dispose" or "re-post to the successor"

The (a)/(b) question, answered from source

What I did NOT establish

Disposition

Disposition note — this thread's own wording, corrected

Consolidation: #5053 closed into this ticket — the same self-heal recycle, caught mid-initialization

The init-gate race, stated precisely

The remedy #5053 names

Structural point worth keeping

Not the same as #4888

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Overlay self-heal staleness verdict recycles every Store/Plugin-bound hub at once, discarding 105 deferred requests mid-burst #4866

Description

Evidence

Activity

rbuergi commented on Sep 22, 2026

The remedy on this ticket is right and the blast radius is the wrong frame — plus what the recycle CANNOT fix

1. It is a drain defect, not a blast-radius defect

2. What a recycle cannot fix, and why that bounds the (a)/(b) question

Disposition

What I did NOT establish

rbuergi commented on Sep 22, 2026

Root cause found, and it is why the discarded work was LOST rather than re-asked — fixed in PR #5272

The mechanism

The fix, and the control on both sides

Why NOT "drain before dispose" or "re-post to the successor"

The (a)/(b) question, answered from source

What I did NOT establish

Disposition

rbuergi commented on Sep 22, 2026

Disposition note — this thread's own wording, corrected

rbuergi commented on Oct 11, 2026

Consolidation: #5053 closed into this ticket — the same self-heal recycle, caught mid-initialization

The init-gate race, stated precisely

The remedy #5053 names

Structural point worth keeping

Not the same as #4888

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions