Overlay self-heal recycle can tear a hub down before its init gates open, discarding accepted SubscribeRequests #5053

What is failing

Two per-issue hubs on the memex-cloud instance (Ops/Issue/fleet-watch__Unobserved and Ops/Issue/client-a__Unobserved, instances of the Hosting/Issue NodeType) accepted SubscribeRequests from the cache hub cache/Gs3WGiD6-ku2Nf6cjfXSAg and deferred them behind their still-closed initialization gates [DataContextInit, MeshNodeInit]. Before those gates opened, each hub's own overlay watcher decided its bound overlay of Hosting/Issue was stale and posted a self-DisposeRequest to re-bind against the current build. At RunLevel=ShutDown the deferred, already-accepted subscriptions were discarded and the sender was answered ShuttingDown — the subscription never lands on any activation of that hub without a re-send.

Probable cause — medium-high confidence

A race between the overlay self-heal recycle and hub initialization: the self-heal path recycles a hub (correctly, for staleness) without checking for work parked behind closed gates, so a hub that is mid-initialization can be disposed with accepted work still deferred. The teardown report itself says accepted work must be drained before a hub goes down — the self-heal DisposeRequest path is not honoring that contract. The why: string names the overlay watcher as the requester, which is what makes this attributable and supports the drain-missing diagnosis. What I cannot tell from the logs is why initialization was slow enough to lose the race — both incidents timestamp to 2026-09-20 ~22:22 UTC, a window that also shows routing back-pressure and messaging timeouts on the same fleet, so general slowdown that night likely widened the race window. Search of the mesh found no existing issue for this specific self-heal-vs-init race; a related shape (direct Dispose() discarding a deferred SubscribeRequest behind the same gates) was filed as MeshWeaver#4888 from incident Admin/_LogIncident/03a2bc711f5e4501 — different teardown path, same missing-drain symptom, worth cross-linking.

Impact

Low. Two occurrences on a single pod (memex-portal-deployment-5485b778c9-99m62), both at the same instant (2026-09-20 22:22 UTC), matching a hub recycles-per-build stampede rather than an ongoing leak. Each discard means one cache-side subscriber's stream never establishes for that hub instance until it resubscrices against the fresh activation — a stale-view risk, not an outage. But the race is structural: any self-heal recycle during a build change can hit every Hosting/Issue instance at once and discard a whole batch of subscriptions.

Where to look

• Start at the MeshWeaver.Messaging.MessageService [DISPOSE-DISCARD] report site (event 7301) — it names the deferral gates and the self-posted DisposeRequest.
• Follow the overlay self-heal path that posts the self-DisposeRequest when the watcher finds a stale overlay (the Overlay self-heal: reason string) — it needs to either wait for open gates / drain deferred work before disposing, or re-post accepted-but-deferred work to the successor hub.
• Compare with the drain-before-dispose contract described in Debugging Disposal, Storms and Leaks and Refused replies during teardown; the rebind-recycle behavior that avoids this shape is discussed in A refused recycle no longer tears the hub down anyway.
• Related filed issue: MeshWeaver#4888 (same log site, direct-Dispose variant).

Evidence

2026-09-20 22:22:16Z memex-portal-deployment-5485b778c9-99m62 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub Ops/Issue/fleet-watch__Unobserved is disposing with SubscribeRequest (id=GIQ_bWYLFUiEWwkfu63J1Q, from cache/Gs3WGiD6-ku2Nf6cjfXSAg) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (Ops/Issue/fleet-watch__Unobserved), i.e. a rebind or self-heal recycle; why: Overlay self-heal: the instance is bound to an overlay of NodeType 'Hosting/Issue' that its own watcher found stale, so the hub is recycled to re-bind against the current build. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-20 22:22:16Z memex-portal-deployment-5485b778c9-99m62 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub Ops/Issue/client-a__Unobserved is disposing with SubscribeRequest (id=nB4JqRhqckOXKsnpZN032w, from cache/Gs3WGiD6-ku2Nf6cjfXSAg) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (Ops/Issue/client-a__Unobserved), i.e. a rebind or self-heal recycle; why: Overlay self-heal: the instance is bound to an overlay of NodeType 'Hosting/Issue' that its own watcher found stale, so the hub is recycled to re-bind against the current build. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.

_{Re-addressed by the current identity function: this incident inherited d2249f800ffc2577 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/b8716c17b178ac35. Recurrences are folded into this issue rather than opening new ones.}