Overlay self-heal recycle sheds gate-deferred SubscribeRequests instead of draining or re-binding them #5126

What is failing

During an overlay self-heal recycle of a per-node hub (an Ops/Logs log-streaming hub bound to an instance of NodeType Hosting/LogEntry whose overlay the watcher found stale), a client SubscribeRequest that was still deferred behind the DataContextInit/MeshNodeInit gates is discarded on teardown instead of being drained or carried over to the re-bound hub. The sender is answered ShuttingDown, so the client knows the subscription never landed — but the streaming hub is then destroyed and re-created, and nothing in the log evidence shows the subscription being re-established against the new hub.

Probable cause

A race between a client subscribing to a log stream and the hub's own watcher posting a self-heal DisposeRequest. The deferred-work accounting (deferred count behind closed gates) works as designed — the message is attributed, not silently dropped — but the recycle path makes no provision for deferred gate-parked work: it sheds it. Confidence: medium. The teardown answer (ShuttingDown) is deliberate, so the protocol supports client retry; whether the cache-side sender (cache/...) retries on ShuttingDown is the part the log cannot show and is the first thing to verify.

Impact

Low. One occurrence on one pod (memex-portal) at a single point in time; the affected work is one log-stream subscription, not data or user-facing requests. However the same log site discarded a SubscribeRequest before (incident Admin/_LogIncident/03a25b3b8a3e219c), so the pattern recurs whenever a rebuild/recycle window overlaps a fresh subscription — during NodeType rebuilds this could shed subscriptions in bulk.

Where to look

The MeshWeaver.Messaging.MessageService dispose path that emits the [DISPOSE-DISCARD] attribution (log event id 7301) — specifically the handling of gate-deferred messages when the teardown was requested by the hub itself (overlay self-heal recycle) — and the cache-side subscription logic that receives the ShuttingDown answer, to confirm whether it resubscribes against the re-bound hub. Possible duplicate of / related to incident Admin/_LogIncident/03a25b3b8a3e219c (same site, same SubscribeRequest discard).

Evidence

2026-09-20 22:26:04Z memex-portal-deployment-75b545bbd-nssvp fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub Ops/Logs/memex-1789415631001716235-6b6b44-klrt7 is disposing with SubscribeRequest (id=oc-MoIzvf06pyKX332A0rg, from cache/Xpi-G1Q-lU6s2Moll9rUcw) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (Ops/Logs/memex-1789415631001716235-6b6b44-klrt7), i.e. a rebind or self-heal recycle; why: Overlay self-heal: the instance is bound to an overlay of NodeType 'Hosting/LogEntry' that its own watcher found stale, so the hub is recycled to re-bind against the current build. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.

_{Opened automatically from Admin/_LogIncident/26f778938d5f9b08. Recurrences are folded into this issue rather than opening new ones.}