Hub teardown discards gate-deferred messages instead of draining them when Orleans deactivates the grain #5520

What is failing

When an Orleans silo enters ShuttingDown and the runtime deactivates a mesh hub grain (reason DirectoryFailure), the hub disposes while a SubscribeRequest is still deferred behind its initialization gates (DataContextInit, MeshNodeInit). The message is never processed and the sender is answered ShuttingDown — so the requesting client's subscription silently never takes effect for that hub instance.

Probable cause

High confidence (read directly from the log attribution): the hub teardown path does not drain deferred, gate-parked messages before dispose. The teardown was triggered externally by Orleans (silo S10.244.2.220:11111 going ShuttingDown, grain directory failure), i.e. a graceful-shutdown/deactivation path, not a crash. The DISPOSE-DISCARD log site exists precisely to record this discard, which suggests the behaviour is known — but the discard itself is the defect (or at least under-mitigated): accepted work should be drained or the deferral should be fast-failed earlier so the client can retry against the re-activated grain instead of receiving a dead-end ShuttingDown reply.

Note this is one instance of a recurring family: dozens of sibling DISPOSE-DISCARD incidents at the same log site have been triaged in the past week, covering SubscribeRequest, UnsubscribeRequest, GetDataRequest and queued-turn discards, across many hubs and pods. This specific incident is not independently urgent; if a tracking issue for the family exists, this can be closed as a duplicate of it.

Impact

Single occurrence, single pod, single client request (SubscribeRequest from cache/NtBUbDE6GUOtvO71iqpNmQ for hub jeandavid.goetz/_Install/Crm) during a silo shutdown window. The immediate user-visible effect is a subscription that did not register (client should recover by resubscribing, and the ShuttingDown answer tells it to). Judged from the numbers this is a low-impact instance of a systemic pattern; the pattern as a whole (work routinely discarded at teardown during rolling deploys/restarts) is the thing worth fixing.

Where to look

Start at the MeshWeaver.Messaging.MessageService dispose/teardown path — the code that emits the [DISPOSE-DISCARD] log (event id 7301) and answers deferred senders with ShuttingDown. Specifically: why deferred messages behind DataContextInit/MeshNodeInit gates are not drained (or rejected promptly) when the hub's RunLevel transitions to ShutDown on grain deactivation, and whether the Orleans deactivation path (OnDeactivateAsync-equivalent) should wait for gate resolution or fast-fail parkers. The framework source was not found in this mesh, so no direct file link is available.

Severity: sev:M — a secondary-path defect with a natural workaround (client retry/resubscribe), but part of a recurring pattern worth a fix.

Evidence

2026-09-23 13:03:07Z memex-portal-deployment-79889596-47trv fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub jeandavid.goetz/_Install/Crm is disposing with SubscribeRequest (id=vj1empZp9EWF5s6xDkxxAg, from cache/NtBUbDE6GUOtvO71iqpNmQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by Orleans deactivating grain jeandavid.goetz/_Install/Crm; why: DirectoryFailure — This activation is being deactivated because server S10.244.2.220:11111:149171855 entered status ShuttingDown and was responsible for this activation's grain directory registration.. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.

_{Re-addressed by the current identity function: this incident inherited d2249f800ffc2577 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/f3c78f7397cbc66d. Recurrences are folded into this issue rather than opening new ones.}