RecycleCascade tears down hubs before deferred SubscribeRequests are drained, discarding accepted work #5117

What is failing

MeshOperations.Recycle (an operator-initiated recycle of Hosting/TriageItem) tears down a hub (Hosting/Triage/ci-failure/...) via RecycleCascade while a SubscribeRequest accepted from the portal cache client is still parked behind the hub's closed initialization gates (DataContextInit, MeshNodeInit). The request is discarded unprocessed and the sender is answered ShuttingDown, violating the documented drain guarantee that accepted work must be drained before a hub goes down.

Probable cause

The RecycleCascade teardown path does not wait for (or reject ahead of the gate-closing) deferred messages that were already accepted but not yet initialized. Medium confidence: the log itself proves the discard happened and that the teardown was cascade-initiated, but the exact interleaving — whether the SubscribeRequest arrived after RunLevel began shutting down, or was deferred and then starved by the cascade — is not distinguishable from this single occurrence.

Impact

Minimal at this fingerprint: a single occurrence on one pod at a single point in time; the client received ShuttingDown and can resubscribe on hub reactivation. However, this is one of 30+ open [DISPOSE-DISCARD] incidents at the same MeshWeaver.Messaging.MessageService log site spanning Subscribe/Unsubscribe/GetData requests across many hubs, suggesting a systemic gap in the shutdown/drain path rather than an isolated race. Likely duplicates/family: Admin/_LogIncident/03a2bc711f5e4501, Admin/_LogIncident/b8716c17b178ac35, and others in the same category.

Where to look

The teardown/deferred-message drain logic in MeshWeaver.Messaging.MessageService (the [DISPOSE-DISCARD] log site, event id 7301), and the RecycleCascade dispose-request handling in MeshOperations.Recycle / NodeTypeRecycleCascade. The fix direction is either draining deferred-but-accepted requests before gate-closed disposal completes, or rejecting them at accept time once RunLevel leaves the running state.

Evidence

2026-09-21 08:44:55Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-education-35199594174 is disposing with SubscribeRequest (id=VqxOdz9y_E6waf9jYNogkg, from cache/wpWb1fSJPUacoVxY6LnltQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-Mm65LBweYUu07nxEzvUZ9w (routed DisposeRequest); why: RecycleCascade from 'Hosting/TriageItem': MeshOperations.Recycle: an operator asked for 'Hosting/TriageItem' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.

_{Re-addressed by the current identity function: this incident inherited d2249f800ffc2577 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/657ae482de37b608. Recurrences are folded into this issue rather than opening new ones.}