Repository navigation
MeshOperations.Recycle discards deferred requests parked behind initialization gates instead of draining them #5106
Description
What is failing
The MeshOperations.Recycle path (the Recycle tool / Compile button) tears a hub down without draining accepted-but-deferred work. When the recycle raced with a portal cache client's pending requests, four messages (GetDataRequest, 2× SubscribeRequest, UnsubscribeRequest) that had been accepted and parked behind the DataContextInit/MeshNodeInit gates were discarded at dispose, and each sender was answered ShuttingDown instead of ever getting its request processed.
Probable cause (moderate confidence)
Recycle stamps a release request and tears the hub down immediately, so the next access reactivates it — but nothing in that path waits for, or flushes, messages already deferred behind the initialization gates. The log text itself states the contract ("accepted work must be drained before a hub goes down"), and Recycle currently violates it. The evidence supports a missing drain/quiesce step in the routed DisposeRequest handling for the Recycle flow; no stack frame was captured, so the exact method to change is not confirmed.
Impact
Low. A single burst of 4 discards on one pod, all at the same instant, against one operator-recycled hub. The affected client is a portal cache circuit whose subscriptions/data fetch were dropped; it received explicit ShuttingDown answers (no silent loss) and the hub reactivates on next access. The user-visible symptom is a stale/blank view on that circuit until it re-subscribes.
Where to look
MeshWeaver.Messaging.MessageService— the DISPOSE-DISCARD log site and the deferred-message queue that gates on[DataContextInit, MeshNodeInit].- The routed
DisposeRequesthandler forMeshOperations.Recycle— where the release request is stamped and teardown begins; this is where a drain-before-close step appears to be missing. - Related but distinct: Hosting/DisposalFlakeAudit documents the September teardown/lifecycle audit (instance-sync cleanup, chore(deps): Bump docker/login-action from 3 to 4 #1513 teardown NACK) — it does not cover the Recycle-vs-deferred-work race, so this is not a duplicate of that work.
Evidence
| Fingerprint | b54e360cb9155f19 |
| Category | MeshWeaver.Messaging.MessageService |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-5998d5fddf-9sfsv |
| Occurrences | 4 |
| First seen | 2026-09-21 08:12:12Z |
| Last seen | 2026-09-21 08:12:12Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Messaging.MessageService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 08:12:12Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-education-35471871959 is disposing with GetDataRequest (id=W-uCGDofg0iHSgchCDIxdA, from cache/wpWb1fSJPUacoVxY6LnltQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-wpWb1fSJPUacoVxY6LnltQ (routed DisposeRequest); why: MeshOperations.Recycle: an operator asked for 'Hosting/Triage/ci-failure/systemorph-meshweaver-education-35471871959' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 08:12:12Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-education-35471871959 is disposing with SubscribeRequest (id=CeCuNnyAEkSjti7Hk8gbMA, from cache/wpWb1fSJPUacoVxY6LnltQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-wpWb1fSJPUacoVxY6LnltQ (routed DisposeRequest); why: MeshOperations.Recycle: an operator asked for 'Hosting/Triage/ci-failure/systemorph-meshweaver-education-35471871959' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 08:12:12Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-education-35471871959 is disposing with SubscribeRequest (id=nYcQ8-v6Nk6HVMXC8u3dhw, from cache/wpWb1fSJPUacoVxY6LnltQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-wpWb1fSJPUacoVxY6LnltQ (routed DisposeRequest); why: MeshOperations.Recycle: an operator asked for 'Hosting/Triage/ci-failure/systemorph-meshweaver-education-35471871959' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 08:12:12Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-education-35471871959 is disposing with UnsubscribeRequest (id=judHwvqNeEGa1q_tZUUlFA, from cache/wpWb1fSJPUacoVxY6LnltQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-wpWb1fSJPUacoVxY6LnltQ (routed DisposeRequest); why: MeshOperations.Recycle: an operator asked for 'Hosting/Triage/ci-failure/systemorph-meshweaver-education-35471871959' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
Re-addressed by the current identity function: this incident inherited d2249f800ffc2577 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/b54e360cb9155f19. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Closes as duplicate of #4888.
Reason: same defect —
MeshWeaver.Messaging.MessageServicediscards work it has already accepted when that work is still parked behind a hub'sDataContextInit/MeshNodeInitgates at teardown, answering the senderShuttingDowninstead of draining it. The log line itself states the contract being violated: accepted work must be drained before a hub goes down.These tickets split by what triggered the teardown, not by what is broken — the trigger set measured in this one burst is: an operator
MeshOperations.Recycle(the Recycle tool / Compile button), aRecycleCascadefrom a NodeType, an overlay self-healDisposeRequest, a stale-build convergence self-recycle (Modules:AutoRecycleOnStaleBuild=true), and a directDispose()on a plugin-install hub (#4888's own sample). One code path, five entry points; folding on the trigger would have produced five tickets for one fix.Evidence migrated to #4888: trigger =
MeshOperations.Recycledirectly. 4 occurrences 2026-09-21 08:12Z. Fingerprintb54e360cb915. Carries nothing mechanistically new; its value is that it names the framework surface (IMeshOperations.Recycle) as a trigger in its own right, which is the surface an operator and the MCPrecycleverb both go through.