Repository navigation
Recycle cascade from TriageItem tears down hubs with deferred subscribe requests still parked behind init gates #5104
Description
What is failing
When an operator recycles Hosting/TriageItem, the RecycleCascade tears down every per-failure TriageItem hub while client subscription messages (SubscribeRequest/UnsubscribeRequest from the portal cache hub) are still parked behind the hubs' DataContextInit/MeshNodeInit gates. The teardown path deliberately discards those deferred messages ([DISPOSE-DISCARD]), so the cache client's subscriptions are silently dropped instead of being drained before shutdown — the drain-before-teardown invariant is not holding for the cascade path.
Probable cause
Medium confidence. The evidence shows a cascading MeshOperations.Recycle issued from Hosting/TriageItem routing DisposeRequests to many descendant hubs at once, and the hub initialization never completed (gates still closed) before the dispose arrived — i.e. the cascade tears hubs down faster than their deferred accept-work can be initialized and drained. Either the cascade should not have taken down hubs that had only just started accepting work, or the dispose path should have waited for gate completion before discarding. It is also possible the cache client is (re)subscribing to hubs that are already mid-teardown, creating a race the [DISPOSE-DISCARD] path catches.
Impact
Bounded, single event: 54 messages discarded within ~300 ms on one pod, all during a single operator-triggered recycle cascade. No user-facing outage evidence, but subscribers whose subscription was discarded lose live updates for those TriageItem hubs until they resubscribe (the hub reactivates on next access).
Where to look
- The recycle cascade attribution:
MeshOperations.Recycle/ RecycleCascade fromHosting/TriageItem(Triage Item NodeType) — how broadly it fans out DisposeRequests. MeshWeaver.Messaging.MessageServicedisposal path: the [DISPOSE-DISCARD] branch that answers deferred senders ShuttingDown instead of draining.- The cache client
cache/wpWb1fSJPUacoVxY6LnltQissuing subscribe/unsubscribe bursts during a cascade.
Evidence
| Fingerprint | e5f1df6baaae6266 |
| Category | MeshWeaver.Messaging.MessageService |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-5998d5fddf-9sfsv |
| Occurrences | 54 |
| First seen | 2026-09-21 08:10:05Z |
| Last seen | 2026-09-21 08:10:05Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Messaging.MessageService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 08:10:05Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-35553726304 is disposing with UnsubscribeRequest (id=TrjkbCyi30aJBTfZjORl7g, from cache/wpWb1fSJPUacoVxY6LnltQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-Mm65LBweYUu07nxEzvUZ9w (routed DisposeRequest); why: RecycleCascade from 'Hosting/TriageItem': MeshOperations.Recycle: an operator asked for 'Hosting/TriageItem' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 08:10:05Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-35553726304 is disposing with SubscribeRequest (id=-USvyuYTnE2NxMu_-djDBw, from cache/wpWb1fSJPUacoVxY6LnltQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-Mm65LBweYUu07nxEzvUZ9w (routed DisposeRequest); why: RecycleCascade from 'Hosting/TriageItem': MeshOperations.Recycle: an operator asked for 'Hosting/TriageItem' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 08:10:05Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-plugins-35543690400 is disposing with SubscribeRequest (id=omcA002ssUW2GR3QjyfSDQ, from cache/wpWb1fSJPUacoVxY6LnltQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-Mm65LBweYUu07nxEzvUZ9w (routed DisposeRequest); why: RecycleCascade from 'Hosting/TriageItem': MeshOperations.Recycle: an operator asked for 'Hosting/TriageItem' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 08:10:05Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-reinsurance-35569819568 is disposing with SubscribeRequest (id=bElBC78ONU63AZaC3EoQCQ, from cache/wpWb1fSJPUacoVxY6LnltQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-Mm65LBweYUu07nxEzvUZ9w (routed DisposeRequest); why: RecycleCascade from 'Hosting/TriageItem': MeshOperations.Recycle: an operator asked for 'Hosting/TriageItem' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 08:10:05Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-reinsurance-35569819568 is disposing with SubscribeRequest (id=RJDaL9SPKkuMsLTYqB3t_w, from cache/Mm65LBweYUu07nxEzvUZ9w) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-Mm65LBweYUu07nxEzvUZ9w (routed DisposeRequest); why: RecycleCascade from 'Hosting/TriageItem': MeshOperations.Recycle: an operator asked for 'Hosting/TriageItem' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
Re-addressed by the current identity function: this incident inherited d2249f800ffc2577 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/e5f1df6baaae6266. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Closes as duplicate of #4888.
Reason: same defect —
MeshWeaver.Messaging.MessageServicediscards work it has already accepted when that work is still parked behind a hub'sDataContextInit/MeshNodeInitgates at teardown, answering the senderShuttingDowninstead of draining it. The log line itself states the contract being violated: accepted work must be drained before a hub goes down.These tickets split by what triggered the teardown, not by what is broken — the trigger set measured in this one burst is: an operator
MeshOperations.Recycle(the Recycle tool / Compile button), aRecycleCascadefrom a NodeType, an overlay self-healDisposeRequest, a stale-build convergence self-recycle (Modules:AutoRecycleOnStaleBuild=true), and a directDispose()on a plugin-install hub (#4888's own sample). One code path, five entry points; folding on the trigger would have produced five tickets for one fix.Evidence migrated to #4888: trigger =
RecycleCascadefromHosting/TriageItem, tearing down every per-failure TriageItem hub while portal-cacheSubscribeRequest/UnsubscribeRequestmessages were still gate-parked; the discard is explicit in the log as[DISPOSE-DISCARD]. 54 occurrences in one burst at 2026-09-21 08:10Z — the highest count of the family. Fingerprinte5f1df6baaae.It also raises a second candidate reading worth keeping on #4888: that the cache client may be (re)subscribing to hubs already mid-teardown, creating a race the discard path merely reports — in which case the fix is at accept time (reject rather than defer) rather than at dispose time.