Repository navigation
Hub recycle discards deferred portal-cache subscribe/unsubscribe behind closed init gates #5119
Description
What is failing
When MeshOperations.Recycle tears a hub down (the Recycle tool / Compile button), any message that is still parked behind the hub's initialization gates (DataContextInit, MeshNodeInit) is discarded with ShuttingDown rather than processed. Here the discarded work is the portal cache's SubscribeRequest and its paired UnsubscribeRequest for the same hub — the subscription bookkeeping on the sender side never lands, so the cache client's view of its subscription state on that hub goes stale after the recycle (the hub is rebuilt on next access, but the old subscription handshake was dropped on the floor).
Probable cause — moderate confidence
A race between operator-initiated recycle and deferred-message gating in MessageService/MessageHub teardown: the recycle (DisposeRequest) won the race against the hub's init gates opening, and the teardown path answers deferred-but-accepted work with ShuttingDown instead of either draining it or reconciling the sender's subscription state. The log explicitly says "accepted work must be drained before a hub goes down" — this is that policy being violated in the recycle path. Evidence is limited to the two log lines (no stack, no exception); I could not locate MessageService source in the mesh, so this is inferred from the message text alone. The lifecycle-race family is documented in Hosting/DisposalFlakeAudit (instance-sync cleanup discarded at teardown), but no existing incident or issue covers this exact subscribe-discard shape.
Impact
Minor. Two occurrences at a single instant on one pod (memex-portal-deployment-…-9sfsv), hub Hosting/Triage/ci-failure/systemorph-meshweaver-reinsurance-35193818678, one operator-triggered recycle. Worst case is a stale cache-side subscription after a hub recycle — no outage, no user-visible failure.
Where to look
The teardown/discard path that emits [DISPOSE-DISCARD] in MeshWeaver.Messaging.MessageService (category id 7301), and how MeshOperations.Recycle's DisposeRequest interacts with messages deferred behind DataContextInit/MeshNodeInit. Compare with the drain-on-dispose work in Hosting/DisposalFlakeAudit. Consider whether the cache client should reconcile subscriptions on a ShuttingDown reply, or whether recycle should quiesce deferred work like ordinary teardown does.
Evidence
| Fingerprint | 7174ac299e9c84d2 |
| Category | MeshWeaver.Messaging.MessageService |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-5998d5fddf-9sfsv |
| Occurrences | 2 |
| First seen | 2026-09-21 08:48:59Z |
| Last seen | 2026-09-21 08:48:59Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Messaging.MessageService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 08:48:59Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-reinsurance-35193818678 is disposing with UnsubscribeRequest (id=JCbAtOJtHkqaQ65ExqLp6Q, from cache/wpWb1fSJPUacoVxY6LnltQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-wpWb1fSJPUacoVxY6LnltQ (routed DisposeRequest); why: MeshOperations.Recycle: an operator asked for 'Hosting/Triage/ci-failure/systemorph-meshweaver-reinsurance-35193818678' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 08:48:59Z memex-portal-deployment-5998d5fddf-9sfsv fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting/Triage/ci-failure/systemorph-meshweaver-reinsurance-35193818678 is disposing with SubscribeRequest (id=VM_i1ZoYIk6NpDJAQPEdCg, from cache/wpWb1fSJPUacoVxY6LnltQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by portal/nodeops-wpWb1fSJPUacoVxY6LnltQ (routed DisposeRequest); why: MeshOperations.Recycle: an operator asked for 'Hosting/Triage/ci-failure/systemorph-meshweaver-reinsurance-35193818678' to be recycled (the Recycle tool / Compile button), which stamps a release request and then tears the hub down so the next access reactivates it. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
Re-addressed by the current identity function: this incident inherited d2249f800ffc2577 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/7174ac299e9c84d2. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Closes as duplicate of #4888.
Reason: same defect —
MeshWeaver.Messaging.MessageServicediscards work it has already accepted when that work is still parked behind a hub'sDataContextInit/MeshNodeInitgates at teardown, answering the senderShuttingDowninstead of draining it. The log line itself states the contract being violated: accepted work must be drained before a hub goes down.These tickets split by what triggered the teardown, not by what is broken — the trigger set measured in this one burst is: an operator
MeshOperations.Recycle(the Recycle tool / Compile button), aRecycleCascadefrom a NodeType, an overlay self-healDisposeRequest, a stale-build convergence self-recycle (Modules:AutoRecycleOnStaleBuild=true), and a directDispose()on a plugin-install hub (#4888's own sample). One code path, five entry points; folding on the trigger would have produced five tickets for one fix.Evidence migrated to #4888: trigger = operator
MeshOperations.Recycle(Recycle tool / Compile button). The discarded work here is the portal cache'sSubscribeRequestand its pairedUnsubscribeRequestfor the same hub — so the sender's subscription bookkeeping is left inconsistent, not merely a subscription missing: the client's view of what it is subscribed to on that hub goes stale in both directions. 2 occurrences 2026-09-21 08:48Z. Fingerprint7174ac299e9c.It also states, correctly at the time, that no existing incident or issue covered this exact subscribe-discard shape — #4888 does, from the plugin-install trigger.