Repository navigation
Stale-build auto-recycle tears hubs down while accepted SubscribeRequests are still deferred behind init gates #5128
Description
What is failing
When a hub self-recycles under stale-build convergence (Modules:AutoRecycleOnStaleBuild=true), it tears itself down while SubscribeRequest messages it had already accepted are still deferred behind the DataContextInit/MeshNodeInit gates. The requests are discarded (never processed) and the senders are answered ShuttingDown — so cache viewers and portal circuits lose a subscription they were told was accepted.
Probable cause
The stale-build recycle path posts a self-DisposeRequest to the hub without first draining (or rejecting) accepted-but-still-deferred work. Medium-high confidence: the log text explicitly attributes the teardown to "Stale-build convergence" (Store/Plugin build v6935 superseding v6932), the discard is a SubscribeRequest parked behind initialization gates, and the framework's own dispose contract says accepted work must be drained before a hub goes down — meaning the convergence recycle (introduced in the opt-in feature) violates the drain contract. The relevant teardown/drain logic is in MeshWeaver.Messaging.MessageService around dispose-entry and deferral handling.
Impact
Low, so far: 5 occurrences, all on one pod (memex-portal-deployment-75b545bbd-gwrc6) in a single instant, tied to one plugin build bump. Affected clients are cache observers and portal circuits (e.g. LearningRoadmap, Hosting hubs) whose subscribe silently fails — they see a ShuttingDown NACK and must re-subscribe. Because auto-recycle runs on every future build publish on this deployment, this will recur with each Store/Plugin release unless drained.
Where to look
MeshWeaver.Messaging.MessageService— dispose-entry handling of deferred messages (the[DISPOSE-DISCARD]site) and whether a recycle-triggered DisposeRequest should wait for gate-deferred work to drain or reject it earlier.- The stale-build convergence path (see Production converges onto new builds by itself) — timing of the self-posted DisposeRequest relative to deferred subscriptions.
- Also relevant: Refused replies during teardown and Debugging Disposal, Storms and Leaks.
Evidence
| Fingerprint | 352e14f1ffea368c |
| Category | MeshWeaver.Messaging.MessageService |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-75b545bbd-gwrc6 |
| Occurrences | 5 |
| First seen | 2026-09-21 09:07:17Z |
| Last seen | 2026-09-21 09:07:17Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Messaging.MessageService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 09:07:17Z memex-portal-deployment-75b545bbd-gwrc6 fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub LearningRoadmap is disposing with SubscribeRequest (id=WXjtqUS390udhXmDNaXaAg, from cache/9mdycNhVAE6S3qduJCUHVA) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (LearningRoadmap), i.e. a rebind or self-heal recycle; why: Stale-build convergence (Modules:AutoRecycleOnStaleBuild=true): NodeType 'Store/Plugin' published build 'Store_Plugin/v6935-s7e280d1-32ff88cdd10c.dll', superseding the bound 'Store_Plugin/v6932-s7e280d1-32ff88cdd10c.dll'. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 09:07:17Z memex-portal-deployment-75b545bbd-gwrc6 fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Hosting is disposing with SubscribeRequest (id=tRF2YYyFUUChcJVT5mxMwg, from cache/Mm65LBweYUu07nxEzvUZ9w) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (Hosting), i.e. a rebind or self-heal recycle; why: Stale-build convergence (Modules:AutoRecycleOnStaleBuild=true): NodeType 'Store/Plugin' published build 'Store_Plugin/v6935-s7e280d1-32ff88cdd10c.dll', superseding the bound 'Store_Plugin/v6932-s7e280d1-32ff88cdd10c.dll'. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 09:07:17Z memex-portal-deployment-75b545bbd-gwrc6 fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub LearningRoadmap is disposing with SubscribeRequest (id=ZBVafT16xU2GmxDHrsIN1A, from cache/Mm65LBweYUu07nxEzvUZ9w) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (LearningRoadmap), i.e. a rebind or self-heal recycle; why: Stale-build convergence (Modules:AutoRecycleOnStaleBuild=true): NodeType 'Store/Plugin' published build 'Store_Plugin/v6935-s7e280d1-32ff88cdd10c.dll', superseding the bound 'Store_Plugin/v6932-s7e280d1-32ff88cdd10c.dll'. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 09:07:17Z memex-portal-deployment-75b545bbd-gwrc6 fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub LearningRoadmap is disposing with SubscribeRequest (id=0_Vtcahxv0GHaa-bqOn0Xw, from portal/MEG50DQchYtHoQHpJfQk6SQ65UjoQxQe29dQaI6Erc4) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (LearningRoadmap), i.e. a rebind or self-heal recycle; why: Stale-build convergence (Modules:AutoRecycleOnStaleBuild=true): NodeType 'Store/Plugin' published build 'Store_Plugin/v6935-s7e280d1-32ff88cdd10c.dll', superseding the bound 'Store_Plugin/v6932-s7e280d1-32ff88cdd10c.dll'. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-21 09:07:17Z memex-portal-deployment-75b545bbd-gwrc6 fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub LearningRoadmap is disposing with SubscribeRequest (id=8gL7hZEljEaqjDXFR-_ZOw, from portal/MEG50DQchYtHoQHpJfQk6SQ65UjoQxQe29dQaI6Erc4) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by itself — a self-posted DisposeRequest (LearningRoadmap), i.e. a rebind or self-heal recycle; why: Stale-build convergence (Modules:AutoRecycleOnStaleBuild=true): NodeType 'Store/Plugin' published build 'Store_Plugin/v6935-s7e280d1-32ff88cdd10c.dll', superseding the bound 'Store_Plugin/v6932-s7e280d1-32ff88cdd10c.dll'. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
Re-addressed by the current identity function: this incident inherited d2249f800ffc2577 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/352e14f1ffea368c. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Closes as duplicate of #4888.
Reason: same defect —
MeshWeaver.Messaging.MessageServicediscards work it has already accepted when that work is still parked behind a hub'sDataContextInit/MeshNodeInitgates at teardown, answering the senderShuttingDowninstead of draining it. The log line itself states the contract being violated: accepted work must be drained before a hub goes down.These tickets split by what triggered the teardown, not by what is broken — the trigger set measured in this one burst is: an operator
MeshOperations.Recycle(the Recycle tool / Compile button), aRecycleCascadefrom a NodeType, an overlay self-healDisposeRequest, a stale-build convergence self-recycle (Modules:AutoRecycleOnStaleBuild=true), and a directDispose()on a plugin-install hub (#4888's own sample). One code path, five entry points; folding on the trigger would have produced five tickets for one fix.Evidence migrated to #4888: trigger = stale-build convergence self-recycle under
Modules:AutoRecycleOnStaleBuild=true— the log attributes the teardown toStore/Pluginbuild v6935 superseding v6932. 5 occurrences 2026-09-21 09:07Z. Fingerprint352e14f1ffea.This trigger is the one that matters most operationally:
AutoRecycleOnStaleBuildis off by default in code but settruefleet-wide by the AKS chart, so on every production portal a genuine assembly-path advance self-recycles hubs — and this defect means each such convergence sheds whatever subscribe work was gate-parked at that instant. It makes the defect a routine consequence of deploying, not an operator-triggered edge case.