Repository navigation
Deferred SubscribeRequests discarded when Orleans silo shutdown deactivates still-initializing hubs #5529
Description
What is failing. Three mesh hubs (e.g. rbuergi/_Install/SocialMedia, Systemorph/Underwriting/LineOfBusiness/_Activity/compile-state) were torn down while still behind their DataContextInit/MeshNodeInit initialization gates, each discarding a deferred SubscribeRequest from cache/… and answering the sender ShuttingDown.
Probable cause. This is the by-design teardown-discard path firing during an Orleans silo shutdown — the log's own attribution says so: the grain was deactivated because server S10.244.3.154:11111:149174495 (a memex-cloud silo) entered ShuttingDown (DirectoryFailure grain-directory eviction). A subscribe landed while the hub was initializing, the silo went down mid-init, and the teardown discarded the parked work. Confidence: high — the attribution added on 2026-09-14 (teardown reports now name who asked) names the cause explicitly, and all three occurrences fall inside a single 7 ms window, i.e. one shutdown event, not a recurring defect. The open question an engineer should close is whether the cache/… sender reliably retries on the ShuttingDown NACK (per Refused replies during teardown) — if it does, this is fully benign.
Impact. Minimal: 3 occurrences, one pod, one 7 ms window during a single silo shutdown (rolling deployment). Senders were told to ask again; no data is lost. Not a release gate.
Where to look. MeshWeaver.Messaging.MessageService — the DISPOSE-DISCARD site in MessageHub.cs, and the deferral/initialization-gate machinery described in Hub Disposal Model and Teardown Layers. Also worth checking whether the cache/YxuIePH430S3wXXs-HRa8A sender (same in all three samples) honors the retry advice.
Note: this is the report class the 2026-09-14 fix deliberately kept filing ("work the platform accepted and did not finish is worth a look"). If silo-shutdown-evicted discard is confirmed benign, consider a targeted suppression for the DirectoryFailure/ShuttingDown attribution shape rather than leaving this as code noise.
Evidence
| Fingerprint | 3ef556deb4edde59 |
| Category | MeshWeaver.Messaging.MessageService |
| Severity | Error |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-5f95fb94dc-pkhnc |
| Occurrences | 3 |
| First seen | 2026-09-23 13:33:38Z |
| Last seen | 2026-09-23 13:33:38Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Messaging.MessageService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-23 13:33:38Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub rbuergi/_Install/SocialMedia is disposing with SubscribeRequest (id=2lTzlnQsd0iZo6Gd88uvMg, from cache/YxuIePH430S3wXXs-HRa8A) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by Orleans deactivating grain rbuergi/_Install/SocialMedia; why: DirectoryFailure — This activation is being deactivated because server S10.244.3.154:11111:149174495 entered status ShuttingDown and was responsible for this activation's grain directory registration.. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-23 13:33:38Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub christoph.hofstetter/_Install/OpenStreetMap is disposing with SubscribeRequest (id=mYRogN9SpEu9uC2IaJXJyQ, from cache/YxuIePH430S3wXXs-HRa8A) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by Orleans deactivating grain christoph.hofstetter/_Install/OpenStreetMap; why: DirectoryFailure — This activation is being deactivated because server S10.244.3.154:11111:149174495 entered status ShuttingDown and was responsible for this activation's grain directory registration.. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-23 13:33:38Z memex-portal-deployment-5f95fb94dc-pkhnc fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub Systemorph/Underwriting/LineOfBusiness/_Activity/compile-state is disposing with SubscribeRequest (id=jg17C03VCUytrI2pAYe2ag, from cache/YxuIePH430S3wXXs-HRa8A) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by Orleans deactivating grain Systemorph/Underwriting/LineOfBusiness/_Activity/compile-state; why: DirectoryFailure — This activation is being deactivated because server S10.244.3.154:11111:149174495 entered status ShuttingDown and was responsible for this activation's grain directory registration.. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
Re-addressed by the current identity function: this incident inherited d2249f800ffc2577 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/3ef556deb4edde59. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 23, 2026
Duplicate of #5424 — same log site/fingerprint; consolidated. (
[DISPOSE-DISCARD]in MessageService when Orleans deactivation / silo shutdown tears down a hub with gate-deferred SubscribeRequests)