Repository navigation
Hub teardown driven by Orleans grain deactivation discards deferred SubscribeRequest instead of draining it #5424
Description
What is failing
The RemoteControl/_Access per-user hub was deactivated by Orleans (grain deactivated due to DirectoryFailure — its responsible server entered ShuttingDown) while a SubscribeRequest was still parked behind its closed initialization gates (DataContextInit, MeshNodeInit). The teardown path discarded the accepted message instead of draining it, logging [DISPOSE-DISCARD]. The sender was answered ShuttingDown.
Probable cause — medium-high confidence
An external teardown (Orleans grain deactivation triggered by server shutdown) reaches the hub's dispose path before its initialization gates have opened. Per Retiring an Activation, a retirement must answer the backlog behind a gate (FailGate) before disposing so the sender gets a definitive reply rather than a discard. Here the deactivation path appears not to run that drain/fail-gate sequence — or runs it after the deferred buffer is already gone — so accepted-but-deferred work is dropped on the floor during a server-shutdown-driven teardown. The evidence (gates named at deferral, explicit deactivation attribution in the log) supports this; whether the gap is in the grain-deactivation-triggered teardown path specifically, versus voluntary shutdown, needs confirmation in code.
Impact
Single occurrence on one pod during a host shutdown at 2026-09-22 23:08 UTC. One client subscribe was dropped; the caller received ShuttingDown and can resubscribe. No data loss, no hang. Low blast radius today, but the invariant "accepted work must be drained before a hub goes down" is violated on this path — the same discard could hit request-response traffic with a worse outcome.
Where to look
src/MeshWeaver.Messaging/MessageService.cs— the[DISPOSE-DISCARD]log site and the deferred-delivery buffer teardown.- The grain-deactivation-driven teardown path (Orleans
DirectoryFailure/ server-ShuttingDownhandling), to confirm whether it runs theFailGate/drain sequence from Retiring an Activation before disposing the hub. - Initialization Gates — deferred-buffer semantics during shutdown.
No existing incident or issue covers this site — first sighting.
Evidence
| Fingerprint | ff2d0c8212295405 |
| Category | MeshWeaver.Messaging.MessageService |
| Severity | Error |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-857546649-vf2q6 |
| Occurrences | 1 |
| First seen | 2026-09-22 23:08:13Z |
| Last seen | 2026-09-22 23:08:13Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Messaging.MessageService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-22 23:08:13Z memex-portal-deployment-857546649-vf2q6 fail: MeshWeaver.Messaging.MessageService[7301]
[DISPOSE-DISCARD] Hub RemoteControl/_Access/amit.kumar_Access is disposing with SubscribeRequest (id=PRmWpKI0Z0mR4Y8Ht0yClw, from cache/JrlWjPZHp0C3Q2wuZIz-sQ) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by Orleans deactivating grain RemoteControl/_Access/amit.kumar_Access; why: DirectoryFailure — This activation is being deactivated because server S10.244.9.143:11111:149122971 entered status ShuttingDown and was responsible for this activation's grain directory registration.. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
Re-addressed by the current identity function: this incident inherited d2249f800ffc2577 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/ff2d0c8212295405. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 23, 2026
I checked this against the code at
39f49494eb. This is not a missingFailGate.Retiring an Activation prescribes
FailGate(…, ErrorType.ShuttingDown)beforeDispose()only for a hub that retires itself after a transient initialization fault. Only then does the retirement have a more specific cause, the database for example, that could lose the race to the generic drain.An Orleans deactivation (
DirectoryFailure: the directory owner's silo enteredShuttingDown) has no cause beyond the teardown itself.MessageHubGrainalready records that cause throughNoteDirectDisposalBy("Orleans deactivating grain …", reason), and your log line carries it. Draining first is not possible either: theSubscribeRequestwas parked behind[DataContextInit,MeshNodeInit], waiting for a bring-up that the deactivation ends. That is the carve-out TeardownLayers → Handler turns documents: when a hub is torn down before it finishes starting, its parked work is answeredShuttingDownand reported. The answer is composed throughShutdownNack.RetryForTheAuthoritativeAnswer(#4866), andMeshNodeStreamCache.IsTransientOwnerFailurere-probes it, so thecache/…sender re-subscribes against the reactivated grain on a live silo.What remains is the level, and it is the same maintainer decision as on #5426: should a bring-up discard of a foreign sender's delivery stay an Error? I have not closed this as a duplicate, because the teardown initiator differs (Orleans, not the overlay self-heal). One change at the same
MessageServicebranch would settle both issues. Recorded in #5442.rbuergi commentedon Sep 23, 2026
Consolidated bot-filed duplicates of this defect (closed not_planned): #5505 #5520 #5529 —
[DISPOSE-DISCARD]in MessageService when Orleans deactivation / silo shutdown tears down a hub with gate-deferred SubscribeRequests. The filer fan-out that minted them (no lookup of an existing issue for the same log site) is fixed in Systemorph/MeshWeaver.Plugins#2290.rbuergi commentedon Sep 23, 2026
Addressed in #5609 (open; this comment will be updated at merge).
This issue fires at the site shared by #4888, #5424 and #5589:
[DISPOSE-DISCARD]for aSubscribeRequestfrom acache/…sender, parked behind[DataContextInit,MeshNodeInit]of a hub that was torn down before its bring-up finished. The teardown here was an Orleans deactivation, directly or throughDirectoryFailure.Why this was a false Error rather than a lost message:
ShutdownNack.RetryForTheAuthoritativeAnswer, carrying the owner banner and the activation tag.SubscribeRequestproducer rides that answer out and asks the NEXT activation:CreateExternalClient→rejectedByRecycle→Resubscribe, theResubscribeerror arm, and the fresh-snapshot re-ask. So nobody was stranded, and the line's "Accepted work must be drained before a hub goes down" was false for this shape.The change. A
[ReaskedOnShutdown]marker goes onSubscribeRequest. A marked request's discard reports at Debug with the same facts (gate, sender, teardown attribution), following the rule #4178 set for self-addressed deliveries. An unmarked request from another sender stays Error, because that waiter may take the NACK as final. The NACK itself is unchanged.Test.
DeferredDeliveryNackedOnDisposeTest.AReaskedRequestDiscardedByABringUpTeardown_IsTeardownNormal_NotAnErroruses the production initiator (NoteDirectDisposalBy("Orleans deactivating grain …")+Dispose()). The existing unmarked case is the control. With the fix line disabled, the new test fails.What stays true after the roll. This fingerprint stops being filed for subscribes. A
[DISPOSE-DISCARD]Error on a post-#5609 image means a NON-re-asked request was discarded, and that one deserves an issue.Not established: whether the teardowns themselves were avoidable.
DirectoryFailureis Orleans deactivating activations whose directory owner silo is stopping, which is expected during a roll.rbuergi commentedon Sep 23, 2026
#5609 is merged:
mainatfe33e13a5b, through the merge queue, with the build and the review check both green. It is not yet running: memex.meshweaver.cloud serves9a61fed964and memex.systemorph.com serves39f49494eb, neither of which contains it. It takes effect when an image containing it is rolled out (an owner step). No manual recycle is needed. Leaving this open until a post-roll reading shows the fingerprint has gone quiet on an image that containsfe33e13a5b.rbuergi commentedon Sep 24, 2026
Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet
What is running.
/api/versionreturns9a61fed964.Ops/Status/memex-cloud(06:42:00Z) shows 4/4 replicas on3.0.0-ci.9263. CD run 9263 built core9a61fed964with Plugins41bcfe968f./api/versionreturns39f49494eb.Ops/Status/memexshows the serving pods on3.0.0-ci.9218, which is core39f49494ebwith Pluginse9d9283abe. A third pod, onci.9260, is unready, so the roll isconverged: false.This issue's fix: #5609 (
fe33e13a5b). It is not an ancestor of either running build (git merge-base --is-ancestor, exit 1).Where it is.
3.0.0-ci.9287(CD run35956222018) is corebf5ac85526with Pluginsc4795a8d5b, and it contains this fix. The registry has sealed it. memex-cloud's self-updater logged[SelfUpdate] handing 3.0.0-ci.9287 to the control lane (https://memex.systemorph…)at 06:47Z. So the roll now waits on the control instance, which is an owner step and was not done here.Still open. It closes on a post-roll reading once a
ci.9287-or-later image serves the namespace this fingerprint fires in.rbuergi commentedon Sep 24, 2026
Consolidated bot-filed duplicates of the
[DISPOSE-DISCARD]site (MeshWeaver.Messaging.MessageService[7301]). All were closed not_planned on 2026-09-24: #4888 #5589 #5626.cache/…sender, parked behind the init gates. They share this issue's fix, fix(hub): a bring-up discard reports at the level of who it strands; a sync sub-hub construction fault is not a shutdown #5609 (fe33e13a5b).61fbbcdaf7), and that was not in memex's39f49494eb.fe33e13a5band61fbbcdaf7are both ancestors of140c5fb7a, which memex-cloud has served since 08:41Z. This issue's post-roll reading now covers the whole site. On a post-fix image, a discard of a request that is NOT re-asked deserves its own issue.#5592 (
BuildupAction … HubDisposingException) is a different site and stays open.rbuergi commentedon Sep 24, 2026
Closing: fix live, no recurrence for ≥ 6 h, positive control present
Image. memex-cloud (memex.meshweaver.cloud) serves
3.0.0-ci.9291./api/versionreturns core140c5fb7ae, andOps/Status/memex-cloud(sampled 14:49Z) shows all 4 replicas on that tag,converged: true. The Plugins half isb3601fecbe: the portal-ai build job of CD run 35970157348 (job 107542528900) logsHEAD is now at b3601fe. The first pods of this ReplicaSet (memex-portal-deployment-589ff8f895) started at 08:41:05Z and 08:42:17Z. More pods of the same ReplicaSet, on the same image, joined later (gqsfn at 14:06Z, vw5jj at 14:10Z, and others). The control instance memex.systemorph.com still serves core39f49494eb.Window and method. I ran
LogsInstanceActions on memex.meshweaver.cloud between 14:43Z and 14:54Z, withsinceMinutes370/375, filtered to this ReplicaSet with| pod=~"memex-portal-deployment-589ff8f895-.+". That is ≥ 6 h of this image only, from 08:41Z to about 14:50Z. Coverage control:Ops/logs-memexcloud-20260924-close6h-c-bootcoverage(warm-up complete) lands 9 lines from 9 pods of this ReplicaSet. The earliest are ghgjt, vn49b and tp564 at about 08:51Z, so the window reaches back to the first boots.Fix: #5609 (
fe33e13a5b). The overlay trigger folded here (#5626) is covered by #5463 (61fbbcdaf7), which is also in the image. It is an ancestor of the running build (git merge-base --is-ancestor, exit 0).|~ "(?i)DISPOSE-DISCARD"(Ops/logs-memexcloud-20260924-close6h-f-disposediscard)|~ "MeshWeaver.Messaging.MessageService"(Ops/logs-memexcloud-20260924-close6h-c-messageservice)The window includes two scale-out joins (14:06Z and 14:10Z) and several pod replacements within the ReplicaSet. Those are the silo membership changes that drove the
DirectoryFailuredeactivations in this issue's sample.Closing as completed. If this shape recurs on an image that contains the fix, reopen and treat it as a new path. Readings were taken read-only through the
memexMCP.