Repository navigation
MessageService.Dispose double-disposes its CancellationTokenSource, faulting grain hub shutdown #2178
Description
What is failing
MessageHubGrain shutdown for the Store/Catalog hub faulted inside MessageService.Dispose() with ObjectDisposedException: The CancellationTokenSource has been disposed. Because disposal threw, the grain took its fallback branch and kept the load context instead of releasing it — so the grain is left in a half-torn-down state and its next activation will reuse stale context.
Probable cause
High confidence: MessageService.Dispose() is being invoked while its CancellationTokenSource is already disposed — a double-dispose (or a dispose-after-cancel-plus-dispose) on the shutdown path from MessageHub.HandleShutdownCore. The stack is unambiguous:
MessageService.cs:1872— the throwing line, insideDispose()MessageHub.cs:2184—HandleShutdownCorecalling into it
Dispose() should be idempotent; guarding the CTS access (null-check / _disposed flag / swallow ObjectDisposedException) is the direct fix. The deeper question is why shutdown reaches Dispose twice — worth checking whether HandleShutdownCore and the grain's deactivation path both dispose the MessageService.
Impact
Low volume so far — 1 occurrence, 1 pod, single instant. But the consequence per occurrence is not cosmetic: the grain explicitly logs that it is keeping the load context when disposal faults, meaning that hub's state does not get cleaned up on shutdown. Subsequent activations of the same grain key (here Store/Catalog) can inherit stale state. Watch for repeat fingerprints on the same grain path.
Where to look
src/MeshWeaver.Messaging.Hub/MessageService.csaround line 1872 — makeDispose()idempotent w.r.t. theCancellationTokenSource.src/MeshWeaver.Messaging.Hub/MessageHub.csaround line 2184 (HandleShutdownCore) — confirmMessageServiceisn't disposed on more than one path during shutdown.MeshWeaver.Hosting.Orleans.MessageHubGrain— the "hub disposal faulted — KEEPING its load context" branch: consider whether faulted disposal should still drop the load context (current behavior arguably makes a transient dispose bug persistent).
Evidence
| Fingerprint | d79c8f1a68873ff6 |
| Category | MeshWeaver.Hosting.Orleans.MessageHubGrain |
| Severity | Error |
| Exception | System.ObjectDisposedException |
| Top frame | MeshWeaver.Messaging.MessageService.Dispose() |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-5b6c4f89f8-97sbj |
| Occurrences | 1 |
| First seen | 2026-08-24 19:36:11Z |
| Last seen | 2026-08-24 19:36:11Z |
Recent log lines
2026-08-24 19:36:11Z memex-portal-deployment-5b6c4f89f8-97sbj fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
Grain Store/Catalog: hub disposal faulted — KEEPING its load context
System.ObjectDisposedException: The CancellationTokenSource has been disposed.
at MeshWeaver.Messaging.MessageService.Dispose() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/MessageService.cs:line 1872
at MeshWeaver.Messaging.MessageHub.HandleShutdownCore(IMessageDelivery`1 request) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/MessageHub.cs:line 2184
Opened automatically from Admin/_LogIncident/d79c8f1a68873ff6. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Aug 25, 2026
Duplicate of #2176 — fixed by #2238.