Repository navigation
MessageHub.Dispose throws OutOfMemoryException while draining subscriptions, leaking the grain's load context #5553
Description
What is failing
When a MessageHubGrain deactivates, MessageHub.Dispose() drains its message subscriptions and throws System.OutOfMemoryException. The grain's OnDeactivateAsync catches the failure and — as designed for partial disposal — keeps the grain's AssemblyLoadContext alive, so the grain's loaded assemblies, subscriptions, and state leak rather than being reclaimed.
Probable cause
The stack shows disposal synchronously draining a subscription set through Rx: ToObservableRecursive on the ImmediateScheduler, fanned out via Merge/SelectMany, inside MessageHub.Dispose() (MeshWeaver.Messaging.Hub/MessageHub.cs:2826). Recursive enumeration of a very large or unexpectedly-unbounded set of subscriptions/coupled streams on the immediate scheduler is the likely allocation blowout — medium confidence; I could not rule out general memory pressure on the pod, since several other OutOfMemoryException incidents were recorded on memex-portal pods this month (see Related incidents). The load-context retention in MessageHubGrain.OnDeactivateAsync (MeshWeaver.Hosting.Orleans/MessageHubGrain.cs:1343) then converts the transient failure into a persistent leak.
Impact
Single occurrence on one pod (memex-portal-deployment-777677694-cc6zz), grain Google/_Entitlements/mkleiner, so far isolated. No user-visible failure is evidenced yet, but each repetition permanently leaks a grain's load context — repeated OOMs at this site would steadily exhaust pod memory. Worth watching in the context of the broader OOM pattern on this deployment.
Where to look
Start at MeshWeaver.Messaging.MessageHub.Dispose() (MessageHub.cs:2826) — specifically whatever builds the observable drained there (the Merge/SelectMany over ToObservableRecursive) and whether the subscription/handler collection it enumerates can be unbounded. Then review MessageHubGrain.OnDeactivateAsync (MessageHubGrain.cs:1343): consider a best-effort teardown of the load context when even partial disposal throws, or at least logging a metric for retained contexts. Architecture background: Message-Based Communication.
Related incidents (likely same underlying memory-pressure cluster)
Admin/_LogIncident/32768bfc4b1eeb3f/72db6ee0331d5217— MessageService OOM in delivery pipelineAdmin/_LogIncident/ac4d85f3e9dce876/add9b566df1fd770— RoutingGrain OOM on directed deliveryAdmin/_LogIncident/d79c8f1a68873ff6— same disposal site,ObjectDisposedExceptioninstead of OOM
Evidence
| Fingerprint | 1ac7248f043f5295 |
| Category | MeshWeaver.Hosting.Orleans.MessageHubGrain |
| Severity | Error |
| Exception | System.OutOfMemoryException |
| Top frame | MeshWeaver.Messaging.MessageHub.Dispose() |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-777677694-cc6zz |
| Occurrences | 1 |
| First seen | 2026-09-23 14:22:25Z |
| Last seen | 2026-09-23 14:22:25Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.MessageHubGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-23 14:22:25Z memex-portal-deployment-777677694-cc6zz fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
Grain Google/_Entitlements/mkleiner: hub disposal failed — KEEPING its load context
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Reactive.Concurrency.ImmediateScheduler.Schedule[TState](TState state, Func`3 action)
at System.Reactive.Linq.ObservableImpl.ToObservableRecursive`1._.Run(IEnumerable`1 source, IScheduler scheduler)
at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
at System.Reactive.Linq.ObservableImpl.Merge`1.Observables.Run(_ sink)
at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector._.OnNext(TSource value)
at System.Reactive.Linq.ObservableImpl.Merge`1.Observables._.InnerObserver.OnNext(TSource value)
at System.Reactive.Linq.ObservableImpl.ToObservableRecursive`1._.LoopRec(IScheduler scheduler)
at System.Reactive.Concurrency.CurrentThreadScheduler.Trampoline.Run(SchedulerQueue`1 queue)
at System.Reactive.Concurrency.CurrentThreadScheduler.Schedule[TState](TState state, TimeSpan dueTime, Func`3 action)
at System.Reactive.Concurrency.Scheduler.ScheduleAction[TState](IScheduler scheduler, TState state, Action`1 action)
at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
at MeshWeaver.Messaging.MessageHub.Dispose() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/MessageHub.cs:line 2826
at MeshWeaver.Hosting.Orleans.MessageHubGrain.OnDeactivateAsync(DeactivationReason reason, CancellationToken cancellationToken) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.Orleans/MessageHubGrain.cs:line 1343
Opened automatically from Admin/_LogIncident/1ac7248f043f5295. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph-com commentedon Sep 23, 2026
Still occurring: 0 more since the last update (1 total).
memex-portal-deployment-777677694-cc6zzRecent log lines
rbuergi commentedon Sep 23, 2026
Duplicate of #5555 — same log site/fingerprint; consolidated. (the same OutOfMemoryException event on pod memex-portal-deployment-777677694-cc6zz, 14:22:25-14:22:38Z — the dispose/drain sites are where the one heap exhaustion surfaced)