Hub init BuildupAction resolves the workspace from a disposed Autofac LifetimeScope, permanently failing the hub

What is failing

During hub initialization, the SubscribeToOwnDeletionInit BuildupAction on mesh data-source hubs (mesh/...) resolves the workspace through the hub's Autofac-backed IServiceProvider. The resolve throws ObjectDisposedException — the LifetimeScope (or one of its parent scopes) is already disposed — and MessageHub.HandleInitialize's .Catch converts that into the hub's FAILED state. The hub then refuses every non-lifecycle request with a typed DeliveryFailure for the rest of its life.

The graceful-failure contract held — no wedge, callers got a fast typed failure instead of the 30s deferral timeout (see Hub Initialization Failure). The defect is that an init BuildupAction raced scope disposal at all.

Probable cause — medium confidence

A lifetime-ordering race between hub initialization and scope teardown:

Both stacks show a resolve from the hub's ServiceProvider inside the HandleInitialize BuildupAction composition (SubscribeToOwnDeletionInit → SubscribeToOwnDeletion), with Autofac reporting the scope already disposed.
The Hub Disposal Model documents that the hub's LifetimeScope (its ServiceProvider) dies during teardown while reactive work can still be in flight. If a DisposeRequest (own-deletion, recycle, host shutdown) lands while the init composition is still pending — behind another slow BuildupAction or inside the startup-timeout budget — the remaining init actions resolve from a dead scope.

The evidence does not distinguish whether the disposed scope is the hub's own (disposed mid-init by the shutdown state machine) or a mesh/host-level parent (disposed at pod shutdown while a child hub still initializes); both produce the identical Autofac error. Either way, the init path must not resolve from a scope it no longer owns.

Impact

Sporadic and low-volume, but not benign when it hits: the affected mesh data-source hub is permanently FAILED, so everything at that hub's address answers with a typed failure until the hub is recreated (session retry / pod restart). One degraded hub per occurrence — degraded functionality for whoever that hub serves, not an outage.

Where to look

src/MeshWeaver.Graph/MeshDataSource.cs — MeshDataSourceExtensions.SubscribeToOwnDeletionInit (line 1043) and SubscribeToOwnDeletion (resolve sites at lines 1455 and 1471). This BuildupAction should tolerate a disposing hub: check IsDisposing/RunLevel before resolving, or classify a disposed-scope ObjectDisposedException as "hub is going away" rather than a faulting init.
src/MeshWeaver.Data/WorkspaceExtensions.cs:84 — GetWorkspace(IMessageHub), the GetRequiredService path seen in one stack.
src/MeshWeaver.Messaging.Hub/MessageHub.HandleInitialize — the BuildupAction composition: consider skipping or cancelling pending init actions once disposal starts, so a disposing hub finishes as Disposed rather than FAILED.

Related, not a duplicate

#1548 (incident Admin/_LogIncident/f2207244d38b45cb) shows the same disposed-LifetimeScope symptom, but on the delivery pipeline (MeshBuilder.Register handler) during shutdown — same root family (resolve-from-disposed-scope racing teardown), different call site. Fixing one will not fix the other.

Evidence


Fingerprint	`aaf0ffe1d2556dc6`
Category	`MeshWeaver.Messaging.MessageHub`
Severity	Error
Exception	`System.ObjectDisposedException`
Top frame	`Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()`
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-5586d5c4cb-msq4l`, `memex-portal-deployment-847cf47cb8-lxggk`
Occurrences	3
First seen	2026-08-23 19:56:27Z
Last seen	2026-08-26 20:01:10Z

Recent log lines

2026-08-23 19:56:27Z memex-portal-deployment-5586d5c4cb-msq4l fail: MeshWeaver.Messaging.MessageHub[0]
      Hub mesh/aqrDopNUCEWwFq7Q2_IVaw initialization failed — a BuildupAction faulted (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.). Hub is now in FAILED state.
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.Resolve(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.Resolve(IComponentContext context, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
         at MeshWeaver.Data.WorkspaceExtensions.GetWorkspace(IMessageHub messageHub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/WorkspaceExtensions.cs:line 84
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletion(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/MeshDataSource.cs:line 1471
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletionInit(IMessageHub…[truncated]
2026-08-26 19:59:33Z memex-portal-deployment-847cf47cb8-lxggk fail: MeshWeaver.Messaging.MessageHub[0]
      Hub mesh/O-DkQNgqxEuvbCaa5Xtxig initialization failed — a BuildupAction faulted (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.). Hub is now in FAILED state.
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletion(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/MeshDataSource.cs:line 1455
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletionInit(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/MeshDataSource.cs:line 1043
         at MeshWeaver.Messaging.MessageHub.<>c__DisplayClass61_0.<Handle…[truncated]
2026-08-26 20:01:10Z memex-portal-deployment-847cf47cb8-lxggk fail: MeshWeaver.Messaging.MessageHub[0]
      Hub mesh/8D4VV48u2E6zwtZuxPStuw initialization failed — a BuildupAction faulted (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.). Hub is now in FAILED state.
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletion(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/MeshDataSource.cs:line 1455
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletionInit(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/MeshDataSource.cs:line 1043
         at MeshWeaver.Messaging.MessageHub.<>c__DisplayClass61_0.<Handle…[truncated]

_{Opened automatically from Admin/_LogIncident/aaf0ffe1d2556dc6. Recurrences are folded into this issue rather than opening new ones.}

Merge pull request #2604 from Systemorph/fix/2444-init-scope-race

fix(#2679): the permission fold resolves its services once, and a ren…

Reopened — this fault is happening again.

Still occurring: 39 more since the last update (42 total).

Last seen: 2026-08-28 20:24:59Z
Pods: memex-portal-deployment-5586d5c4cb-msq4l, memex-portal-deployment-847cf47cb8-lxggk, memex-portal-deployment-54bcd4f496-r2tr7, memex-portal-deployment-d669887c7-46cnl, memex-portal-deployment-d669887c7-zxrmk, memex-portal-deployment-5f9fc4fbfc-ms8jc, memex-portal-deployment-c9d4dd69-ml8hb, memex-portal-deployment-c9d4dd69-v6bnc

Recent log lines

2026-08-28 19:30:08Z memex-portal-deployment-c9d4dd69-v6bnc fail: MeshWeaver.Messaging.MessageHub[0]
      Hub mesh/_0N6QM_FJkmuZzMOHW-psg initialization failed — a BuildupAction faulted (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.). Hub is now in FAILED state.
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletion(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/MeshDataSource.cs:line 1455
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletionInit(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/MeshDataSource.cs:line 1043
         at MeshWeaver.Messaging.MessageHub.<>c__DisplayClass61_0.<Handle…[truncated]
2026-08-28 19:36:13Z memex-portal-deployment-c9d4dd69-v6bnc fail: MeshWeaver.Messaging.MessageHub[0]
      Hub mesh/pfxxlwo2AkiMBRdKoWpE2w initialization failed — a BuildupAction faulted (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.). Hub is now in FAILED state.
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletion(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/MeshDataSource.cs:line 1455
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletionInit(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/MeshDataSource.cs:line 1043
         at MeshWeaver.Messaging.MessageHub.<>c__DisplayClass61_0.<Handle…[truncated]
2026-08-28 20:24:59Z memex-portal-deployment-c9d4dd69-v6bnc fail: MeshWeaver.Messaging.MessageHub[0]
      Hub mesh/EUclV-5M3UechbVfWNUIpA initialization failed — a BuildupAction faulted (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.). Hub is now in FAILED state.
      System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
         at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
         at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
         at Autofac.Core.Lifetime.LifetimeScope.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
         at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
         at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
         at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
         at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletion(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/MeshDataSource.cs:line 1455
         at MeshWeaver.Graph.MeshDataSourceExtensions.SubscribeToOwnDeletionInit(IMessageHub hub) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Graph/MeshDataSource.cs:line 1043
         at MeshWeaver.Messaging.MessageHub.<>c__DisplayClass61_0.<Handle…[truncated]

Closing as redundant — the defect is fixed in current origin/main, and the fault has been dormant 20 days.

Dormant. Last seen: 2026-08-28 20:24:59Z (the reopen comment above, matching Admin/_LogIncident/aaf0ffe1d2556dc6) — 20 days quiet as of 2026-09-18. Log ingest is live (incidents updating 2026-09-18T05:42Z) and the portals have not rolled in that window, so it stopped under the same running image.

Fixed — with the code as evidence. Core origin/main = 229930bf2f29845323e0f350e73829b1bdb4928e. The fix landed as PR #2604 (fix/2444-init-scope-race, merge 89f886275a, git merge-base --is-ancestor 89f886275a origin/main → yes) and is live in the file today:

git -C <core> grep -n "IsTerminatedByScopeTeardown(ex)" origin/main -- src/MeshWeaver.Messaging.Hub/MessageHub.cs
→ 773:  if (IsShuttingDown || this.IsTerminatedByScopeTeardown(ex))

MessageHub.HandleInitialize's .Catch now recognises "the fault chain carries ObjectDisposedException and the hub's own scope no longer resolves" as a shutdown outcome: LogDebug, no FAILED residue, gate opened so the imminent disposal flows. So the two things this ticket reported — the Error-level "initialization failed … Hub is now in FAILED state" line, and the permanently-FAILED hub answering every later request with a typed DeliveryFailure — are both gone for this shape.

The root cause named in the fix matches this ticket's evidence exactly: an out-of-band scope teardown (host root container disposed on a Host.StartAsync abort — the #2443 aborts of 2026-08-26 — or an ancestor Autofac scope disposing the hub's child scope) flips Autofac's disposed flag before the disposer reaches the tracked hub instance, so a BuildupAction resolving in that window sees a dead scope while IsShuttingDown is still false. A disposed scope is proof the hub is going away; an ObjectDisposedException from an unrelated disposed dependency, scope alive, still faults the init.

It is pinned by a deterministic repro (ScopeDisposedDuringInitializationTest, RED before / GREEN after), and the classifier was since generalised into the shared probe-gated ScopeTeardown used by init, routing, the permission fold and the layout error path (#2444/#2638/#2679). The call site this ticket pointed at also hardened independently: src/MeshWeaver.Graph/MeshDataSource.cs now wraps the own-node subscription setup in a try/catch that reports via TryLogWarning instead of escaping.

If the fault fires again the recurrence bot reopens automatically with fresh evidence — that is the designed behaviour, and it is why closing a dormant incident is safe rather than a bet.

Navigation Menu

Repository navigation

Hub init BuildupAction resolves the workspace from a disposed Autofac LifetimeScope, permanently failing the hub #2444

What is failing

Probable cause — medium confidence

Impact

Where to look

Related, not a duplicate

Evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Hub init BuildupAction resolves the workspace from a disposed Autofac LifetimeScope, permanently failing the hub #2444

Description

What is failing

Probable cause — medium confidence

Impact

Where to look

Related, not a duplicate

Evidence

Activity

systemorph-com commented on Sep 17, 2026

rbuergi commented on Sep 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions