Repository navigation
NodeType recycle cascade aborts 11 of 21 legs with ObjectDisposedException after its own wave disposes the shared DI scope #5123
Description
What is failing
The node-type recycle cascade (Recycling a node type now recycles everything built on it) tried to recycle the NodeType Hosting/Deployment and the wave broke down after 10 of 21 legs: 10 instance addresses were recycled, but the remaining 11 legs (Hosting/Deployment, Hosting/Backup, Hosting/DeploymentStatus, Hosting/FleetConsole, Hosting/InstanceAction, Hosting/InstanceRequest, and ~5 more) could not enumerate their live instances because the Autofac LifetimeScope behind the service provider had already been disposed (ObjectDisposedException). Those legs' live instance hubs were NOT recycled and must be recycled by hand.
Probable cause
Use-after-dispose of a captured DI scope inside the cascade, triggered by the cascade itself. The enumeration of each type's instances is deferred (a lazy leg in NodeTypeRecycleCascade.DependencyNetwork, MeshWeaver.Graph/Configuration/NodeTypeRecycleCascade.cs), resolving through MeshService.Query<T> / SubscribeScopedObservable against a captured AutofacServiceProvider. Recycling the first 10 addresses tears down hubs; once one of them owns the shared scope the remaining legs resolve through, every subsequent leg hits an already-disposed scope — exactly the observed "first legs succeed, then everything fails" pattern. High confidence (~85%): the exception comes from the DI container itself, fires mid-cascade, and all 11 failures share the identical disposed-scope message. Sibling incidents d3faa573fbb24ab8 and 8bd12828ac02105b show the same defect in smaller waves (1 failing leg) an hour earlier, at 08:05–08:51.
Impact
One occurrence on one pod (memex-portal-deployment-75b545bbd-gwrc6) today. Low operational blast radius, but functionally significant: an unknown number of instance hubs under Hosting/* are serving the assembly they were born with instead of release 3019 (ecda297ebe9a4cc492b431681ffb8747) and won't pick it up until recycled by hand — the exact stale-state problem the cascade feature exists to eliminate. The burst fired right after the 08:52 release request, so self-update waves of the Hosting tree are the trigger.
Where to look
MeshWeaver.Graph/Configuration/NodeTypeRecycleCascade.cs—DependencyNetwork, the per-type lazy instance-enumeration legs.MeshWeaver.Hosting/MeshService.cs—Query<T>resolving from a possibly-disposedAutofacServiceProvider.MeshWeaver.Messaging.Hub/ImpersonationScopeExtensions.cs—SubscribeScopedObservable, where the scoped observable defers resolution.
Fix direction: snapshot the full enumeration of live instance hubs eagerly, before any hub in the wave is torn down, or catch ObjectDisposedException per leg and re-resolve against a live scope instead of declaring the leg INCOMPLETE.
Duplicates / related
Same defect already filed as issue #5099 (incident Admin/_LogIncident/d3faa573fbb24ab8, filed 15:56 UTC) — same log site, same ObjectDisposedException, same pod, ~1 hour earlier. This ticket should be linked as a duplicate or merged: the fix in one place covers all three incidents (d3faa573fbb24ab8, 8bd12828ac02105b, this one).
Evidence
| Fingerprint | dcab7e9d1f7ec9a0 |
| Category | MeshWeaver.Graph.Configuration.NodeTypeRecycleCascade |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-75b545bbd-gwrc6 |
| Occurrences | 1 |
| First seen | 2026-09-21 08:52:26Z |
| Last seen | 2026-09-21 08:52:26Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Graph.Configuration.NodeTypeRecycleCascade. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 08:52:26Z memex-portal-deployment-75b545bbd-gwrc6 fail: MeshWeaver.Graph.Configuration.NodeTypeRecycleCascade[0]
[RecycleCascade] Hosting/Deployment: the recycle is INCOMPLETE. 10 address(es) were recycled, but 11 enumeration leg(s) could not be read, so an unknown number of live hubs keep serving the assembly they were born with and must be recycled by hand: the instances of 'Hosting/Deployment' could not be enumerated (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.), so its live instance hubs are NOT recycled | the instances of 'Hosting/Backup' could not be enumerated (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.), so its live instance hubs are NOT recycled | the instances of 'Hosting/DeploymentStatus' could not be enumerated (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.), so its live instance hubs are NOT recycled | the instances of 'Hosting/FleetConsole' could not be enumerated (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.), so its live instance hubs are NOT recycled | the instances of 'Hosting/InstanceAction' could not be enumerated (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.), so its live instance hubs are NOT recycled | the instances of 'Hosting/InstanceRequest' could not be enumerated (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has…[truncated]
Opened automatically from Admin/_LogIncident/dcab7e9d1f7ec9a0. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Closes as duplicate of #5099.
Reason: same defect — use-after-dispose of the cascade's captured DI scope, same throw site, same "first legs succeed then every later leg fails" shape.
Evidence migrated to #5099 — this is the worst-case sample and the one that establishes the blast radius: recycling NodeType
Hosting/Deploymentbroke down after 10 of 21 legs; the remaining 11 legs (Hosting/Deployment,Hosting/Backup,Hosting/DeploymentStatus,Hosting/FleetConsole,Hosting/InstanceAction,Hosting/InstanceRequestand ~5 more) could not enumerate their instances, so an unknown number ofHosting/*instance hubs kept serving the assembly they were born with instead of release 3019 (ecda297ebe9a4cc492b431681ffb8747). All 11 failures share the identical disposed-scope message. The trigger is named: the burst fired right after the 08:52Z release request, so self-update waves over the Hosting tree are what provoke it — and the Hosting tree is exactly what an operator relies on to observe and drive a roll. Confidence ~85%. 1 occurrence, podmemex-portal-deployment-75b545bbd-gwrc6. Fingerprintdcab7e9d1f7e.The mechanism this sample pins down: recycling the first 10 addresses tears down hubs; once one of them owns the shared scope the remaining legs resolve through, every subsequent leg hits an already-disposed scope. That makes the failure rate-dependent on how many legs the wave has — a large cascade is more likely to break than a small one, which is the opposite of what an operator would assume.