Repository navigation
NodeType recycle cascade gives up on instance enumeration when its captured DI scope is already disposed, leaving instance hubs on stale builds #5099
Description
What is failing
The new node-type recycle cascade (see Recycling a node type now recycles everything built on it) tried to recycle the NodeType Hosting/TriageItem and could not: when it tried to enumerate the type's live instance hubs, the Autofac LifetimeScope behind the service provider had already been disposed (ObjectDisposedException thrown from AutofacServiceProvider.GetService). The cascade aborted that leg, recycled 0 addresses, and left the type's live instance hubs serving the assembly they were born with — they must now be recycled by hand. This is the new "one wave" feature silently not reaching the instances it was built to reach.
Probable cause
Use-after-dispose of a captured DI scope inside the cascade. The sibling incident 8bd12828ac02105b captured the same burst with a full stack: the enumeration is a lazy lambda in NodeTypeRecycleCascade.DependencyNetwork (MeshWeaver.Graph/Configuration/NodeTypeRecycleCascade.cs ~line 223) that resolves through MeshService.Query<T> (MeshWeaver.Hosting/MeshService.cs ~line 307) via SubscribeScopedObservable (MeshWeaver.Messaging.Hub/ImpersonationScopeExtensions.cs ~line 210). By the time the deferred enumeration runs, the scope has been disposed — most likely because an earlier leg of the same recycle wave tore down the hub whose scope the enumeration was capturing. High confidence (~80%): the exception is thrown by the DI container itself, ruling out a data-layer error, and it fired exactly during the cascade.
Impact
Low so far — a single occurrence today on one pod. But it is functional, not noise: instance hubs of the affected type keep stale activations until manually recycled, which is precisely the stale-state problem the cascade feature was shipped to eliminate. If it recurs on every large recycle, stale hubs accumulate silently.
Where to look
MeshWeaver.Graph/Configuration/NodeTypeRecycleCascade.cs—DependencyNetwork, the lazy instance-enumeration lambda (~line 223).MeshWeaver.Hosting/MeshService.cs—Query<T>(~line 307) resolving from a possibly-disposedAutofacServiceProvider.MeshWeaver.Messaging.Hub/ImpersonationScopeExtensions.cs(~line 210), where the scoped observable defers resolution.
Fix direction: enumerate the live instances eagerly (before any hub teardown the wave itself triggers), or catch ObjectDisposedException and re-resolve against a live scope instead of reporting the leg INCOMPLETE and giving up.
Duplicates / related
Sibling incident Admin/_LogIncident/8bd12828ac02105b is the same failure 1 ms earlier at the same log site (the enumeration-failed detail message vs. this cascade-summary message) — a human should merge the two tickets. Possibly the same disposed-scope family as incidents 0857ac25 (HostedHubsCollection), 04ba38cd (LayoutAreaHost rendering), 03a2... dispose-discard messages.
Evidence
| Fingerprint | d3faa573fbb24ab8 |
| Category | MeshWeaver.Graph.Configuration.NodeTypeRecycleCascade |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-75b545bbd-gwrc6 |
| Occurrences | 1 |
| First seen | 2026-09-21 08:05:16Z |
| Last seen | 2026-09-21 08:05:16Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Graph.Configuration.NodeTypeRecycleCascade. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 08:05:16Z memex-portal-deployment-75b545bbd-gwrc6 fail: MeshWeaver.Graph.Configuration.NodeTypeRecycleCascade[0]
[RecycleCascade] Hosting/TriageItem: the recycle is INCOMPLETE. 0 address(es) were recycled, but 1 enumeration leg(s) could not be read, so an unknown number of live hubs keep serving the assembly they were born with and must be recycled by hand: the instances of 'Hosting/TriageItem' could not be enumerated (ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.), so its live instance hubs are NOT recycled
Opened automatically from Admin/_LogIncident/d3faa573fbb24ab8. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph-com commentedon Sep 21, 2026
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-75b545bbd-gwrc6Recent log lines
systemorph-com commentedon Sep 21, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-75b545bbd-gwrc6Recent log lines
systemorph-com commentedon Sep 21, 2026
Still occurring: 1 more since the last update (4 total).
memex-portal-deployment-75b545bbd-gwrc6Recent log lines
systemorph-com commentedon Sep 21, 2026
Still occurring: 1 more since the last update (5 total).
memex-portal-deployment-75b545bbd-gwrc6Recent log lines
systemorph-com commentedon Sep 21, 2026
Still occurring: 2 more since the last update (7 total).
memex-portal-deployment-75b545bbd-gwrc6Recent log lines
rbuergi commentedon Sep 21, 2026
Consolidation 2026-09-21 — 2 tickets folded here
Folded as duplicates: #5100 #5123. Three incidents, one defect: the cascade's captured Autofac
LifetimeScopeis disposed by the cascade's own wave, so the deferred instance-enumeration lambda throwsObjectDisposedExceptionand those legs recycle nothing.Mechanism, from #5123: recycling the first N addresses tears down hubs; once one of them owns the shared scope the remaining legs resolve through, every subsequent leg fails. So the failure is worse the larger the wave — the opposite of what an operator would assume. Measured:
Hosting/Deploymentbroke down after 10 of 21 legs, leaving 11 (Hosting/Deployment,Hosting/Backup,Hosting/DeploymentStatus,Hosting/FleetConsole,Hosting/InstanceAction,Hosting/InstanceRequest, ~5 more) unrecycled and serving their birth assembly instead of release 3019 (ecda297ebe9a4cc492b431681ffb8747).Trigger: the burst fired right after the 08:52Z release request — self-update waves over the
Hosting/*tree — and that tree is exactly what an operator uses to observe and drive a roll.Why this is
sev:H: recycle-after-deploy is the mandated last step of a change set and the only surface that makes a running activation re-read its configuration. This defect makes that remedy silently partial: the cascade reports its work, recycles 0 (or 10 of 21) addresses, and the instance hubs keep serving the old assembly. A primary path is broken and the workaround — recycling each instance by hand — exists but requires knowing which legs aborted. Not raised beyond H: the aborts are logged at fail level, so it is not wholly silent, and there is no evidence yet of a user-visible wrong answer.docs: R3's hoist is cosmetic when the service's lifetime is keyed to …
rbuergi commentedon Sep 22, 2026
Fixed in #5163 — not closing:
sev:Hcloses on production verification, never on a mergeRoot cause, and it is not quite what the issue proposed
The issue's "probable cause" was right about the family and about the file, and it named the fix direction as "enumerate the live instances eagerly (before any hub teardown the wave itself triggers), or catch
ObjectDisposedExceptionand re-resolve". The first was already done and is why the bug was confusing; the second would not have worked.MessageHub.HandleDisposeruns theRecycleCascadeseam and then callsDispose(), on the stated reasoning that the network is derived "while this hub is still whole enough to compute it". That holds for the derivation's synchronous prologue and for nothing after it:DependencyNetworkresolved its three services eagerly, in that prologue — the "enumerate eagerly" remedy, already in the code;Concat, so the later legs subscribe on a pool thread afterDispose()completed andHostedHubsCollection.CloseScopeWhenDisposedclosed the hub's Autofac scope.There is no race to lose — by construction the later legs run against a dead scope.
Why hoisting could not fix it, which is the transferable part.
IMeshServiceis registeredAddScoped(PersistenceExtensions.cs:720,:797), i.e. per hub. The eagerly-capturedMeshServiceholds the dying hub and reaches back through it on everyQuery<T>:StampViewer→CaptureContext()→hub.ServiceProvider.GetService<AccessService>()(MeshService.cs:98, from:347). So hoisting relocates the throw into the service's own body, where no call-site hoist can reach it.Doc/Architecture/DisposedScopeAndDyingHubsR3 (merged in #5159) prescribes "hoist, never guard" on the explicit premise of mesh-lifetime singletons — true there, and not true here.The fix
DependencyNetworkenumerates through the mesh's read-issuing hub (MeshExtensions.ReadIssuingHub), a survivor: hosted by the mesh hub so the definition hub's teardown cannot touch it, off the router, and not the node-CRUD execution hub, so these reads are not queued behind every write in flight. The cascade already applied "issue it from a survivor" to its outboundDisposeRequests; this is the read half it had missed.Control, both sides
ARecycleCascadeEnumeratesFromASurvivorTest— 3 cases:var reader = hubrestoredBoth failures cite this issue's own string verbatim:
ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.One arrives as 2Incompletelegs (the production shape: composed while the hub is whole, subscribed after it died), one thrown synchronously (called after it died). The case that passes on the broken build is the calibration case (hub alive) — which is what shows the other two are not failing for an unrelated reason.Two things keep them from going vacuous: the precondition is asserted (a resolve from the dead hub must throw, so a change in scope-closing order cannot make the cases pass having exercised nothing), and every case asserts the instance path is in the network rather than that the answer is merely "complete" — an empty network is complete too, and "complete, 0 addresses" for every input is the silent form of this very defect.
On the duplicate
The issue asks for
Admin/_LogIncident/8bd12828ac02105bto be merged with this one. Agreed — same failure 1 ms earlier at the same log site: that is the per-legEnumerating the instances of {Type} failedline, this is the cascade summary. One defect, one fix.What closing this needs
A roll, then against the running portal:
Hosting/TriageItemis this incident's own type);[RecycleCascade]line reports a complete cascade with a non-zero address count, and noObjectDisposedExceptionin any leg detail.I have not rolled anything and cannot — live portals are read-only to me. Leaving this open for that verification.
rbuergi commentedon Sep 22, 2026
Status: the fix (#5163) is on the running image. The positive verification is still owed
7a39fb2412, merged 2026-09-22 03:53Z) is contained in3da3423c4b. That is what memex.systemorph.com and memex.meshweaver.cloud both answer on/api/versionnow.main,NodeTypeRecycleCascade.DependencyNetworkresolvesIMeshService/AccessServicefrom the mesh's read-issuing hub (hub.GetMeshHub().ReadIssuingHub()), a survivor, rather than from the dying definition hub. This is the correct answer to the "hoisting a per-hub-scoped service holds the dying hub" trap.ARecycleCascadeEnumeratesFromASurvivorTestpins it: 2 of 3 red with the old reader, per the comment above.Not done: the closing read. Recycle
Hosting/TriageItemon the control instance, then confirm a[RecycleCascade] … address(es) in its dependency networkline with a non-zero count and noObjectDisposedExceptionleg. I could not reach the control instance's MCP this session (503), and a recycle is a live-portal write that should come from the operator.rbuergi commentedon Sep 23, 2026
Status 2026-09-23 ~18:20Z: no change. The fix (#5163) is live and the positive read is still owed
3.0.0-ci.9218(core39f49494).7a39fb2412(fix(mesh,hosting): a recycle and a bake must ask a survivor, not the thing that is dying (#5099, #5073) #5163) is contained in it.Hosting/TriageItem. It is a live-portal write on the control instance, and a roll there (9218 → 9260) is in progress and not converging (Ops/Status/memexat 16:34Z: 1/2 ready,converged: false). Also, my session on the control instance's MCP expired during this pass.The closing act (unchanged, the operator's): once
memexhas converged, recycle the NodeTypeHosting/TriageItem, then read the portal log (LogsInstanceAction,sinceMinutescovering the recycle) and confirm two things:[RecycleCascade] Hosting/TriageItem … address(es)line with a non-zero count;ObjectDisposedException.An
INCOMPLETEline means the fix does not hold.Not established: nothing new since the 09-22 19:20Z comment. No recurrence has been folded since 2026-09-21 08:50:58Z, and as that comment says, that absence is not evidence.
rbuergi commentedon Sep 25, 2026
Triage re-read 2026-09-25 ~15:30Z. Running builds (
/api/version): memex.systemorph.com =39f49494eb(ci.9218), memex.meshweaver.cloud =140c5fb7ae(ci.9291). Ancestry was checked withgit merge-base --is-ancestoragainstorigin/main. Both portals are still held at the V58 migration: the ci.9332 pods crash-loop onDbVersionGate db_version=57 < expected 58. Nothing will roll until a governed Reconcile ofDeployments/memexruns the migration Job.Status: the fix is deployed, and the verification is still owed. #5163 (
7a39fb2412) is contained in both running builds. The closing act is unchanged and is an operator's: recycle the NodeTypeHosting/TriageItemon the control instance, then use aLogsread to confirm a[RecycleCascade] … address(es)line with a non-zero count and noObjectDisposedExceptionleg. I did not do it, because the control instance is mid-stuck-roll.rbuergi commentedon Sep 26, 2026
Post-roll verification, memex.systemorph.com, 2026-09-26 ~05:55Z. memex serves
3.0.0-ci.9332= core4028fd6468(/api/version), Plugins half1470fbf3(CD run 9332 portal-ai jobHEAD is now at 1470fbf; same ref as the/healthpublication-seal line). The DB is at V58:memex-migration-63loggedDatabase migration completed. Version: 58at 2026-09-25 17:58:03Z. The roll is not converged:Ops/Status/memexat 05:48Z reads 1/3 ready; only5dbb7df97b-dktrd(booted 04:06:39Z) serves. So every post-roll window below is about 1 h 50 min on one serving replica. All reads areLogsInstanceActions underOps/Actions/logs-memex-20260926-postroll-*.Still open: the closing act was attempted and refused on permission.
7a39fb2412) is in4028fd6468.recycleMCP tool onHosting/TriageItemon the control instance and got:Recycle requires Update permission on the target node … Ask someone with write access to the node (or a platform admin) to do it.Nothing was recycled, so there is no[RecycleCascade]line to read.recycletool takes noreasonargument. A recycle issued through it would therefore logreason not stated by the caller.Closing act (owner, someone with Update on
Hosting/TriageItem): recycle it, then run aLogsread forRecycleCascadecovering the recycle. Look for a non-zeroaddress(es)count and noObjectDisposedExceptionleg. The roll is not converged (1/3 ready), so prefer to do this after it converges, or the cascade will reach only one replica's activations.rbuergi commentedon Sep 26, 2026
Verified live on memex (ci.9406 = core cdda28d, which carries #5163), 2026-09-26 15:20Z.
The governed
RecycleactionOps/Actions/recycle-memex-20260926-triageitem-5099(Plugins #2399), maintainer-approved and run as system, recycled NodeTypeHosting/TriageItem:[RecycleCascade] Hosting/TriageItem: 180 address(es) in its dependency network: a non-zero count.recycleIncomplete: []: no leg gave up on instance enumeration, which was this issue's failure.recycled 'Hosting/TriageItem': the dispose was posted and a FRESH activation answered a read (version 5716).Done, 8/8 steps.That meets the closing condition in the triage comment: a non-zero address count with no incomplete leg. Closing.
Not established: I did not run a separate Loki read for an
ObjectDisposedExceptionaround 15:20Z. The action's own enumeration completing with no lost leg is the evidence.