Repository navigation
fix(hub): track and cut the per-hub DI and type-metadata cost — a burst of hub activations can OOM a 12 GiB portal replica #7385
Description
The per-hub cost of DI and type metadata has no open tracker, and on 2026-10-23 it turned one UI click into an out-of-memory crash of a portal replica.
Evidence (governed AnalyzeDump, control instance, Ops/Actions/i7082-analyzedump-hwx6z-heapstat, page Ops/Dumps/i7082-analyzedump-hwx6z-heapstat; memex pod hwx6z, dump written 2026-10-23T08:03:15Z, read with core main cdf9bf0):
- Managed heap 12,319,278,588 bytes, 203.1 M objects. Gen2 11.08 GiB, compacted; SOH free 0 %, so this is live retained memory and not fragmentation.
- Autofac child-scope rows: 5.78 GiB (50.3 %). MiddlewareDeclaration 2.44 GB (50.8 M objects); ExternalComponentRegistration 1.35 GB (12.06 M), plus its NoOpActivator 0.29 GB; ResolvePipelineBuilder 0.50 GB (12.4 M); Service[] 0.40 GB; ConcurrentQueueSegment+Slot[] 0.45 GB.
- Per-hub TypeRegistry rows: 1.49 GiB (13 %). TypeDefinition 2.79 M with its Lazy/Lazy/Func/LazyHelper; ConcurrentDictionary<String,TypeDefinition>+Node 4.69 M.
- No application data type is in the top 30.
- 2,568 Orleans activation message loops were live. That is about 4,700 external registrations and about 1,090 type definitions per activation, and the heap grew about 3.6 MB per activation (2.1 to 11.3 GiB). That matches the ~5 MB per node hub in MeshWeaver#6623.
Code path: ServiceProviderExtensions.SetupModules calls BeginLifetimeScope(c => PopulateMeshWeaverWithOwners(...)) for every hub. A configuration action gives the child scope its own registry, and that registry wraps each parent registration it touches in an ExternalComponentRegistration with a ~4-stage pipeline. MessageHubConfiguration also builds a TypeRegistry per hub, which holds its own TypeDefinition per registered type. #5925 only shares the display metadata.
This is defect 3 of Doc/Architecture/PortalHeapIsHubs ("a hub costs ~390 KB, ~173 KB of it duplicated framework metadata"). That page measured 262 external registrations per hub in September. Its trackers MeshWeaver#3321 and #3432 are closed, and #6623 (sev:L) covers only boot-activated hubs. The trigger on 2026-10-23 is fixed by MeshWeaver.Plugins#4112 (the mark-all-read fan-out). Fixing the trigger leaves any burst of a few thousand hub activations, from any caller, able to exhaust a 12 GiB heap, because each hub costs megabytes of identical framework metadata.
Not established: the MessageHub count in this dump (MessageHub is not in the 30 rows kept). So it is not known whether registrations per hub grew since September or hubs per activation grew. Retention by referrer (no gcroot) is also not established; the attribution to hub scopes is from types and code, not from a root walk.
Analysis comment: MeshWeaver#7082 issuecomment-6382995136.
Where: submitted from the portal Feedback module on memex.systemorph.com; no main node attached.
Owner basis: the named symbols ServiceProviderExtensions.SetupModules, PopulateMeshWeaverWithOwners and MessageHubConfiguration are MeshWeaver core — they are the subject of Systemorph/MeshWeaver PRs #7027 and #7131 — and every tracker the report cites (#3321, #3432, #5925, #6623, #7082) is a Systemorph/MeshWeaver issue; #6623 measured the same ~5 MB per node hub.
Related: PR #7131 (fix(hub): a hub scope no longer re-declares its ancestors' instances or the mesh's type definitions) is in flight on this exact cost — this issue is the open tracker the report asks for.
Origin: https://memex.systemorph.com/Feedback/_Submissions/fb-82f9c96856254c5a0837d7953fb48eb5 (submitted on memex.systemorph.com)
filed by the triage agent on behalf of rbuergi via systemorph-com
Triage routing: filed in Systemorph/MeshWeaver — agent: Systemorph/MeshWeaver — Symbols ServiceProviderExtensions.SetupModules / PopulateMeshWeaverWithOwners / MessageHubConfiguration are MeshWeaver core (subjects of MeshWeaver PRs #7027 and #7131); all cited trackers (#3321, #3432, #5925, #6623, #7082) are Systemorph/MeshWeaver issues..
Activity
rbuergi commentedon Oct 27, 2026
The two cost rows the dump measured are addressed on main:
ExternalComponentRegistration,MiddlewareDeclaration,ResolvePipelineBuilder): fix(hub): a hub scope no longer re-declares its ancestors' instances or the mesh's type definitions (#7082) #7131 (merged 2026-10-24) stops a hub scope from re-declaring its ancestors' instances, and refactor(di): replace Autofac with MeshWeaver's own container, MeshServiceProvider (#7145) #7147 (merged 2026-10-25) removes Autofac entirely in favour of the per-hub overlayMeshServiceProvider.TypeDefinitionrows: fix(types): a hub's type registry is a parent-chained overlay — only overrides are stored #7167 (merged 2026-10-25) turns a hub's type registry into a parent-chained overlay that stores only overrides.The triage note's "#7131 in flight" is stale. The running image
94cfc71adbcontains all three. Awaiting post-roll verification: a heap reading (per-activation growth, or a governed AnalyzeDump) on the running image that shows the per-hub framework metadata gone. Leaving open.rbuergi commentedon Oct 29, 2026
Deployed; behaviour not verifiable read-only. Leaving open.
0de4227604(compare:ahead). Running build, read 2026-10-29 16:40–17:05Z: memex.systemorph.com, memex.meshweaver.cloud and control.systemorph.com all answer/api/version= core0de4227604. PerOps/Status/<deployment>on the control instance every replica runs image 3.1.12025 (control:memex-control:3.1.12025), built by core main-cd run 12025 with the plugins host ate560f85c6d. The sealed plugins module publication on both memex portals is7c7e0fe555(/health→publication-seal; 15 of 43 packages are still carried at an older sealed build).rbuergi commentedon Nov 1, 2026
Verified on production with the instrument this issue names (a governed AnalyzeDump on a post-fix image). Closing.
The reading:
Ops/Dumps/t1031-5388-analyzedump-dsfp6on the control instance. It is a dump of memex podmemex-portal-deployment-86f575cbd7-dsfp6, written 2026-10-30 19:03Z and analysed 2026-10-31 17:39Z.The build: the pod's own warm-up line (
Ops/Actions/iss7385-dsfp6-commit, 2026-10-30 12:19:53Z) readsplatform build 3.1.12166 … commit g1a63da9724. Compared against1a63da9724, the merge commits of #7131 (8478c800), #7147 (2cc600e3) and #7167 (f531a285) all returnahead. So the pod ran all three fixes.What the dump shows, against the original reading from
hwx6z:hwx6z(pre-fix, 2026-10-23)dsfp6(3.1.12166, 2026-10-30)MiddlewareDeclaration,ExternalComponentRegistration,ResolvePipelineBuilder, …)TypeDefinition+ConcurrentDictionary<String,TypeDefinition>For scale:
dumpasynccounts 89 Orleans activation message loops indsfp6. At thehwx6zrates, 89 activations would carry roughly 85 MB ofMiddlewareDeclarationand roughly 50 MB ofTypeDefinitionrows. Either would rank in the top 40, and neither does.Not established:
MessageHubitself is not in the 40 rows kept, so per-hub cost was not computed directly.A different retention is in this dump: 12.3 GB on the heap, 48.8 % of it
Free, about 2.42 M retainedGetDataRequestdeliveries and theirHubStreamProviderFactorySelectManychains, and aStackOverflowExceptioninSelectMany…InnerObserver.OnError. That is a content-collection resolution cycle, not the per-hub framework metadata this issue tracks. The fix is in #7763.