Portal managed heap leaks ~215 MB/h through 267 gen2 collections, with ZERO sync/bake events — the cause #2194 is not tracking

Filed at #2194's own instruction — "if no wave occurred, it is (2) and needs its own RCA". #2194 is titled for sync/bake recompile waves; this is the cause that is actually running, and it has zero sync/bake events.

The measurement

memex-cloud, replica …-5nqbz, born 2026-09-03T17:51Z, read from its own runtime watchdog line:

time (UTC)	managed heap	gen2 collections
06:55	6 856 MB	342
09:47	7 981 MB	410
12:48	8 660 MB	454
15:41	8 744 MB	537
18:36	9 371 MB	575
21:51	10 065 MB	609

+3.2 GB in 14.9 h (~215 MB/h) across 267 full collections, and the post-collection floor never returns to any earlier level. A cache the GC has simply not trimmed does not survive 267 gen2 collections. This is retention.

It is not the #2194 mechanism

Counted over that replica's entire 28-hour life (186 137 lines), with info:-level lines confirmed shipping (2 791 of them) so a zero is a real zero:

Stale-build offer:                          0
Stale-build convergence:                    0
EnrichWithNodeType: self-heal recompile #   0
BatchBake:                                  0
KEEPING its load context                    0
NOT unloading collectible context           0
Grain … deactivating: reason=               0

No publication, no bake, no recompile, no supersession. Modules:AutoRecycleOnStaleBuild is armed on both namespaces and has never fired because nothing has ever been stale.

It does not track load — it tracks uptime

replica	age	working set	log lines / 30 min
`fxbhd`	28 h	11 443 Mi	512 — and zero of every hot family
`5nqbz`	28 h	14 096 Mi	1 914
`zn7nf`	15.7 h	4 134 Mi	2 476 — the most of any replica
`lgwrk`	8.3 h	1 384 Mi	16

The near-idle replica holds nearly three times the memory of the busiest one. So "the load is simply heavy" does not explain it, and neither does the unanchored [CrossSchema] fan-out traffic (Doc/Architecture/CrossSchemaFanOutElimination, #2640) — that is a CPU multiplier on this curve, not its source.

The two candidate retainers, and why the first is not sufficient

A — no hub or grain is ever deactivated. deactivating: reason= is 0 across five sampled replicas and 514 687 log lines. Every hub activated since boot is still activated, so every ALC lease taken at bind time (MeshDataSource.cs:1017-1035 — a lifetime lease; NodeAssemblyLoadContext.Dispose defers Unload() while any lease is held) is still held. Nothing superseded is needed for that to accumulate.

B — lazy first-compile, because bundle coverage is failing. From the boot lines:

DynamicTypePreWarmer: ADOPT-ONLY boot complete … adopted=9 uncovered=368 of 377 dynamic NodeType(s); nothing was compiled at boot
368 dynamic NodeType(s) were NOT covered by the prebuilt bundles and will each pay a Roslyn compile on FIRST ACCESS

replica	booted	adopted / total	compile activities since boot
`fxbhd`	09-03 17:49	8 / 377	35
`5nqbz`	09-03 17:51	9 / 377	167
`zn7nf`	09-04 06:06	86 / 399	1
`q8v6s`	09-04 07:10	96 / 399	6 254
`lgwrk`	09-04 13:34	174 / 407	0

Coverage improves with later boots as bakes land — 8/377 → 86/399 → 174/407 across 09-03 17:49 → 09-04 13:34 — so that lane is moving. But it is not enabled: grep -rn RequirePrebuilt deploy/ returns nothing that sets it, and neither live ConfigMap carries the key, so Modules:RequirePrebuilt is in effect on no mesh. (#2193, the design of record, is closed; the remaining named blocker is Systemorph/MeshWeaver.Plugins#1262.) But B alone does not explain the curve: zn7nf logged 1 compile activity and still reached 4.1 GB, and fxbhd logged 35 and reached 11.4 GB. B is a multiplier (see q8v6s: most compiles, highest CPU, fastest growth); A is the shape that is present on every replica without exception.

That is as far as log evidence can take it, and I want to be explicit that it stops there: A and B are candidates, not a proven retainer. Neither is established as the root cause by this data.

Why it becomes an outage rather than a restart

limits.memory is 16Gi and nothing sets DOTNET_GCHeapHardLimit[Percent] (audited by key name across inline env:, memex-portal-config and every envFrom Secret on both namespaces — no value read), so .NET's container default gives the managed heap a 12 GiB ceiling. The runtime defends that with back-to-back blocking gen2 rather than letting the kubelet restart the container: 0 OOMKills across 7 replicas over 28 h, against GC pause share 0.66 sustained for 2 h 15 on 09-03 with /alive answering Healthy throughout.

Full derivation: Doc/Architecture/WhyAGcBoundPodStaysInRotation (#3319).

What would actually name the retainer

Log evidence cannot distinguish A from B or find a third. A gcdump / heap dump from a fat replica would, in one read — retained-bytes by type immediately says whether the roots are hub state, ALCs, cached streams, or something unlisted.

🚨 That needs an ephemeral debug container on a production pod (kubectl debug --profile=sysadmin), which is a production mutation and a maintainer decision. Two caveats if it is authorised: attach the probe to a healthy, non-draining pod — Doc/Architecture/ImagePairSkew records that kubectl debug silently creates nothing on a pod already carrying a deletionTimestamp, printing its "Targeting container" line while ephemeralContainerStatuses stays empty — and pick a fat replica that is not the one hosting a hub someone is using, since the probe competes for CPU with a process already spending two thirds of its time in GC.

(There is no Doc/Architecture/ProfileALivePortalPod page in this repo at ff8c5f1cc; the profiling recipe ImagePairSkew refers to lives outside it. Worth writing down here if this route is taken.) Nothing in this issue has been mutated. The cheap alternative — read dotnet_gc_* off Prometheus — is not available: the portal emits those metrics (AddRuntimeInstrumentation() is on) but the OTel collector's metrics pipeline exports only file/metrics, a rotating JSON file on the node, so nothing queries them. Routing that pipeline to Prometheus is the durable version of the same read and needs no pod mutation at all.

Suggested order

Route the OTel metrics pipeline to Prometheus (deploy/aks/manifests/observability/otel-collector-config.yaml) — it costs no pod restart and turns "read the heap" into a query instead of a dump.
Take one heap dump from a fat replica, or read dotnet_gc_* once (1) is live, and name the retainer.
Fix that. Candidate A (hubs never deactivating) would be the durable fix and is owned by nothing today; candidate B shrinks on its own as bake coverage improves, and disappears when Modules:RequirePrebuilt is actually enabled somewhere.

Not this

Raising limits.memory, lowering the GC hard limit, recycling pods on a timer, or deleting pods daily are all band-aids — the pod-deletion stopgap has been applied by hand at least four times and is recorded as a stopgap on #2194. The question is why 267 full collections free nothing, not how to get more headroom.

Refs #2194 · #2640 · #3319 · Systemorph/MeshWeaver.Plugins#1262

docs(nodetypes): correct 'convergence is the missing piece' with the …

docs: name the portal heap retention root — 8,419 of 9,393 hubs are s…

The retainer is named: 8,419 of 9,393 hubs are `sync/{clientId}`

This issue said the log evidence stopped where it stopped — "A and B are candidates, not a proven retainer" — and that a heap dump would name the retainer in one read. Took the dump. It is neither candidate.

Two --type Heap dumps from memex-cloud/…-gs6j6 (26 h, 6 859 MiB working set) through an ephemeral --profile=sysadmin container, 2026-09-04 22:32Z and 22:55Z. Read-only throughout; both dump files deleted from the pod afterwards and their absence verified; the pod was Running/RESTARTS 0 before and after.

What the heap is

Total 49,827,806 objects, 5,544,881,838 bytes    (Free 2,031,989,944  →  3.51 GB live)

MT               Count      TotalSize  Type
7f9d080a7478     9,386      2,928,432  MeshWeaver.Messaging.MessageHub
7f9d0953b320       962        123,136  MeshWeaver.Hosting.Orleans.MessageHubGrain

🚨 SOS dumpobj and gcroot segfault on this process's dumps, so the addresses came out of a 40-line ClrMD walk (MessageHub → Configuration → Address → Segments[0]), built and run inside the same debug container:

TOTAL MessageHub: 9393
--- by address type segment ---
    8419  sync
     184  MeshWeaver
     144  Ifrs17
      67  Crm
      61  Underwriting
      47  Planning
      26  ReinsuranceDemo      … (the tail is partition names — node hubs)
--- sample full addresses ---
   sync/oeTOuFSg1kSZgDMysMaQ4Q
   sync/QP8kKCTa2k-bgSYV8-5moQ
   Plugins/OpenAI

8,419 of 9,393 — 89.6 % — are sync/{clientId}. The ~974 remaining node hubs match the 962 MessageHubGrain activations almost exactly, so the grain-hosted working set is not what grows: each node hub carries ~8.6 synchronization sub-hubs. At the ~390 KB a hub retains, that is ≈3.2 GB of the 3.51 GB live heap.

Why 9,386 hubs is 3.5 GB and not 300 MB

Every hub owns its own Autofac child ILifetimeScope and its own TypeRegistry. Per hub, from the dump:

Type	Count	Bytes	Per hub
`Autofac…MiddlewareDeclaration`	10,001,898	480 MB	1,066
`Autofac…ExternalComponentRegistration`	2,455,373	275 MB	262
`Autofac…ResolvePipelineBuilder`	2,534,453	101 MB	270
(+ `Service[]`, `NoOpActivator`, `Slot[]`, `ServiceRegistrationInfo`, …)
Autofac subtotal		≈1.17 GB	≈128 KB
`MeshWeaver.Messaging.Serialization.TypeDefinition` (+ its `Lazy`/`Func` fan-out)	848,844	≈0.41 GB	90

≈1.58 GB of the 3.51 GB live heap — 45 % — is identical framework metadata held 9,386 times. Nothing in that table is application data.

The retaining root, with the chain

HostedHubsCollection.messageHubs (HostedHubsCollection.cs:29) — no idle sweep, no TTL, no cap, no LRU. The one removal is hub.RegisterForDisposal(h => messageHubs.TryRemove(h.Address, out _)) (:203). A grain-hosted hub disposes only in MessageHubGrain.OnDeactivateAsync (:890-892) — whose first line (:864, LogInformation) is the deactivating: reason= this issue counted at zero.

And the two halves are the same objects: each sync/ hub's stream fires a HeartBeatEvent every 45 s (SyncStreamOptions.cs:18 → JsonSynchronizationStream.cs:1062 → MeshExtensions.cs:537) into GrainKeepAliveCallback → DelayDeactivation(10 min) (MessageHubGrain.cs:560). A 13× over-provision, and the heartbeat is itself a grain call so it also resets Orleans' own idle clock. The population that fills the heap is the same population that guarantees nothing can ever go idle.

All three candidates falsified — by measurement, not argument

Candidate	Refutation
A — ALC leases held open because nothing deactivates	`dotnet.assembly.count` = 451 (6.9 GB pod) / 475 (14.0 GB pod); 450 distinct mapped `.dll` in `/proc/1/maps`. The fattest replica carries 24 more assemblies, not thousands.
B — lazy first-compile from bundle-coverage failure	Same numbers. A real CPU cost and still #2193's lane; not this memory.
GC fragmentation / committed-not-returned (not in the issue, but the honest alternative)	On `5nqbz` at 14.0 GB: `gen2 heap size 10,251,252,048` with `gen2 fragmentation 1,075,128` — 0.01 % free. Live, reachable data at the 12 GiB ceiling.

That last row answers this issue's central question directly: 267 full collections free nothing because there is nothing to free.

🚨 A tooling trap that would have sent the next reader the wrong way

dotnet-gcdump on this process stopped at exactly 10,000,000 objects and reported a 568 MB heap — one sixth of the truth — with a type table skewed to whatever it reached first. It reports no error. Had I trusted it, the conclusion would have been "the live set is small, so this is fragmentation", which is precisely the wrong answer. dumpheap -stat's Total … objects line is the number to trust.

What I am deliberately NOT claiming, and the next measurement

I did not fix it, because the disposal code is correct and the dump cannot tell me which way it fails. SynchronizationStream.Dispose() completes the store, walks its composite synchronously and disposes its hub (SynchronizationStream.cs:2198-2251). So a surviving sync/ hub is either:

(a) still referenced — a subscriber that really is attached, or something that never let go; or
(b) asked to dispose and never finished — MessageHub.Dispose() posts ShutdownRequest(Quiescing) and returns, and the callback that removes the registry entry runs in the ShutDown phase, several action-block turns later, bounded by nothing.

A type histogram cannot separate (a) from (b), and they call for opposite fixes. Guessing here is how this issue's predecessor spent a month on the convergence lane.

The next measurement is one small extension of the same ClrMD walk: histogram MessageHub.RunLevel over the 8,419 sync/ hubs. A large group at >= ShutDown/Disposing is (b) — a teardown-completion defect, bounded, and the fix is that the registry entry is released on the request rather than the completion. A large group at Started with a live stream is (a), and the question moves to a ClrMD root-walk from one such stream (SOS gcroot will crash the same way dumpobj did).

The uncomfortable third possibility is worth stating: if most are legitimately live, then the portal's working set is simply larger than its ceiling, and the ~390 KB-per-hub DI cost is the whole story rather than a multiplier.

Committed

Doc page Doc/Architecture/PortalHeapIsHubs — the full measurement, the retaining root with file:line, the three falsifications, the two separated defects, the open question above, and the counter recipe that reproduces the discrimination without a dump (gen2 heap size vs gen2 fragmentation settles retention-vs-fragmentation in one read; dotnet.timer.count is a cheap proxy for live hubs — 11,387 at 6.9 GB, 37,796 at 14.0 GB).

PR: #3325.

_{Measured 2026-09-04 against memex-cloud on memex-aks-rg/memexaks-cluster, read-only throughout. No cluster object mutated; no secret value read, printed or stored.}

docs: name the portal heap retention root — 8,419 of 9,393 hubs are s…

Followed the chain three levels further — the corpse-holder is `SynchronizationStream<T>.Hub`

The previous comment named the population (8,419 of 9,393 hubs are sync/) and said the next measurement was a RunLevel histogram. Ran it, and the two after it. Same replica, three more dumps, all read-only; every dump file deleted from the pod afterwards.

1. The population is not homogeneous

TOTAL MessageHub: 9398
    6925  sync  RunLevel=1Started
    1495  sync  RunLevel=6Dead
     974  <node-or-other>  RunLevel=1Started
       4  sync  RunLevel=0Starting

Dead is the enum's terminal state — "the hub is fully disposed and inert" (MessageHubRunLevel.cs:22). 1,495 sync/ hubs completed their entire shutdown and are still reachable. At ~390 KB retained each that is ≈580 MB of pure corpse, and unlike the Started population it admits no benign reading.

2. Who holds them — one pass over all 51,562,911 objects

dead hubs = 1496
--- referrers of DEAD sync hubs ---
   19448  MessageHub+<>c__DisplayClass188_0        ⎫
   13464  MessageHub+<>c__DisplayClass59_0         ⎪ the hub's OWN internals —
    1496  MessageHubConfiguration                  ⎬ exactly 1496 of each,
    1496  MessageService                           ⎪ they die with it
    1496  HierarchicalRouting / RouteConfiguration ⎭
    1485  MeshWeaver.Data.Serialization.SynchronizationStream<MeshWeaver.Mesh.MeshNode>
      11  MeshWeaver.Data.Serialization.SynchronizationStream<System.Text.Json.JsonElement>
       1  ConcurrentDictionary<Address, IMessageHub>+Node
--- GC roots pointing directly at a dead hub ---
direct-root hits = 0

🚨 HostedHubsCollection.messageHubs is NOT the corpse-holder — it did its job. Exactly one of the 1,496 is still in a ConcurrentDictionary<Address, IMessageHub>; the RegisterForDisposal removal at :203 fired for the other 1,495. (This corrects my previous comment, which named the registry as the retaining root. It is where the LIVE hubs sit; it is not what keeps the dead ones alive.)

🚨 The corpse-holder is SynchronizationStream<T>.Hub — 1,485 + 11 = 1,496, one stream per dead hub, exactly. No GC root points at a dead hub directly; every one is alive purely transitively, through the stream that created it.

3. And the stream was never told

SynchronizationStream total=8461 disposed=11
    4219  <MeshWeaver.Mesh.MeshNode>           live
    3193  <MeshWeaver.Data.EntityStore>        live
     992  <MeshWeaver.Data.InstanceCollection> live
      11  <System.Text.Json.JsonElement>       DISPOSED

Only 11 of 8,461 streams are disposed. SynchronizationStream.Dispose() sets isDisposed before it disposes the hub, so a stream reading false never ran Dispose(). Put the two together:

≈1,485 SynchronizationStream<MeshNode> instances are NOT disposed, and their Hub is Dead.

A sync/ hub is a hosted hub: its parent disposes it during DisposeHubsReactive when the parent goes down — a Blazor circuit ending, a DisposeRequest, a recycle — while the stream that created it is owned by a workspace somewhere else and is never notified. The stream keeps isDisposed == false, keeps a strong reference to the corpse and everything the corpse roots (its Autofac ILifetimeScope, its TypeRegistry), and keeps being handed out as usable.

The codebase already knows the creation-time half of this hazard — SynchronizationStream.cs refuses to hand out a stream whose host is winding down, because "a consumer cannot even detect the corpse: ISynchronizationStream exposes no liveness/disposal member", and that cost a production NRE inside LayoutAreaHost's constructor. The post-creation half — the hub dying under a stream already handed out — has the same root and is not covered.

Note this also explains the zero deactivating: reason= count without contradiction: these hubs are hosted sub-hubs, disposed by their parent's teardown, never MessageHubGrain activations.

So: three defects, and I am deliberately not fixing them blind

#	Defect	Size	Status
1	A hosted hub dies under a stream that is never told	1,496 hubs, ≈580 MB	Unambiguous. `ISynchronizationStream` has no liveness member, so nothing can detect it.
2	`Started` `sync/` hubs never released	6,925 hubs, ≈2.7 GB	Ambiguous — see below
3	~390 KB retained per hub, ~173 KB of it duplicated DI + type-registry metadata	45 % of the live heap	Real regardless of 1 and 2

What is still inference, stated plainly:

What kills the 1,496. Measured: Dead while the stream is undisposed, so not via Stream.Dispose(). Inferred: the parent. The confirming measurement is a log correlation, not a dump — count [DISPOSE-CONTAINER] … lifetime scope closed (HostedHubsCollection.cs:338, Debug) for sync/ addresses over a window against the growth in the Dead population.
Whether the 6,925 Started hubs are garbage at all. Consistent with correct behaviour (8,450 undisposed streams for subscribers really attached) and with streams abandoned without Dispose(). The discriminator is the same one-pass referrer walk retargeted at their streams.

🚨 Fixing (1) alone recovers ~580 MB of ~3.5 GB. Worth doing, and it is the only one I would call ready — but (2) is where the rest is, and it needs the walk above before anyone writes code. Guessing here is how the predecessor issue spent a month on the convergence lane.

Committed

All five measurements are in Doc/Architecture/PortalHeapIsHubs (#3325, and the follow-up that adds these three). The page also carries two tooling traps paid for here: dotnet-gcdump silently truncated at exactly 10,000,000 objects and reported a 568 MB heap (a sixth of the truth), and SOS dumpobj/gcroot segfault on this process — anything that follows a field must be ClrMD, which works fine inside the debug container since the SDK image has internet.

_{Measured 2026-09-04 22:32–23:14Z against memex-cloud on memex-aks-rg/memexaks-cluster, read-only throughout. Specimen pod Running / RESTARTS 0 before and after. No cluster object mutated; no secret value read, printed or stored.}

Carried from #2194 on close: the alerts that would watch this leak are authored and INERT

Both working-set alerts now exist in deploy/aks/scripts/values.observability.yaml — PortalReplicaWorkingSetDiverged and, as of #3319, PortalHeapEnteringGcDefenceBand (per namespace, no sibling ratio, thresholded at the GC hard limit). Neither is armed.

GET /api/v1/alerts   → active alerts: 0
GET /api/v1/rules    → rule groups: 0
helm list -A         → loki   monitoring   revision 1   updated 2026-05-31

That file has never been applied to this cluster, so the detector for the curve this issue tracks has been blind since the day it merged, and #3319 changes nothing about that — it improves what would fire, not whether anything is armed.

The action is a helm upgrade of the loki release in ns monitoring. It touches no portal pod in either namespace — no rollout, no restart. It is a production mutation and therefore an operator decision, which is why it is recorded here rather than done.

🚨 The only evidence it worked is GET /api/v1/rules returning a group naming both alerts. git grep is evidence of authorship and never of arming — that conflation is what made #2194 read as instrumented for a month while it was blind.

It pairs naturally with step 1 of the suggested order above: routing the OTel metrics pipeline to Prometheus would let an alert read dotnet_gc_heap_size against the hard limit directly, instead of inferring it from a kubelet-side working-set number that cannot distinguish managed heap from native memory.

Re-measured 2026-09-05 12:38Z — the curve is unchanged, and the blocker on this issue is gone

The GRAFANA_PW dependency was never necessary. Loki's API is reachable in-cluster and
unauthenticated, so az aks command invoke + curl reads it directly — no Grafana, no password, no
new secret:

az aks command invoke -g memex-aks-rg -n memexaks-cluster \
  --command "curl -s http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/labels"

(Grafana itself was publicly exposed at grafana.systemorph.com on a shared admin password for 95
days; that ingress is now removed and both Grafana and Loki sit behind an internal-only ingress on
10.42.0.11, VPN-reachable. Unrelated to the leak, but found while chasing this.)

The measurement, on a completely different replica set

Every replica in the original table has been replaced or restarted since. Container uptime measured
from the last start (lastState.terminated.finishedAt where a restart occurred), working set from
kubectl top:

pod	container uptime	working set	MB/h
`fxbhd`	42.8 h	11 889 Mi	278
`gs6j6`	40.2 h	10 580 Mi	263
`zn7nf`	30.5 h	4 998 Mi	164
`q8v6s`	29.5 h	4 615 Mi	157
`xrgsx`	21.6 h	3 330 Mi	154
`lgwrk`	21.6 h	2 815 Mi	130
`5nqbz`	12.0 h	3 894 Mi	325

Pearson r(uptime, working set) = 0.875 across all seven, and the mean rate is 210 MB/h
against the 215 MB/h this issue was filed on. Two days, a different replica set, the same number.
That is the strongest form of confirmation available short of a dump: the original measurement was
not an artefact of one pod.

fxbhd at 11 889 Mi is ~99 % of the 12 GiB managed-heap ceiling the issue derives.

The "no OOMKill" finding still holds — I checked, because it looked like it had changed

Three replicas now show restarts=1, which the original table did not. None is an OOMKill:

pod	last termination
`5nqbz`	`Completed`, exit 0
`lgwrk`, `xrgsx`	`Unknown`, exit 255, both at 2026-09-04T15:01:17Z (same instant — a node-level event, not the app)

So the mechanism in WhyAGcBoundPodStaysInRotation is intact: the runtime defends the ceiling with
blocking gen2 rather than letting the kubelet recycle the container. A restart count is not evidence
against it.

What this does and does not settle

It settles that the leak is live, reproducible and uptime-linked — candidate A's shape (nothing
ever deactivates, so every ALC lease is held for the process lifetime) predicts exactly this
correlation, and candidate B does not (5nqbz has the highest rate on the shortest uptime).

It does not name the retainer. Nothing in the logs can, as this issue already concluded. That
still needs the gcdump from a fat replica — fxbhd or gs6j6 are the candidates, and both are
comfortably above the threshold where retained-bytes-by-type would be unambiguous.

Sizing defect (1) — and ruling out the fix that looks obvious

Not implementing this today: it is a reference-dropping change on a hot path and deserves a clean
pass with tests. But I measured what it involves, so the next attempt does not re-derive it.

"Just close the lifetime scope" is not the missing piece

MessageHub deliberately does not dispose its own ServiceProvider — the call is present and
commented out, with the reason recorded beside it:

//await ((IAsyncDisposable)ServiceProvider).DisposeAsync();
// … Configuration.ParentHub re-resolves from ParentServiceProvider, which is often disposed by
// the time we get here … Never call DI from a disposal path.

Closing it is delegated instead: OwnsServiceProvider is read by HostedHubsCollection.Add,
"the only place that both knows the scope exists and can act strictly after this hub is terminally
down". So the scope is closed for these hubs — which is consistent with the earlier finding that
HostedHubsCollection removed 1 495 of 1 496. Adding a scope-dispose to the hub's own teardown would
re-introduce exactly the ObjectDisposedException that comment is guarding against, and would not
free the bytes anyway.

What actually retains, therefore

Closing an Autofac scope does not null the fields on an object that outlives it. The retained graph
is stream → dead hub → its own resolved state (TypeRegistry, and whatever else was resolved onto
the hub at construction). The only way to reclaim it is to drop a reference, which is what makes
this delicate rather than fiddly.

The two places a reference could be dropped, measured

candidate	shape	cost
`SynchronizationStream.Hub`	`public IMessageHub Hub { get; private set; }` — settable, so it can be nulled in `Dispose()`	32 `Hub.` uses in a 2 501-line file against only 4 `isDisposed \|\| Hub is null` guard sites — so ~28 call sites need auditing, not one
`MessageHub.ServiceProvider` / `.TypeRegistry`	both `public { get; }` — get-only	changing them to settable and nulling on disposal touches a public surface that `hub.ServiceProvider.GetRequiredService<T>()` uses everywhere; riskier than the stream side

So the stream side is the right target, and the work is the ~28 unguarded Hub. call sites plus a
test that reproduces corpse-holding (dispose a hosted sync/ hub under a stream nobody disposed,
assert the stream releases it).

One thing that must not be lost

Dispose() sets isDisposed, completes the store, clears sharedReduceCache and disposes
streamDisposables — but it never clears Hub. So even the streams that are disposed keep
their hub reachable. Whatever the fix, Dispose() clearing the field is part of it, not only the
hub-death path.

🚨 Defect (1)'s fix cannot be "drop the `Hub` reference" — that was tried, and it cost a production NRE

I picked this up to implement defect (1) and stopped before writing code, because the file records
that the predecessor of the current constructor did exactly the proposed change and it was
reverted. The reasoning is in SynchronizationStream.cs around the refusal path, and it is worth
quoting rather than paraphrasing:

The predecessor of this code answered that by building a DEAD stream — isDisposed, Store completed,
Hub = null! — and documenting that "every code path that touches Hub goes through
TryGetActiveHub". NO CONSUMER HONOURED THAT … And a consumer cannot even detect the corpse:
ISynchronizationStream exposes no liveness/disposal member. Handing one out is a contract
violation by construction — it cost a production NRE inside LayoutAreaHost's constructor
(Stream.Hub.ServiceProvider…) on the overlay self-heal recycle window, which escaped to the
subscriber as a TERMINAL DeliveryFailure.

Re-measured today, because the sizing on this issue is scoped to the wrong file

The sizing above ("32 Hub. uses against 4 guard sites in a 2 501-line file, so ~28 call sites")
counts only inside SynchronizationStream.cs. Measured across src/:


`TryGetActiveHub` uses outside `SynchronizationStream.cs`	0
stream `.Hub` dereferences across `src/` (conservative regex, excluding that file)	47
`ISynchronizationStream.Hub` declaration	`IMessageHub Hub { get; }` — non-nullable

Inside the file the position is actually good, and better than the sizing suggests: the write paths
already funnel through TryGetActiveHub (Update bails via SignalDisposedToProducer), DeliverMessage,
OnNext and RegisterForDisposal each carry isDisposed || Hub is null, and the FailStartup /
OpenGate pair is guarded with a deliberately loud else-branch. Only OwnerVersion() and the two
Build*ChangeItem helpers read Hub unguarded, and all three are reachable only after Update's
guard has passed.

The exposure is entirely outside the file, through a contract that promises non-null — which is
precisely why the in-file discipline did not save the predecessor.

So the prerequisite is the contract, not the field

The blocker both this issue and that comment name is the same sentence: ISynchronizationStream
exposes no liveness member. Until a consumer can ASK whether the stream is still alive, dropping the
reference converts a memory leak into a null-dereference on the layout-area render path — a strictly
worse failure, and one this repository has already shipped once.

A staged shape that does not re-cause it:

Expose liveness on ISynchronizationStream — the internal IStreamLivenessSource
(Source / IsDisposed / IsFaulted) and StreamLiveness.IsUsable already compute exactly this,
including walking the reduce chain; today it is internal and consumers cannot reach it.
Migrate the 47 dereference sites to obtain the hub through something that can answer "no".
Only then drop the reference — at which point Hub-is-null is a state the contract admits
rather than a lie.

Step 1 is a public-surface addition and step 2 is a mechanical sweep with a real cost, so this is a
larger change than the sizing implies. I have not started it; flagging so nobody re-derives the trap.

Also worth correcting

StreamLiveness.IsUsable already treats current.Hub is not { } hub as dead, so detection is
solved and only release is missing — the issue reads as though both were open.

🤖 Filed from Claude Code.

🚨 Correction: do NOT implement defect (1) as I sized it — that approach was already tried and cost a production NRE

meshweaver-64 caught this before writing code. My sizing comment above is wrong in a way that could
have walked someone into repeating a production incident, so treat this as superseding it.

What I got wrong

I sized the fix as "Hub is { get; private set; } so it CAN be nulled; 32 Hub. uses against 4
guard sites, ~28 call sites to audit". That count is scoped to SynchronizationStream.cs alone,
and the exposure is almost entirely outside it.

SynchronizationStream.cs records the previous attempt, in the constructor's refusal path:

The predecessor of this code answered that by building a DEAD stream — isDisposed, Store
completed, Hub = null! — and documenting that "every code path that touches Hub goes through
TryGetActiveHub". NO CONSUMER HONOURED THAT: grep -rn TryGetActiveHub src found only this
file, while ~96 sites dereference ISynchronizationStream.Hub, which the interface declares
NON-NULLABLE. And a consumer cannot even detect the corpse: ISynchronizationStream exposes no
liveness/disposal member. […] it cost a production NRE inside LayoutAreaHost's constructor
(Stream.Hub.ServiceProvider…) on the overlay self-heal recycle window, which escaped to the
subscriber as a TERMINAL DeliveryFailure. A page that subscribed during a recycle got "this
failed forever" instead of "ask again".

So "null the field, audit the call sites" is not an untried idea with a manageable cost. It is the
thing that was tried, and the in-file discipline is precisely what failed to hold — because the
contract promises non-null to ~96 callers who have no way to ask.

Also correcting myself: detection is NOT open

My earlier comment reads as though both detection and release were missing. Only release is.
StreamLiveness.IsUsable already treats current.Hub is not { } hub as dead, checks
RunLevel > Started and MessageHub { IsDisposing: true }, and walks the reduce chain via
IStreamLivenessSource.Source. The cache already refuses to serve a corpse. What is missing is
letting it go.

The prerequisite is the CONTRACT, not the field

The staged shape, which is materially bigger than what I sized:

Expose liveness on ISynchronizationStream. IStreamLivenessSource (Source / IsDisposed /
IsFaulted) and StreamLiveness.IsUsable already compute the answer, chain-walk included — they
are merely internal. Making a consumer able to ask is what the predecessor lacked.
Migrate the dereference sites to an accessor that can answer "no".
Then drop the reference.

Step 1 is a public-surface addition, so it is a maintainer decision, not something to start off
the back of an issue comment. I am not implementing it, and I would not recommend anyone do so
without that call being made explicitly.

My apologies for the earlier sizing — I measured one file and reported it as the cost of the change.

Step 1 of 3 has landed — #3380 merged

SynchronizationStreamLiveness.IsUsable(stream) / stream.TryGetHub() are now public, so a consumer
can ask whether a stream is alive. Behaviour is unchanged; this is purely the contract.

Deliberately kept as an extension, not a member on ISynchronizationStream — StreamLiveness
records two reasons the interface is internal and neither has expired: adding a member to a public
interface breaks every downstream implementer (these ship as packages), and IStreamLivenessSource.Source
is not a consumer-facing concept (it exists so the reduce chain is walked in exactly one place). Both
new methods delegate to StreamLiveness.IsUsable and add no logic — that type exists because three
hand-copied predicates diverged (#1455), and a wrapper that re-implemented the check would be the fourth.

The guard is reflection-based on purpose: MeshWeaver.Data grants InternalsVisibleTo("MeshWeaver.Data.Test"),
so a test that merely called the methods would compile and pass unchanged if the surface reverted to
internal. Falsified rather than assumed — with the class flipped to internal the project still
builds and the other three tests still pass; only TheSurfaceIsGenuinelyPublic goes red.

What remains on this issue

Migrate the 47 dereference sites onto TryGetHub(), which can answer null.
Then clear Hub on disposal — reclaiming the retained stream → dead hub → resolved state graph,
which is the actual leak this issue is about.

🚨 Doing 3 before 2 is the production NRE, in the order it happened: the predecessor nulled the field
and documented that everything went through TryGetActiveHub, which was private to one file while ~96
sites dereferenced a property the interface declares non-null. LayoutAreaHost's constructor NRE'd
during a recycle window and the subscriber got a terminal DeliveryFailure.

The design and the ordering are now committed as Doc/Architecture/StreamLivenessAndTheHubReference,
so this does not have to be re-derived from issue comments again.

Leaving this issue OPEN — the leak is not fixed until step 3.

docs(#3321): name the dependent half and why no gate can see it

fix(#3321): the reachability assertion waits for the teardown it meas…

Step 3 is merged — the stream releases its hub

Core: fix(#3321): step 3 — a disposed stream releases its hub #3427 → main (f41f8bda)
Dependent: Systemorph/MeshWeaver.Plugins#1405 → that repo's main (cdfaef28), merged first

Closing manually: the merge queue does not fire Closes links.

🚨 The description on this issue was a sixth of the fix

"SynchronizationStream.Dispose() clears Hub" is how step 3 has been written up throughout. Read
against the dump this issue already carries, that reclaims 11 of 1 496:

SynchronizationStream total=8461 disposed=11
   1485  <MeshNode>     hub RunLevel=Dead, stream NOT disposed
     11  <JsonElement>  hub RunLevel=Dead, stream DISPOSED

A sync/ hub is a HOSTED hub — the parent disposes it during its own teardown while the stream that
created it, owned by a workspace elsewhere, is never told. Dispose() never runs for those 1 485.
So the release happens at both ends through one idempotent ReleaseHub(): from Dispose(), and
from a hook the constructor registers on the sub-hub's own disposal. That hook is registered after
streamDisposables is hooked onto the hub, because a hub's composite disposes in registration order
and the stream's own registrants must still see a bound Hub.

The 25 sites, and the one that changed the design

A fresh sweep found 29 stream-receiver .Hub occurrences in src/; four must stay (two ARE the
predicate, one is already ?., one is a liveness check). The other 25 — the 20 named on this issue,
plus the three LayoutAreaHost reads step 2 called "hub-turn confined" (that premise expired with
this change), Workspace.EvictClientSubscriptions, and WorkspaceExtensions.ApplyChanges which
step 2's list missed behind a null-forgiving !.

The 12 patch reducers were the design problem. A reducer's signature is … → ChangeItem<T>: it
must return a change, so it has no absent value. But all of them are invoked from exactly ONE
place — ToChangeItem, the only call site of ReduceManager.PatchFunction — and null is what that
method already returns for a stream with no PatchFunction. So the reducers refuse through
RequireHub()'s TRANSIENT HubDisposingException and the gateway maps that one type to its own
modelled absence. Not a swallow, and not a catch (NullReferenceException).

🚨 Three things this took that are worth carrying forward

1. PRESENCE is not LIVENESS. The first draft guarded every owner-side site with TryGetHub()
(IsUsable). DataSource.Initialized is Task.WhenAll(streams.Select(s => s.Hub.Started)) — and a
stream whose initial load FAULTED is not IsUsable, so it dropped out of the WhenAll,
Initialized completed successfully, and a hub whose data source had thrown went on answering
requests. DataContextFaultedInitBeforeStreamHubBoundTest caught it: the #2625 wedge, re-introduced
by a guard that looked like a tightening of safety. Owner-side guards now use the presence
accessor, which matches the isDisposed || Hub is null refusal their own gate already applies.

2. The reachability test was FLAKY and its green was luck. Hub.Dispose() only starts the
teardown, so on return the hub is still rooted by its own in-flight shutdown. Collecting immediately
passed in a full-suite run — where surrounding tests happened to give it time — and failed the
moment the test ran alone: 1 failed in isolation against 485 passed in the suite, same binary. It
now awaits DisposalCompleted before collecting.

3. No gate could have told anyone about the dependent half. This is shape 7 — a public member
whose BEHAVIOUR changes behind an unchanged signature — and the Cross-repo pair gate resolves
removed types and members, of which this diff has zero. A by-hand sweep of all six satellites, on
the receiver type and including in-mesh C# inside .json node strings, found 11 production
dereferences in MeshWeaver.Plugins's Blazor view layer and zero elsewhere. Every one reads
Stream!.Hub.Something — the ! is on Stream, not on .Hub, so the NRE moves one level
right past the operator that suppresses the warning and the compiler stays silent. One sits inside a
bare catch { }, where it would have rendered a wrong chart instead of throwing.

Falsification

Measured by disabling only the two ReleaseHub() call sites — every guard and accessor left in
place, so the experiment isolates exactly what step 3 does:

Suite	release disabled	release enabled
`StreamReleasesItsHubTest`	4 failed, 2 passed	6 passed
`TornDownStreamCallSitesTest` (step 2's)	4 failed, 1 passed	5 passed
`StreamCacheLivenessTest` (Plugins, without the `Hub is null` arm)	1 failed — `TimeoutException`, 30 s	3 passed, 833 ms

The live controls passed both ways in all three — a release that fired unconditionally would satisfy
every negative at once.

What is NOT claimed

This releases the 1 496 Dead-hub corpses — defect (1) on this issue, ~580 MB of a 3.5 GB live
heap. Defect (2) — the 6 925 Started sync/ hubs, ~2.7 GB — is untouched and still open, and
this issue's own analysis says the discriminator for it is a referrer walk that has not been run. So
expect the curve to flatten, not to vanish; the honest next measurement is a fresh dump against a
replica running a build that carries f41f8bda, compared on the SynchronizationStream total / disposed and RunLevel=Dead counts this issue already established. If it warrants a follow-up,
that is a new issue about defect (2), not a reopening of this one.

Committed

Doc/Architecture/StreamLivenessAndTheHubReference carries the whole design — the 11-vs-1 485
split, the two-ended release with its registration-order constraint, the 29/4/25 site table with a
reason per group, the measured presence-vs-liveness finding, the dependent-half sweep with per-repo
counts, and the falsification numbers. Plus a What's New entry (Category: Fix).

Defect (2) is now tracked on its own: #3432.

It carries the measured population (6 925 Started sync/ hubs, ≈2.7 GB of a 3.51 GB live heap,
from the same five dumps as this issue), the 1 496 / 11 / 1 485 split that made step 3 need two
hooks rather than one, and — explicitly — the list of things that are not established, so the
next session treats the mechanism as a hypothesis rather than a finding.

It also names the outcome that would falsify it: if the referrer walk finds diffuse referrers
matching live subscribers, this is per-hub COST (≈1.58 GB of duplicated Autofac + type-registry
metadata across 9 386 hubs), not retention — a different issue and a different fix.

🚨 And the thing not to misread: step 3 released ≈580 MB of ≈3.5 GB, so the curve should flatten,
not vanish. A still-rising curve is not step 3 having failed.

Portal managed heap leaks ~215 MB/h through 267 gen2 collections, with ZERO sync/bake events — the cause #2194 is not tracking #3321

Description

The measurement

It is not the #2194 mechanism

It does not track load — it tracks uptime

The two candidate retainers, and why the first is not sufficient

Why it becomes an outage rather than a restart

What would actually name the retainer

Suggested order

Not this

Activity

rbuergi commented on Sep 4, 2026

The retainer is named: 8,419 of 9,393 hubs are sync/{clientId}

What the heap is

Why 9,386 hubs is 3.5 GB and not 300 MB

The retaining root, with the chain

All three candidates falsified — by measurement, not argument

🚨 A tooling trap that would have sent the next reader the wrong way

What I am deliberately NOT claiming, and the next measurement

Committed

rbuergi commented on Sep 4, 2026

Followed the chain three levels further — the corpse-holder is SynchronizationStream<T>.Hub

1. The population is not homogeneous

2. Who holds them — one pass over all 51,562,911 objects

3. And the stream was never told

≈1,485 SynchronizationStream<MeshNode> instances are NOT disposed, and their Hub is Dead.

So: three defects, and I am deliberately not fixing them blind

Committed

rbuergi commented on Sep 4, 2026

Carried from #2194 on close: the alerts that would watch this leak are authored and INERT

rbuergi commented on Sep 5, 2026

Re-measured 2026-09-05 12:38Z — the curve is unchanged, and the blocker on this issue is gone

The measurement, on a completely different replica set

The "no OOMKill" finding still holds — I checked, because it looked like it had changed

What this does and does not settle

rbuergi commented on Sep 5, 2026

Sizing defect (1) — and ruling out the fix that looks obvious

"Just close the lifetime scope" is not the missing piece

What actually retains, therefore

The two places a reference could be dropped, measured

One thing that must not be lost

rbuergi commented on Sep 5, 2026

🚨 Defect (1)'s fix cannot be "drop the Hub reference" — that was tried, and it cost a production NRE

Re-measured today, because the sizing on this issue is scoped to the wrong file

So the prerequisite is the contract, not the field

Also worth correcting

rbuergi commented on Sep 5, 2026

🚨 Correction: do NOT implement defect (1) as I sized it — that approach was already tried and cost a production NRE

What I got wrong

Also correcting myself: detection is NOT open

The prerequisite is the CONTRACT, not the field

rbuergi commented on Sep 6, 2026

Step 1 of 3 has landed — #3380 merged

What remains on this issue

rbuergi commented on Sep 6, 2026

Step 3 is merged — the stream releases its hub

🚨 The description on this issue was a sixth of the fix

The 25 sites, and the one that changed the design

🚨 Three things this took that are worth carrying forward

Falsification

What is NOT claimed

Committed

rbuergi commented on Sep 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The retainer is named: 8,419 of 9,393 hubs are `sync/{clientId}`

Followed the chain three levels further — the corpse-holder is `SynchronizationStream<T>.Hub`

≈1,485 `SynchronizationStream<MeshNode>` instances are NOT disposed, and their `Hub` is `Dead`.

🚨 Defect (1)'s fix cannot be "drop the `Hub` reference" — that was tried, and it cost a production NRE