Compile activity hubs are not retired when the compile finishes — keep-alive heartbeats still route to them and OOM under compile-storm hub churn

What is failing

A system-generated HeartBeatEvent keep-alive (answer-suppressed, sent from the cache hub) was still being routed to the per-node hub of a compile activity 49 minutes after that compile had completed successfully — the activity node shows status Succeeded with a ~1s runtime, yet the ping (routing path cache → mesh → compile-…) was still being delivered to it nearly an hour later. Processing that delivery in MessageHub.HandleMessageAsync threw System.OutOfMemoryException while walking the delivery rule chain.

The heartbeat itself is a fixed-size ping — this OOM is not about the message. It is the pod's exhausted managed heap surfacing at the next allocation, which happened to land inside the message pipeline (and the ~1.8s the delivery had already spent before failing says the hub was struggling before it threw).

Probable cause — moderate confidence

Two defects jointly supported by the evidence:

Activity hubs are not retired when the activity completes. The compile finished in about a second, but keep-alive traffic was still routable to its hub long after. Whatever activates hubs for _Activity nodes does not release them at terminal status, so finished activities keep a live, routable hub.
Hub churn from this morning's compile storm exhausted the pod. Crm/Stage alone saw more than a hundred compile-* activities created in roughly a 15-minute window this morning (the node listing truncates at 200 even counting only this space), and compile activity nodes accumulate with no cleanup — the oldest visible go back to at least Aug 28. If each retained activity keeps a live hub with subscriptions and caches, that is a slow leak, and the heartbeat to a stale hub is simply where the next allocation failed. Corroborating hub-lifecycle stress in the same window: hubs stuck in ShutDown being force-torn-down "so children/subscriptions cannot leak" (#1140) and per-node hub initialization timeouts (#1174).

Not supported by the evidence: a defect in heartbeat handling itself, or an oversized message. Also rule out with container metrics: the pod already being at its memory limit for unrelated reasons — an OOM burst in the memex-cloud namespace earlier today (#3044) shows memory pressure is not confined to this component.

Impact

A single dropped keep-alive that nobody was waiting on — routing deliberately answers heartbeats with nothing (routing answer-once policy) — so no functional loss and not an outage. It matters as a stability signal: every compile storm multiplies the set of stale, still-routable activity hubs, each of which is a periodic heartbeat destination, and the same allocation failure will recur on any pod whose heap is already stressed.

Where to look

src/MeshWeaver.Messaging.Hub/MessageHub.cs — HandleMessageAsync (line 750 in the recorded stack) is where the OOM surfaced; the allocation is in the rule chain it walks.
Hub lifecycle for Activity nodes — who activates a hub for _Activity/compile-*, and why it is not disposed/unsubscribed when the activity reaches a terminal status.
Retention for _Activity children — compile activity nodes accumulate without cleanup; a retention sweep plus deactivation-on-terminal-status would bound the hub population.
Container/heap metrics for the affected pod around the failure window, to separate compile-storm hub churn from a pre-existing memory limit.
Related but not duplicates (different log sites): #1140 router traffic + hub disposal deadlocks, #1174 activity-hub init timeouts, #3044 earlier OOM burst in ReportFailure/Post.

Evidence


Fingerprint	`92c0e9442e275f65`
Category	`MeshWeaver.Messaging.MessageService`
Severity	Error
Top frame	`MeshWeaver.Messaging.MessageHub.HandleMessageAsync(IMessageDelivery delivery, AsyncDelivery[] ruleChain, CancellationToken cancellationToken)`
Namespace	`memex`
Pods	`memex-portal-deployment-c7f5ff965-vk9zq`
Occurrences	1
First seen	2026-09-02 11:13:52Z
Last seen	2026-09-02 11:13:52Z

Recent log lines

2026-09-02 11:13:52Z memex-portal-deployment-c7f5ff965-vk9zq fail: MeshWeaver.Messaging.MessageService[0]
      An exception occurred during the processing of {"$type":"MessageDelivery\u00601[HeartBeatEvent]","message":{"$type":"HeartBeatEvent"},"id":"6CZNlJmWikO3TP5DhjncGg","accessContext":{"$type":"AccessContext","objectId":"system-security","name":"system-security","email":"","roles":[]},"routingPath":["cache/8N2zsv00RkqXfR_Zqr7GpQ","mesh/8N2zsv00RkqXfR_Zqr7GpQ","Crm/Stage/_Activity/compile-202609021024566204c48055472004402b49f4d77fe64bb89"],"sender":"cache/8N2zsv00RkqXfR_Zqr7GpQ","target":"Crm/Stage/_Activity/compile-202609021024566204c48055472004402b49f4d77fe64bb89","properties":{"AnswerSuppressed":true}} after 1849ms. Exception: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at MeshWeaver.Messaging.MessageHub.HandleMessageAsync(IMessageDelivery delivery, AsyncDelivery[] ruleChain, CancellationToken cancellationToken) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/MessageHub.cs:line 750
         at MeshWeaver.Messaging.MessageHub.<>c__DisplayClass72_0.<MeshWeaver.Messaging.IMessageHub.HandleMessageAsync>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/MessageHub.cs:line 827
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run(). Address: Crm/Stage/_Activity/compile-202609021024566204c48055472004402b49f4d77fe64bb89.

_{Opened automatically from Admin/_LogIncident/92c0e9442e275f65. Recurrences are folded into this issue rather than opening new ones.}

Root-caused, and the hypothesis in the body needs one correction. Draft PR: #3116.

Activity hubs ARE retired at terminal status. That release exists and works (#1435/#1324):
ActivityLogAppender.Append fires IMeshNodeStreamCache.ReleaseIfUnwatched on the terminal
write, and NodeTypeCompilationActivity — which owns every compile-<ts> node — is on that
path. CompileActivityHubRetentionTest measures the bound. So no retention sweep and no
deactivation-on-terminal-status is needed; both already exist.

What actually happened is one layer down, and it is why the release found nothing to release.
A warm mirror has exactly two reclaimers and both are keyed on the cache's _streams entry: the
10-minute idle sweep iterates it, and ReleaseIfUnwatched looks the path up in it. Drop the
entry while its upstream sync stream is still attached and neither can see the stream again — and
the stream is what carries the 45 s HeartBeatEvent.

MeshNodeStreamCache.ResetFailureState — the change-feed invalidation, which runs on every
mesh change event for a path, including the activity's own terminal write — did exactly that. It
is the fourth site that evicts a faulted entry; the other three go through EvictFaultedEntry,
which detaches and disposes the upstream (the rule #1202 established, and whose remarks call it
"the single teardown behind all three re-probe triggers … so they cannot drift"). This one kept a
hand-rolled copy that disposed only the Rx hydration and reported UpstreamReleased: false
hard-coded.

Its comment justified that with "a terminally errored upstream has already torn down its own
keep-alive" — true of a NotFound, the one fault class that disposes the stream's keepAlive.
The class that matters here is the transient one: an Orleans-idle-collected grain or a 60 s
request timeout, where the node exists and is momentarily unreachable, so the stream stays live
and keeps heart-beating. Under compile-storm load that fault is common, the change events are
continuous, and the orphan is only ever cleaned up if that exact path is read again — which a
finished compile activity never is. Hence 49 minutes, and hence "for the process lifetime".

Fix: route that eviction through EvictFaultedEntry like the other three. Pinned by
ChangeFeedResetReleasesUpstreamTest, which reproduces the real transient fault and carries a
control arm (the same state released through the already-correct ReleaseIfUnwatched must report
an upstream) so it cannot pass vacuously. Fails before, passes after. Written up in
Doc/Architecture/MeshNodeStreamCache → Dropping the entry without the upstream orphans a
heartbeat nothing can reach.

Two things from the body that are confirmed not the cause: heartbeat handling itself, and an
oversized message. One thing worth a separate issue: several terminal _Activity writers bypass
ActivityLogAppender.Append entirely and so never fire ReleaseIfUnwatched —
ActivityLogLogger.PublishSnapshotLocked (every kernel/script/markdown/test run),
CodeNodeType.FailActivity, BuildProtocolDriver.FinishActivity. Those fall back to the
10-minute idle sweep, so they are a bounded retention rather than the unbounded leak fixed here.

Fixed by #3116.

🚩 This issue's stated cause was wrong, and the remedies it proposed already exist

Activity hubs are retired at terminal status. ActivityLogAppender.Append fires IMeshNodeStreamCache.ReleaseIfUnwatched on the terminal write (#1435/#1324), NodeTypeCompilationActivity — owner of every compile-<ts> node — is on that path, and CompileActivityHubRetentionTest measures the bound. So both remedies suggested here (a retention sweep, deactivation-on-terminal-status) were already in place. The Crm/*/_Activity compile nodes named in the body are also gone from the live mesh, so the recorded numbers had drifted.

The actual cause, one layer down

A warm mirror has exactly two reclaimers, both keyed on the cache's _streams entry: the idle sweep iterates it, ReleaseIfUnwatched looks the path up in it.

MeshNodeStreamCache.ResetFailureState — the change-feed invalidation, which runs on every mesh change event for a path, including the activity's own terminal write — is the fourth site that evicts a faulted entry. The other three go through EvictFaultedEntry, which detaches and disposes the upstream sync stream. This one kept a hand-rolled copy that disposed only the Rx hydration and hard-coded UpstreamReleased: false.

Its comment justified that with "a terminally errored upstream has already torn down its own keep-alive" — true only of a NotFound. For a transient fault (idle-collected grain, 60 s timeout: the node exists, it is momentarily unreachable) the stream stays live with its 45 s HeartBeatEvent running, gets orphaned, and is invisible to both reclaimers forever — unless that exact path is read again, which a finished compile activity never is.

Hence 49 minutes. Hence one leak per compile.

Fix: call EvictFaultedEntry(path, "invalidate"). No timer, no watchdog, no widened bound.

🚨 The pattern, because it is the third instance today

EvictFaultedEntry's own remarks claim it is "the single teardown … so they cannot drift". It had drifted — into a fourth copy that did half the work. The same shape produced #3021 (the audit stamp inlined twice, silently losing data) and #3061 (AccessControl.md documenting a gap as a deliberate non-decision while the pre-flight and the delivery gate had each written their resolution inline). A comment asserting "the single X" is a claim about the past, not an invariant — and it becomes the reason nobody re-checks.

Not the same as #3109

#3109 is about probe hubs refusing hosted-child creation during disposal (ContentTypeRegistration.ProbeRegister + IsShuttingDown at BuildupAction boundaries). Different subsystem, different symptom, no overlap with the heartbeat path. Two remedies for one defect was a mistake made once today; this is not that.

Verification

ChangeFeedResetReleasesUpstreamTest reproduces the real transient fault via a hijacked owner and asserts the upstream is released, with a control arm that releases the same state through the already-correct ReleaseIfUnwatched so the subject cannot pass vacuously. Without the fix the subject fails on exactly that assertion while the control passes. MeshWeaver.Hosting.Test 302/302, MeshWeaver.Documentation.Test 202/202, both full and unfiltered.

Recorded in Doc/Architecture/MeshNodeStreamCache.md. Follow-up filed as #3117 — three terminal _Activity writers bypass ActivityLogAppender.Append and fall back to the 10-minute idle sweep; bounded retention rather than this unbounded leak, which is why it is separate.

Reopened — this fault is happening again.

Still occurring: 1 more since the last update (2 total).

Last seen: 2026-09-19 06:28:52Z
Pods: memex-portal-deployment-58dbb6d98d-8msww, memex-portal-deployment-c7f5ff965-vk9zq

Recent log lines

2026-09-02 11:13:52Z memex-portal-deployment-c7f5ff965-vk9zq fail: MeshWeaver.Messaging.MessageService[0]
      An exception occurred during the processing of {"$type":"MessageDelivery\u00601[HeartBeatEvent]","message":{"$type":"HeartBeatEvent"},"id":"6CZNlJmWikO3TP5DhjncGg","accessContext":{"$type":"AccessContext","objectId":"system-security","name":"system-security","email":"","roles":[]},"routingPath":["cache/8N2zsv00RkqXfR_Zqr7GpQ","mesh/8N2zsv00RkqXfR_Zqr7GpQ","Crm/Stage/_Activity/compile-202609021024566204c48055472004402b49f4d77fe64bb89"],"sender":"cache/8N2zsv00RkqXfR_Zqr7GpQ","target":"Crm/Stage/_Activity/compile-202609021024566204c48055472004402b49f4d77fe64bb89","properties":{"AnswerSuppressed":true}} after 1849ms. Exception: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at MeshWeaver.Messaging.MessageHub.HandleMessageAsync(IMessageDelivery delivery, AsyncDelivery[] ruleChain, CancellationToken cancellationToken) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/MessageHub.cs:line 750
         at MeshWeaver.Messaging.MessageHub.<>c__DisplayClass72_0.<MeshWeaver.Messaging.IMessageHub.HandleMessageAsync>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/MessageHub.cs:line 827
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run(). Address: Crm/Stage/_Activity/compile-202609021024566204c48055472004402b49f4d77fe64bb89.
2026-09-19 06:28:52Z memex-portal-deployment-58dbb6d98d-8msww fail: MeshWeaver.Messaging.MessageService[0]
      An exception occurred during the processing of {"$type":"MessageDelivery\u00601[HeartBeatEvent]","message":{"$type":"HeartBeatEvent"},"id":"jcYtIPuBrkeRI4RUs1mzMQ","accessContext":{"$type":"AccessContext","objectId":"system-security","name":"system-security","email":"","roles":[]},"routingPath":["cache/TJ6z4gfxn0upvFcOmR78QQ","mesh/TJ6z4gfxn0upvFcOmR78QQ","ReinsuranceDemo/_Policy"],"sender":"cache/TJ6z4gfxn0upvFcOmR78QQ","target":"ReinsuranceDemo/_Policy","properties":{"AnswerSuppressed":true}} after 1942ms. Exception: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at MeshWeaver.Messaging.MessageHub.HandleMessageAsync(IMessageDelivery delivery, AsyncDelivery[] ruleChain, CancellationToken cancellationToken) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/MessageHub.cs:line 1041
         at MeshWeaver.Messaging.MessageHub.<>c__DisplayClass90_0.<MeshWeaver.Messaging.IMessageHub.HandleMessageAsync>b__0() in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/MessageHub.cs:line 1141
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run(). Address: ReinsuranceDemo/_Policy.

_{Re-addressed by the current identity function: this incident inherited 92c0e9442e275f65 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

docs(logwatch): one log site holds TWO buckets, and a re-addressed in…

Closing — the titled defect is fixed and still fixed; the reopen is a different fault at the same log site

1. The fix, as the code stands today (not as the PR described it)

src/MeshWeaver.Hosting/MeshNodeStreamCache.cs → ResetFailureState ends in the shared teardown:

// The SHARED teardown, never a copy of it — see the remarks. EvictFaultedEntry
// applies the same IsFaulted gate this site used to apply inline, and adds the two
// things the copy was missing: the upstream detach + disposal (#3110) and the
// pair-exact claim that re-parks on a lost race.
EvictFaultedEntry(path, "invalidate");

That is #3116 (merge bcdc8086b744b0110f8a35a294f3d16e1b8fc431, 2026-09-02 14:27Z), present on
origin/main at MeshNodeStreamCache.cs:877, with the control test still in the tree at
test/MeshWeaver.Hosting.Test/ChangeFeedResetReleasesUpstreamTest.cs (real transient fault, plus the
control arm through the already-correct ReleaseIfUnwatched so the subject cannot pass vacuously).
Nothing has reverted it, and ActivityLogAppender.cs:170 still carries #3110 by name in its remarks.
The release on terminal status that the body said was missing is also still there —
ActivityLogAppender.cs:217, if (status.IsTerminal() && cache?.ReleaseIfUnwatched(activityPath) == true).
The bounded-retention follow-up #3117 is closed as completed (2026-09-02 19:51Z).

2. The reopen cannot be about activity hubs — the address is a METHOD, not a defect

The reopening occurrence is

2026-09-19 06:28:52Z  target "ReinsuranceDemo/_Policy"  after 1942ms  OutOfMemoryException
   at MeshWeaver.Messaging.MessageHub.HandleMessageAsync(...) MessageHub.cs:line 1041

ReinsuranceDemo/_Policy is a live partition satellite hub, not an _Activity/compile-* node. There
is no finished activity in it, so it is not evidence about retiring one.

Why it landed here anyway — reproduced locally, no cluster read, per
Doc/Architecture/LogWatchTriage → "The exact instrument: RECOMPUTE the fingerprint":

SHA256("MeshWeaver.Messaging.MessageService\n0\n"
     + "MeshWeaver.Messaging.MessageHub.HandleMessageAsync(IMessageDelivery delivery, "
     + "AsyncDelivery[] ruleChain, CancellationToken cancellationToken)")[..8]
  = 92c0e9442e275f65        ← this incident, exactly

Note what is not in that payload: the exception type. The discriminator for this burst was the
top frame alone, so 92c0e9442e275f65 addresses every fault of every type that surfaces inside
MessageHub.HandleMessageAsync — the single most-travelled method in the mesh. Both occurrences it
holds are heartbeats to different hubs that OOM'd while the rule chain was walked; the 2026-09-02
one happened to be a compile activity, which is how the issue got its title.

3. What the 2026-09-19 occurrence actually is — and it is shared with #3045

Same pod, two minutes apart:

time	incident	target	allocation that failed
06:28:52Z	`92c0e9442e275f65` (this issue)	`ReinsuranceDemo/_Policy`	the rule-chain walk
06:30:42Z	`1b405a782123de6b` (#3045)	`Google/_Access/anuj_Access`	`Task.AssignCancellationToken` — a continuation object, tens of bytes

Pod memex-portal-deployment-58dbb6d98d-8msww, one exhausted managed heap, surfacing at whatever
allocated next, filed as two incidents because the two log sites have different categories. A
~40-byte allocation failing is not a payload problem and not a hub-lifecycle problem; it is the heap
being gone.

That condition has its own live evidence on the same namespace and its own mechanism in this repo:
Admin/_LogIncident/8aaed8b9442edb3a — memex-cloud's process_progress health check reporting
Unhealthy / GC-bound: …, still updating 2026-09-19T14:26Z — and
src/MeshWeaver.Deployment.Contract/DeploymentContent.cs on AutoRecycleOnStaleBuild:

Off, every publication wave leaves serving pods on their old assemblies behind the Recycle banner
until the type-hosting silos go GC-bound at 17–20 GB while still Ready (MeshWeaver#2192).

So the remedy for the residue is a recycle/roll policy on the instance record, not a change to hub
retirement — which already works, per the two call sites quoted above. (Note for whoever reads the
prose in Doc/Architecture/MeshNodeStreamCache and KernelHubOptions: the
CompileActivityHubRetentionTest they name is not in core's test/ tree — it travelled with the
kernel/AI work — so nothing here rests on it.)

4. If this reopens again, one read settles it

Read the incident's newest samples[] entry and look at the target. …/_Activity/compile-*
⇒ genuine regression of this issue, reopen the investigation. Anything else ⇒ the fold, and the fault
is whatever exhausted that pod's heap. Since 2026-09-19 ~11:41Z the portal also re-addresses reports,
so the successor node (reporterFingerprint/foldedFrom naming 92c0e9442e275f65) is where the
traffic now lands — sweep content.category:*MessageService*, because content.foldedFrom:<id> does
not match an array member and answers 0 either way.

The mechanics above are now committed in Doc/Architecture/LogWatchTriage (PR #4924) rather than
left in this thread.

_{Measured 2026-09-19 by Claude Opus 5 — control instance reads (read-only), GitHub REST, and the

identity function run locally against origin/main.}

review(#4924): the 2026-08-09 payload table was missing its two frame…

Navigation Menu

Repository navigation

Compile activity hubs are not retired when the compile finishes — keep-alive heartbeats still route to them and OOM under compile-storm hub churn #3110

What is failing

Probable cause — moderate confidence

Impact

Where to look

Evidence

🚩 This issue's stated cause was wrong, and the remedies it proposed already exist

The actual cause, one layer down

🚨 The pattern, because it is the third instance today

Not the same as #3109

Verification

Closing — the titled defect is fixed and still fixed; the reopen is a different fault at the same log site

1. The fix, as the code stands today (not as the PR described it)

2. The reopen cannot be about activity hubs — the address is a METHOD, not a defect

3. What the 2026-09-19 occurrence actually is — and it is shared with #3045

4. If this reopens again, one read settles it

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Compile activity hubs are not retired when the compile finishes — keep-alive heartbeats still route to them and OOM under compile-storm hub churn #3110

Description

What is failing

Probable cause — moderate confidence

Impact

Where to look

Evidence

Activity

rbuergi commented on Sep 2, 2026

rbuergi commented on Sep 2, 2026

🚩 This issue's stated cause was wrong, and the remedies it proposed already exist

The actual cause, one layer down

🚨 The pattern, because it is the third instance today

Not the same as #3109

Verification

systemorph-com commented on Sep 19, 2026

rbuergi commented on Sep 19, 2026

Closing — the titled defect is fixed and still fixed; the reopen is a different fault at the same log site

1. The fix, as the code stands today (not as the PR described it)

2. The reopen cannot be about activity hubs — the address is a METHOD, not a defect

3. What the 2026-09-19 occurrence actually is — and it is shared with #3045

4. If this reopens again, one read settles it

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions