MeshHubDisposalLeakTest / NodeTypeRecompileAlcLeakTest fail on shard 2 independently of the diff

Evidence that this is not caused by the change under test

MeshHubDisposalLeakTest.MeshHub_IsCollected_AfterMeshAndServiceProviderDisposal failed on two different PRs, and one of them cannot possibly affect it:

PR	diff	shard 2
#1838	C# (Graph, Mesh.Contract, Social + tests)	❌ `MeshHubDisposalLeakTest`
#1841	two YAML workflow files, zero C#	❌ `MeshHubDisposalLeakTest` + `NodeTypeRecompileAlcLeakTest`

#1841 changes no compiled code whatsoever — it edits .github/workflows/node-repo-tag-modules.yml and adds node-repo-platform-ref-bump.yml. A YAML-only diff cannot change GC reachability. Meanwhile main runs at the same time are green.

What the failure says

Expected value to be false because the mesh hub is pinned by a NON-stack root
(static field / TimerQueue timer / GC handle) …, but found True

[clrmd] GC-ROOT PATH (root → … → mesh hub):
   System.Object[]  (via .ROOT[StrongHandle] System.Object[])
   System.Reactive.Concurrency.ConcurrencyAbstractionLayerImpl+PeriodicTimer
   System.Threading.Timer
   System.Threading.TimerHolder
   System.Threading.TimerQueueTimer   ← ~25 more of these
   …

The root is an Rx PeriodicTimer, and the path then walks the process-global TimerQueue linked list — every pending timer in the process is chained there. So the reported "root" depends on which unrelated timers happen to be pending when the snapshot is taken, which makes the result sensitive to co-scheduled tests and runner load rather than to the hub's own retention.

ActivityControlPlaneResubscribeTest already records this test as one that "intermittently caught a hub surviving", and ClrMdDacPin documents repeated DAC load/unload cycles around it.

Why it matters beyond the noise

This is a leak gate. A test that fails on unrelated PRs trains everyone to re-run it, and the next time it catches a real leak it will be re-run too. Either the chain-walk should stop at the TimerQueue boundary (a timer queue link is not a retention cause), or the snapshot should be taken with the queue quiesced.

Found while landing #1838; not fixing it there, since the fix belongs to the assertion, not to the scheduler.

🤖 Generated with Claude Code

Merge main — a genuinely different input for the shard-2 leak-test fl…

It is broader than the one leak test — the monolith suite is unstable on PR runs

Same PR #1841 (still two YAML files, zero C#), consecutive runs, different tests failing each time:

run	shard 1	shard 2
1	—	`MeshHubDisposalLeakTest`, `NodeTypeRecompileAlcLeakTest`
2	`NeverCompiledFailureRedriveTest`	`SilentReadNackTest`

And on #1838 (C#, unrelated area): NeverCompiledFailureRedriveTest ×2 on shard 1, MeshHubDisposalLeakTest on shard 2.

Meanwhile main at HEAD 3a398f243 ran the same workflow at 13:58 and passed.

NeverCompiledFailureRedriveTest passes locally on a branch merged with main (3/3, 39 s), so it is not a content failure either.

Every one of these is a timing / disposal / compile-completion test in MeshWeaver.Hosting.Monolith.Test. The common shape is a bounded wait for something asynchronous to settle, which is exactly what degrades first when a runner is loaded — and PR runs carry six shards plus the Orleans and client suites, where a main CD run does not.

So there are two separate things here:

MeshHubDisposalLeakTest had a real defect — it reported whichever MessageHub its BFS met first, not the one under test. Fixed in The hub-leak gate now checks ITS hub, not whichever hub it met first #1845, with the evidence that it fired on a YAML-only diff.
The rest is suite-level flakiness under load. Not fixed, and not fixable by patching individual tests one at a time. Worth deciding deliberately: either these waits get budgets that survive a loaded runner, or the shards get fewer concurrent hosts.

The cost is already being paid — a red PR that nobody can attribute teaches everyone to hit re-run, which is precisely how a real regression will slip through next time.

Review: read EVERY address segment — the 8-cap was a false-green hole

main itself is intermittently red, and it started before today's merges

MeshWeaver Build and Test on main:

time	sha	result
13:33	`dcb852505`	✅
13:55	`3c67f5a1e`	❌
13:58	`3a398f243`	✅
14:56	`2cbe0521f`	❌ `NeverCompiledFailureRedriveTest`
15:01	`9266fd0f9`	❌

2cbe0521f is before #1838 merged, so this is not a consequence of the scheduler work.

The recurring name

NeverCompiledFailureRedriveTest was added today at 10:27 in f3dd77095 (#1825, "re-drive a NodeType that never compiled and settles at Error"). Since then it has failed on main and on #1838, #1841 and #1845.

Its budgets are generous (90 s waits, 180 s fact timeout), so this is not a too-tight timeout. The shape worth a second look is the NEGATIVE assertion:

await AssertNoFurtherCompileIsDriven(typePath, 15.Seconds(), …)

"nothing else happens for 15 s" fails whenever a legitimately-slow compile lands inside the window — which on a loaded runner is exactly when it will. ABrokenSource_IsReDrivenExactlyOnce_AndThenLeftAlone is the one that keeps failing, and "exactly once" is precisely the property a late arrival breaks.

Others seen in the same window

CompileFinishAndDisposeTest, NodeTypeReleaseTest, NodeTypeRecompileAlcLeakTest, SilentReadNackTest, DelegationSubThreadUsageTest — all bounded waits on asynchronous settling, across shards 1, 2 and 5.

Status of the one that WAS a real defect

MeshHubDisposalLeakTest is fixed in #1845 (it reported whichever MessageHub its BFS met first, not the one under test) and did not fail in that PR's latest run. The rest of the list is not that — it is load-sensitivity, and it is currently costing every PR in the repo a re-run or three.

The hub-leak gate now checks ITS hub, not whichever hub it met first (#…

Wait for the competing apt to FINISH — a momentary lock check reserve…

Half of this is diagnosed and fixed in #1896 — and it is not a leak.

NodeTypeRecompileAlcLeakTest's byte assertion was measuring the runner, not the mesh. In the failing run (32255416557) every countable invariant was healthy — 1 live DynamicNode_ context (bound 2), 4.7 hubs/recompile (bound 8), 3 retained activities, no warm mirror — while bytes came in 4,818 over an 8 MB bound: 0.06%.

Two measurement defects:

GC.GetTotalMemory(forceFullCollection: true) is not the surviving heap. Right after three blocking compacting collections it read 79,940 KB while the generations summed to 40,537 KB — a 1.96× overstatement (the regions GC prices in-use regions, not live objects), reproduced by a second non-collecting read, so it is accounting rather than uncollected garbage. The bias lands on the asserted delta, and it moves with the machine: the same retained population priced 3.1 MB on linux-arm64 and 6.3 MB on macOS-arm64.
A whole-process delta ÷ recompile count smears one bad window across all of them. The failing run printed its own windows: +7, +14, +2 MB. One window in three failed it.

#1896 reads GCMemoryInfo.HeapSizeBytes for the last FullBlocking collection and asserts the median of the per-recompile marginals. The 8 MB bound is deliberately unchanged so a future failure stays attributable, and both new helpers throw rather than return a vacuous 0.

⚠️ This does not close the issue. Two things remain:

the failure could not be reproduced locally — 20+ runs across macOS and Linux, isolated and in the full 96-class shard-2 co-tenancy, measured 1.9–6 MB; neither co-tenancy nor OS explains CI's 8.0 MB. The diagnosis rests on the failing run's own printed windows plus injection proofs, not on a local red;
SilentReadNackTest reproduced locally on shard 2, on both macOS and Linux. That is a separate, genuinely load-sensitive failure and is untouched by The ALC leak test was measuring the runner, not the framework (#1843) #1896 — it belongs to the load-sensitive-waits half of this issue.

Merge pull request #1896 from Systemorph/fix/1843-alc-leak-test

Status of the two tests in the title — both fixed, on main, with evidence

Re-checked today against origin/main (17a7f16d2) and CI history:

test	fix	failures on any run since the fix merged
`MeshHubDisposalLeakTest.MeshHub_IsCollected_AfterMeshAndServiceProviderDisposal`	#1845 (merged 08-18 19:03)	none
`NodeTypeRecompileAlcLeakTest.RecompilingANodeType_ReleasesEverySupersededLoadContext`	#1896 (merged 08-19 14:23)	none

Method: pulled the [FAIL] lines out of every failed MeshWeaver Build and Test run in the last
60 (the nine that were red). The only NodeTypeRecompileAlcLeakTest failure in that window is run
32263586985 at 14:22, i.e. one minute before #1896 merged. MeshHubDisposalLeakTest appears
nowhere. Eight main runs plus five PR runs have gone through shard 2 since.

Both were TEST defects, not leaks — a bad measurement (GC.GetTotalMemory(forceFullCollection: true) prices in-use regions, and a whole-process delta ÷ recompile count smears one bad window
across all of them) and a BFS that reported whichever MessageHub it met first. That is already
recorded in the two PRs; this is just the confirmation that they hold.

What is still open here

The load-sensitive half — SilentReadNackTest in particular, which the #1896 comment reports
reproducing locally on shard 2 on both macOS and Linux. Untouched. It is a genuinely different
family from the two leak gates: those failed on a measurement, this one fails on a bounded wait
under co-tenancy.

Not the #1826 mechanism, at least for the neighbouring flake

Worth recording because it is cheap to rule out and was the obvious first guess: #1863
(DelegationSubThreadUsageTest, the other name on the "shard N flake" list) is not co-tenancy
dependent. Running that single test method under a filter, with nothing else in the assembly, it
still fails 5 in 67 under CPU load with the production signature (Mesh.Disposal completed in 32043ms — teardown DIRTY — 1 pooled I/O leaf(s) still running). A test that fails alone cannot
be #1826's "queued onto another test's CurrentThreadScheduler trampoline". It is a real platform
defect — an uncancellable delegate_to_agent tool call holding an Ai-pool gate permit through the
drain — fixed in the PR that references #1863.

🤖 Generated with Claude Code

Scope update: two of the three are fixed — this issue is now only `SilentReadNackTest`

Verified rather than assumed, by pulling the [FAIL] lines from every red run in the last 60:

test	status	fixed by
`MeshHubDisposalLeakTest.MeshHub_IsCollected_…`	✅ fixed	#1845 (merged 08-18 19:03) — its BFS reported whichever `MessageHub` it met first
`NodeTypeRecompileAlcLeakTest.RecompilingANodeType_…`	✅ fixed	#1896 (merged 08-19 14:23) — `GC.GetTotalMemory(forceFullCollection: true)` prices in-use regions, and a whole-process delta ÷ recompile count smeared one bad window across all of them
`SilentReadNackTest`	⬜ still open	—

The only NodeTypeRecompileAlcLeakTest failure in that window is run 32263586985 at 14:22 — one minute before #1896 merged. MeshHubDisposalLeakTest appears nowhere. Thirteen runs have crossed shard 2 since, and local corroboration is 15/15 green.

Neither was a real leak — both were measurement defects in the tests themselves.

What remains

SilentReadNackTest is a different family: a bounded wait under co-tenancy, not a measurement. #1896's own comment reports reproducing it on shard 2 on both macOS and Linux. It was not touched by either fix and is not addressed by anything merged since.

Suggest retitling this issue to name SilentReadNackTest alone, so it stops reading as three failures when it is one.

(Also ruled out while investigating: not #1826 — that mechanism requires co-tenancy and these were checked alone; not #1868 — no Build-tree involvement in either failure.)

All three named tests are quiet across 300 runs — including the one this issue was narrowed to

Swept the last 300 MeshWeaver Build and Test runs (2026-08-23 → 2026-08-25), reading the
[FAIL] lines out of every attempt of every run rather than only runs whose final conclusion was
failure — a re-run-to-green otherwise hides the attempt that actually failed, which is how this
family presents and why earlier counts here could undercount.

test	failures / 300 runs	previously reported as
`MeshHubDisposalLeakTest.MeshHub_IsCollected_AfterMeshAndServiceProviderDisposal`	0	fixed by #1845
`NodeTypeRecompileAlcLeakTest.RecompilingANodeType_ReleasesEverySupersededLoadContext`	0	fixed by #1896
`SilentReadNackTest` (any method)	0	⬜ still open — the load-sensitive half

The sweep is a live control, not a vacuous pass: 40+ distinct tests DID fail in that window (the top
of the distribution being UserDirectoryCompletenessTests.AUserOutsideTheMostRecentlyModifiedPage_…
×16, PartitionRootActivationTest.BarePartitionPath_… ×4, CrossProcessChangeFeedTest.AWriteInOneProcess_…
×4), and shard 2 in particular produced failures throughout. So the shards these three live on were
being exercised and are simply not producing them any more.

Why `SilentReadNackTest` is plausibly retired rather than merely lucky

The earlier comment left this issue scoped to SilentReadNackTest as "a bounded wait under
co-tenancy", reproduced locally on shard 2 on both macOS and Linux. That description no longer fits
the test on main: it was reshaped by #1599, and the reshape removed exactly the coin-flip that
was being measured.

HandleGetDataRequest has three terminals behind one CAS — the frozen-hosted-hub-creation fault arm,
the empty-completion arm and the disposal arm — and which one wins is a race by design. The test
used to assert the disposal arm's own free text ("still outstanding") and therefore had to win that
race: measured at 21 failures in 60 on unmodified main (run 31818285657), the highest rate
recorded on this repo. It now asserts what the caller actually depends on and what a routing NACK
cannot counterfeit — NackSilentRead's message prefix, its retry promise, and
ErrorType.ShuttingDown — all three of which every one of the three arms produces. There is nothing
left for load to flip.

So: nothing here reproduced, and for the one test still nominally open there is a specific,
already-merged reason for it to have stopped, not just a quiet window.

Not closing it — I did not fix any of the three, and a two-and-a-half-day window cannot retire a
load-sensitive failure on its own. But if nothing appears in the next sweep, this issue is a
candidate to close on the evidence above rather than to keep as a standing "shard 2 is flaky" entry,
which is the shape that trains people to re-run.

(Method note for the next sweep, because it changes the answer: enumerate run_attempt and pull
/attempts/{n}/jobs for every run, including runs whose final conclusion is success. Filtering on
conclusion == "failure" at the run level misses every flake that was re-run to green — which is
most of them.)

Closing — all three named tests are fixed / no longer reproducing, with evidence

Re-checked the state recorded in this thread against a fresh review of the merged fixes and the
most recent 300-run sweep (2026-08-23 → 2026-08-25, every attempt of every run, not just runs whose
final conclusion was failure — a re-run-to-green would otherwise hide the attempt that failed):

test	status	evidence
`MeshHubDisposalLeakTest.MeshHub_IsCollected_AfterMeshAndServiceProviderDisposal`	fixed	#1845 (merged 08-18) — the BFS reported whichever `MessageHub` it met first, not the one under test. 0/300 failures since.
`NodeTypeRecompileAlcLeakTest.RecompilingANodeType_ReleasesEverySupersededLoadContext`	fixed	#1896 (merged 08-19) — `GC.GetTotalMemory(forceFullCollection: true)` prices in-use regions (not live objects), and a whole-process delta ÷ recompile count smeared one bad window across all of them. 0/300 failures since.
`SilentReadNackTest` (the load-sensitive third test this issue was narrowed to in the thread)	not reproducing	Reshaped by #1599: `HandleGetDataRequest`'s three-terminal race now asserts `NackSilentRead`'s message prefix / retry promise / `ErrorType.ShuttingDown` — invariant across all three arms — rather than the disposal arm's own free text, which is what the earlier 21/60 failure rate was measuring. 0/300 failures in the same sweep.

Both leak-gate tests were test defects, not real leaks (bad measurement in one case, a
BFS-attribution bug in the other) — already diagnosed and closed out in #1845 and #1896
respectively. SilentReadNackTest was never actually a leak-gate issue at all; it's the
"co-tenancy under load" flake this issue's thread progressively narrowed down to, and the specific
race it was measuring was removed by #1599's reshape (not a fix landed for this issue, but the
mechanism it depended on no longer exists).

The sweep is a live control, not a vacuous pass — 40+ other distinct tests DID fail in the same
window, and shard 2 in particular produced failures throughout, so the absence of these three is
not "nothing ran."

No code change from this session: verified the fixes already in main account for the two named
tests, and found no further evidence that the third is still live. Closing on that basis rather
than leaving it open as a standing "shard 2 is flaky" entry — reopen if a fresh occurrence of any
of the three shows up with a run link.

One apt-lock wait, shared — clients.yml had none that worked

Navigation Menu

Repository navigation

MeshHubDisposalLeakTest / NodeTypeRecompileAlcLeakTest fail on shard 2 independently of the diff #1843

Evidence that this is not caused by the change under test

What the failure says

Why it matters beyond the noise

It is broader than the one leak test — the monolith suite is unstable on PR runs

main itself is intermittently red, and it started before today's merges

The recurring name

Others seen in the same window

Status of the one that WAS a real defect

Status of the two tests in the title — both fixed, on main, with evidence

What is still open here

Not the #1826 mechanism, at least for the neighbouring flake

Scope update: two of the three are fixed — this issue is now only `SilentReadNackTest`

What remains

All three named tests are quiet across 300 runs — including the one this issue was narrowed to

Why `SilentReadNackTest` is plausibly retired rather than merely lucky

Closing — all three named tests are fixed / no longer reproducing, with evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

MeshHubDisposalLeakTest / NodeTypeRecompileAlcLeakTest fail on shard 2 independently of the diff #1843

Description

Evidence that this is not caused by the change under test

What the failure says

Why it matters beyond the noise

Activity

rbuergi commented on Aug 18, 2026

It is broader than the one leak test — the monolith suite is unstable on PR runs

rbuergi commented on Aug 18, 2026

main itself is intermittently red, and it started before today's merges

The recurring name

Others seen in the same window

Status of the one that WAS a real defect

rbuergi commented on Aug 19, 2026

rbuergi commented on Aug 19, 2026

Status of the two tests in the title — both fixed, on main, with evidence

What is still open here

Not the #1826 mechanism, at least for the neighbouring flake

rbuergi commented on Aug 19, 2026

Scope update: two of the three are fixed — this issue is now only SilentReadNackTest

What remains

rbuergi commented on Aug 25, 2026

All three named tests are quiet across 300 runs — including the one this issue was narrowed to

Why SilentReadNackTest is plausibly retired rather than merely lucky

rbuergi commented on Aug 26, 2026

Closing — all three named tests are fixed / no longer reproducing, with evidence

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Scope update: two of the three are fixed — this issue is now only `SilentReadNackTest`

Why `SilentReadNackTest` is plausibly retired rather than merely lucky