Repository navigation
FrameworkStaleProactiveRebuildTest fails on three unrelated PRs while main stays green — a load-widened race blocking the merge train #1368
Description
MeshWeaver.Hosting.Monolith.Test.FrameworkStaleProactiveRebuildTest.FrameworkStaleNodeType_ProactivelyRebuilds_WithoutInstanceActivation has failed on three unrelated PRs within a few hours, while main is green across every merge in the same window.
| PR | subject | shares code with the test? |
|---|---|---|
| #1359 | batch indexing/import progress writes | no |
| #1364 | silent-read ShuttingDown NACK |
no |
| #1366 | bake lease membership takeover | adjacent (bake), plausible |
Only #1366 is even arguably related. Three independent branches hitting the same assertion is not three coincidences.
Two more single sightings in the same window, listed in case they share a cause rather than as separate reports:
MeshWeaver.Graph.Test.LayoutAreaRetrievalTest.LayoutAreasUnifiedReference_MatchesTheTypedRequest(fix: a GetDataRequest whose stream goes terminal without a value is NACKed, not left hanging (#1362) #1364)MeshWeaver.Hosting.Monolith.Test.StaleActivationDurableFirstSeedTest.ReactivationViaRead_ServesDurableTruth_AndWritesNothingBack(fix: key the synced-query cache on the query set, not the id alone (#1311) #1325)
Neither test is new
FrameworkStaleProactiveRebuildTest came in with #464 (e0e8ab91b); StaleActivationDurableFirstSeedTest with af7836c44. So this is a pre-existing test newly destabilised, not a test shipped broken.
The leading hypothesis, stated so it can be falsified
CI was running ~7 concurrent workflows during this window (an unusually deep merge train). A test whose name is "proactively rebuilds without instance activation" is asserting that something happens on its own within some bound — exactly the shape that fails first when runners are contended. The same load-widened-window pattern showed up twice elsewhere today.
If that is right, the defect is the test's dependence on a bound rather than on a condition, and the cure is to wait on the actual signal (stream.Where(...).FirstAsync().Timeout(...)) rather than on elapsed time.
🚨 What must not happen: do not raise the bound, do not add a retry, and do not re-run to see whether it passes. The repo's standing rule is that flakes are real races; a test that fails under load and passes when idle is reporting something true about the code or about itself. Establish which before changing anything.
Two candidate roots worth checking first
- The test genuinely races — a bounded wait rather than a condition wait. Cheapest to check, and the most likely given the name.
- Today's bake changes moved the ground under it. fix(#1347): retire the pre-run NodeType bake image, and stop demoting source cycles to last #1357 retired the pre-run bake image entirely and fix(#1347): a NodeType bake that could not RUN no longer reports as one that passed #1361 changed the readiness gate to distinguish a faulted enumeration from an empty one.
FrameworkStaleProactiveRebuildis squarely in that area. Main passing does not clear this — main may simply be winning the race on an idle runner.
DOTNET_PROCESSOR_COUNT=4 is known to reproduce CI-shaped races on the maintainer's macOS host and is the obvious first experiment.
Why it matters now
This is currently the binding constraint on the merge train — three otherwise-ready PRs are held by it, and every re-queue costs a full CI cycle and re-samples the same race.
Activity
rbuergi commentedon Aug 13, 2026
Fourth sighting, same shape, on #1377 (run 31660165091, shard 4, commit
abed8d44):MeshWeaver.AI.Test.CodeCellOutputCaptureTest.Progress_Of_Succeeded_Activity_Renders_Output_For_PublicRead_Viewer— 1 fail / 1906 pass, the failing project taking 32s.Recording it because it fits the pattern rather than as a new report:
test/MeshWeaver.Hosting.Monolith.Test(deletesNodeTypeAssemblyLeakTest, adds a hub-count measurement toNodeTypeRecompileAlcLeakTest).MeshWeaver.AI.Testis a separate assembly in a separate process.FrameworkStaleProactiveRebuildTest,LayoutAreaRetrievalTest,StaleActivationDurableFirstSeedTest, and this one) across four unrelated PRs — which argues further against "each of these tests has its own race" and for the shared cause named in the issue.Not re-run, per the standing rule.
🤖 Generated with Claude Code
rbuergi commentedon Aug 13, 2026
At least two of these are #890 victims, not a race in this test
Cross-linking with evidence rather than a hunch, because it changes what to look for here.
A sweep of 1,471 CI runs / 6,814 shard artifacts for #890 (a compile-emit NRE that poisons a test process: once it fires, every later NodeType compile in that process NREs and takes ~25–50 s instead of ~1 s) found 14 occurrences between 2026-08-07 and 08-12. Two of them are runs cited in this issue:
31617758554type/ProactiveStale1cbf60e2f1b44cc4a6c52bba970cefa4—Compile failure for … NullReferenceException31649727371CodeEditRecompile/CompileActivityLogred, shardexit=124In both,
FrameworkStaleProactiveRebuildTestran entirely after the first NRE in that process — all 13 of its trace lines are downstream of the poison. Andtype/ProactiveStale*is precisely the NodeType this test rebuilds, so the rebuild it asserts on did not "race": Roslyn's emit threw and the compile never produced an assembly.That fits this issue's shape exactly — fails on unrelated PRs while
mainstays green. #890 ismain-carried and load-dependent (it hit a push tomaintoo), it does not correlate with any branch's diff, and it strikes whichever compile-dependent test happens to run after the poison.FrameworkStaleProactiveRebuildTestis compile-dependent, so it is a natural victim.What this does and does not mean. It does not prove every failure listed here is #890 — I only checked the runs whose artifacts still exist, and two matched. It does mean that before hunting a load-widened race in this test, it is worth checking the shard's
collected-logs/_meshweaver-test-trace.logforIf that is present before this test starts, the failure belongs to #890 and no change here will fix it.
Also relevant to the
exit=124sightings: of everyexit=124shard in that sweep, 10 of 14 are #890, 3 are pre-instrumentationHosting.Monolith.Testtimeouts (one confirmed to be #890), and 1 is an unrelated Cosmos.Test timeout. The timeout is a consequence of the poison — six failed compiles at ~50 s each blow the 8-minute cap.Full sweep, the falsified premises, and what has been ruled out by experiment: #890. Diagnostics to make the next occurrence self-describing: #1378.
🤖 Generated with Claude Code
Merge pull request #1370 from Systemorph/fix/1368-framework-stale-reb…