Repository navigation
Six of the last ten CI failures were unrelated single integration tests — a flake population taxing every second PR #1384
Description
Six of the last ten failed CI runs across all branches failed on a different single integration test, none related to its PR. That is not three coincidences — it is a population, and it is costing roughly a CI cycle on every second PR.
The shape
The failures concentrate in two areas:
- Live-model tests.
DelegationSubThreadUsageTest.DelegatedSubThread_RecordsUsageUnderRealModel_AndPricesNonZero(aTimeoutExceptionagainstclaude-haiku-4-5) andOrleansSubThreadAutoResumeTest(asserting a live model produced ≥3 streaming snapshots; it saw 2). Both assert on the behaviour of an external service: its latency, and how it chooses to chunk a stream. Neither is a property of this repo, and no change here can make them deterministic. - Orleans activation paths. Assertions of the "expected to have happened by now" shape.
Observed evidence of the population: DelegationSubThreadUsageTest failed one PR's first run and the same diff went fully green on the next run; six distinct tests failed across ten runs on unrelated branches.
Why this needs its own issue rather than per-test triage
These are not on the known-unrelated list, so each one costs a fresh investigation to attribute — and the attribution is always the same answer. Worse, the cost is not just time: a red run that is not the PR's fault trains everyone to re-run reds, which is exactly the habit that hides real races. Two genuine defects were found this week because someone declined to re-run (a store-key defect where the test was green only when production was broken, and a NACK that slandered healthy hubs). That discipline stops paying if half of all reds are noise.
Known populations this is NOT
Distinguish before attributing anything here:
- CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 — a bare
NullReferenceExceptionfrom every compile in a shard, usually endingexit=124 TIMEOUT. That accounts for essentially the entireHosting.Monolith.Testtimeout population. - CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here) #613 —
FutuRe.Test exit=139 SIGSEGV, an upstream CoreCLR background-GC fault. - A per-node instance serves its NodeType's NodeTypeDefinition as its own Content after a stale-assembly recycle #1379 (fixed) —
GATE FAILED — tests: Store/Catalogon the plugin gate, previously ~1 in 20.
The six here are none of those.
What would settle it
- Decide whether live-model assertions belong in the gating suite at all. Testing real pricing and real streaming has genuine value, but an assertion whose outcome depends on an external API's latency and chunking cannot gate a merge. A separately-scheduled run that is allowed to fail without blocking a PR would keep the coverage and remove the tax. Note
~3 streaming snapshotsis an assumption about model behaviour, not a contract — the model is free to chunk differently on any given call. - Quantify the rest. A sweep of failed shards over the last week, bucketed by test, would say whether the Orleans-activation half is one race or several — the same technique that turned CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 from "a flaky project" into 14 attributable occurrences with a rising rate.
- Do not fix any of these by raising a timeout or a retry count. For the live-model ones the bound is not the problem; for the activation ones a widened bound hides a real race.
Provenance
Surfaced while landing #1373/#1374 (export defects). Both PRs hit it — one on its first run, greening on the next with an unchanged diff — which is what prompted the scan across branches rather than treating it as a local nuisance.
Activity
rbuergi commentedon Aug 13, 2026
The sweep (item 2), and a falsified premise (item 1)
Bucketed every failed
MeshWeaver Build and Testrun since 2026-08-08 — 300 runs — by failingtest name, from each shard's
Test failedannotations, deduped per run.The shape is real and steady — just over half of all reds are a single test — and bigger than
"six of ten". But the 51 recent singletons spread across 30 distinct tests, so the answer to
"is the Orleans-activation half one race or several" is: several. Nothing in the tail shares a
root with anything else in it.
Last 29 h, occurrences | distinct branches:
FutuReAnalysisTest.EuropeRe_AnnualReport_EmbeddedChartsNo renderer is registered for area KeyMetrics, preceded by'$type':'AnalysisContent' is NOT registered in this hub— a dynamic-NodeType registration race. NOT #613 (that isexit=139 SIGSEGV, a different signature in the same project)SkillAutocompleteTest.Autocomplete_ListsBuiltInSkills_FromCatalogFrameworkStaleProactiveRebuildTest.…ProactivelyRebuilds…mainThreadAgentIntegrationTest.FullFlow_…OrphanedStartingExecutionRecoveryTest,NodeTypeModelProbeTest,LogIncidentIngestAccessContextTest,CrossHubPatchAtomicityTestTwo big buckets over the full 4.4 days are already dead and should not be re-chased:
OrleansBrokenNodeTypeAccessTest(16, all 08-09/08-10) andStaleStampRootBindingTest(6, all08-10). #890 / #613 / #1379 excluded throughout, not double-counted.
Item 1: the two named tests are not live-model tests
Neither calls a model. Both build a fake in-process
IChatClient.DelegationSubThreadUsageTest—claude-haiku-4-5is a string used to look up aModelPricingrate; the seeded provider endpoint is literallyhttps://example.invalid/v1.OrleansSubThreadAutoResumeTest— the sub-agent emits a fixed 320-char report in 5-char chunks20 ms apart (≈1.3 s through
ThreadExecution'sSample(100 ms)⇒ ≈13 writes).≥3 snapshotsis an assertion about this repo's streaming path, not about model chunking.
So there is nothing to move out of the gating path, and the coverage keeps its bar. Both were
ordinary in-repo races; #1385 fixes them with deterministic waits, no bound raised, no assertion
loosened:
GetMeshNodeStream({subThread}/_Usage/{model})— a POINTread of a not-yet-existing node. That
NotFoundopensMeshNodeStreamCache's storm-breakerwindow on the exact path
RecordUsagewas about to accumulate into, and the breaker fast-failswrites too. Runs
31666562025/31645997822log the chain in order:RouteMessage: NotFound … /_Usage/claude_haiku_4_5→[TokenUsage] RecordUsage failed … No node found→ 45 spolling a satellite that exists with zero tokens. The test's own read broke the write it was
waiting for. Fixed by reading through the live children query (empty-on-absent), the primitive
ThreadTokenUsageTest.WaitForUsagealready documents after AI.Test is not parallel-safe: cross-class shared state (repro in 100s via DOTNET_PROCESSOR_COUNT=4) #1040.IsExecuting: false && Messages.Count >= 2.IsExecutingisStatus is StartingExecution or Executing, so it is false for the INITIALIdlestate — the same snapshot that satisfied the earlierMessages.Countwait, replayed bythe shared handle. Run
31667783996's two snapshots were"Allocating agent..."and"Generating response..."— framework placeholders, zero agent output. Fixed by closing thewindow on the response cell reaching
ThreadMessageStatus.Completed, which cannot be truebefore the round starts (16 snapshots locally now, was 2).
Follow-up worth its own issue
The storm breaker fast-failing writes is by design, and it relies on the change-feed
ResetFailureStateto heal a path after a realCreateNode. In these runs the window was stillopen when
RecordUsage's post-createUpdateran — eitherRecordNegativelatched a probefailure that landed after the create's invalidation, or the invalidation never reached it. I could
not discriminate from the CI logs and did not guess at a fix. Repro recipe: point-read
{thread}/_Usage/{model}while absent, run a usage-emitting round, assert the satelliteaccumulated. Production is not currently exposed —
ThreadTokenChipreads_Usageonly throughthe children query; the test was the sole point-reader.
fix: two CI flakes that were neither flaky nor live-model (#1384)
rbuergi commentedon Aug 13, 2026
Correction to my own sweep: it systematically UNDER-counts, and #1385's first CI run showed how
The bucketed numbers above count only failures recorded in a trx. They therefore miss every
failure inside a project whose host was killed — and that is a large, separate slice:
Those 58 are the crash/timeout/build/plugin-gate population. When a project hits the 8 min
wall-clock cap, xUnit is killed mid-run: completed tests stream green, in-flight and not-yet-run
tests silently vanish from the count, and the shard fails only via the exit-marker gate — which
emits no
Test failedannotation and therefore no test name. So the singleton figures are afloor, not a census.
#1385's own first CI run is a clean demonstration. It went red on shard 1 with:
No
Test failedannotation anywhere, and the per-shardTest Results (shard 1)reporter saidpass. Only the raw log names the culprit —
ReleaseRequestWatcherHighWaterTest. BurstOfTwoReleaseTriggers_SecondTriggerIsNotLost, which burned 50 s failing on a corrupt dynamicassembly:
"because the format is invalid" on a just-emitted assembly is a torn/partial read of the compiled
file, not a compile error. It is worth checking whether the assembly emit path has the same
non-atomic-write hazard that
AtomicFileWrite(added toFileSystemStorageAdapter/FileSystemVersionStoreon main today) was introduced to fix for JSON — the assembly write doesnot appear to go through it. This is plausibly the same root as the other live NodeType-compile
entries in the tail (
FrameworkStaleProactiveRebuildTest, andFutuReAnalysisTest's'$type':'AnalysisContent' is NOT registered→No renderer is registered for area KeyMetrics),but I have not proven that and am not asserting it.
Two process notes this run also confirms:
fix, not investigate — merged main and re-ran; the failing project is untouched by Two CI flakes that were neither flaky nor live-model — plus the bucketed sweep (#1384) #1385's diff
(shards 0 and 5, which carry the two tests it changes, both passed).
ReleaseRequestWatcherHighWaterTestnever appears in the bucket tables above because itsproject timed out. Any future sweep of this population needs to parse the exit markers and the
raw shard logs, not just the annotations.
fix: a third unsound wait — OrleansChatTest's "text stopped growing" …
Merge pull request #1385 from Systemorph/fix/1384-flake-sound-waits
rbuergi commentedon Aug 13, 2026
Null result on the two largest live buckets — plus a mechanism for the FutuRe one, and two corrections to the tail
I went after the buckets in the order the tail ranks them. I did not reproduce any of them on this
macOS host, so no fix is proposed from a guess. What I did get is a concrete mechanism for the
FutuRe bucket that also explains why it is CI-only, and two corrections to the tail itself.
What I tried, and what it cost
FutuReAnalysisTest.EuropeRe_AnnualReport_EmbeddedCharts(8|3)/tmp/MeshWeaverFutuReTests/.mesh-cachere-pointed per run, verified genuinely cold: the run re-emittedFutuRe_LocalAnalysis_*,FutuRe_GroupAnalysis_*,FutuRe_BusinessUnit_*,FutuRe_LineOfBusiness_*), atDOTNET_PROCESSOR_COUNT=4, at=2, and once through CI's native xUnit v3 host (dotnet MeshWeaver.FutuRe.Test.dll) rather thandotnet testCodePackageInstallTest(fresh, see below)DOTNET_PROCESSOR_COUNT=4, ReleaseThe local wall-clock is the tell: the whole FutuRe suite finishes in 21–29 s here, while the CI
failure is a 50 s wait that never satisfies. The window is not CPU count — it is the cold Roslyn
compile being slow enough on a 2-core runner to cross a 5 s threshold that never gets crossed
here. See below.
Mechanism for the FutuRe bucket (not proven — but it predicts every observed symptom)
NodeTypeEnrichmentHelpers.CreateCompilationInProgressConfiguration— the compilation-in-progressoverlay added by 74eddc6 (2026-08-11, "live compile progress instead of a silent park") —
registers the progress page for
Overviewand nothing else:So when an instance hub activates while its NodeType is still Pending/Compiling — after the 5 s
InFlightOverlayGrace— it settles onto a hub on which every area exceptOverviewhas norenderer, and
LayoutDefinition.BuildNotFoundControlserves:That is verbatim the reported failure string. The companion warning is the same cause seen from
the other side: the overlay composes the mesh DEFAULT node-hub config, so
LocalAnalysisConfig's.WithContentType<AnalysisContent>()never runs on that hub — hence'$type':'AnalysisContent' is NOT registered in this hub. One cause, both symptoms.Why CI-only: the overlay only engages if the compile is still in flight at 5 s.
FutuRe/LocalAnalysiscompiles in well under a second on this host even from cold, so the grace never expires and the
overlay never appears — which is exactly why three cold-cache configurations here are green.
Why it does not simply heal:
ArmOverlaySelfHeal's fast path needs the type node's version toadvance past the one captured at overlay time; its fallback is
SelfHealGrace = 45 s, andGetControl's first wait isMath.Max(timeoutSeconds, ActivationBudgetSeconds)= 50 s. A healon the grace path has to land a recycle, a re-activation and a re-render inside the remaining ~5 s.
That is the razor's edge the 8 failures sit on, and it is consistent with them being intermittent
on the same three branches rather than deterministic.
The fix I would propose (deliberately not shipped from here — unverifiable without the repro,
and it touches a live feature): give the overlay a catch-all so every area of an overlaid instance
renders the progress page instead of a false "Area not found". The sanctioned shape already exists
in the tree —
KernelContainer.cs:87,layout.WithView(ctx => !layout.HasNamedRenderer(ctx.Area), …),with the
HasNamedRendererguard that stops a catch-all wiping named areas. This is worth doingindependent of the flake: today a deep link to any non-
Overviewarea during a framework-bumpsweep tells the user the area does not exist, which is precisely the silent-park failure 74eddc6
set out to remove — just relocated from
Overviewto everywhere else.**Area not found**frame into atransient progress frame (and, on Ok, a
RedirectControl).FutuReAnalysisTest.IsAreaNotFound—the "keep waiting, this is transient" filter — must learn both, or the bucket converts from a
timeout into a latched-wrong-frame failure on
Should().Contain("Total Premium").Correction 1 —
StaleStampRootBindingTestis NOT deadThe tail lists it as 6 occurrences "all 08-10, do not re-chase". It failed again today, on
main, run31683003325(commit891aae06a, 08-13T08:39Z), shard 4 — and with a differentsignature, so it is a second root wearing the same test name:
WaitForRootReadypings the recycled root until it answers and gives up gracefully afterRootReadyTimeout. Here the root did not re-activate within 20 s and was stillRunLevel=Deadafterwards — i.e. a hub-teardown wedge, not a stale-stamp binding problem. Note the graceful
.Catchis what converts that wedge into a later, differently-named test failure.Correction 2 — the "format is invalid" cluster survived #1387
#1387 fixed the non-atomic publish in both
IAssemblyStoreimplementations. The same signaturerecurred on #1387's own merge commit — run
31683641381, commite45466ee6, 08-13T08:47Z:That path is the compilation cache, not the assembly store —
{cacheDir}/{nodeName}_{ticks}_{guid}/,i.e. a directory published by
EmitToDiskWithRetry, whose publish is already atomic(
.staging-…dir →Directory.Move) and whose lambda wiring I checked is correct(
CompileToDiskAsyncreally does pass the staging dir through). So this occurrence is notexplained by #1387's defect and not by a torn write at that site. There is a second, still-open
root behind "format is invalid". Its post-publish verification is only
File.Exists && Length > 0,which is weaker than "the image is complete" — worth a look, but I could not prove it.
One latent defect found while looking (no flake attached)
MeshNodeCompilationService.CompileToReleaseAsync+LoadAndExtractConfigurationsFromReleaseAsynchave no callers anywhere in
src/ortest/— butCompileToReleaseAsyncdoesDirectory.CreateDirectory(releaseFolder)thenFile.Create(dllPath)+compilation.Emit(...)straight onto the final name, and
releaseFolderisGetReleaseFolderPath={cacheDir}/{nodeName}_{releaseKey}— which matches the live{nodeName}_*globTryGetLatestCachedDllPathenumerates, ordered by directory write-time (so a folder being writtenright now sorts newest), while
IsReleaseValidis satisfied by a 0-byte DLL. That is the exacthazard
EmitToDiskWithRetry's own comment describes and guards against, unguarded, publishingexecutable bytes. It causes nothing today because it is dead — but it sits next to the corrected
path as a working-looking template. Delete it or route it through the same staging+move.
rbuergi commentedon Aug 13, 2026
Follow-up: one root found and fixed — #1408
My comment above ends on a null result. It was written 20 minutes too early: the bucket I could not
reproduce locally then reproduced itself on my own PR, and handed over the stack trace.
#1402 is a workflow-only change — it cannot touch a test outcome — and it went red on
StaleStampRootBindingTest.StaleStampSelfTypedRoot_RootServesItsTypesArea(run31686258015,09:22Z), 40 minutes after the same test failed on
main(run31683003325, 08:39Z). That is thebucket "Correction 1" above says is not dead. This time the log named the cause:
Pin()andLoadNodeAssembly()guarded on the same_disposedflag, andDispose()raised iton entry — before draining pins. So a scan could pin the context (proving it was live), have
Dispose()flip the flag one instruction later and then block waiting for that very scan, andget an
ObjectDisposedExceptionthrown at it against a LoaderAllocator that was provably stillalive.
CompileResultFromAssemblycatches it, writesCompilationStatus.Error, and the watcherparks the type with no retry — so a millisecond-wide race causes permanent damage. The #1387
reasoning exactly, from the opposite side: there the bytes were incomplete, here they were perfect
and we refused to read them.
Fixed in #1408 by splitting the flag into
_unloading(Dispose started — refuse new scans) and_disposed(drain over — refuse loads), with the relaxation applying only to a pinned caller soan unpinned one keeps today's strict behaviour. Deterministic fail-before/pass-after test; the
before-run reproduces the CI message verbatim.
MeshWeaver.Graph.Test1098/1098,MeshWeaver.PluginCatalog.Test270/270,MeshWeaver.FutuRe.Test59/59.Still open from this sweep, unchanged by #1408 and worth separate issues:
this one:
[PackageInstaller] root ShopA never answered after its recycle,RootReadyTimeoutburned in full (20 s), the root still
RunLevel=Deadafterwards. Different failure, same test.compilation-in-progress overlay registers a view for
Overviewonly, so every other area of anoverlaid instance serves a false "Area not found"). Not reproducible on this host; three
cold-cache configurations green.
already atomic. Second root still unexplained.
rbuergi commentedon Aug 13, 2026
Index: where each bucket from the sweep now lives
Short pointer comment, because the table in the sweep above is what a future triager will read and two of its rows are now wrong.
SkillAutocomplete6|4FrameworkStaleProactiveRebuild4|4StaleStampRootBindingTest— "6, all 08-10 … do not re-chase"main08:39Z, and a PR 09:22Z) and carries two distinct rootsObjectDisposedException, PARKS the type) → fixed in #1408 · root B (recycle/teardown wedge) → #1405FutuReAnalysisTest.EuropeRe_AnnualReport_EmbeddedCharts8|3OrleansBrokenNodeTypeAccessTest16Plus two things the sweep could not have known:
e45466ee6, run 31683641381), from the compilation-cache path — whose publish is already atomic — not from the assembly store A NodeType assembly can be loaded mid-write — 'format is invalid' parks the type, and the emit path skips AtomicFileWrite #1387 fixed. A NodeType assembly can be loaded mid-write — 'format is invalid' parks the type, and the emit path skips AtomicFileWrite #1387 is closed, so this has its own home: "format is invalid" on a fresh NodeType assembly survived #1387 — a second root in the compilation-cache path #1412.NodeRepoInstanceOrderingTestfailed with a signature byte-identical to Flake: PluginDependencyOrderTest.InstallingTheClosureInOrder_LandsTheDependentTyped times out when the Dependent/Item hub recycles mid-install #1405's, down toMeshNodeStreamExtensions.cs:678. Filed as a comment on Flake: PluginDependencyOrderTest.InstallingTheClosureInOrder_LandsTheDependentTyped times out when the Dependent/Item hub recycles mid-install #1405, which is the family issue.On the sweep being a floor
Your correction about the 58 trx-less runs holds, and I hit the same wall from the other side: the two highest-ranked live buckets were not reproducible on an M-series Mac — FutuRe went 59/59 green in three configurations (verified-cold NodeType cache at
DOTNET_PROCESSOR_COUNT=4, at=2, and through CI's native xUnit v3 host), andCodePackageInstallTestwent 270/270. Both need the 2-vCPU runner's interleaving.flake-repro.ymlis the instrument for that and it is under-used — it loops one test on the real runner. I am using it right now as a controlled A/B (same filter, same iteration count,mainvs a branch) to attribute a failure rather than argue about it. That pattern generalises to the rest of this tail: for any bucket that will not reproduce locally, dispatch it on both arms instead of re-running the PR.Merge pull request #1408 from Systemorph/fix/1384-pinned-scan-parks-n…
rbuergi commentedon Aug 13, 2026
A sweep that parses exit markers — and the largest live bucket REPRODUCED on a dev host
Two things this adds to the thread: the sweep redone the way your own correction says it has to be
done (raw shard logs + exit markers, not annotations), and — for the first time in this issue — a
local reproduction with a measured rate of the bucket that sweep ranks first.
1. The sweep, with exit markers
33 failed
MeshWeaver Build and Testruns since2026-08-13T00:00Z. For every failedRun tests (shard N)job I pulled the raw log and parsed BOTH[FAIL]test names and the[CI] <project> exit=<n> <VERDICT>markers.[FAIL](annotation-visible)Exit markers: 11 ×
Hosting.Monolith.TestTESTFAIL, 6 ×PluginCatalog.Test, 4 ×Hosting.Orleans.Test,3 ×
AI.Test, 2 ×Graph.Test, 1 ×Threading.Test, 1 ×InstanceSync.Test, 1 ×exit=124 TIMEOUT(#890), 1 ×
exit=139 SIGNAL(#613) — plus 3 of a verdict the annotation sweeps never had a name for:That is the trx-less population being caught in the act rather than inferred: the shard does have a
trx, its failures are attributable, and the host still died afterwards. Worth its own look —
the named failures in those runs are the ones below, so whatever kills the host is not what fails the
tests.
2. Rank by BRANCH SPREAD, not by count
Raw counts mislead here, because a PR whose own diff breaks a test contributes several occurrences on
one branch. Filtering the window's buckets that way:
DynamicTypePreWarmerTest×3 testsfix/1391-nosources-default-queries, twice — that PR's own regression, not a flakeStoreLevelWriteConflictTestfix/1284-activity-node-shape; asserts"ActivityLog", got"Activity"— that PR's own renameLayoutAreaRetrievalTestFrameworkStaleProactiveRebuildTest.…ProactivelyRebuilds…NodeTypeCompileParkTest×3,ReleaseRequestWatcherHighWater,LinkedInProfileLayoutArea,FrameworkStaleRecord…fix/1384-flake-sound-waits— a single shard blow-up, not six bucketsStaleActivationDurableFirstSeedTest.ReactivationViaRead_ServesDurableTruth_AndWritesNothingBackOrleansGrainTeardownStragglerTestPluginDependencyOrderTestSo the largest genuinely live, cross-branch bucket in the window is
StaleActivationDurableFirstSeedTest.ReactivationViaRead_ServesDurableTruth_AndWritesNothingBack—fix/1311-…00:28Z,fix/1362-silent-read-nack00:38Z,fix/1284-activity-node-shape11:28Z.Not #890 (named test,
exit=1, no NRE), not #613, not #1405's recycle wedge (it completes in ~2 s).3. It reproduces — 1 in 60
All three CI occurrences carry the identical assertion, to the word:
i.e. a pure read-reactivation wrote one extra row to storage, which is exactly the #590 invariant
the test exists to pin. On an M-series Mac, native xUnit v3 host,
DOTNET_PROCESSOR_COUNT=2:in a loop → 1 failure in 60 iterations (≈1.7 %), same assertion, same
2vs3. ~3.5 s periteration, so a full measurement is minutes, not a CI cycle. This is the first bucket in this issue
that does not need the 2-vCPU runner.
4. …and it is a Heisenbug with respect to logging
Raising
MeshWeaver.Graph.MeshNodeTypeSource/MeshWeaver.Graph.SaveMeshNodeHandler/MonotonicWriteGuardStorageAdaptertoDebugin the test'sbin/…/appsettings.json(thesanctioned way — no src log-level edits) and re-running the same loop went 160 iterations clean,
under more machine load than the run that reproduced it. At p = 1/60 that is P ≈ 7 %. So the
window this race lives in is narrow enough that the debug writes close it — which is worth knowing
before anyone spends a day log-diving it.
5. What the failure needs in order to be attributable
Gated.WriteCount(path)counts writes that reached the innermost adapter; the assertion reports onlythe count. "found 3" says a write happened and nothing about which, and the two candidate
mechanisms are distinguishable purely by the version it carried:
OwnNodeCache.PersistedSnapshotreference-identity suppression inMeshDataSource.InstallPersistenceSamplerfailing open. Note that stamp is a singlelast-write-wins slot written by
MeshNodeTypeSource.BuildInstanceCollectionon every acceptedown-node emission, while the sampler consumes emissions asynchronously — so two accepted emissions
in the activation window are enough for the first to stop matching the slot.
writesBeforewas sampled — whichwould have to have passed the monotonic guard, so it is the less likely of the two.
I am carrying a one-line change to
GatedReadStorageAdapterthat recordsv{version}/{name}perwrite and puts the trace in that assertion message, so the next occurrence — local or CI — names the
offender instead of leaving it at "found 3". Happy to send it separately if it is wanted before the
root cause.
Provenance
Found while landing #1412 (the "format is invalid" second root — PR #1423). Sweep script, raw logs
and the loop harness are reproducible from the recipe above.
rbuergi commentedon Aug 13, 2026
Follow-up on the
StaleActivationDurableFirstSeedTestbucket: the extra write is a version MINT, and all three CI failures agreeCorrecting and sharpening my comment above before anyone spends time on it.
The one number that identifies the write
The test advances durable truth out of band to
Version = 7000and then does nothing but a read.All three CI failures log the same line during the reactivation:
7001, not 7000 — in
31654549526,31655126436and31694736256alike. Nothing in the test everwrites 7001. So the extra write the assertion catches is not an echo of the durable row: it is a
freshly minted revision,
NextVersion(7000), produced inside the activation window and thenpersisted by the own-node persistence sampler. That also explains the absence of any
[MonotonicWriteGuard] CONFLICTline in those runs — 7001 > 7000 sails through the guard, so thewrite reaches the innermost adapter and is counted.
Two things follow, and both narrow the hunt a lot:
v1, lose to the guard,and never reach the adapter (
WriteCountcounts only writes that arrive);PersistedSnapshotreference-identity miss on the seed either — that wouldre-persist 7000, not mint 7001.
The mint has to come from
MeshNodeTypeSource.UpdateImpl's update re-stamp:NextVersion(7000)requiresalreadyAdvanced == false, i.e. a genuine CONTENT change arriving ata version that is not above the one already held — the diff a few lines up compares content with
!ex.Equals(nv with { Version = ex.Version }), so a version-only difference is already excluded. Inother words: something committed a node whose CONTENT differs from the seeded one while carrying the
SAME version 7000.
The candidate that fits the shape is the enrichment graft.
Initializeseeds the workspace withGraftRoutingEnrichment(durableRow)—durableRow with { HubConfiguration = routing.HubConfiguration }— while every other route to the own node (notably the storage change-feed reconcile in
MeshDataSourceExtensions.SubscribeToOwnDeletion, which commitsnotification.Entityverbatim)carries the RAW row. Grafted and raw differ by that delegate and by nothing else, and record
Equalscounts the delegate, so raw-vs-grafted at the same version reads as a content change →re-stamp → 7001 → the sampler persists it. I have NOT proven that leg fires here, which is why I am
posting it as the lead rather than a fix.
Reproduction, honestly
DOTNET_PROCESSOR_COUNT=2, native xUnit v3host,
-class StaleActivationDurableFirstSeedTest), the exact CI assertion.MeshNodeTypeSource/SaveMeshNodeHandleratDebug, and a further 259 clean with logging back at baseline but awrite-recording line added to
GatedReadStorageAdapter. At p = 1/60 those are P ≈ 7 % and ≈ 1 %.So: the bucket does reproduce on a dev host, which is new for this issue — but the window is
narrow enough that a log write or an extra
ConcurrentQueue.Enqueueon the write path closes it.Anyone measuring a fix here needs to measure the BEFORE on the same tree they measure the AFTER on,
and should expect a few hundred iterations per arm.
The cheap thing that would settle it
Gated.WriteCount(path)reports a count; the assertion says "found 3" and stops. Recordingv{version}/{name}per write inGatedReadStorageAdapterand putting the trace into that assertionmessage turns the next occurrence — CI or local — into a direct answer:
v7000/durable-advance, v7001/durable-advanceconfirms the mint,v1/createdwould refute it and point back at thedispose-flush. It is ~15 lines in a test-only helper. Say the word and I will send it on its own.
rbuergi commentedon Aug 13, 2026
The grafted-vs-raw leg does NOT fire — it cannot. And the mint has a much better candidate, one line away.
Taking up the lead from the comment above ("I have NOT proven that leg fires here, which is why I am posting it as the lead rather than a fix"). It is refuted, and refuting it points straight at something else.
1. Refuted —
MeshNode.Equalsdeliberately excludesHubConfigurationThe lead rested on this step:
MeshNodedoes not use compiler-synthesized record equality. It hand-writesEquals, and the delegate is explicitly left out —src/MeshWeaver.Mesh.Contract/MeshNode.cs:387-412, with the exclusion stated in its own doc comment:GraftRoutingEnrichmentchanges exactly one field (durableRow with { HubConfiguration = routing.HubConfiguration }), and that field is the oneEqualsignores. SoUpdateImpl's update filteris false for a grafted/raw pair at the same version: not an update, no re-stamp, no mint. Verified rather than reasoned — a new fact asserts it in exactly that shape (
MeshNodeContentEqualityTest.RoutingGraft_VersusRawRow_AtSameVersion_IsNotAContentChange, PR below), so a later switch to synthesized equality cannot silently reintroduce the mint.Worth keeping: the premise of the lead was right. Grafted and raw genuinely do meet —
InMemoryStorageAdapter.WritestripsHubConfigurationbefore storing and before publishing (:98-99), so every change notification carries a raw row while the activation seed carries a grafted one. It just isn't a content change.Also refuted, same evidence: the stale v1 routing snapshot cannot reach
UpdateImpleither. The routing-supplied own-node stream has exactly two consumers — the ctor enrichment capture (MeshNodeTypeSource.cs:156-162, which only writes_latestRoutingNodeand never touches the workspace) andInitialize's Concat leg, which is gated byAcceptOwnNodeEmission. There is no third subscriber. SoUpdateImplis not where 7001 comes from at all.2. What does mint 7001 — the change-feed reconcile, via
UpdateOwnMeshDataSource.cs:1394-1398, thestorage.Changesreconcile inSubscribeToOwnDeletion:GetMeshNodeStream()with no path is the OWN node ⇒UpdateOwn, which mints unconditionally (MeshNodeStreamExtensions.cs:839-842):With
current= the pre-recycle node (v1) andupdated= the notification'snewNode@7000 ⇒NextVersion(7000)= 7001, carryingName = "durable-advance". That is the number, from a single expression.Every gate on the way is passed by design:
newNode.Version == lastSelfWrite.Value—lastSelfWriteis the pre-recycle version (1), ≠ 7000;current.Version >= newNode.Version— 7000 > 1, so it adopts, which is its whole purpose;UpdateOwn's no-op gates (:817-818) — Name and Version differ;OwnNodeCache.PersistedSnapshotreference identity — the adopted node is a freshwith-copy;MonotonicWriteGuard— 7001 > 7000 sails through, which is why noCONFLICTline appears in any of the three CI runs.Then
.Do(pending.Track)sits before theSample(MeshDataSource.cs:1168), soPendingOwnSaveholds node@7001 immediately and either the 200 ms sample tick orFlushPendingOwnSaveat dispose writes it. One write, which is exactly the "expected 2, found 3" the assertion reports.Which hub? The old one.
MessageHub.Disposeonly postsShutdownRequest(Quiescing)and returns;delSub— a subscription that can write — is not torn down untilDisposeImplat the ShutDown phase.RecycleOwnernever awaits disposal, so the test's out-of-bandStorage.Write(…Version = 7000)lands while the recycled hub's reconcile is still listening.IsolatedChangeFeeddoes not replay, so the new hub can never see that notification — which is why this is the old hub's doing and why it is a race at all.3. This is now reproduced locally, and the timing confirms the account
1 failure in 250 iterations on the unmodified tree at today's
main(4e8e2efd7, ~130 commits past the last measurement),DOTNET_PROCESSOR_COUNT=2, native xUnit v3 host,loglevel=Warningthroughout — the CI assertion, verbatim,expected 2 … found 3. Same recipe as the earlier 1-in-60; the two rates are not distinguishable at these sample sizes, so read it as "still roughly 1 in 100–200", not as a change.The reproduction settles the sequencing, because of when the drop line fires:
421 ms in, immediately after the recycle and the out-of-band write — and ~2 s before the reactivation's assertions. The floor is already 7001 at that point, i.e. the mint happened around the out-of-band write, not during the reactivation. That matches the old-hub reconcile exactly, and it does not match anything on the new hub's activation path (whose seed cannot be above 7000 unless something already wrote 7001).
4. Why the number was over-read (mea culpa on my own earlier comment)
The comment above treats 7001 as strongly diagnostic:
That inference assumed the mint site was
UpdateImpl. ThroughUpdateOwnthe arithmetic is different and less specific:NextVersion(Math.Max(current.Version, updated.Version))yields 7001 for anycurrent ≤ 7000— v1 gives 7001 just as v7000 does. So 7001 never discriminated "a content change at the held version" from "adopting 7000 from below". It only ever saidMath.Max(...) == 7000.5. The root cause, stated as a shape
Adopting an already-durable external write is an observation, and an observation must not mint. The reconcile exists to move the in-RAM node forward to match a write someone else already persisted; routing that through
UpdateOwnstamps the in-RAM node one revision above the durable row with identical content, and the persistence sampler then writes that phantom revision back. That is an activation/observation-window write-back — the very #590 shape this test pins, arriving through the door next to the one it is watching.The correct shape already exists a few lines away, in
BuildInstanceCollection: adopt at the durable version and stampOwnNodeCache.PersistedSnapshotso the sampler suppresses the echo.AdoptDurableTruth's mint is different and should stay — there the point is to get above a durable row that refused your write.I am not shipping that change here. It sits in the middle of the cross-hub write-loss family (
CrossHubPatchAtomicityTest's lagged-echo defence is the sameUpdatelambda), and at ~1-in-200 a fix needs a few hundred iterations per arm to be worth anything. Whoever takes it should measure BEFORE and AFTER on the same tree, atloglevel=Warning— and know that instrumenting the write path closes the window (aConcurrentQueue.Enqueuethere went 259 iterations clean).Two secondary contributors worth separate consideration either way:
RecycleOwnerdoes not await disposal, and a subscription that can write survives until the ShutDown phase.6. Naming the offender without closing the window — PR #1428
"found 3" is unattributable, and the obvious fix (log every write) is the one thing that provably destroys the race. It is also unnecessary: the durable row IS the last write that reached storage, and the test already reads it — two lines after the assertion that fails. Moving that read ahead of the assertion puts the answer in the failure message at zero cost to the write path:
v7000/'durable-advance'→ an unchanged re-persist (a lossless echo)v7001/'durable-advance'→ the mint-on-adopt abovev7001/'created'→ a stale pre-recycle snapshot re-stamped over durable truthA read cannot perturb the race, and taking it before the count is sampled only ever widens the window for a late write to be caught.
PR: test-only, no production change, no assertion loosened, no bound raised.
rbuergi commentedon Aug 13, 2026
Follow-up: the mechanism is now pinned deterministically — no race, no recycle, 100 % of the time in 2.5 s
My comment above proposed the
storage.Changesreconcile as the mint site but rested on a timing argument. It does not need one. Strip the race out entirely and the defect is unconditional.The experiment: keep the hub LIVE (no recycle, no gate, no reactivation) and advance durable truth out of band, so the live hub adopts an external write through its own change-feed reconcile. Observation is a read; it must not mint and must not write back.
Written at 5000, the hub holds 5001, and the durable row is now 5001 too — it wrote the phantom revision back. Reproduces every time, in 2.5 s, at
loglevel=Warning. (writes 1 → 3is create + my own out-of-band write + one hub write-back — theBe(writesBefore)assertion in my probe was sloppy about that baseline; the version assertion and the durable row reading 5001 are the decisive parts and neither depends on the count.)So the chain is confirmed end to end, from a single expression:
NextVersion(5000)= 5001; the node is a freshwith-copy soOwnNodeCache.PersistedSnapshotdoes not suppress it;.Do(pending.Track)sits before theSample, so the sampler (or the dispose flush) persists it;MonotonicWriteGuardaccepts it because 5001 > 5000.What this changes about #1384
The defect is not a race. The race only decides whether it is observed.
ReactivationViaRead_ServesDurableTruth_AndWritesNothingBackrecycles the owner before the out-of-band write, so whether the write-back happens depends on whether the notification still finds a live reconcile subscription — andMessageHub.Dispose()merely postsShutdownRequest(Quiescing)and returns, tearingdisposables(including that write-capable subscription) down only later at the ShutDown phase, which the test never awaits. That is the ~1-in-250 window. In production there is no such window at all: any hub that is simply live when another writer advances its row takes the phantom mint unconditionally.Which also means the blast radius is wider than one flaky test: every cross-writer convergence mints a spurious revision and writes it back. Two mirrors, a migration, a repair job, GitSync, or a second silo advancing a node all leave the observer's copy one revision above durable truth and then persist that. It inflates version history with content-identical revisions and it is an activation/observation-window write-back of exactly the #590 shape.
Repro, ready to paste
Drop into
StaleActivationDurableFirstSeedTest(it reuses that class'sGated/Storage/ReadDurablehelpers) and run-method …StaleActivationDurableFirstSeedTest.LiveHub_AdoptingAnExternalDurableWrite_MintsNothing_AndWritesNothingBack:It is not in PR #1428 — a knowingly-red test does not belong on a branch. It should land with the fix, as the fail-before/pass-after.
The fix, and why I am not shipping it here
The reconcile's job is to move the in-RAM node forward onto a revision someone else already made durable. That is an adopt-at, not a mint-above. The shape already exists a few lines away in
MeshNodeTypeSource.BuildInstanceCollection: take the row at its own version and stampOwnNodeCache.PersistedSnapshotso the sampler suppresses the echo.AdoptDurableTruth's mint must stay as-is — there the point is to climb above a durable row that refused your write.I am leaving it to whoever owns this area, because the
Updatelambda in question is alsoCrossHubPatchAtomicityTest's lagged-echo defence (the concurrent cross-mirror data-loss family), and changing adopt semantics there is a scope call rather than a mechanical fix. The good news is that it no longer needs the flake to validate: fail-before/pass-after on the deterministic test above, then the flake arm only as a regression check.Measured for whoever picks it up — same tree (
4e8e2efd7),DOTNET_PROCESSOR_COUNT=2, native xUnit v3 host,loglevel=Warningthroughout:ReactivationViaRead_…unmodifiedLiveHub_AdoptingAnExternalDurableWrite_…rbuergi commentedon Aug 13, 2026
Split the root cause out to #1432 so it does not stay buried in this sweep thread — same convention as #1405 / #1411 / #1412.
#1432: the
storage.Changesreconcile adopts an already-durable external write throughUpdateOwn, which mints unconditionally, so the hub lands one revision above durable truth and the persistence sampler writes that phantom revision back. Deterministic, 2.5 s, no recycle and no race — the repro is in the issue.This bucket (
StaleActivationDurableFirstSeedTest.ReactivationViaRead_…, measured 1 failure / 250 iterations on4e8e2efd7atloglevel=Warning) is one observation of #1432, made intermittent only by whether the change notification still finds the recycled hub's write-capable subscription alive —Dispose()postsShutdownRequest(Quiescing)and returns, andRecycleOwnernever awaits the teardown.So the row for this bucket: root cause → #1432; PR #1428 carries only the equality pin (refuting the graft lead) and a discriminator that names the offender without touching the write path.
Merge pull request #1428 from Systemorph/fix/1384-grafted-vs-raw
22 remaining items
rbuergi commentedon Aug 26, 2026
Every named member of the population now has a merged root cause or an owner — but the rate this issue exists to measure has not been re-measured, so it stays open
Following the 13:04Z comment, which enumerated the population as six named tests and put agents on four. All four landed root-cause fixes between 14:30Z and 14:46Z. None widened a bound, added a retry, or weakened an assertion, and each shipped a guard or a failing-first repro:
LiveQueryHandoffDropTest(the one redding three unrelated PRs at once)IEnumerable.ToObservable()defaults toCurrentThreadScheduler, whose[ThreadStatic]trampoline flag makes emissions queue instead of run when aTaskresumes inline inside an Rx pipeline.Initialis then never emitted — no error, no completion, no row. Fixed at all 12 sites on the storage/query read path (includingPersistenceService.ListChildPaths, "the walk the PORTAL actually uses"), plusBareToObservableOnReadPathGuardratcheting it to zero. Measured cold, whole-assembly,--cpus 4: 3 failures / 13 runs → 0 / 40.CrossProcessChangeFeedTestMonotonicWriteGuardthen refused a legitimate write andAdoptDurableTruthrebased an owner that never edited anything. New test 10/10 red with the fix reverted, 10/10 green with it.MeshWeaver.Threading.Testexit=124wedgeCatch(...).Repeat()over a source that completed without emitting, so the 200 ms backoff never ran: ~54 kHz synchronous re-subscribe, inside the assertion's own.ToTask()and therefore out of reach of.Within(),[Fact(Timeout)],methodTimeoutand the harness watchdog alike. 3 sites +UnthrottledResubscribeInTestRatchetGuardwith a deliberately empty allow file.CrossHubPatchAtomicityTestMergeGuardlogic defect.MergeGuardwas measured refusing only the always-conflictinglastModifiedfield, never a data key — so the "data loss on the cross-hubstream.Updatepath" concern did not materialise.The remaining two are owned, not orphaned.
PartitionRootActivationTestandRoutingBackpressureShapeTest— the ones I flagged as unticketed in the 08:07Z and 09:48Z comments — are both already on #2346 ("shard-0 Orleans suite is red across unrelated PRs"), which is open and actively worked and which frames exactly the concentration those comments described.OrleansGrainTeardownStragglerTest(#2301) is closed;OrleansChatHistoryTest(#2291) is open and owned.🚨 Why this does not close
Four merged root causes is not a measured rate. The fixes landed 14:30–14:46Z; roughly an hour of CI has run against them, which is nowhere near the 300-run window the 2026-08-13 sweep used, and a sweep taken now would overwhelmingly sample the pre-fix period. The number this issue was opened on — "just over half of all reds are a single unrelated test" — has not been re-taken, and it is the only thing that can close it.
The closing criterion, concretely: re-run the 08-13 bucketing over a window that starts after
2026-08-26T14:46Z, once there are enough runs to be worth bucketing (a day or two). If the single-unrelated-test share has collapsed, this closes on that measurement. If it has not, the sweep names whatever replaced the six.rbuergi commentedon Aug 26, 2026
The measurement, re-taken — item 2 of this issue, on a fresh window
Since the comment above said the rate had not been re-taken, here it is. Method matters because the previous sweeps under-counted for two specific reasons, both now covered:
MeshWeaver Build and Testrun since 2026-08-21 (878) via the workflow-runs API, then pulled each failed shard job's own annotations (Test failed (shard N)). 🚨 TheTest Results (shard N)reporter jobs are absent from many recent runs, so a sweep that reads the reporter silently misses those runs entirely.run_attempt > 1and 34 now readsuccess, hiding an earlier red — invisible to any run-level query. All figures below are attempt-level, recovered with?filter=all.Window 2026-08-23T00:00Z → 2026-08-26T15:22Z (3.6 d, all branches)
failuremainpush: 33 / 169 = 19.5 %So the shape this issue is named for is now 38 % of reds, not the ~60 % of "six of the last ten". It persists; it is no longer the majority.
83 distinct tests failed, but 61 appeared on only ONE branch (96 occurrences) — those are overwhelmingly the PR's own fault. 13 tests appeared on ≥3 distinct branches, accounting for 117 occurrences and 49 of the 68 solo-cause attempts. That cross-branch set is the population, and distinct-branch-count is an objective proxy for "unrelated" that does not require reading each diff.
The cross-branch population, and two things in it that are NOT flakes
Hosting.Test.LiveQueryHandoffDropTestAI.Test.AiContentPackDriftTest.EveryFile_StillHashesToItsPinnedReconciliationPointHosting.PostgreSql.Test.UserDirectoryCompletenessTests.AUserOutsideTheMostRecentlyModifiedPage_…Hosting.Monolith.Test.CrossProcessChangeFeedTestHosting.Orleans.Test.OrleansMeshTests.HubWorksAfterDisposal(HubFactory)Hosting.Orleans.Test.PartitionRootActivationTestHosting.Orleans.Test.OrleansChatHistoryTest.ColdStart_…Hosting.Monolith.Test.UserDirectoryContentShapeTest(typed)Hosting.Orleans.Test.OrleansGrainTeardownStragglerTestHosting.Orleans.Test.OrleansMeshTests.HubWorksAfterDisposal(Kernel)Hosting.Orleans.Test.OrleansNodeChangePropagationTest.Delegation_NodeChanges_PropagateFromSubThreadHosting.Monolith.Test.PandasExplorerLayoutAreaTest🚨 The date column disqualifies the top two from this issue's taxonomy, and a raw occurrence count would have promoted both:
AiContentPackDriftTest— 18 occurrences across 16 branches, all on 2026-08-25 and never again. A single-day systemic drift against a pinned hash, already resolved. Not a race.LiveQueryHandoffDropTest— 20 occurrences across 15 branches, all on 2026-08-26. Its whole life is inside the window: it did not exist as a problem when this issue was filed, and it is fixed as of Fix a live query's scope walk being queued on a foreign Rx trampoline and never running (#2377) #2389 (14:32Z).Two members of the durable core are new to this issue and have no ticket I could find:
UserDirectoryCompletenessTests(16 occ / 12 branches, 3 days) andOrleansNodeChangePropagationTest. The rest map onto #2346 / #2291 / the four fixed today.The failure mode this issue's taxonomy does not name at all
22 of 178 failing attempts had a shard fail with ZERO test-name annotations — the host died, so no test can be blamed and no sweep that buckets by test name sees them:
MeshWeaver.Threading.Testexit=124 TIMEOUTmain×5)Hosting.Monolith.Test(both halves)exit=124 TIMEOUTmain×3)Hosting.Orleans.Testexit=139 SIGSEGVThe
Threading.Testrow is exactly the population #2390 measured and fixed today (its own sweep: first 2026-08-25T23:24Z, 400 runs back to 08-22, none before) — independent corroboration from a different data path. TheHosting.Monolith.Testtimeouts predate it and are still unattributed here.Not established, stated rather than glossed
cancelledand were not inspected. A shard can fail and then be cancelled before the reporter runs, so every count above is a floor..trxwas written. Only project + exit classification are established.Still open. The post-fix window is ~1 h old, so none of this measures whether today's four merges moved the rate; it measures the rate through them. Re-take it over a window starting after
2026-08-26T14:46Zin a day or two — that is the number that closes this.rbuergi commentedon Aug 26, 2026
Resolution status, 2026-08-26 end of day
Six of the eight named members are closed, and — the part worth recording — every one turned out to be a real framework defect, not test flakiness.
.ToObservable()bindsCurrentThreadScheduler, whose[ThreadStatic]trampoline makes emissions queue instead of run when aTaskresumes inline in an Rx pipeline. Fixed at 12 sites; cold-run failure rate 23% → 0/70. In production this silently empties a children listing — for ever.MonotonicWriteGuardthen refused a legitimate write andAdoptDurableTruthrebased an owner that had never edited. The "1.3% flake" was which write won..Catch(…).Repeat()where the source completed without emitting, so the backoff never ran — 460,000 re-subscribes in 8.45 s, synchronous, out of reach of every harness timeout.CancellationTokenSource.Cancel()runs callbacks synchronously on the caller's thread — and those callbacks are whole pooled-subscription teardowns. Mesh teardown ran application clean-up inline, unbounded and unlogged.Success.Refs, notCloses, because it does not explain every cast memberSharedOrleansFixtureInvalidProgramExceptionWhat this says about the issue's own thesis
The premise — "unrelated single integration tests" — was right about the symptom and wrong about the cause. They were not unrelated. Three of the six are the same shape (an Rx pipeline that stalls, batches, or re-subscribes because of how a scheduler or a queue behaves under inline resumption), and several share a deeper one: a comment asserting a guarantee the code never delivered. #2305's write queue documented "the next queued Update sees post-patch state" with nothing implementing it; #2394's
Drainwas bounded by a timeout that covered only the step after the unbounded one.Also corrected during the sweep, and worth keeping: the earlier measurement here was understated, because
gh run listreports only a run's latest attempt — every failure someone re-ran to green vanishes from the sample. Recovering attempts across 878 runs surfaced 34 hidden reds and moved "solo unrelated test" from ~60% to 38.2%. When surveying CI history, enumerate attempts, not runs.Remaining
Two open members (#2346, #2291), plus #2421 (progress arriving as one batch — under measurement to split a real defect from a racy test) and #2436 (two sibling TOCTOU port helpers). Leaving this issue open until those land, since it is the population tracker and the population is not empty.
rbuergi commentedon Aug 28, 2026
Two more data points from today (2026-08-28), both single unrelated tests
Adding these here rather than filing separately — they are the population this issue is about, and one of them is a new member worth naming.
1.
MeshNodeLanguageServiceTest.ScriptCompletions_FilterByTypedPrefix_NotJustTheAlphabet— red on main at 12:32Z (run33171489378, shard 2). Main was green immediately before (33170783055) and after (33172973648,33173945145,33174881868), on merges that do not touch the language service. Single test, single occurrence, no diff to blame.2.
DataContextInitFaultedTest.FaultedInit_ReachesTerminalFailedState_AndAnswersFast— red on PR #2611's merge commit at 12:52Z (run33172441705, shard 4). Filed separately as #2614 because it is new and therefore worth pinning while the trail is warm: the test landed on main the same day (5204e0fd4, #2528). Measured 25/25 green onorigin/mainlocally underDOTNET_PROCESSOR_COUNT=4, and #2611's diff is confined toMeshWeaver.Graph's NodeType compilation while the test exercisesMeshWeaver.Data's DataContext init — no path between them.What today suggests about the shape
Both fit the profile in the issue body — a single unrelated integration test, on a shard, once — but neither is a live-model test. #2614's is a 15-second
.Timeouton a path that should answer immediately (WithInitialData(() => Observable.Throw(...))faults synchronously at subscribe). If that class of bound-under-load turns out to be a second generator alongside the live-model one, it wants a different remedy: raising it would be a band-aid, because the assertion exists precisely to catch a wedge, so a widened bound would hide the thing it was written to find.Cost this session: one full CI cycle on #2611, which is the tax this issue quantifies.
rbuergi commentedon Aug 29, 2026
Fresh measurement — population still nonzero, ~1 flake-shaped red every ~12 h post-outage
Same sweep as the sibling issues just closed/updated: every
MeshWeaver Build and Testrun onmain, 2026-08-26T07:38Z → 2026-08-28T20:46Z (121 non-cancelled attempts, 19 failures), eachfailure read from its shard's own annotations.
Excluding the known 08-26 morning merge-pile-up cluster (9 failures in 2.3 h, 07:58–10:11Z — a
separate, already-discussed incident), the post-outage window is 2026-08-26T10:11Z →
2026-08-28T20:46Z: 104 valid attempts, 11 failures:
MeshOperationsnot found)using MeshWeaver.AI)Memex.AppHostimport)OrleansChatHistoryTest.ColdStart_AgentSeesAllPreviousMessagesOrleansGrainTeardownStragglerTestMeshNodeLanguageServiceTest…(now#2616)DataContextInitFaultedTest…(now#2614)HOST_CRASHEDon shard 25 of 11 are this issue's shape (a single unrelated integration test/host, nothing else on the
shard) — roughly one every 12 h over the 58.5 h post-outage window. The population is not empty.
Leaving open.
One structural note for next time this is swept
MeshWeaver.Plugins#786("The AI engine moves in", merged 2026-08-28T11:59:20Z) relocated theAI-specific Orleans tests (
OrleansGrainTeardownStragglerTest,OrleansChatHistoryTest,OrleansNodeChangePropagationTest,OrleansSubThreadRoutingTest,OrleansAutoExecuteTestandsiblings) out of this repo entirely, into
MeshWeaver.Plugins/src/MeshWeaver.AI.Orleans.Test. Twoof the five singletons above (
ColdStart…,GrainTeardownStraggler…) are tests that can nolonger fail in this repo's CI — closed
#2500today partly on that basis (#2346stays open — it has outlived fixes before, see that thread). If thisissue's population drops after 08-28T12:00Z, that migration is probably why; if a member with one
of those names shows up again, it belongs in
MeshWeaver.Pluginsnow, not here.rbuergi commentedon Aug 29, 2026
The sweep #1384 asked for: 1200 runs, every attempt, bucketed by test
All numbers below are MEASURED from the GitHub Actions API unless a line says INFERRED.
Method. All 1200
MeshWeaver Build and Testruns from 2026-08-22T12:40Z to 2026-08-29T07:41Z. For every run that failed or carriesrun_attempt > 1, I enumerated every attempt (376 run/attempt pairs) — a re-run-to-green otherwise hides the attempt that failed — pulled all 290 failedRun tests (shard N)job logs (48 MB) and extracted[FAIL]lines and the real[CI] <project> exit=N <TAG>result lines.Raw counts: 290 failed shard jobs · 371 (test, job) failure occurrences · 158 distinct test names.
Run-level failure rate
The issue's "every second PR" was written at the 08-26 peak. It has since roughly halved.
🚨 The headline: a third of the population is victims, not flakes
This is the finding that changes how the list should be read.
10 cascade events produced 67 of 371 occurrences (18.1%) and touched 50 of 158 distinct test names (31.6%).
InvalidProgramExceptionA single #890 event takes down 2–10 test names at once (max 10, on
fix/install-seed-cap-outruns-its-fallback08-27). Those 37 names includeCodeEditRecompileTest×6,NodeTypeReleaseGateTest×3,NodeTypeCompileParkTest×3,CompileLeafStabilityTest×2,LinkedInTelemetryImportTest,PandasExplorerLayoutAreaTest,StaleStampRootBindingTest… i.e. exactly the "unrelated single integration test" texture the issue describes. Each one costs a fresh attribution investigation, and the answer is always the same. This is a ~6× blame amplifier: 9 real events wearing 37 names.Same shape as this issue's own #2291 finding, at scale. INFERRED: a meaningful share of the "six of the last ten" that prompted this issue were cascade victims rather than distinct flakes.
Ranking by frequency
AI.Test.AiContentPackDriftTest.EveryFile_StillHashesToItsPinnedReconciliationPointHosting.Test.LiveQueryHandoffDropTest.A_notification_snapshotted_before_Initial_…PostgreSql.Test.UserDirectoryCompletenessTests.AUserOutsideTheMostRecentlyModifiedPage_…Orleans.Test.PartitionRootActivationTest.BarePartitionPath_NoMeshNode_RespondsToPingOrleans.Test.OrleansGrainTeardownStragglerTest.GrainLifetimeCallbacks_AfterActivationIsInvalid_…Monolith.Test.CrossProcessChangeFeedTest.AWriteInOneProcess_ReachesTheOtherProcessesLiveMirror_…Orleans.Test.OrleansChatHistoryTest.ColdStart_AgentSeesAllPreviousMessages(#2291)AI.Test.AgentToolCancellationTest.GetVersions_ParkedOnAStalledStore_UnwindsWhenTheRoundIsCancelledOrleans.Test.OrleansNodeChangePropagationTest.Delegation_NodeChanges_PropagateFromSubThreadFutuRe.Test.FutuReAnalysisTest.EuropeRe_AnnualReport_EmbeddedCharts_ShouldRenderViaPathResolutionMonolith.Test.PandasExplorerLayoutAreaTest.Explorer_NoPythonNode_RendersToolbarAndDegrades…MemexTemplate.Test.MemexTemplateGeneratorTest.*(7 tests, one branch — not a flake, a broken branch)Monolith.Test.MeshNodeLanguageServiceTest.ScriptCompletions_FilterByTypedPrefix_…(#2616)Distinct-branch count is the flake discriminator, not occurrence count. Rows 12–18 fail 5× each on one branch — that is a broken branch, correctly reported. Row 1 fails on 19 branches — that is a ledger/gate problem hitting everyone.
Ranks 2, 4 and 6 already have landed fixes on
origin/main(verified by grepping the code onto main, not by PR title):LiveQueryHandoffDropTest→4bdcd5392+001a68e04(LiveQueryHandoffDropTest fails post-warmup-fix (e2080c4fa) — new information per #2338's own ask #2377), landed 08-26. All 20 occurrences are on 08-26; zero since.PartitionRootActivationTest→2d12824a4/dc3e889e9+042ae4bfd(Flake: OrleansMeshTests.HubWorksAfterDisposal — shard-0 Orleans suite is red across unrelated PRs #2346), landed 08-26.CrossProcessChangeFeedTest→639fc2aca/123a78868(CrossProcessChangeFeedTest intermittently reproduces the PRE-FIX signature it was written to detect #2008), landed 08-26.So the ranking is dominated by bursts that were found and fixed. Ranking on the full window over-weights them. The list that actually costs re-runs now is:
Still failing on/after 08-27:
FutuReAnalysisTest.EuropeRe_AnnualReport_EmbeddedCharts_ShouldRenderViaPathResolutionOrleansGrainTeardownStragglerTest.GrainLifetimeCallbacks_AfterActivationIsInvalid_…MeshNodeLanguageServiceTest.ScriptCompletions_FilterByTypedPrefix_…(#2616)AgentToolCancellationTest.GetVersions_ParkedOnAStalledStore_…PartitionRootActivationTest.BarePartitionPath_…DataContextInitFaultedTest.FaultedInit_ReachesTerminalFailedState_AndAnswersFastOrleansNodeChangePropagationTest.Delegation_NodeChanges_PropagateFromSubThreadExit-code census (deduped per job)
exit=1 TESTFAILexit=1 MASKED(host died after streaming results)exit=124 TIMEOUT— 12Threading.Test, 9Hosting.Monolith.Test (part N/2), 1PluginCatalog.Testexit=139 SIGNAL(#613) — 2 Orleans, 2 FutuReHosting.Monolith.Testtimeout population". My first pass appeared to refute that — I greppedMeshWeaver.Hosting.Monolith.Test exit=124and got zero. That was my bug: since sharding, the label isMeshWeaver.Hosting.Monolith.Test (part 2/2), so the exact-string grep silently returned nothing. Corrected count is 9. Anyone re-running this sweep must match the(part N/M)suffix or they will conclude a live population is extinct.Also worth naming: the single largest timeout bucket is
MeshWeaver.Threading.Test(12), not Monolith — a burst confined to 08-25/08-26, entirely on shard 5, zero since. Not #890 (different project, no NRE in those logs). Unattributed; I am not filing it as it appears self-resolved, but ifThreading.Testshard-5 timeouts return, this is the prior.What I recommend follows from the numbers
Data and scripts are reproducible from the Actions API; method is described above so anyone can re-run it.
rbuergi commentedon Aug 29, 2026
Addendum: the cross-repo entry, and the theme that runs through all of it
Module bundle (MeshWeaver.Markdown.Export)— RESOLVED, and it was never a flakeFolding in the one the ranking above does not cover (it lives in Plugins'
Plugin Catalog CI, not this workflow). Two occurrences on two unrelated PRs — Plugins #854 (aMeshWeaver.Hosting.PostgreSql-only change) and #860 — both onPixelFidelityOfferedOnlyWhenAvailableTest.MarkdownExportDialog_NeverOffersPixelFidelity, 60 s observable timeout against an 864 ms – 1.06 s local runtime.Two hypotheses were on the table: runner starvation in the 40-way matrix, or a genuine race. It was the race, and it was a product bug.
ExportDocumentLayoutArea.RenderExportdidPost(...)and thenhub.Observe(delivery).Subscribe(...)— the overload that registers the response subject after the request is in flight. An unmatched correlation id is dropped (HandleCallbacks→RESPONSE_ARRIVED_NO_SUBJECT→delivery.Processed()), so the enriched control never arrives and the dialog stays on its seed forever. The request targets the hub's own address, the shortest and most often synchronous response path there is.The discriminator was inside the failing class itself, and it is worth reusing: the sibling test waits for a value the seed already has, so a lost reply cannot fail it; the failing test waits for
NodeName, which only the reply can supply. Starvation would have hit both. Only one failed. That asymmetry ruled out hypothesis 1 without needing a green-on-main control — which, as noted, does not exist for that job.Fixed and merged as Plugins #872, with a ratchet guard that reds on the reverted source. Real users could hit the same lost reply.
The theme: four of the five things I looked at were diagnostics that could not be read
This is the part I would carry forward from the whole sweep, because it is why the flake population is expensive out of proportion to its size.
BELOW-ROSLYNwas unearned — 9 of 9 verdicts (PR #2663)IoPoolDrain: pool '{PoolName}' …ILoggerafterMesh.Dispose(); 0 of 294 dispose windows carry any ILogger line (PR #2665)DynamicTypePreWarmer, false, and it retired a live axis without testing itPlus the one I introduced myself and had to correct: grepping
Hosting.Monolith.Test exit=124returned zero because sharding renamed the label to(part N/M), which would have "proved" a live population extinct.Every one of these makes a real defect look like a flake, or a flake look like nothing at all. That is the mechanism behind this issue's central complaint — the attribution cost, not the failure count. My concrete recommendation for #1384 stands and sharpens:
ILogger. It is structurally unreadable there. This is a general rule, not a MeshNodeLanguageServiceTest teardown DIRTY — one pooled I/O leaf still ignores cancellation after #2598 (#2665 names the pool on the next occurrence) #2616 detail.Status of what I touched
src/MeshWeaver.AI.Orleans.Test(~28 files, ~7,200 lines) runs in no CI job at all since 2026-08-28, thoughai-engine/SKILL.md:26claims it runs inportal-hosts. "No failures since 08-26" for anything in that suite is not evidence of health. Details on #2291.rbuergi commentedon Aug 29, 2026
The Monolith.Test host CRASHES are a distinct population member, and two of four name one test
Four
MeshWeaver.Hosting.Monolith.Testhost deaths today, on diffs that cannot reach the host (aconcurrency:block, a Python script, workflow/script-only changes). The classifier reportsMASKED: host crashed AFTER streaming results— the trx records fewer failures than the exit code implies, so the process died during or after teardown having already reported.Two of the four name the same test, ~2 h apart, on core #2690 (run
33268909413) and #2693 (run33274198115):Why that name is worth taking literally
The test's own remarks describe a mechanism that is hard to distinguish from the crash:
So the test deliberately leaves a read outstanding against an already-disposed stream, then tears the owner down. That is the exact shape of a use-after-dispose — and the hypothesis worth testing is that the scenario escapes its own scope rather than an unrelated straggler killing the shard.
What rules out the obvious explanation
MonolithMeshTestBasealready joins the disposal signal —WaitWithProgressAsyncawaitsMesh.DisposalCompleted(bounded,ConfigureAwait(false)), thenIoPoolRegistry.DrainAll()cancels and joins offloaded I/O, thenAsyncDisposeQueue.DrainAsync. So "the test base does not wait for disposal" is disproven; whatever survives is surviving past a join that exists.The teardown's own comment says why that is possible:
DisposalCompletedcovers the hub's action blocks and message round-trips, and explicitly not ThreadPool leaves. A read parked against a disposed data-source stream is neither an action block nor an IIoPool leaf, so it may be covered by neither.The cheap next step
TestPhaseTraceis written for exactly this: the post-dispose phases are traced individually (DISPOSE_IOPOOL_DRAIN_START/DONEwith pool names and residual counts,DISPOSE_ASYNC_QUEUE_DRAINED clean=…), and the comments note it is all a killed host leaves behind — the mesh's log sink stops capturing atMesh.Dispose(), so ILogger output is structurally absent. Reading that file from either crashed run should say which phase the process died in, and a named residual pool would distinguish "a leaf we know about" from "the outstanding read".Recording rather than chasing: this is core's, and the two runs above are the evidence to start from. Flagging it here because a host crash is materially worse than a flaky assertion — it kills the shard and masks every other result in it, so it is the population member most likely to be hiding others.
Separately: the ~30 s assertion flakes in this population are one root, filed as #2700 (a hand-written 30 s bound repeated ~2,679 times). These crashes are not that — different mechanism, and they should not be folded into it.
rbuergi commentedon Aug 30, 2026
Another data point (2026-08-30, PR #2807 — a docs-only diff, three markdown files): run 33322993649, shard 1,
MeshWeaver.Hosting.Monolith.Test (part 1/2) exit=124 TIMEOUT (8m wall-clock cap hit). Not a hang: the host started 16:41:21Z and tests were still STARTING at 16:48:52Z (CodeEditRecompileTest.CompileTool_UnchangedSources_ForcesFreshRun…) when the cap fell at 16:49:21Z — 184 class args in that part, and the main runs before and after it were green. Capacity on the shard, not a test defect; the diff cannot reach the project.rbuergi commentedon Aug 30, 2026
Correction to my comment above: the sweep session identified this as #2747 —
Hosting.Monolith.Testsplit in 2 puts each half at ~332 s against the 480 s per-project cap (69%), so a slightly slower shard tips it over; PR #2800 splits it into 3 and adds a headroom guard. Not a capacity flake in the abstract — a known long pole with no headroom check.rbuergi commentedon Aug 30, 2026
Closing: both halves of this population were answered, and one of them left the repo
This issue asked for three things. Measured on today's
main:1. "Decide whether live-model assertions belong in the gating suite at all" — moot in this repo
Both named carriers are gone from core:
The AI suite moved to
MeshWeaver.Pluginswith the engine (#2276). The one core survivor isMeshWeaver.Portal.E2E.Test/ChatDelegationTest.cs, which is an e2e test and not part of the gating unit suite this issue was about. So the question — "can an assertion whose outcome depends on an external API's latency and chunking gate a merge?" — is now aMeshWeaver.Pluginsquestion and should be asked there if it still bites.2. "Quantify the rest" — done, and it produced named issues rather than a vague population
This was the substantive ask, and it was carried out at length:
canary=BELOW-ROSLYNverdict, 54 of 371 test-failure occurrences (14.6%) across 37 distinct blamed test names — oneEmitpoisoning a whole process. Tonight it also gained its first earned rather than inferred verdict, and the finding that the fault is total and permanent, not intermittent.ChecksReportedsignal), Race: RolePlay/Story install has a CreateOrUpdateNodeRequest outstanding at its own hub disposal (the gate now reports it Inconclusive, #2649) #2454 (a handler that entered, exitedProcessedand never replied — fix in fix: an upsert always answers (#2454), and CD gains a force-rebuild button (#2622) #2825).3. "Do not fix any of these by raising a timeout or a retry count" — held, and vindicated
Worth recording, because it was tested this evening and nearly failed. A 6× teardown regression in one project presented as four failing tests with a rotating cast, all green in isolation. Raising the enclosing timeout would have turned the suite green and shipped it. What found it was timing the whole project against
main, one variable at a time. The rule in this issue is why that was not done.Why close rather than keep as an umbrella
Its own diagnosis — "these are not three coincidences, it is a population" — was right, and the value was in converting the population into attributable, individually-actionable issues. That has happened. What remains open (#890, #2700) are those issues, each with measurements this one could not have carried. Keeping a bucket open above them adds a second place to look without adding information.
Reopen if a new unattributed population appears — the technique recorded here (bucket every failed shard over a window, rank by test) is the reusable artifact and it worked twice.
fix(sync): report the give-up count from one read, not three (#1384 r…
Merge pull request #3040 from Systemorph/fix/1384-resync-wedge-surfaces
fix: the resync give-up must not count an ack that overtook its own a…