Repository navigation
CI: 8 distinct tests each failed exactly once on 2026-08-05, all pass locally — environment-wide instability, not per-test flakiness #826
Description
Across the PR runs of 2026-08-05 (mostly on #823/#825 and the #784/#794 era), eight different tests each failed exactly once in CI and passed on every local run against the same commit and on CI re-run:
EducationStartExerciseTest(also seen as a real regression on Node CRUD off the router: handling ops for your own node ≠ being a work queue #794 — the one-time flakes elsewhere are distinct occurrences)ExportImportAccessControlTestInboxToolIntegrationTestCrossPartitionRealUserQueryTests.UnpinnedStructuredQuery(twice, different runs)OrleansDynamicCompilationTestFutuReAnalysisTest.EuropeReCessionLayoutAreaTest.Cession_Trace_HubConfigurationContentCollections.NodeHubContentCollectionTest.CollectionNamedArea_UrlDecodesEncodedPathSegments
The pattern — a different test each time, one occurrence each, never reproducible locally or on re-run — points at shard-environment instability (CPU starvation on loaded runners, timing-sensitive Within(...) budgets, shared-fixture contention), not at eight independently flaky tests. Per the project's own standard, a flake is a defect: either the assertion budgets don't fit the runner envelope, or there is a genuine timing race that only the slow environment exposes.
Suggested angles, in order of expected yield:
- Correlate the eight failures with runner load / shard duration — every failure was on a shard already past ~6–10 min.
- The dispatch flake-repro from the activity-writes work reproduced only at DEFAULT log level (Debug masks the race) — the same masking may apply here; a targeted soak of these eight at default level on a throttled runner would separate budget-starvation from real races.
- If budget-starvation: scale
Within(...)budgets by an environment factor on CI rather than raising them globally (local runs should stay tight so real regressions surface fast).
Not blocking anything today — filed so the pattern is recorded while the data is fresh instead of being rediscovered one flake at a time.
🤖 Generated with Claude Code
Activity
rbuergi commentedon Aug 7, 2026
Status update from the flake-cluster campaign (2026-08-07):
nodeType:Markdown limit:10query asserted a specific seeded row that concurrent sibling writes page out (459b440 + 7a9b159/7f989fb41). Core test suite is flaking under parallel shards — two unrelated tests failed docs-only PRs within an hour #834 closed on this.[WATCHDOG-SUSPEND]when wall vs monotonic diverge — a suspended/stalled runner window can no longer masquerade as a test hang.Remaining seven one-time failures (EducationStartExercise, ExportImportAccessControl, InboxToolIntegration, OrleansDynamicCompilation, FutuReAnalysis.EuropeRe, CessionLayoutArea, ContentCollections UrlDecodes) stay open under this issue pending the load-correlation/soak angle; the compile-wedge family has its own root-cause thread in #612.
🤖 Generated with Claude Code
rbuergi commentedon Aug 8, 2026
Another sample of the pattern, 2026-08-08, PR #927 run 31238512895 (branch synced with main at 1dc26f8): shard 4
MeshWeaver.AI.Test.CrossHubPatchAtomicityTest.ConcurrentCrossHubPatches_DoNotDropAQueuedMessageand shard 1MeshWeaver.Query.Test.ObservableQueryIntegrationTests.ScopeDescendants_NotifiesOnAllDescendantChangeseach failed exactly once; both pass locally on the same head (Release, first attempt). The PR's diff touches only NodeTypeCompilationHelpers watcher subscriptions + ActivityControlPlaneExtensions — no overlap with either test's machinery. Consistent with this issue's environment-wide-instability shape rather than per-test flakiness.rbuergi commentedon Aug 8, 2026
Second sample today, PR #924 run 31240189056 (shard 5):
MeshWeaver.NodeOperations.Testran 85/85 passed, 0 failed, 25.9s — then the host exited 2 in teardown ('completed-only results were published'). The tests-pass-host-exits-nonzero teardown-straggler signature, no failing test to triage. Diff under test touches only the C# file serializer + one Hosting.Test file — no overlap.rbuergi commentedon Aug 8, 2026
Second hit of the SAME test today — this one is a specific flake, not the 'each failed exactly once' shape this issue tracks.
MeshWeaver.AI.Test.CrossHubPatchAtomicityTest.ConcurrentCrossHubPatches_DoNotDropAQueuedMessagehas now failed on two unrelated PRs within ~2 hours:Both pass locally on the same head (Release, first attempt — I ran the #927 one directly). Neither diff goes anywhere near cross-hub patching. Two occurrences of one named test on unrelated diffs is a reproducible race in that test or in the cross-hub patch path it exercises, and it deserves its own issue rather than being folded into this one's environment-instability bucket — filing separately if it recurs a third time. Recording here so the pattern is visible alongside the other samples.
rbuergi commentedon Aug 8, 2026
A SECOND named test is now recurring across unrelated PRs — the pattern is two specific races, not broad instability.
MeshWeaver.Query.Test.ObservableQueryIntegrationTests.ScopeDescendants_NotifiesOnAllDescendantChanges:MeshWeaver.AI.Test.CrossHubPatchAtomicityTest.ConcurrentCrossHubPatches_DoNotDropAQueuedMessage(reported above):So today produced two tests × two occurrences each, always the same shard per test, on four diffs that touch none of the machinery involved, and both pass locally on the same head. That is the signature of two reproducible races, not the 'eight distinct tests, once each' environment wobble this issue was opened for.
Both are actively costing merges — every hit blocks a green PR and burns a CI cycle. Suggest splitting them into their own issues (one per test) so each can be root-caused against its own subsystem: cross-hub patch queuing for the first, subtree change-notification for the second. Happy to file both if that is the preferred shape.
rbuergi commentedon Aug 8, 2026
Split the two recurring named tests into their own issues, each with the evidence and a starting analysis: #945 (CrossHubPatchAtomicityTest — shard 4, PRs #927/#930) and #946 (ObservableQueryIntegrationTests.ScopeDescendants — shard 1, PRs #927/#940).
For #945 I ruled out the obvious suspect: the owner-side merge IS atomic (
ApplyMeshNodePatchInTurndoes read-merge-write insideprimary.Updateon the action-block turn), so sibling keys cannot be lost there. What remains is the bounded ACK path in the same function — a 20 s wait for the commit emission plus a 10 s flush, both reachable on a saturated shard — and whether an expired ack turns into a retry that re-diffs against a stale mirror base.Neither reproduces locally (18 cores, both pass comfortably), so the next step is instrumenting those paths rather than more re-runs.
This issue keeps its original scope: the genuinely one-off, environment-wide instability from 2026-08-05.
rbuergi commentedon Aug 8, 2026
A third cross-PR flake locus today:
MeshWeaver.Hosting.Monolith.Test.StaleActivationSeedRollbackTest.PostRecycleWrite_LandsAboveTheDurableVersion_AndDoesNotRollTheStoreBack, failing on two unrelated PRs — #932 (partition-id validation) and #933 (token accounting). Neither diff touches version seeding or activation.Failure shape:
Expected 2 to be greater than 7000 because the node's revision counter is monotonic across activations— i.e. the post-recycle write landed at version 2, so the hub reactivated on a stale snapshot instead of durable truth.Main itself is healthy (last four completed main runs all green), so this is not a main regression. Worth noting the failure is in the same family as #590/#648/#667 (activation seed vs durable truth) which were fixed earlier today — this may be a residual window in that area rather than pure test flakiness, and the assertion is a genuine invariant (a monotonic revision counter), not an over-strict proxy.
Syncing both PRs onto current main and re-running; recording here so the pattern is visible if it recurs. Today's tally of cross-PR flake loci: #945 (cross-hub patch, shard 4), #946 (subtree notification — ROOT-CAUSED AND FIXED, PR #955), #618 (Orleans shard 0, two tests), and now this one.
rbuergi commentedon Aug 8, 2026
Flake tally across the last 40
MeshWeaver Build and Testruns (2026-08-08)11 of 40 runs failed (~27%). Tallying the failing test names across those 11 runs separates the recurring flake from one-off incidents — the raw count is misleading because two incidents contributed ~15 names between them.
Hosting.Monolith.Test.StaleActivationSeedRollbackTest.PostRecycleWrite_LandsAboveTheDurableVersion_AndDoesNotRollTheStoreBackHosting.Monolith.Test.StaleActivationDurableFirstSeedTest.PostRecycleWrite_WhenDurableSeedLagsTheRoutingCache_LandsAboveDurableTruthHosting.PostgreSql.Test.ProvisioningBoundaryValidationTest.EnsurePartitionProvisioned_ErrorsLoud_OnMalformedIdHosting.Orleans.TestassemblyQuery.TestQueryAsyncIntegrationTests/ObservableQueryIntegrationTestsGroupMembershipLiveRevocationTests,MeshHubDisposalLeakTest,TwoWaySyncTest.NoOpUpdate_…,CrossHubPatchAtomicityTest.ConcurrentCrossHubPatches_…What is NOT a flake (excluded from the above conclusion)
TypeLoadExceptionon a leakedDynamicNode_*assembly poisoningReferencedAssemblyProviderat everyTestClusterstartup in that process. That is Orleans test host: corrupt/evicted DynamicNode assembly poisons every TestCluster.DeployAsync (TypeLoadException 'format is invalid') — whole-project cascade #618, and it re-greened on the next run after a merge with main.ProvisioningBoundaryValidationTest×2 is a REAL failure, not a flake — aStartsWith('_')exemption I added toOwnsPartitionProvisioningValidatorcontradicts the test's expectation that_accessis junk. Being fixed on PR A junk id could still provision a schema the router would never route to #932; the underlying cause looks to be a caller creating aSpacewith id_Access.The actual recurring flake
StaleActivation*— 4 occurrences across 2 sibling tests, both asserting the same invariant (a post-recycle write must land above the durable version and not roll the store back). Nothing else recurs more than twice. This is the one worth root-causing; the remaining singletons are a thin tail.Why this matters beyond the tally
At a ~27% run failure rate a clean PR needs multiple attempts to go green by luck alone. PR #935 (a Blazor
Style/Classforwarding fix) failed three consecutive runs on three different unrelated tests —CrossHubPatchAtomicityTest, thenMeshHubDisposalLeakTest, thenTwoWaySyncTest— none of which its diff touches. That is the cost being paid on every PR, not an occasional annoyance.rbuergi commentedon Aug 8, 2026
A fifth distinct one-off today:
TrackedChangeProjectionTest— and it is characterized, not just observedMeshWeaver.Content.Test.TrackedChangeProjectionTest.RevertingAProjectedChange_PutsTheTextBack_AndIsItselfVersionedfailed once on shard 4 (PR #932, run31273996116).It was investigated rather than assumed-flaky, and ruled out as caused by that PR four independent ways:
MeshWeaver.Content.Testproject is green locally on that exact commit — 249 passed / 1 skipped / 0 failed, the same 250 CI ran.mainruns are green, including the commit merged into the branch.test/MeshWeaver.Content.Test/is touched by the diff, and the test's partition istest— so the PR's newIsValidPartitionSegmentbootstrap gate is a no-op for it.The shape — this is the useful part
FromHistoryreturned 0 changes afterWaitForVersionshad already observed ≥2 rows. So the test waited on the right condition, saw it satisfied, and the subsequent projection read still came back empty. That is a read-your-writes / projection-lag race, not a timing-too-short problem — widening a bound would not have helped, and the wait it already does is the correct kind (condition, not sleep).Worth noting the family resemblance to the two
StaleActivation*failures that dominate this issue: all three are "a write is durably visible by one read path and not yet by another." Whoever picks up theStaleActivation*race should check whether this is the same defect seen through a different projection before treating it as separate work.Disposition
Left untouched deliberately — no retry, no widened bound, no skip. PR #932 stays blocked on it rather than being merged red.
rbuergi commentedon Aug 8, 2026
Correction to my previous comment —
TrackedChangeProjectionTestis NOT a flakeI characterized
RevertingAProjectedChange_PutsTheTextBack_AndIsItselfVersionedabove as a one-off projection race in the same family asStaleActivation*. That was wrong, and the better answer is much more actionable.It is pre-existing on
mainand DETERMINISTIC in isolation. Run alone on a clean detached worktree atorigin/main(621558c4b) it fails identically every time — a byte-identical assertion failure atTrackedChangeProjectionTest.cs(188). It passes only when its sibling test runs first, which is why it normally hides: in a full-project run the ordering that rescues it is the common case, so it surfaces only when sharding or filtering happens to isolate it.So this is a test-order dependence (shared/leaked state between the two tests, or a setup step the sibling performs that this test relies on), not a load-sensitive race. It does not belong in the flake tally I posted above, and it should not be waiting on the
StaleActivation*investigation — it has a deterministic repro today:The assertion is on
ChangeProjection.FromHistory(...)immediately afterWaitForVersions(path, 2)has observed ≥2 version rows: the rows land, but the projection computes no change from them.This currently blocks PR #932, which was verified clean two independent ways — the gate it adds was instrumented and never fired during this test (its first path segment is
test, which passes validation), and the failure reproduces onmainwithout that gate existing at all.rbuergi commentedon Aug 8, 2026
Root-caused and fixed in #932 — confirming your correction: this is not a flake.
TrackedChangeProjectionTest.RevertingAProjectedChange_PutsTheTextBack_AndIsItselfVersionedfails deterministically when run alone and passes only when its sibling runs first. Instrumented both orderings:The version rows are identical. The sibling leaks nothing the test consumes — it warms the mesh so reconciliation lands before the read. The difference is which node snapshot the assertion projected from.
WaitForVersionswaits on the version store; the node is a different store and had not caught up (stream.Updatereturns the optimistic node, storage stamps the reconciled version a moment later). The helper took whichever snapshot was cached via a bare.Where(n => n is not null).Take(1)— the pre-edit v1.ChangeProjection.FromHistoryprojects only versions strictly older than the node it is handed, sooldercame back empty →[]→ "expected a single item, found 0", with every version row already in the store.Not a product bug:
FromHistorydocuments that the caller re-subscribes when the version moves, andCollaborativeMarkdownViewdoes exactly that underSwitch, so the live view self-corrects within an instant. Only a one-shot assertion is exposed.Fixed by reading the node at the newest recorded version (
NodeAtVersion) — the same discipline the test already used for its later revert assertion. No sleep, no widened bound, no retry, no reordering. Now passes run alone; fullMeshWeaver.Content.Testis 249 passed / 1 skipped / 0 failed.Worth noting the general shape for other suites: waiting on version rows does not imply the node has caught up. A bare
Take(1)onGetMeshNodeStream(path)right after an edit can hand you the pre-edit snapshot.rbuergi commentedon Aug 8, 2026
The recurring
StaleActivation*pair is root-caused — it is a real product write race, not a test defect. PR #968 (open, not merged).The sharper of the two signatures is the one that names it:
StaleActivationDurableFirstSeedTestfails at the line immediately after its own out-of-bandStorage.Write(… Version = 9000)— that write acks, and the very next durable read returns the create-time version 1. The store was genuinely rolled back.StaleActivationSeedRollbackTest'sExpected 2 to be greater than 7000is the same event one step later: a hub reactivating on the rolled-back row seeds v1 and mints 2.Cause.
MonotonicWriteGuardStorageAdapterrecorded its per-path high-water mark in a.Do(Observe)hung off the completed inner write — i.e. after the row was already mutated. Every backend publishesIStorageAdapter.Changesfrom inside the write (InMemoryStorageAdapter.Write:_nodes[path] = nodethen_changes.OnNext(...)), and the framework's own topology puts writers on that feed (every per-node hub reconciles its own node from it). A stale snapshot presented in that window carries a version at-or-below the stale mark, skips verification entirely, and overwrites the newer row. The next write is then accepted for a perfectly good reason — verification confirms the rolled-back row really is older — so acked writes die silently. That is theVersion=12/sk-v6 → Version=2/sk-v0production shape the guard exists to prevent.Fix: claim the mark before mutating the row. Two lines.
Verification is a deterministic pin, not the flaky test passing:
MonotonicWriteGuardWindowTestenters the window by construction (it subscribes to the framework's ownIStorageAdapter.Changesand issues the stale write from that notification — no sleeps, no timeouts, no retry). Fail-without →[durable] v1/stale-snapshotwith the stale write reporting no error; pass-with → refused,[durable] v7000/durable-advance. Both run on the tree merged with current main.Not proven, stated plainly: the CI flake itself never reproduced locally (25 warm-mesh iterations, 20 cold-mesh iterations, and a
DOTNET_PROCESSOR_COUNT=2run, all with an instrumented innermost storage adapter that saw no sub-advance write). The cause comes from the CI evidence plus the guard's own decision table — in-process, a v1 write can only land over a v9000 row via that window — and is pinned by the deterministic test. Which component issues the stale write in CI is still unidentified; the fix does not depend on it.Related but distinct: this is not the
TrackedChangeProjectionTestshape (WaitForVersionswaiting on the version store while the node lives in another). That one is a read-visibility skew across two stores where nothing is lost. This is a write defect in a single store — the test'sReadDurablegoes down the sameIStorageAdapterthe write did, and the stale write was accepted, so the row was mutated backward rather than merely observed stale. Two defects, not one.Residual, filed as not-fixed here: the mark is per-process, so the guard still cannot see another replica's writes, and
PostgreSqlStorageAdapter's upsert has no version condition (ON CONFLICT … DO UPDATE SET version = EXCLUDED.version, unconditional). On the multi-replica portals nothing enforces monotonicity for a fresh process's first write to a path. The airtight fix is a store-level compare-and-set; it needs a Postgres instance to verify and deserves its own issue.Sequence the pin's durable read on the forced stale write, so it cann…
rbuergi commentedon Aug 9, 2026
Disposition sweep of the original eight — 4 explained, 4 still occurring and named
Swept 600
MeshWeaver Build and Testruns = 640 run-attempts, 2026-08-04T12:35Z → 2026-08-09T00:50Z, reading every attempt rather than run conclusions (39 runs in the window carry a re-run, so a conclusion-based sweep would have missed attempt-1 reds that later went green).Coverage was verified, not assumed. 480 of the 640 attempts actually executed the test shards (the rest reused an already-green tree); per-shard executions were 467–477, so effectively every test project ran ~470–480 times in the window. All 247 failing
Run tests (shard N)job logs fetched non-empty. The only unfetchable logs are the 373Test Results*reporter jobs, which carry no test output — no shard log silently read as clean.Aggregate for context: 156 / 480 attempts (32%) had ≥1 failing test; 19 more had a shard red with zero failing tests (host/teardown exit — the "tests pass, host exits non-zero" signature).
Two incidents excluded first — a raw name tally would be badly wrong
31137723323(2026-08-07T01:20)Collection fixture type 'MeshWeaver.Hosting.PostgreSql.Test.PostgreSqlFixture' threw— the #977 family31171377098(2026-08-07T10:45,feat/log-watch-triage)YamlDotNet.Core.SemanticErrorException : While scanning a plain scalar value, found invalid mappingwhile activatingIWorkspace— a branch-local content breakBetween them these two supply 5 of the 10 raw
ExportImportAccessControlnames, 13 of the 15 rawInboxToolIntegrationnames, 3 of the 4 rawOrleansDynamicCompilationnames and 3 of the 10 rawUnpinnedStructuredQuerynames. Every occurrence counted below is an isolated singleton — 1 or 2 failing tests in the entire run.Per-test disposition
EducationStartExerciseTest.StartExercise_Learner_…fix/mesh-crud-off-router(30923573112,30939056795,31001137070attempts 1 and 2). A re-run reproduced it, so it never had the "passes on re-run" shape. Failure is deterministic:Access denied: user 'learner-user' lacks Create permission on 'TestCourse/Module1/Ex1'. Zero hits on any other branch in 404 test-executing attempts since 2026-08-05.ExportImportAccessControlTestmain:31000712491(08-05),31078761170(08-06,docs/deployment-skills),31153699826(08-07),31170758886(08-07),31279216108(2026-08-08T21:24,main). Signature is constant:Expected the observable to emit a value within 60s… The observable emitted nothing at all.InboxToolIntegrationTest.CheckInbox_TwoCallsBackToBack_SecondReturnsEmpty31001137070(08-05),31244333631(2026-08-08T06:37).Expected "(no new messages)" to contain "only one"— the first CheckInbox saw nothing. 127 attempts since the last hit.CrossPartitionRealUserQueryTests.UnpinnedStructuredQuery31084037396last). The three later "hits" are phantoms of thePostgreSqlFixturethrow above. Zero genuine occurrences in the 370 test-executing attempts since.OrleansDynamicCompilationTest31024795628(ColdStart_CompileVia… : Expected value to be true because Object reference not set to an instance of an object). The other 3 are phantoms of theTwoSiloCacheUpdateFixturethrow — that family is #618 (closed). Clean across 392 test-executing attempts since.FutuReAnalysisTest.EuropeRe31033460533(08-05,…Search_ShouldReturn8LoBs),31244442523(2026-08-08T06:40,…Overview_ShouldRender). The recent one is a leaked callback:left Observe subscriptions pending past the Quiescing budget … 1 pending callback(s) after 2.00s: …=CreateNodeRequest@mesh/…(2738ms).CessionLayoutAreaTest30944058400(08-04,MotorXL_LayoutArea_ReturnsContent,main),31050462577(08-05,Cession_Trace_HubConfiguration). Same leaked-callback / Quiescing-budget signature as #6 — not an independent defect. Clean across 385 test-executing attempts since.NodeHubContentCollectionTest.CollectionNamedArea_UrlDecodesEncodedPathSegments31001137070,31050462577(08-05),31111317689(08-06),31244331775(2026-08-08T06:37). Signature: the area emits aMarkdownControlwith emptyMarkdownand never the real content within 30 s.Members added to this issue after it was opened, for completeness:
StaleActivation*(both siblings)31279179600@ 2026-08-08T21:23, i.e. before #968 merged (23:01Z). Zero in the 24 test-executing attempts since. Residual = #971.CrossHubPatchAtomicityTest31278416685@ 21:04, before #975 merged (23:45Z). Zero in the 30 attempts since.ObservableQueryIntegrationTests.ScopeDescendantsTrackedChangeProjectionTestNodeAuthorshipPersistenceTests(×4)31276902526(08-08T20:26) and31286186569(2026-08-09T00:27), plus the 412-name fixture throw.MeshHubDisposalLeakTest31277556232,main). Measured unaffected by #613/#967, per the recorded lead (it hand-rolls teardown, sobase.DisposeAsync()never runsioPools.DrainAll()).The four that remain are two signatures, not four
Observecallback past the 2 s Quiescing budget:FutuReAnalysisTest.EuropeRe(live),CessionLayoutAreaTest(dormant since 08-05, identical message). ACreateNodeRequestthat takes 2 738 ms to answer.ExportImportAccessControlTest(nothing emitted at all in 60 s),InboxToolIntegrationTest(firstCheckInboxreturns(no new messages)),NodeHubContentCollectionTest.UrlDecodes(MarkdownControlwith emptyMarkdown). This is the same family as everything already root-caused on this issue — Post-recycle writes rolled the store back: the monotonic guard claimed its mark after the row, not before #968, A junk id could still provision a schema the router would never route to #932, A view catching up got handed a snapshot older than what it had already applied #975 were all "durably visible by one path, not yet by another."The issue's own hypothesis, tested
The filed hypothesis was runner-load / environment-wide instability. Measured, no signal: runs started within ±5 min of a failing attempt — median 3.0, mean 3.4; identical to the whole population (median 3.0, mean 3.4). Run-start concurrency is a proxy for runner-pool contention, not a direct measure, but it is the one the hypothesis predicted would separate them and it does not.
The stronger evidence is the campaign's own record: every member of this issue that recurred turned out to be a real defect — #834, #968, #975, #955, #932, #977, #618. None survived as environment wobble. "60 s and nothing emitted at all" and "a
CreateNodeRequestpending 2 738 ms" are not budgets fitting an envelope badly; they are things not completing.Recommendation: do not close yet — and retire the framing when you do
The "eight distinct tests, once each, environment instability" premise is disproven, so the issue should not stay open on that basis. But four named members are still failing unrelated PRs and
main, and closing now would drop them:ExportImportAccessControlTest— the most active; hitmainat 2026-08-08T21:24 (31279216108).NodeHubContentCollectionTest.CollectionNamedArea_UrlDecodesEncodedPathSegments— 4 hits, 3 different shards.FutuReAnalysisTest.EuropeRe— quiescing-budget leak; would also cover the dormantCessionLayoutAreaTest.InboxToolIntegrationTest.CheckInbox_TwoCallsBackToBack_SecondReturnsEmpty— 2 hits.Cleanest shape, following the precedent this issue already set with #945/#946: file these four (or three — #3 and the Cession sibling are one signature) as their own issues, then close #826 with the disposition table above as its record.
MeshHubDisposalLeakTestneeds a home too — it is unfixed, recurring, and explicitly not covered by #967.Also worth noting the post-fix windows for #968 and #975 are still short — 24 and 30 test-executing attempts, both clean. Consistent with fixed; not yet proof. Another day of
mainruns will settle it.🤖 Generated with Claude Code
rbuergi commentedon Aug 9, 2026
Closing — the premise is disproven and every member now has a disposition
A sweep of 600 runs / 640 run-attempts (2026-08-04 → 2026-08-09) settled this. Method notes, because they changed the answer: every attempt was read rather than run conclusions (39 runs in the window carry a re-run, so a conclusion-based sweep misses attempt-1 reds), all 247 failing shard logs were verified non-empty, and two mass incidents had to be excluded before any tally meant anything — run
31137723323(412 names, one cause:Collection fixture type 'PostgreSqlFixture' threw) and run31171377098(616 names across all six shards, one cause: a branch-localYamlDotNet … invalid mapping). Those two alone supplied most of the raw name count for three members.The hypothesis in the title is dead
This was filed as "8 tests each failed exactly once, all pass locally" — i.e. environment wobble. That is wrong, and measurably so: runs started within ±5 min of a failing attempt come out at median 3.0 / mean 3.4 — identical to the whole population. And every member that ever recurred turned out to be a real defect: #834, #968, #975, #955, #932, #977, #618. None was a wobble.
Disposition of all 8
CrossPartitionRealUserQueryTestsEducationStartExerciseTestAccess denied … lacks Create permissionStaleActivation*CrossHubPatchAtomicityTestOrleansDynamicCompilationTestExportImportAccessControlTestNodeHubContentCollectionTest.CollectionNamedArea_UrlDecodesEncodedPathSegmentsInboxToolIntegrationTest.CheckInbox_TwoCallsBackToBack_SecondReturnsEmptyFutuReAnalysisTest.EuropeRe(+ dormantCessionLayoutAreaTest)The four live residues collapse to two signatures, which is why they are filed as two issues rather than four tickets or one vague cluster:
MeshHubDisposalLeakTestis filed there too — 5 occurrences on 2026-08-08 alone including onmain, and explicitly not fixed by Unload node ALCs at the END of teardown, not before the I/O drain (#613) #967, with the recorded lead that the test hand-rolls its teardown and so never runsDrainAll().One honest caveat
The post-fix windows for #968 and #975 are only 24 and 30 test-executing attempts, both clean. That is consistent with fixed — it is not yet proof. If either name reappears, reopen the specific issue rather than this one.
Closing: the cluster hypothesis is disproven, and nothing here is now untracked.