CI wedge: CodeEditRecompileTest StaleBarrierType first compile ends Error, re-trigger not picked up (sub-case b) — shard timeout exit=124 #612
Description
Occurred on PR #611 CI run 30004790036 (shard 1), on a branch whose diff is causally excluded from the failure (SelfUpdateHostedService is registered by no test mesh, and the new test never ran in the killed process per the shard phase trace).
Evidence from the shard-1 _meshweaver-test-trace.log:
- Monolith.Test started 12:07:34;
CodeEditRecompileTestbegan wedging at its second CTOR (12:08:04 → 12:09:19, first 75s watchdog kill); every subsequent class CTOR accounted for until the 12:15:34 wall-clock kill (exit=124, no trx). - The test's own triage diagnostics:
WaitForLatestRelease stuck for TestData/StaleBarrierType: known=(null), MIRROR: Status=Error, Latest=(null), INDEX Release children: [(none)], RE-TRIGGER: STILL STUCK after re-trigger (sub-case b: persistent clobber / dead subscription). - So the FIRST compile of StaleBarrierType ended
CompilationStatus.Errorunder CI load, and the forced re-trigger (RequestedReleaseAt+Force) produced nothing for 25s — release watcher dead/clobbered. That wedged the whole class (6 tests × 50–75s) plusReleaseRequestWatcherHighWaterTest, blowing the 6-min cap.
This is the compile-wedge family previously addressed in #580/#583/#584 — this occurrence shows a residual order/load-sensitive race (sub-case b) that will intermittently red any branch, including main. Needs its own pinned repro + root-cause fix; starting point: test/MeshWeaver.Hosting.Monolith.Test/CodeEditRecompileTest.cs (WaitForLatestRelease, ~lines 880–920), first-compile-Error path and the release-watcher subscription lifecycle.
🤖 Generated with Claude Code
Activity
rbuergi commentedon Aug 7, 2026
Fresh occurrence with a new data point — PR #884 run 31153582625 shard 2 (2026-08-07 06:37-06:40Z), branch causally excluded (diff = ClrMD leak-test files only, and those tests passed on the shard):
Note
ReqAt=andHandled=are EMPTY after a re-trigger that wroteRequestedReleaseAtthrough the same mirror handle — i.e. the re-trigger write never became visible on the MIRROR at all. That narrows sub-case b: it is not (only) the owner-side release watcher that is dead — the cross-hub mirror handle the test (and any client) uses appears to have stopped applying/receiving state after the first compile endedError. The wedge then took out the whole class plus OverlaySelfHealInstanceRecycleTest, LinkedInTelemetryImportTest and PandasExplorerLayoutAreaTest on the shard (five failures, same run).🤖 Generated with Claude Code
rbuergi commentedon Aug 7, 2026
Root cause found and fixed — correcting my earlier comment.
The 'dead mirror' reading of the PR-884 occurrence was wrong: the diagnostic's MIRROR probe runs BEFORE the re-trigger (both CI occurrences show identical empty pre-trigger values), so empty
ReqAt=/Handled=proves nothing about the mirror. The trigger chain is fully alive — the re-trigger stamps, un-parks, and dispatches a REAL fresh compile every time.The actual mechanism (sub-case b = deterministic re-failure, not a dead subscription):
MeshNodeCompilationService.SnapshotSourcesraced the authoritativeDirectSourceProbeagainst the cached synced source query withMerge(...).FirstAsync(). The cached query replays its latest set synchronously on subscribe, while the probe can never answer before its ~1s chunk quiet window — so a cached query that latched empty (a missed source-create update under CI load: the stale-synced-query class) always won the race with an empty set. The compile consumed ZERO sources → the configuration lambda's CS0103 is deterministic →Status=Error+ park. Every retry re-ran the same race and re-failed identically — permanently wedged, exactly the CI picture (Matched Code nodes (0)in both occurrences' compile logs). The probe leg already refused to emit empty for this very reason; the cached leg had no such guard.Fix:
RaceSourceSnapshot— first NON-EMPTY answer wins immediately; EMPTY only by consensus of both legs (source-less configuration-only types still compile); never-emitting legs still hit the outer Timeout. Pinned by a deterministic unit test + an end-to-end repro of the exact wedge (latched-empty cache → first compile Error → source created → fresh trigger must produce a release), fail-without/pass-with verified; compile-family suite 24/24.Second, distinct defect observed in the same PR-884 shard (every compile in the process — including the deliberately-invalid pre-Roslyn
BrokenType— failing with a bare NRE message): currently undiagnosable because every sink reduced the fault to.Message. The fix also makes non-Roslyn aborts carry exception TYPE + STACK to the activity log / node error / logger, so the next occurrence names its throw site. Filed separately.PR incoming.
🤖 Generated with Claude Code
Merge pull request #892 from Systemorph/fix/612-empty-source-snapshot…