Repository navigation
Compile source snapshot: an empty cached answer can no longer win the race (#612) - #892
Merged
Merged
Conversation
… race Fixes #612 (sub-case b). The compile's point-in-time source snapshot raced the authoritative DirectSourceProbe against the cached synced source query with Merge(...).FirstAsync(). The cached query replays its latest set SYNCHRONOUSLY on subscribe while the probe cannot answer before its ~1s chunk quiet window — so a cached query that had latched EMPTY (a missed source-create update under CI load) always won with an empty set. The compile then consumed ZERO sources, the configuration lambda failed CS0103 deterministically, CompilationStatus=Error parked the type — and every retry, including the un-parking RequestedReleaseAt re-trigger, re-ran the same race and re-failed identically. That is the 'RE-TRIGGER: STILL STUCK (sub-case b: persistent clobber / dead subscription)' CI wedge: not a dead subscription — a persistent deterministic re-failure. Fix: RaceSourceSnapshot combinator — first NON-EMPTY answer from either leg wins immediately (healthy cache keeps zero probe latency, the #690 guard); EMPTY settles only by consensus of both legs, so source-less configuration-only NodeTypes still compile; a never-emitting leg still falls to the outer Timeout. Also: non-Roslyn compile aborts now surface exception TYPE + STACK on the activity log, the node's CompilationError names the exception class, and the terminal LogWarning carries the exception object — the recurring CI first-compile NRE (PR-884 run 31153582625) was undiagnosable because every sink reduced it to .Message. The next occurrence will name its throw site. Tests (CodeEditRecompileTest): deterministic pin of the race (SourceSnapshot_StaleEmptyCachedAnswer_MustNotOutraceDirectProbe), empty- consensus + healthy-cache contracts, and an end-to-end repro of the exact CI wedge (FirstCompileError_LatchedEmptySourceQuery_FreshTriggerMustRecompileFromDirectRead) — fail-without/pass-with verified; compile-family suite 24/24 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a CI wedge in NodeType compilation where an empty cached synced-query source snapshot could always “win” a Merge(...).FirstAsync() race against the authoritative direct probe, leading to deterministic zero-source compiles, CompilationStatus=Error, and unrecoverable re-trigger loops under load.
Changes:
- Replaces the
Merge(...).FirstAsync()source snapshot race with a newRaceSourceSnapshotcombinator that only allows non-empty results to win immediately; empty can only settle by consensus (both legs complete without sources). - Improves non-Roslyn compile failure diagnosability by recording exception type + stack in the activity log, including the exception type in
CompilationError, and logging the exception object to preserve stacks. - Adds deterministic unit + end-to-end tests that pin the stale-empty-cache race and the exact “first compile error → re-trigger still stuck” CI wedge signature; upgrades
WaitForLatestReleasediagnostics to distinguish dead mirror vs dead watcher vs deterministic re-failure.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| test/MeshWeaver.Hosting.Monolith.Test/CodeEditRecompileTest.cs | Adds deterministic pins for the source-snapshot race and an end-to-end repro; enhances WaitForLatestRelease diagnostics with post-trigger probing. |
| src/MeshWeaver.Graph/Configuration/NodeTypeCompilationHelpers.cs | Improves compile failure logging: activity log now includes exception type + stack for non-Roslyn aborts; node error summary includes exception type; logger receives exception object. |
| src/MeshWeaver.Graph/Configuration/MeshNodeCompilationService.cs | Introduces RaceSourceSnapshot and updates SnapshotSources to prevent empty cached answers from outracing the direct probe. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| public static class EmptyRaceLayoutAreas | ||
| { | ||
| public static UiControl Overview(LayoutAreaHost host, RenderingContext _) | ||
| => Controls.Html("<div id='marker'>EMPTYRACE_V1</div>"); |
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #612 (sub-case b) — the CI wedge where a first compile ends
Errorunder load and no re-trigger ever recovers, taking out whole shards (runs 30004790036, 31153582625).Mechanism:
MeshNodeCompilationService.SnapshotSourcesraced the authoritativeDirectSourceProbeagainst the cached synced source query withMerge(...).FirstAsync(). The cached query replays its latest set synchronously on subscribe, while the probe cannot answer before its ~1 s chunk quiet window — so a cached query that had latched empty (a missed source-create update under CI load) always won the race. The compile consumed zero sources → deterministic CS0103 →Status=Error+ park — and every retry, including the un-parkingRequestedReleaseAtre-trigger, re-ran the same race and re-failed identically. "Sub-case b: persistent clobber / dead subscription" was actually a persistent deterministic re-failure; the watcher, trigger chain, and mirror are all healthy (verified in both CI dumps: triggers stamped, fresh compile activities minted,Matched Code nodes (0)in the compile log).Changes
RaceSourceSnapshotcombinator (unit-testable): first NON-EMPTY answer from either leg wins immediately (healthy cache keeps zero probe latency — the Compile discovery: read sources directly instead of through the live synced query #690 regression guard); EMPTY settles only by consensus of both legs (source-less configuration-only NodeTypes still compile); a never-emitting leg still falls to the outerTimeout. The probe leg already refused empty wins — this applies the same rule to the cached leg.{TypeName}: {Message}on the node'sCompilationError, and the exception object on the terminalLogWarning— the recurring process-wide CI compile NRE (CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890) was undiagnosable because every sink reduced it to.Message; its next occurrence will name its throw site.WaitForLatestReleasediagnostics upgraded: pre-trigger MIRROR printsCompilationError; new POST-trigger MIRROR probe discriminates dead-mirror vs dead-watcher vs deterministic re-failure.Tests
SourceSnapshot_StaleEmptyCachedAnswer_MustNotOutraceDirectProbe— deterministic pin of the race (fails on old semantics).SourceSnapshot_ProbeSilentAndCachedEmpty_SettlesEmptyConsensus/SourceSnapshot_HealthyCachedNonEmpty_WinsWithoutWaitingForProbe— companion contracts.FirstCompileError_LatchedEmptySourceQuery_FreshTriggerMustRecompileFromDirectRead— end-to-end repro of the exact CI wedge (fails on old semantics with the CI-identical terminal picture).Verification
Fail-without/pass-with confirmed for both pins;
CodeEditRecompileTest11/11 and the compile-family suite 24/24 green; re-verified 11/11 after merging current main;dotnet build -c Release -warnaserrorclean on MeshWeaver.Graph and Hosting.Monolith.Test.Related follow-ups filed: #890 (distinct process-wide compile NRE), #891 (watchers die silently on own-stream OnError).
No What's New entry: internal reliability fix; the user-visible effect (compile no longer wedges) rides the bug fix.
🤖 Generated with Claude Code