Starved cross-silo source discovery produces FALSE CompileErrors that the bake gate reads as image regressions #1218

Activity

1. rbuergi

   added a commit that references this issue on Aug 11, 2026

   Merge pull request #1226 from Systemorph/fix/1218-unestablished-sourc…

   ...

   9398713
2. rbuergi commented on Aug 12, 2026

   

   rbuergi

   on Aug 12, 2026

   ContributorAuthor

   More actions

   Status check — proposals 1 and 2 are done on main; only proposal 3 remains.

   Shipped in PR #1226 (+827 lines, incl. SourceDiscoveryUnavailableTest.cs and SourceSnapshotEstablishmentTest.cs):
   src/MeshWeaver.Graph/Configuration/SourceSnapshot.cs (IsEstablished, Established/Empty/Unavailable) plus MeshNodeCompilationService.cs:580-582, 596-680, 777-861 (SnapshotSourcesCore, RaceSourceSnapshot, DirectSourceProbe) → SourceDiscoveryUnavailableException → CompilationStatus.Unavailable → PreWarmStatus.TimedOut → non-gating. So an unestablished source set can no longer be reported as a CompileError, which was this issue's core ask.

   Proposal 3 is not done. The 60 s SubscribeRequest timeouts (~68 min on a 237-type sweep) are inherent to the activation-path bake, and batch bake — which performs no activations — is still hard-coded off at src/MeshWeaver.Hosting/DynamicTypePreWarmerHostedService.cs:190 (var batchBake = false;). That flip is open PR #1238.

   Narrowing this issue to proposal 3; it closes when #1238 lands. #1213 closed as covered by the same fix.
3. rbuergi commented on Aug 12, 2026

   

   rbuergi

   on Aug 12, 2026

   Last edited by rbuergi

   ContributorAuthor

   More actions

   🚨 Do not close this on the starvation fix alone — there is a second, independent route to the same phantom CS0246.

   Found while fixing #1253 (PR #1279). DirectSourceProbe is the other leg of RaceSourceSnapshot; the cached leg it races already reads as System, so the asymmetry was invisible.

   MeshService.Query stamps request.UserId at call time, inside a Defer that DelaySubscription has already moved to a ThreadPool tick. So the stamp is null and GetEffectiveUserId resolves that to Anonymous.

   Why that is worse than it sounds: an empty Anonymous answer was harmless — the probe's own .Where filtered it out. But a partial Anonymous answer is both Established and non-empty, so it wins the race, and Roslyn is handed a short source set. That is exactly this issue's phantom CS0246 signature, arriving by a different mechanism than starved cross-silo discovery — and it specifically fits the cross-partition shared= detail reported here, since those are the sources most likely to be partially readable as Anonymous.

   So the accounting for this issue is now:

   • proposals 1 and 2 — done (PR fix(compile): never compile against a source set that could not be established (#1218) #1226, SourceSnapshot + SourceDiscoveryUnavailableException)
   • proposal 3 — done (PR Batch bake is the default again for a gated bake — verified on all three portals #1238, batch bake default-on)
   • this second route — NOT done, and not covered by either

   PR #1279 fixes the five compile-path read sites for #1253's symptom, but the Anonymous-wins-the-race behaviour in DirectSourceProbe deserves its own verification here before this issue is closed.

   (Reposting: an earlier version of this comment lost its final sentence to a shell-quoting slip.)
4. rbuergi

   closed this as completedin #1279on Aug 12, 2026
5. rbuergi commented on Aug 12, 2026

   

   rbuergi

   on Aug 12, 2026

   ContributorAuthor

   More actions

   Verified: the second route is fully covered by #1279 (merged, de54cd002). No further change needed for it.

   Re-read on merged main, MeshNodeCompilationService.DirectSourceProbe now reads:

   return Observable
       // Observable.Using INSIDE DelaySubscription, never around it: the resource factory
       // then runs on the delayed subscribe — the very tick that calls mesh.Query …
       .Using(() => accessService?.ImpersonateAsSystem() ?? Disposable.Empty, _ => probe)
       .Catch(…)
       .DelaySubscription(subscriptionDelay ?? SourceStallProbeDelay);

   The ordering is the part that matters, and it is right: DelaySubscription schedules the
   subscribe onto a ThreadPool tick, the Using resource factory runs inside that subscribe, and
   the per-leg Observable.Defer(() => mesh.Query…) calls run inside the same subscribe — which is
   exactly when MeshService.Query stamps request.UserId from AccessService. The stamp is now
   System instead of null→Anonymous, so the probe can no longer produce an ESTABLISHED-but-PARTIAL
   set that wins RaceSourceSnapshot, and both legs of the race read under the same identity.

   Swept for a third route to the same shape and found none:

   • the cached leg (GetSourceCollection) already impersonates System;
   • the sourcesOverride route bypasses the race entirely (SourceSnapshot.Established(override)),
     but its only producer is NodeTypeBatchBake.BakeOne, which is System-scoped end-to-end and
     already refuses an unestablished empty set;
   • NodeTypeCompilationHelpers.RunCompile passes sourcesOverride: null and wraps the compile in
     Observable.Using(ImpersonateAsSystem, …).

   One gap worth recording, not code: no test pins the probe's identity. #1279's
   CompileIncludeSystemScopeTest asserts the @@-include read, not that the probe reads as System —
   so a regression here would be silent again. Cheap to add later (the probe would need internal
   visibility plus a two-namespace RLS fixture where Anonymous can read one source location and not
   the other); flagging rather than doing it, since the fix itself is in.