Repository navigation
Compile discovery: read sources directly instead of through the live synced query - #690
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR changes the NodeType compile “source snapshot” path to prefer a direct IMeshService.Query read (point-in-time snapshot) and only fall back to the cached synced query when the direct read yields no sources, addressing long stalls caused by missed Initial emissions in synced-query subscriptions on large meshes.
Changes:
- Reordered compile source discovery to attempt a direct mesh read first, then fall back to the cached synced source query.
- Updated the direct-probe helper to accept a configurable subscription delay and adjusted logging/error messaging for the new flow.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+508
to
511
| return DirectSourceProbe(ntDef, selfPath, TimeSpan.Zero) | ||
| .Concat(Observable.Defer(() => ResolveSources(ntDef, selfPath, null).Take(1))) | ||
| .FirstAsync() | ||
| .Timeout(bound) |
Comment on lines
557
to
+561
| /// compile never issues it. Failures complete EMPTY rather than erroring — this is a fallback | ||
| /// racing a primary, and it must never be the thing that fails a compile. | ||
| /// </summary> | ||
| private IObservable<IEnumerable<MeshNode>> DirectSourceProbe( | ||
| NodeTypeDefinition? ntDef, string selfPath) | ||
| NodeTypeDefinition? ntDef, string selfPath, TimeSpan? subscriptionDelay = null) |
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
Test Results (shard 1)781 tests - 403 781 ✅ - 402 2m 30s ⏱️ - 5m 6s Results for commit d800d75. ± Comparison against base commit 799cc1f. This pull request removes 403 tests.♻️ This comment has been updated with latest results. |
Contributor
Test Results 60 files - 1 60 suites - 1 30m 51s ⏱️ - 3m 25s Results for commit d800d75. ± Comparison against base commit 799cc1f. This pull request removes 403 tests.♻️ This comment has been updated with latest results. |
…synced query
A compile needs a POINT-IN-TIME snapshot of its sources. It was taking that snapshot
from the cached SYNCED query, which exists to deliver LIVE updates — a guarantee a
compile does not need and, on a large mesh, pays 45 seconds for.
Measured on memex, same box, same data, same four queries a Store/Coupon compile
issues:
via IMeshService.Query (what MCP `search` uses) ~0.25s counts 16 / 4 / 1
via the cached synced query (what compiles used) 45.20s
45.20s is one SyncStreamOptions.HeartbeatInterval to within 200ms: the synced
subscription misses its Initial and idles until the periodic heartbeat re-delivers.
Roslyn itself is 2.83s, so 97% of every compile was that wait — and types then
crossed the 60s settle window, which is how every plugin root came to serve the
"did not settle" fallback on 2026-07-27.
Everything else was eliminated by experiment BEFORE writing this, which is the part
the earlier attempts skipped:
- the compile path itself 0ms discovery on a fresh mesh (issue #686)
- topology 0ms with ONE silo and with TWO (real AdoNet cluster,
both silos Active) — silo count is not the cause
- scale / ~90 schemas the same queries answer in 0.25s ON memex
- the heartbeat schedule #684 shipped and measured no change (48s → 57s)
- pre-warming enabled on memex, degraded 7 of 16 roots, rolled back
So: direct read FIRST, synced query kept only as a fallback when the direct read
finds nothing. Never the other way round. #682 kept the cache primary and raced a
delayed probe against it — that probe never fired once in production, and this
ordering is exactly what removes the dependency on winning a race.
647/647 Graph, 8/8 Hosting.Monolith compile/pre-warm integration tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rbuergi
force-pushed
the
fix/discovery-direct-query
branch
from
July 29, 2026 00:02
d800d75 to
e004eed
Compare
rbuergi
added a commit
that referenced
this pull request
Jul 29, 2026
Second half of unbreaking main. #690 (de86b03) made MeshWeaver.Hosting.Monolith.Test exceed CI's 6m wall-clock cap (exit=124) on every run since it landed. CI reports that cap as "likely fixture/init hang". It is not a hang — the suite completes; it just got slower. Measured on one box, CompileLeafStabilityTest: before #690 (cached query primary) 4s #690 as merged (probe, then cached) 52s ← 13× this change (true race) 4s full suite: 7m50s -> 3m13s (CI cap 6m) WHY IT GOT SLOWER. #690 chained the direct probe ahead of the synced query with Concat. Concat is SEQUENTIAL: the synced query is not even subscribed until the probe COMPLETES. So every compile paid the probe's full latency up front — and the probe cannot answer faster than SourceProbeQuietWindow, because each leg has to watch its chunked Initial go quiet before it may emit. That is a ~1s floor per compile that the previous code never paid: the probe was DelaySubscription'd and on a healthy mesh never ran at all. That floor lands on far more calls than "compiles". The disk cache is keyed by effectiveLastModified, which comes from DiscoverSourceMaxLastModified -> SnapshotSources — so source discovery runs BEFORE the cache can be consulted, and a cache HIT pays it too. In a suite that touches the same NodeTypes repeatedly the cache-hit path is the common case, which is why a 1s floor cost minutes. Merge starts both sources NOW and FirstAsync takes whichever answers first, so: * healthy synced query -> answers immediately, probe latency never on the path * stalled synced query -> probe answers in a read + quiet window, not 45s #690's production win is kept; neither source is privileged, so the fast path does not depend on winning a race (#690's stated objection to #682). Merging is safe precisely because of #690's own `.Where(count > 0)`: a probe that finds nothing stays SILENT and completes, so it can never beat the cached query with an empty set and compile against no sources. That filter is also the likelier reason #682's probe "never fired" in production — not the 3s delay alone. NOT done: shrinking SourceProbeQuietWindow to 50ms, which also takes 52s -> 7s. That window exists so a chunked Initial can settle; the code's own comment warns a partial source set "compiles WRONG (missing files -> phantom errors)". Nor raising the 6m cap — the cap did not change, the compile path got 13x slower. Verified: Hosting.Monolith.Test 419/421 in 3m13s (2 skipped, 0 failed), including CompileSourceSnapshotWedgeTest which covers the stalled-snapshot path #690 exists to fix; Graph builds clean under -c Release -warnaserror. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 29, 2026
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The measurement this is built on
Same box (memex), same data, the exact four queries a
Store/Couponcompile issues:IMeshService.Query— what MCPsearchuses45.20 s is one
SyncStreamOptions.HeartbeatIntervalto within 200 ms: the synced subscription misses itsInitialand idles until the periodic heartbeat re-delivers. Roslyn itself is 2.83 s, so 97% of every compile was that wait — and types then crossed the 60 s settle window, which is how every plugin root came to serve the "did not settle" fallback on 2026-07-27.A compile needs a point-in-time snapshot. It was taking it from a mechanism whose purpose is live updates — a guarantee it does not need and, on a large mesh, pays 45 s for.
What this changes
Direct read first, synced query kept only as a fallback when the direct read finds nothing. Never the other way round.
#682 kept the cache primary and raced a delayed probe against it. That probe never fired once in production. This ordering removes the dependency on winning a race — the fast path is unconditional.
Everything eliminated by experiment first
This is the part my earlier attempts skipped, and why I trust this one more:
Testing
647/647 Graph; 8/8 Hosting.Monolith compile + pre-warm integration tests (these drive real Roslyn compiles through this exact path).
Deploy check
After rollout, the verdict is one measurement: a cold
Store/Couponcompile should fall from ~48 s to a few seconds, and core #685's phase line will print the discovery figure directly —⏱ Source discovery took Xms. If that number is still tens of seconds, this is wrong too and should be reverted rather than explained.🤖 Generated with Claude Code