Compile: stop source discovery idling 90s on a missed Initial, and pin recursive compilation by rbuergi · Pull Request #682 · Systemorph/MeshWeaver

rbuergi · 2026-07-27T21:32:24Z

The profile

Straight from the Store/Plugin compile activity node that preceded today's outage:

20:46:02.486  Invoking compiler…
20:47:32.673  Source query: … (all nine, one timestamp)
20:47:35.288  Compiled assembly written

phase	time
source discovery	90.19 s
Roslyn compiling 30 files	2.6 s
write + release	0.2 s

97% of the compile was waiting, not compiling. And 90.19 s is 2 × SyncStreamOptions.HeartbeatInterval (45 s).

Root cause — a lost wakeup, not a lock

When the cached synced source query's subscription misses its Initial, nothing re-requests it. The read makes no progress until the sync stream's periodic heartbeat re-delivers content. Miss it twice and a 3-second compile takes a minute and a half — which is exactly how Store/Plugin blew the 60 s settle window and left every plugin root serving the "did not settle" fallback.

This failure mode has bitten before: one screen above the fix, an existing comment documents the identity-loss variant — "the source reads run unauthenticated and stall until the next 45s sync-stream heartbeat — the ~42s freeze." Same signature, different trigger.

The fix

Race the cached query against a direct, uncached mesh read of the same expanded queries:

the probe is subscription-delayed (3 s), so a healthy compile — cache hit or one cold storage read — never issues it and pays nothing;
a stalled compile answers in about a read instead of 45 or 90 seconds;
Amb takes whichever speaks first and drops the other;
when the probe wins it logs at Warning — "compiles are slow" was invisible until someone read microsecond timestamps out of an activity node.

The probe accumulates chunks rather than taking the first emission: a query's Initial arrives chunked, and a partial source set compiles wrong — the missing file surfaces as a phantom Roslyn error pointing at code that is perfectly fine.

Fully reactive: Defer / CombineLatest / Scan / Throttle / Take / Select / Catch. No await, no ToTask, no blocking bridge.

Recursive compilation tests

The incident ran through the cross-type path: Store/Plugin declares shared=@Store/Coupon/Source, shared=@Store/Order/Source, shared=@Store/BillingProfile/Source — which is what turns discovery into nine live queries. RecursiveCompilationTest pins both halves:

Expansion — a shared reference yields both the folder node and its subtree, for every referenced type; deterministic; no self-recursion; a non-shared type does not leak in; matching agrees with expansion (the compiler and the Sources menu use the same resolver).
Chunk accumulation — chunked Initial accumulates instead of truncating, re-delivery is idempotent, removal drops, empty/null changes are inert.

Worth stating plainly: these pass on main. The recursion is correctly specified — what failed is the execution of those queries stalling. The tests are there so a regression in either half is caught, and so the next person profiling a slow compile starts from a pinned contract.

Testing

644/644 Graph tests pass.

Follow-ups not in this PR

Why the Initial is missed in the first place (cold subscription racing the source-update burst) — this makes it survivable, it does not remove the race.
Per-stage compile timing in the activity log, so this is measurable without microsecond arithmetic.
A readiness gate / deploy-time precompile — worth revisiting after this, on real numbers: a 3 s compile changes that calculus completely.

🤖 Generated with Claude Code

…n recursive compilation Profiled from the Store/Plugin compile activity log that preceded today's outage: 20:46:02.486 Invoking compiler… 20:47:32.673 Source query: … (all nine, one timestamp) 20:47:35.288 Compiled assembly written 90.19s of source discovery. 2.6s of Roslyn. 97% of the compile was waiting, not compiling — and 90.19s is 2 x SyncStreamOptions.HeartbeatInterval (45s). That is a LOST WAKEUP, not a lock: when the cached synced source query's subscription misses its Initial, nothing re-requests it, so the read makes no progress until the sync stream's periodic heartbeat re-delivers content. Miss it twice and a 3-second compile takes a minute and a half — which is how a type blew the 60s settle window and left every plugin root serving the "did not settle" fallback. The same signature is already documented one screen up in this file ("stall until the next 45s sync-stream heartbeat — the ~42s freeze") for an identity-loss variant, so this failure mode has bitten before. Fix: race the cached query against a direct, uncached mesh read of the SAME expanded queries. The probe is subscription-delayed, so a healthy compile — cache hit or one cold storage read — never issues it and pays nothing; a stalled one answers in about a read instead of 45 or 90 seconds. Amb takes whichever speaks first. When the probe wins it says so at Warning, because "compiles are slow" was invisible until someone read microsecond timestamps out of an activity node. The probe ACCUMULATES chunks rather than taking the first emission: a query's Initial arrives chunked, and a partial source set compiles WRONG — the missing file surfaces as a phantom Roslyn error pointing at code that is fine. Fully reactive throughout: Defer/CombineLatest/Scan/Throttle/Take/Select/Catch. No await, no ToTask, no blocking bridge. Also adds RecursiveCompilationTest for the cross-type (shared=@Other/Source) path that this incident ran through — Store/Plugin reaches into three other types, which is what makes discovery nine live queries. It pins expansion (both folder and subtree forms, every referenced type, deterministic, no self-recursion, matching agrees with expansion) and chunk accumulation (chunked Initial accumulates, re-delivery idempotent, removal drops, empty/null inert). Notably these pass on main: the recursion is correctly SPECIFIED — what failed is the execution of those queries stalling. 644/644 Graph tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Self-review of the fix found a way to make things worse than the bug. Every leg of the uncached probe degrades to an empty list on error, so a probe that hit a cold partition or a permission wall would report "no sources", beat the cached query after 3s, and compile the type against NOTHING — a green compile producing an empty assembly, which is strictly worse than the 90s stall it exists to dodge. The probe now stays silent unless it actually found sources; the cached query and the outer Timeout decide from there. Pinned by EmptyDiscovery_IsNotAValidSourceSet. 645/645 Graph tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

rbuergi · 2026-07-27T21:37:22Z

Reproduced on a warm, healthy mesh — one heartbeat, exactly

Triggered a compile of Store/Coupon on memex just now (site healthy, nothing wedged, cache warm). Its activity log:

21:35:47.712  Compile started for Store/Coupon
21:35:47.712  Invoking compiler…
21:36:32.909  Source query: … (all nine, one timestamp)
21:36:35.736  Compiled assembly written
21:36:35.972  end

phase	time
source discovery	45.20 s
Roslyn compiling 23 files	2.83 s

45.20 s is one HeartbeatInterval (45 s), to within 200 ms. The outage compile was 90.19 s — two.

This settles two things:

Not a cold-start artifact. A warm, healthy mesh pays a full heartbeat on every compile. The outage wasn't a freak event; it was this constant plus a 60 s settle window, which any type with enough sources will eventually cross.
It's the cross-type path. Store/Coupon matched 23 Code nodes across four types (Store/Coupon, Store/Plugin, Store/Order, Store/BillingProfile) via shared= — nine expanded queries, which is why RecursiveCompilationTest in this PR pins that expansion.

Expected after this change: the probe answers at ~3 s (its subscription delay) instead of 45 s, so the same compile lands in about 6 s wall-clock — and the Warning line says the probe won, making the underlying lost-wakeup visible instead of silently costing 45 s a time.

I'll post the same measurement taken after deploy.

Copilot

Pull request overview

This PR targets a production compile outage mode where source discovery can stall for ~90s when a cached synced query misses its Initial, by adding a delayed “direct/uncached” probe as a fallback and pinning recursive compilation behavior with new tests.

Changes:

Add a delayed, uncached IMeshService.Query-based source probe raced (Amb) against the cached synced source snapshot to avoid long idle waits on a missed Initial.
Add a pure accumulator (ApplyQueryChange) and a new test suite to pin recursive source-query expansion and chunked-Initial accumulation behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
`src/MeshWeaver.Graph/Configuration/MeshNodeCompilationService.cs`	Races cached source snapshot vs. delayed uncached probe; adds accumulator for chunked query changes.
`test/MeshWeaver.Graph.Test/RecursiveCompilationTest.cs`	Adds tests for shared-source expansion and chunked `Initial` accumulation behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    internal static ImmutableDictionary<string, MeshNode> ApplyQueryChange(
+        ImmutableDictionary<string, MeshNode> acc, QueryResultChange<MeshNode> change)
+    {
+        if (change?.Items is not { Count: > 0 } items)
+            return acc;
+        foreach (var node in items)
+        {
+            if (string.IsNullOrEmpty(node?.Path))
+                continue;
+            acc = change.ChangeType == QueryChangeType.Removed
+                ? acc.Remove(node.Path)
+                : acc.SetItem(node.Path, node);
+        }
+        return acc;
+    }


+        if (queries.Length == 0)
+            return Observable.Empty<IEnumerable<MeshNode>>();
+
+        var mesh = hub.ServiceProvider.GetService<IMeshService>();
+        if (mesh is null)
+            return Observable.Empty<IEnumerable<MeshNode>>();


+                    mesh.Query<MeshNode>(MeshQueryRequest.FromQuery(q))
+                        .Scan(ImmutableDictionary<string, MeshNode>.Empty, ApplyQueryChange)
+                        .Throttle(SourceProbeQuietWindow)
+                        .Take(1)
+                        .Select(map => (IReadOnlyCollection<MeshNode>)map.Values.ToList())
+                        .Catch((Exception _) => Observable.Return<IReadOnlyCollection<MeshNode>>([]))))


+                }))
+            .Catch((Exception _) => Observable.Empty<IEnumerable<MeshNode>>())
+            .DelaySubscription(SourceStallProbeDelay);


github-actions · 2026-07-27T21:49:29Z

Test Results (shard 3)

12 files + 1 12 suites +1 4m 28s ⏱️ + 1m 56s
974 tests +59 970 ✅ +59 4 💤 ±0 0 ❌ ±0
1 037 runs +59 1 033 ✅ +59 4 💤 ±0 0 ❌ ±0