fix(bake): an empty source set needs a positive witness — #3663's three-hour readiness stall by rbuergi · Pull Request #3698 · Systemorph/MeshWeaver

rbuergi · 2026-09-08T08:24:27Z

The stall, and the one number that explains it

BatchBake logs the size of every source-discovery pass. Every boot on memex (memex.systemorph.com — the namespace #3663 opens with), 09-07 22:40Z → 09-08 07:37Z:

boot (UTC)	Code nodes resolved	`compileErrors=`	readiness
09-07 22:40:02	1236	2	granted
09-07 22:43:42	1236	2	granted
09-08 00:31:22	1145	6	REFUSED
09-08 03:32:37	1237	2	granted
09-08 03:35:30	1237	2	granted
09-08 07:34:44	1241	2	granted
09-08 07:36:57	1241	2	granted

Denominator: 29 boots across both portals in 15 hours; exactly one refused, and three of the six that granted readiness ran the same image. 2 is that portal's standing baseline of already-broken types, which MarkOutcome correctly refuses to let gate. The four extra are the entire Doc partition's NodeTypes — 4 of 4, not 4 of 40. memex-cloud never shows it (6837 → 6841 nodes; compileErrors= steady at 4–5 either side of 8059); its simultaneous stall was the 10-second pending_module_activation check over 687 set records, i.e. #3676, a different defect conflated into the same issue.

The defect

On the short pass those four types matched ZERO sources. Roslyn was handed each type's generated provider file alone and emitted a completely honest CS0246 'CessionData' could not be found about a type declared in that very NodeType's own Source/. The gate did exactly its job on a genuine CompileError.

The invariant written to stop precisely this (Assemble → SourceDiscoveryFailedException, #1216, whose doc comment even quotes the identical CS0246 'ScopeLibrary' shape) did not fire:

if (matched.Count == 0 && pending.DeclaresSources && !pending.SourcesKnownDeleted)

DeclaresSources is false for a type on the DEFAULT queries — which is very nearly the whole population, exactly as DynamicTypePreWarmer.ClassifyCompileFailure had already had to be corrected to say in #1391 ("an empty Sources does not mean configuration-only — it means uses the DEFAULT queries"). The same conjunct sat here in the opposite direction. Verified on the live node: Doc/DataMesh/SocialMedia/Profile has no sources and a populated currentSourceVersions naming its 2 files — the mesh's own record said "two source files" while the pass matched none, and nothing read it.

The change

NodeTypeBatchBake.DiscoveryUnestablished is now the one predicate, and it asks a positive witness rather than the absence of one. PendingType carries KnownSourceCount (CurrentSourceVersions?.Count) in place of the boolean, so all three shapes stay distinguished:

0 (explicitly empty) — the mesh agrees; established, classifies NoSources, still must not gate (Content-broken NodeTypes no longer stall rollouts; a baking silo no longer captures traffic #1204);
n > 0 — the record contradicts the zero match; unestablished, batch abandoned for the activation-driven sweep;
null (absent) — no witness; falls back to the pre-existing declared-queries rule, unchanged.

It cannot launder a real regression: one matched source makes the set established, the compile runs, and a genuine failure gates as before.

Why the stall lasted three hours — worth recording

RetractRegression is level-triggered on the type recompiling, and nothing recompiled (the content never changed, so the park registry's source-change retry never fired). What ended it was the kubelet: startupProbe is failureThreshold: 1080 × periodSeconds: 10 = exactly 3 h, and the container died at 03:29:52Z having started at 00:30:23Z (2 h 59 m 29 s). #3676 did not clear it — falsified, not assumed: the recovery is at 03:33Z, 47 minutes before #3676 merged, on 3.0.0-ci.8059, which cannot contain a later commit. Had #3676 been the cause the recovery would have had to postdate that merge and arrive on a later image.

Verification

EmptySourceSetNeedsAWitnessTest (4): the regression case; Content-broken NodeTypes no longer stall rollouts; a baking silo no longer captures traffic #1204's corroborated-empty case in both declared and default form; the absent-snapshot fallback in both directions; and three shapes of "any matched source is always established".
Control arm: the same test class against the pre-🚨 Every rollout since 3.0.0-ci.8059 stalls at the bake gate: #3628's MeshNodeContentDegradedException fires during the NodeType bake for instance nodes typed by their own NodeType's source (4 Doc types 'regressed') #3663 predicate — 1 of 4 fails (the regression case, Expected: True / Actual: False), the other 3 pin unchanged semantics on both.
MeshWeaver.Hosting.Test 461/461 with the fix; 460/461 with the control arm, the single failure being that one case.
DocumentationLinkIntegrityTest + WhatsNewEntryIntegrityTest 2/2.
dotnet build -c Release -warnaserror, one project per invocation, 0 Warning(s) 0 Error(s): MeshWeaver.Hosting, MeshWeaver.Documentation, MeshWeaver.Hosting.Monolith, MeshWeaver.Hosting.Orleans, MeshWeaver.Hosting.Test, MeshWeaver.Documentation.Test.

Left open deliberately (not this PR)

The same boot logged ShippedPrebuiltBundles: bundle Doc.zip: adopted 4/4 prebuilt assembly(ies) at 00:31:08, and ten seconds later the sweep enumerated 204 of 209 … need building — 5 already on the share and recompiled them anyway. Adoption seeding and the sweep's store probe disagree on a cold boot; that is what put these four on the compile path at all. It costs time, not correctness, and is recorded in the doc page rather than fixed here.

Doc: Doc/Architecture/SourceSetEstablishment — the three shapes of zero, the measurement above, and the two LogQL queries to re-measure with. What's New (Fix): 2026-09-08-a-short-read-no-longer-condemns-a-healthy-node-type.

Pairs-with: none — no public type or member removed; the changed record is private, the new predicate is internal.
Implementers: none — no interface member added.
Mirror-sync: none — no localization catalog key added or reworded.

🤖 Generated with Claude Code

…ee-hour readiness stall The batched bake's source-discovery invariant only fired for NodeTypes that DECLARE their own source queries. Very nearly none do — the defaults (namespace:{path}/Source scope:subtree nodeType:Code) are how the population is authored, exactly as ClassifyCompileFailure had to be corrected to say in #1391 — so the check that exists to stop a short discovery pass becoming a compile verdict was switched off for almost every type in a real mesh. Measured on memex.systemorph.com: the boot at 2026-09-08 00:31:22Z resolved 1145 Code nodes where its six neighbours on that portal resolved 1236/1236/1237/1237/ 1241/1241 — three of them on the same image. All four Doc-partition NodeTypes matched ZERO sources, Roslyn was handed each type's generated provider file alone, and emitted a genuine-looking CS0246 about a type declared in that NodeType's own Source/. The gate recorded four regressions and refused readiness for the startup probe's full 3 h (1080 x 10 s) until the kubelet restarted the container. Denominator: 29 boots across both portals in 15 hours, one refusal. DiscoveryUnestablished now consults the positive witness the record already carries: CurrentSourceVersions with n > 0 entries contradicts a zero match, so the pass answers "I don't know" and the batch falls back to the activation-driven sweep. All three shapes of the snapshot stay distinguished — explicitly empty still classifies as NoSources and still must not gate (#1204); absent still falls back to the pre-existing declared-queries rule. No bound widened, no retry added, the gate's Regressed rule untouched. Doc: Doc/Architecture/SourceSetEstablishment. What's New (Fix): 2026-09-08-a-short-read-no-longer-condemns-a-healthy-node-type. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The invariant fixes what a short pass may CONCLUDE; it does not explain why that one pass was short, and the page must not read as if it did. Records the leading candidate (RunQuery's one-second quiet window is the only completion signal available — QueryResultChange<T> carries no terminal marker), the measurement that would settle it (per-query change-event count and largest inter-chunk gap, short pass vs complete pass on the same portal), and the reason widening the window is the wrong response. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

🟢 Approval recommended

The change is narrowly scoped, test-backed, and the new source-set invariant aligns with existing CurrentSourceVersions semantics to prevent false regressions without weakening genuine compile gating.

Pull request overview

This PR fixes a rollout-readiness stall caused by batched NodeType source discovery occasionally resolving an empty/short source set and incorrectly treating the resulting compile errors as true image regressions. It makes “zero matched sources” require a positive witness from the NodeType’s persisted CurrentSourceVersions snapshot before concluding the type genuinely has no code.

Changes:

Refines batch-bake “empty source set” detection by introducing KnownSourceCount (from CurrentSourceVersions?.Count) and centralizing the invariant in NodeTypeBatchBake.DiscoveryUnestablished.
Adds a focused regression test suite covering the three snapshot shapes (absent/empty/populated) and ensuring any non-zero match still yields an established set.
Documents the incident and the invariant (Architecture page + What’s New entry) and links it from the Architecture index.

File summaries

File	Description
test/MeshWeaver.Hosting.Test/EmptySourceSetNeedsAWitnessTest.cs	Adds regression/behavioral tests for the new “empty source set needs a witness” invariant.
src/MeshWeaver.Hosting/NodeTypeBatchBake.cs	Implements the new `KnownSourceCount` witness and the `DiscoveryUnestablished` predicate; updates batch failure logging/error text accordingly.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-08-a-short-read-no-longer-condemns-a-healthy-node-type.md	Adds a user-facing Fix entry describing the symptom and the corrected behavior.
src/MeshWeaver.Documentation/Data/Architecture/SourceSetEstablishment.md	Adds an architecture note capturing the invariant, the three “zero” shapes, and re-measurement queries.
src/MeshWeaver.Documentation/Data/Architecture.md	Registers the new architecture page in the “Node types” section.

Review details

Files reviewed: 5/5 changed files
Comments generated: 0
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

A zero needs a denominator. count_over_time over the same 15-hour window: memex 1065 'NodeType bake regressed' lines from ONE stream (cbztk boot 0, one per 10-second readiness probe for 2h58m), memex-cloud 0. The non-zero arm is what makes the zero arm mean something — across two portals in fifteen hours, exactly one container boot ever put the bake gate into Regressed, and the simultaneous memex-cloud stall did not involve this gate at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-08T08:38:49Z