feat(#3703,#3704): publish the two bake verdicts on /health by rbuergi · Pull Request #4015 · Systemorph/MeshWeaver

rbuergi · 2026-09-11T08:10:23Z

Closes #3703 · Closes #3704

The gap, which was never either issue's own mechanism

#3703 and #3704 are both blocked on the SAME thing: the reading that would settle them exists only in a boot log, and log access on this fleet is break-glass. Neither could be confirmed nor closed by anything an operator is authorised to run.

Cold boot: 78 prebuilt assemblies adopted, the sweep's store probe counts 5 ten seconds later — 197 compiles instead of ~20, and the upstream half of #3663 #3703 — fix: the bake sweep no longer rebuilds what this process just adopted (#3703) #3928's fix (StampSuperseding / OverlayOnto / ClassifiedFromLocalAdoption / fromlocaladoption) is present and running in both portal images. Its confirming reading is a boot log line with no portal surface.
Unexplained: one boot's batched source discovery came back 91 Code nodes short — RunQuery completes on a 1-second quiet window and QueryResultChange<T> has no terminal marker #3704 — diag(#3704): a short source read now names its own mechanism #3799's ChunkTiming and the Largest inter-chunk gap … = {Share}% line are running on both portals, so an empty result would now mean "no wide gaps" rather than "not deployed". Again, a Loki query with no portal route.

This publishes both verdicts on /health, which is public, unauthenticated, and — because it is composed by the process, as the system — past RLS. That is the property an RLS-filtered search sweep does not have, and it is what turns two permanently-unmeasurable issues into two measurable ones.

Verified against the live portals first (2026-09-11 06:49Z)

memex.meshweaver.cloud 3.0.0+6231c4da · memex.systemorph.com 3.0.0+45306a33.

Claim	Verdict
`/health` is public and unauthenticated	Holds — HTTP 200 with no credential on both
`content-types` and `pending_module_activation` answer per-replica type loadability	Holds
Repeated calls sample different replicas	Holds — 10 calls to memex.meshweaver.cloud returned 2 distinct bodies (7 × 671 B, 3 × 873 B) with disjoint readings: 3 vs 7 untypable NodeTypes, 6 vs 3 pending modules, no overlap
`Admin/_LogIncident` lives only on memex.systemorph.com and covers memex-cloud	Holds — `namespace:Admin/_LogIncident scope:subtree` is truncated-at-limit there and returns 0 on memex.meshweaver.cloud; incident `0a24845deb486a56`, read on the control instance, carries `"namespace": "memex-cloud"`
It is invisible to an unscoped search	Holds — same portal, same moment: `nodeType:LogIncident` → 0, `namespace:Admin/_LogIncident scope:subtree` → truncated

One brief claim did NOT survive, and it matters for the doc: the zsh $ref:path trap does not fail only toward "absent". Measured here on zsh 5.9 — "$ref:src/…" expands to bare origin/main (the :s modifier eats the path) and git cat-file -e on a commit object exits 0, i.e. a false PRESENT; test/ → mainest/… and helm/ → originelm/… exit 128, a false ABSENT; deploy/ is correct by luck. src/ — the prefix a core session reaches for most — is the false-present one, and a false present on "does main already carry this?" means skipping work that was never done. The skill entry records the measured table, not the half.

Where each surface lives — this did NOT span repos

Memex.Portal.ServiceDefaults is in core (memex/aspire/), and BOTH portal hosts reference it, so the checks reach the Monolith and the Distributed image without touching MeshWeaver.Plugins. State in MeshWeaver.Hosting (registered in AddMeshCatalog, beside ContentDegradationRegistry — the registration whose container topology is proven to reach a host-side check, since content-types is readable off a live portal today); the IHealthCheck in ServiceDefaults, exactly as ContentTypeHealthCheck does it.

Registered unconditionally, and that is deliberate: nodetype_bake sits behind if (gateBake), which is right for a readiness gate and exactly wrong for an instrument — one absent precisely where pre-warming is off answers nothing about the deployments that most need it.

🚨 The entry that cannot be a gate that cannot fail

/health prints only non-Healthy entries, so a census entry answering Healthy-and-silent would be byte-identical on the wire to one never registered — the very ambiguity these issues are stuck in. ProbeEndpoints.CensusTag makes a tagged entry print its reading whatever its status:

bake-report — Degraded when there is no report at all ("measured NOTHING … NOT a clean bake") and when the snapshot predated this process's adoptions; Healthy and still printed when clean, numbers and all.
source-discovery — Healthy when no pass ran (the normal state of a warm replica; degrading on it would be a check that can never pass) and it prints that in words; Degraded when a gap reached the window.

The tag is not a probe tag: publishing a number can never restart a pod or evict it from the Service, and there is a test for that.

Controls, in both directions, each proved falsifiable

16 tests, all green. Each was reverted and observed to redden on its own assertion and nothing else.

Revert	Reddens	On its own assertion
census line → `if (entry.Status == Healthy) continue;`	`ACleanCensusReading_IsPrintedAlthoughItIsHealthy`, `NoDiscoveryPass_IsHealthy_AndStillPrintsThatNothingWasMeasured` (2 of 7; the other 5 pass)	"a CLEAN bake reading did not appear on /health … Body was:\nHealthy" — the body is literally the ambiguity
`IsClean(null)` → `true`	`NoReportIsNotACleanReport_AndTheSentenceSaysSo`, `NoBakeReportAtAll_IsDegraded_AndSaysItMeasuredNothing`, `ADegradedCensus_ReachesNeitherPostStartupProbe`	"a replica that published NO bake report reported clean. That is the missing-instrument-reads-as-a-pass defect…"

The third entry there is the vacuity guard working as designed: ADegradedCensus_ReachesNeitherPostStartupProbe asserts its own precondition first — "the precondition of this test — a Degraded census entry — is not present, so the assertions below would hold vacuously" — and that is the line that fired, so the probe assertions can never pass on an empty premise.

Paired positive/negative controls, so no "nothing reported" assertion stands alone: AHealthyCheckWithoutTheCensusTag_StaysSilent (the tag is opt-in and self still does not print) is paired with the clean-census case that does print; NarrowGaps_ExonerateTheCompletionRule_AndPointUpstream asserts NotContain("mechanism #3704 names") and is paired with AGapReachingTheWindow_IndictsTheCompletionRule; ANarrowPassAfterAWideOne_DoesNotEraseTheEvidence pins that the high-water reading only moves up. The three pre-existing ProbeSeparationTest cases still pass, so probe separation is undisturbed.

Build + test evidence

0 Error(s), 0 Warning(s) on -c Release -warnaserror, one project per invocation: MeshWeaver.Hosting, Memex.Portal.ServiceDefaults, MeshWeaver.Hosting.Test, Memex.Portal.Shared.Test, MeshWeaver.Documentation.Test. BakeCensusRegistryTest 9/9 · HealthCensusTest 7/7 · ProbeSeparationTest 3/3 · documentation guards 15/15 (DocumentationLinkIntegrityTest, ArchitectureTopicMapTest, WhatsNewEntryIntegrityTest, ProbeSemanticsGuard, MigrationWorkloadModelGuard).

Work products committed

Architecture/MeasuringALivePortalReadOnly — /health promoted to the first-choice, non-break-glass instrument: what each entry answers, why an absent entry is not a clean one, the census exception, replica sampling, and what is still dark. Plus the incident store's three misreadings (one portal only; namespace load-bearing; a frozen occurrences can mean RENAMED — the fingerprint's third part is the masked message, so a re-wording mints a new node).
Architecture/OperatingFromThePortal + AGENTS.md — the "can THIS replica load NodeType X" claim corrected. It is mostly answered now; what remains dark is named (content-types records a degradation only when a read degrades, so a type nobody has opened appears in no list).
Architecture/ReadingCiSignals — the org Actions-budget refusal, and why re-running is the correct action there and nowhere else on that page.
.claude/skills/worktree/SKILL.md — the zsh $ref:path trap, with the measured per-prefix table.

Declarations

Pairs-with: none — this change removes no public type and no public member. ResolveSources's existing 5-argument signature is preserved as a forwarder and the registry arrives on a new OVERLOAD, precisely so module bundles built on a previous platform keep binding. Swept MeshWeaver.Plugins at origin/main for <see cref> references to NodeTypeBatchBake.ResolveSources that a new overload could make ambiguous (CS0419): the only hits are generated bin/**/*.xml build artefacts, no source.

Implementers: none — no member added to a public interface and no abstract member added to a public abstract class. The two new registries are sealed classes; the two new health checks implement the framework's own IHealthCheck.

Mirror-sync: none — no localization key added and no existing value re-worded. The new strings are a machine- and operator-facing diagnostic payload on /health (plain text, read by curl, the kubelet and the control instance's Sample action), not UI a viewer reads on screen; every existing entry on that endpoint — content-types, bundle_adoption, required_modules — is an untranslated English literal for the same reason.

🤖 Generated with Claude Code

Both issues were blocked on the same gap and neither on its own mechanism: the reading that would settle them existed ONLY as a boot log line, and log access on this fleet is break-glass. So neither could be confirmed nor closed by anything an operator is authorised to run. Core now records both readings in process and ServiceDefaults prints them on ProbeEndpoints.Health — public, unauthenticated, composed by the process and therefore past RLS, which is the property an RLS-filtered `search` sweep does not have. - NodeTypeBakeReportRegistry (#3703): the bake report's totals, per-state breakdown and ClassifiedFromLocalAdoption, published beside the count of adoption stamps this process holds — the OTHER half of the 78-vs-5 pair the issue was filed as, so the two can never again be read as a contradiction. Recorded at BOTH report sites (the adopt-only probe every boot runs, and the compiling sweep). - SourceDiscoveryRegistry (#3704): per discovery query, the folded-change count, what each delivered, where the fold settled and the LARGEST inter-chunk gap against the completion window — the discriminator the issue asks for. The 50% threshold moves onto the registry so the log's warning and the health verdict read ONE constant and cannot disagree about a pass. 🚨 The property that makes publishing them worth anything: a missing measurement may not read as a clean one. /health prints only non-Healthy entries, so a census entry that answered Healthy-and-silent would be byte-identical on the wire to one that was never registered — the exact ambiguity the two issues are stuck in. ProbeEndpoints.CensusTag makes a tagged entry print its reading whatever the status; bake-report is Degraded when there is NO report; source-discovery stays Healthy when no pass ran (the normal state of a warm replica, so degrading on it would be a check that cannot pass) and prints that in words. The census tag is not a probe tag, so publishing a number can never restart a pod or evict it from the Service. Registered unconditionally, deliberately: nodetype_bake sits behind `if (gateBake)`, which is right for a readiness gate and exactly wrong for an instrument — one that is absent precisely where pre-warming is off answers nothing about the deployments that most need it. ResolveSources takes the registry through a new OVERLOAD rather than an extra parameter: adding a parameter replaces a method's signature and breaks module bundles built on a previous platform. Docs: MeasuringALivePortalReadOnly gains /health as the first-choice, non-break-glass instrument (measured today: 10 calls to memex.meshweaver.cloud returned 2 disjoint bodies, so one call answers about ONE replica you did not choose), plus the incident store's three misreadings. OperatingFromThePortal's "can THIS replica load NodeType X" row and AGENTS.md's matching claim are corrected — mostly answered now, and what is still dark is named. ReadingCiSignals gains the org Actions-budget refusal (~2 s, zero steps, BlobNotFound logs, the reason only in the annotation). The worktree skill gains the zsh unbraced `$ref:path` trap, which fails toward PRESENT for src/ and toward ABSENT for test/ and helm/. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-11T08:17:46Z

Test Results (shard 1)

508 tests ±0 508 ✅ ±0 44s ⏱️ -8s
1 suites ±0 0 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 5cf38a6. ± Comparison against base commit 27ea9fd.

github-actions · 2026-09-11T08:18:43Z

Test Results (shard 0)

256 tests ±0 256 ✅ ±0 2m 4s ⏱️ +32s
1 suites ±0 0 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 5cf38a6. ± Comparison against base commit 27ea9fd.

github-actions · 2026-09-11T08:18:45Z

Test Results (shard 3)

390 tests ±0 390 ✅ ±0 52s ⏱️ -1s
2 suites ±0 0 💤 ±0
2 files ±0 0 ❌ ±0

Results for commit 5cf38a6. ± Comparison against base commit 27ea9fd.

Copilot

🟡 Changes recommended

Unresolved moderate findings affect measurement correctness, public diagnostic exposure, state atomicity, and payload bounds.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This pull request publishes NodeType bake and source-discovery verdicts through public /health, replacing log-only measurements with census-tagged health output.

Changes:

Adds mesh-scoped registries and census health checks.
Captures bake and discovery measurements during runtime.
Adds tests and operational, architecture, and changelog documentation.

File summaries

File	Summary
`test/MeshWeaver.Hosting.Test/BakeCensusRegistryTest.cs`	Tests bake registry verdicts and thresholds.
`test/Memex.Portal.Shared.Test/HealthCensusTest.cs`	Tests health census publication and probe separation.
`src/MeshWeaver.Hosting/SourceDiscoveryRegistry.cs`	Stores and formats discovery measurements. Moderate (3 votes): Redact public query details to prevent information disclosure. Moderate (1 vote): Publish discovery data and high-water state atomically. Moderate (1 vote): Bound entries and query lengths with an omitted-count indicator. Moderate (1 vote): Serialize or CAS high-water updates to prevent lost evidence.
`src/MeshWeaver.Hosting/Persistence/PersistenceExtensions.cs`	Registers diagnostic registries.
`src/MeshWeaver.Hosting/NodeTypeBatchBake.cs`	Captures source-discovery timing. Moderate (1 vote): Record failed or attempted passes separately from legitimate no-work states.
`src/MeshWeaver.Hosting/NodeTypeBakeReportRegistry.cs`	Stores bake reports and verdicts.
`src/MeshWeaver.Hosting/DynamicTypePreWarmer.cs`	Publishes bake and discovery reports.
`src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-11-health-now-reports-what-the-portal-measured-and-what-it-did-not.md`	Adds the changelog entry. Nit (1 vote): Align the entry’s `Category` with the intended Fix/Feature classification.
`src/MeshWeaver.Documentation/Data/Architecture/ReadingCiSignals.md`	Documents CI signal and rerun guidance.
`src/MeshWeaver.Documentation/Data/Architecture/OperatingFromThePortal.md`	Updates portal-based diagnostic guidance.
`src/MeshWeaver.Documentation/Data/Architecture/MeasuringALivePortalReadOnly.md`	Documents live `/health` measurement procedures.
`src/MeshWeaver.Documentation/Data/Architecture.md`	Updates the architecture index.
`memex/aspire/Memex.Portal.ServiceDefaults/SourceDiscoveryHealthCheck.cs`	Exposes discovery status through health checks. Moderate (2 votes): Distinguish an unregistered registry from a registered-but-empty instrument.
`memex/aspire/Memex.Portal.ServiceDefaults/ServiceDefaults.cs`	Registers census health checks and output behavior.
`memex/aspire/Memex.Portal.ServiceDefaults/ProbeEndpoints.cs`	Defines census tagging semantics.
`memex/aspire/Memex.Portal.ServiceDefaults/BakeReportHealthCheck.cs`	Exposes bake status through health checks.
`AGENTS.md`	Updates operational guidance.
`.claude/skills/worktree/SKILL.md`	Documents the zsh reference-path trap.

Review details

Suppressed comments (5)

src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-11-health-now-reports-what-the-portal-measured-and-what-it-did-not.md:3

The PR description says the What's New entry is Category: Fix, but this new entry declares Category: Feature. Please align the metadata with the intended classification (the repository guidance distinguishes user-noticeable fixes from features), because the published changelog will otherwise contradict the PR declaration.

Category: Feature

src/MeshWeaver.Hosting/NodeTypeBatchBake.cs:429

The registry is recorded only after Throttle(...).Take(1) settles successfully. A provider error or timeout therefore reaches the ResolveSources fallback without recording an attempted/failed pass, and the empty registry is then rendered as Healthy "no pass ran". That makes a failed discovery indistinguishable from a warm replica that had no work; record the failure/attempt state separately from the legitimate no-work state.

                .Do(nodes => chunks.Report(logger, bounded, nodes.Count, QueryQuietWindow, discovery))

src/MeshWeaver.Hosting/SourceDiscoveryRegistry.cs:103

The registry publishes the per-query map before it publishes widest, while the health check reads those two fields in separate calls. A request that lands between these writes can include a newly recorded wide-gap pass but see the old widest (or null), and therefore report Healthy and omit the indictment. Publish one immutable state containing both values atomically, then have the check read that state once.

        byQuery[pass.Query] = pass;
        // Read-compare-write without a CAS loop on purpose: a lost race here can only lose the
        // high-water mark for ONE pass whose per-query entry is recorded anyway, and discovery
        // passes are issued sequentially by one sweep. A gate would buy nothing and cost the rule
        // this repository does not bend.

src/MeshWeaver.Hosting/SourceDiscoveryRegistry.cs:149

Describe concatenates every distinct query's full diagnostic line, but the distinct set is built from all pending types' authored source/test queries, not just the fixed global queries. Thus both entry count and line length are unbounded, and every public /health request allocates and returns the whole payload; a large/custom mesh can make this check expensive or exceed health-consumer limits. Cap the published entries/query lengths and report an explicit omitted count.

        return $"{passes.Count} discovery query/queries measured on this replica; {verdict}. "
               + $"Widest ever: {widest?.Line ?? "(none)"}. Most recent per query: "
               + string.Join("; ", passes.Select(p => p.Line));

src/MeshWeaver.Hosting/SourceDiscoveryRegistry.cs:106

Record updates this mesh-scoped singleton from separate discovery subscriptions, but the read/compare/write below is not atomic. If two passes read the same old widest value and the narrower pass assigns after the wider one, Widest drops the stronger evidence and /health no longer reports the true high-water gap. Use a CAS loop (or equivalent serialization) for this update.

        var current = widest;
        if (current is null || pass.GapSharePercent > current.GapSharePercent)
            widest = pass;

Files reviewed: 18/18 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        var registry = services.GetService<SourceDiscoveryRegistry>();
+        var passes = registry?.Snapshot() ?? [];


+        $"'{Query}' settled at {Settled} node(s) from {Chunks} change(s) "
+        + $"({Items} item(s) delivered) in {ElapsedMs.ToString("F0", CultureInfo.InvariantCulture)}ms; "
+        + $"largest inter-chunk gap {LargestGapMs.ToString("F0", CultureInfo.InvariantCulture)}ms "
+        + $"= {GapSharePercent.ToString("F0", CultureInfo.InvariantCulture)}% of the "
+        + $"{WindowMs.ToString("F0", CultureInfo.InvariantCulture)}ms completion window";


github-actions · 2026-09-11T08:20:50Z

Test Results (shard 2)

1 273 tests ±0 1 271 ✅ ±0 3m 49s ⏱️ -5s
3 suites ±0 2 💤 ±0
3 files ±0 0 ❌ ±0

Results for commit 5cf38a6. ± Comparison against base commit 27ea9fd.

github-actions · 2026-09-11T08:22:18Z

Test Results (shard 4)

1 956 tests ±0 1 956 ✅ ±0 4m 46s ⏱️ +13s
4 suites ±0 0 💤 ±0
4 files ±0 0 ❌ ±0

Results for commit 5cf38a6. ± Comparison against base commit 27ea9fd.

github-actions · 2026-09-11T08:24:15Z

Test Results (shard 5)

6 files ± 0 6 suites ±0 6m 51s ⏱️ -15s
2 715 tests +16 2 523 ✅ +16 192 💤 ±0 0 ❌ ±0
2 719 runs +16 2 527 ✅ +16 192 💤 ±0 0 ❌ ±0

Results for commit 5cf38a6. ± Comparison against base commit 27ea9fd.

github-actions · 2026-09-11T08:24:35Z

Test Results

17 files ± 0 17 suites ±0 19m 10s ⏱️ +18s
7 098 tests +16 6 904 ✅ +16 194 💤 ±0 0 ❌ ±0
7 102 runs +16 6 908 ✅ +16 194 💤 ±0 0 ❌ ±0

Results for commit 5cf38a6. ± Comparison against base commit 27ea9fd.

Copilot AI lite review requested due to automatic review settings September 11, 2026 08:10

meshweaver-cloud Bot enabled auto-merge September 11, 2026 08:10

Copilot started reviewing on behalf of rbuergi September 11, 2026 08:11

View session

Copilot AI reviewed Sep 11, 2026

View reviewed changes

meshweaver-cloud Bot added this pull request to the merge queue Sep 11, 2026

Merged via the queue into main with commit ac76174 Sep 11, 2026
35 checks passed

rbuergi deleted the worktree-agent-ae9df72e0f457ac16 branch September 13, 2026 10:27

Navigation Menu

Repository navigation

feat(#3703,#3704): publish the two bake verdicts on /health - #4015

feat(#3703,#3704): publish the two bake verdicts on /health#4015
meshweaver-cloud[bot] merged 1 commit into
mainfrom
worktree-agent-ae9df72e0f457ac16

rbuergi commented Sep 11, 2026

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		var registry = services.GetService<SourceDiscoveryRegistry>();
		var passes = registry?.Snapshot() ?? [];

Conversation

rbuergi commented Sep 11, 2026

The gap, which was never either issue's own mechanism

Verified against the live portals first (2026-09-11 06:49Z)

Where each surface lives — this did NOT span repos

🚨 The entry that cannot be a gate that cannot fail

Controls, in both directions, each proved falsifiable

Build + test evidence

Work products committed

Declarations

Uh oh!

github-actions Bot commented Sep 11, 2026

Test Results (shard 1)

Uh oh!

github-actions Bot commented Sep 11, 2026

Test Results (shard 0)

Uh oh!

github-actions Bot commented Sep 11, 2026

Test Results (shard 3)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (5)

Uh oh!

github-actions Bot commented Sep 11, 2026

Test Results (shard 2)

Uh oh!

github-actions Bot commented Sep 11, 2026

Test Results (shard 4)

Uh oh!

github-actions Bot commented Sep 11, 2026

Test Results (shard 5)

Uh oh!

github-actions Bot commented Sep 11, 2026

Test Results

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants