From c87a07674518cbe3c3a0e4c94d70fd8365147211 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20B=C3=BCrgi?= <6334612+rbuergi@users.noreply.github.com> Date: Mon, 21 Sep 2026 15:31:55 +0200 Subject: [PATCH 1/5] docs: a dynamic NodeType registers its content type only where an instance activates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plugins#2178 and #2180 both report a Hosting type that one replica cannot type. Both attribute it to a declined prebuilt bundle. Measured on two live portals, that is not what is happening, and the difference decides the fix. A dynamic NodeType's content CLR type enters IMeshContentTypeRegistry as a side effect of its hub configuration being built, which happens when a per-node hub cold-activates. A per-node hub is a single activation cluster-wide. DynamicTypePreWarmer's adopt-only pass ASKS, NEVER BUILDS (the default for every deployment), and its compiling sweep reports a type the shared store already holds as AlreadyBaked and "never activated". So on every replica after the first, a dynamic type whose few instances activate elsewhere has a usable assembly and no registration — and every read of one degrades. ContentTypeRegistrationSweep's own reason for skipping dynamic types carried the false half of this: "a dynamic type with zero instances has no payload carrying its discriminator, so there is nothing to degrade". True for zero, false for few. Corrected in place, with the measurements. Measured 2026-09-21, read-only: - memex.systemorph.com: bake-report Healthy, 229/230 usable, live record census clean (0 untyped, 0 foreign) — while content-types named Hosting/DeploymentStatus and Store/Plugin as untypeable on that replica. - memex.meshweaver.cloud: 8-11 untypeable types per replica, a DIFFERENT set on each, Hosting/Deployment among them; the 3 types with no usable assembly are all in partition MeshWeaver, disjoint from every named type. - The discriminator is instance count, not bundle state: Hosting/InstanceAction has >=200 instances and is untypeable nowhere; Hosting/DeploymentStatus has 5 and Hosting/Deployment has 4, and both are untypeable now. - Store/Plugin reads buildProvenance: AdoptedVerified on the live framework identity — adopted, never compiled, never activated. Also records the fix that must NOT ship, because it was built and measured: calling ProbeRegister from the degrade seam obtains the configuration through EnrichWithNodeType, which triggers the compilation chain and WRITES the shared NodeType record (the stale-Ok self-heal flips it to Pending). A read seam doing that lets every non-owning replica compile and re-stamp one row the whole deployment shares — the cross-stamp NodeTypeLiveRecordCensus exists to detect. Control on each side, full MeshWeaver.Hosting.Test suite: clean main 861/861 green; with the seam wired, ANodeTypesSourcesWaitForItsBundleTest failed "Expected AdoptedVerified ... but found Compiled". No behaviour change here. Registration belongs with the pre-warmer's AlreadyBaked branch as a post-sweep background trickle; that trade (the ~13.5 s of assembly opening #1660 removed from boot) is a maintainer decision. Co-Authored-By: Claude Opus 5 (1M context) --- .../DynamicContentTypeRegistration.md | 176 ++++++++++++++++++ .../ContentTypeRegistrationSweep.cs | 27 ++- 2 files changed, 201 insertions(+), 2 deletions(-) create mode 100644 src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md diff --git a/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md new file mode 100644 index 0000000000..d811e578a8 --- /dev/null +++ b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md @@ -0,0 +1,176 @@ +--- +Name: Dynamic Content Type Registration +Category: Architecture +Description: A dynamic NodeType's content CLR type registers only as a side effect of one of its instances activating in THIS process — so a type whose few instances live on another replica is untypeable here, with a usable assembly, a clean bake and a clean census. The mechanism, the measurements that separate it from a declined bundle, and the on-demand registration that closes it. +Icon: +--- + +# Dynamic Content Type Registration + +**A replica can hold a NodeType's assembly, count it baked, report a clean census — and still be +unable to type a single node of it.** Not because the bundle was declined, not because the bytes +are missing, but because nothing in that process ever *registered* the CLR type. This page is +about the third cause, which is the common one, and which no instrument named until now. + +## The mechanism + +A dynamic NodeType — in-mesh `Source/*.cs`, compiled by Roslyn at runtime — declares its content +type inside its own hub configuration. The CLR type enters the process-wide +`IMeshContentTypeRegistry` as a **side effect of that configuration being built** +(`MeshDataSource.WithContentType`), and the configuration is built when a **per-node hub +cold-activates**. Three consequences follow, and none of them is obvious from any one of them: + +1. **A per-node hub is a single activation cluster-wide.** A node's hub lives on exactly one + silo. Building it registers the type *there*. +2. **Every other replica reads the same nodes through its own `MeshNodeStreamCache`**, receives + the payload as JSON, asks its own registry for the `$type`, and is told nothing. The content + stays a bare `JsonElement`. +3. **Nothing at boot closes the gap.** `DynamicTypePreWarmer`'s adopt-only pass *asks, never + builds*, and it is the default path for every deployment. Where the compiling sweep does run, a + type the shared assembly store already holds is reported `AlreadyBaked` and — in the warmer's + own words — *"never activated"*. That is the normal case for **every replica after the first**, + and for every replica of a roll onto an identity that is already baked. + +So the exposure is precisely: **a dynamic NodeType with FEW instances, all of whose instances +happen to activate elsewhere.** A type with many instances gets activated on most replicas sooner +or later and registers itself. A type with zero instances has nothing to read. A type with four — +`Hosting/Deployment`, `Hosting/DeploymentStatus`, `Store/Subscription`, `PG3Reporting/Fund` — is +the one that breaks, and it breaks differently on each replica. + +### The premise that was false + +`ContentTypeRegistrationSweep` registers static definitions at boot and deliberately excludes +dynamic ones, for two good reasons (the per-NodeType boot cost the CI content bake removed, and +the loader's corrupt-file self-heal tripping on an adopted-but-not-yet-loadable bake) and one +that does not hold: + +> A dynamic type registers the moment an instance hub activates — and a dynamic type with **zero +> instances** has no payload carrying its discriminator, so there is nothing to degrade. + +True for zero. False for few. The sentence quietly assumes one process. + +## What it looks like from outside + +It looks like a declined bundle, and it is not one. `content-types` on `/health` says: + +> N node type(s) whose content this replica cannot type — the module that declares the type is not +> loaded here (**its prebuilt bundle was declined or its compiled assembly is not on this +> replica**), so their pages render empty + +Both stated causes are real, and neither was what was measured. The discriminating readings, taken +on two live portals on 2026-09-21: + +| instrument | memex.systemorph.com (core `746b4e48`) | memex.meshweaver.cloud (core `026442ff`) | +|---|---|---| +| `bake-report` | **Healthy** — 229 of 230 types carry a usable assembly | Healthy — 242–302 of 366 with a verdict, all usable | +| the ONE / THREE types with **no** usable assembly | `BinaryClickerV2` (CompileError) | 3 × CompileError, all in partition `MeshWeaver` | +| live record census | **clean** — 0 untyped, 0 foreign, 230 of 230 | clean on each replica sampled | +| `content-types` | `Hosting/DeploymentStatus ×30`, `Store/Plugin ×8` | 8–11 types per replica, **a different set on each**, incl. `Hosting/Deployment` | + +The provenance of a named type closes it: `Store/Plugin` on the control instance reads +`compilationStatus: Ok`, `compiledFrameworkVersion: se271838…` (the live identity) and +**`buildProvenance: AdoptedVerified`** — adopted from the share for exactly this framework, so the +pre-warm reported it `AlreadyBaked` and never activated it. The assembly is verified present and +the type is unregistered: those are not in tension, they are the same sentence read twice. + +🚨 **The two sets are DISJOINT.** The types with no usable assembly are not the types that cannot +be typed, and the types that cannot be typed all have usable assemblies. That single comparison +falsifies both stated causes and is the cheapest way to recognise this failure — it needs one +`/health` body and no cluster access. + +### Reading the instrument correctly + +- **An entry's PRESENCE is a live verdict, not history.** `ContentTypeHealthCheck` calls + `ContentDegradationRegistry.Unresolved`, which re-asks the registry per entry at probe time + (both routes: the NodeType path and the stored `$type`). A boot-race entry disappears the moment + its type registers. Only the **×count** is cumulative since boot. +- **An entry's ABSENCE is not a clean bill.** A degradation is recorded only when a READ degrades. + A type nobody has opened on this replica appears in no list, however unregistered it is. +- **Repeated `/health` calls sample different replicas.** Four calls returned four different sets; + that variation *is* the finding, not noise. The per-replica form with no guesswork is the + control instance's `Sample` action, but it truncates each health body — the untruncated bodies + come from calling the public `/health` several times. + +## The consequence that is not cosmetic + +The control plane is written in types that live in the mesh. When the replica hosting the +self-update inbox cannot type `Hosting/Deployment`, `SelfUpdateRouting`'s record lookup returns +**0 rows** while the rows are in the table and every other hub lists them. The announcement is +consumed and dropped with no error and no log line naming a cause — the thing that would report +the failure is written in the type that failed. That is +[Plugins#2178](https://github.com/Systemorph/MeshWeaver.Plugins/issues/2178), a 24-hour CD stall; +[Plugins#2180](https://github.com/Systemorph/MeshWeaver.Plugins/issues/2180) is the same shape one +type over. + +🚨 **A restart does not fix it and neither does a recycle.** A fresh activation re-reads, but what +it *finds* is the same shared store and the same skip: `AlreadyBaked`, never activated, never +registered. Rolling forward "fixes" it only by accident — a new framework identity makes the +prebuilts stale, which forces a local compile, and a local compile activates the hub and registers +the type. That is why the remedy looked like a build-artifact remedy and why it kept coming back. +See [Stale State Until Recycle](../StaleStateUntilRecycle) for the general rule this is an +instance of, and for why a second recycle proves nothing the first did not. + +## Closing it — and the fix that must NOT be shipped + +🚨 **The obvious fix is unsafe, and this is the most useful thing on this page.** The tempting +move is to call the existing `ContentTypeRegistration.ProbeRegister` from the degrade seam: the +place that already *reports* the gap would *close* it, on demand, once per type, at zero boot cost. +That was built and it works — and it must not ship, because of what obtaining the configuration +costs. + +Getting a dynamic type's hub configuration means `IMeshNodeHubFactory.ResolveHubConfiguration` → +`NodeTypeEnrichmentHelpers.EnrichWithNodeType`, and that path is **not a read**: + +- it is handed the `IMeshNodeCompilationService` and **triggers the compilation chain** when the + type's sources are ahead of its build; +- it **writes the shared NodeType record** — the stale-`Ok`-with-null-assembly self-heal flips the + record to `Pending` to force a recompile; +- it **arms the rebind watcher** for that address. + +A NodeType record is ONE row the whole deployment shares. Letting a read seam take that path means +every non-owning replica that happens to read a node independently compiles and re-stamps the +record — which is precisely the mid-roll cross-stamp `NodeTypeLiveRecordCensus` was built to +DETECT as a fault, and the hazard behind the leaving-hub and per-replica-compile rules. Measured: +with the seam wired that way, `ANodeTypesSourcesWaitForItsBundleTest` failed with *"Expected +AdoptedVerified … but found Compiled"* — a held, correctly-adopted type driven into a compile by +nothing but a read. + +**So registration belongs with the component already sanctioned to activate dynamic types on this +replica: the pre-warmer.** Its `AlreadyBaked` branch is where the skip happens, it runs once per +process, it is sequenced after bundle seeding, and it already holds the bake's verdict that the +type has usable bytes here. The shape that fits the two original objections is a **registration-only +pass over the `AlreadyBaked` types, as a paced background trickle AFTER the sweep settles** — off +the readiness path, so the per-NodeType boot cost the content bake removed stays removed, and after +the bake has settled, so no probe meets an adopted-but-not-yet-loadable bundle. That pass covers +every adopted type rather than only the ones somebody read, which also closes the blind spot below. + +The cost is real and is a judgement call, not a detail: it is the ~13.5 s of assembly opening that +#1660 removed from boot, moved to a background trickle. That trade wants the maintainer's decision, +which is why this page describes the shape rather than shipping it. + +## What this does not establish + +- **No fix is shipped and neither issue is closed.** What is established is the mechanism, the + measurements that separate it from a declined bundle, and that the cheap fix is unsafe. +- **No production verification.** The acceptance measurement for any fix is positive, not an + absence: on a replica reporting a type under `content-types`, that entry disappears from the next + `/health` probe while `bake-report` is unchanged, and the type's record still reads + `buildProvenance: AdoptedVerified` afterwards — a fix that heals by compiling has re-stamped the + shared record and is the unsafe shape wearing a green tick. +- **The residual population is not bounded.** Every dynamic NodeType this replica adopted rather + than compiled is unregistered here until something reads one; the measurement can name only the + ones that HAVE been read. On the control instance that is 229 of 230 types adopted, and 2 named. +- **Whether a type also fails to register after a genuine local compile** was not tested; every + case measured was an adoption. + +## See also + +- [NodeType Compilation](../NodeTypeCompilation) — how an in-mesh type becomes an assembly. +- [Module Build Architecture](../ModuleBuildArchitecture) — the bake, the share, and why a second + replica finds the first one's work. +- [A Content Verdict Is Per Node](../AContentVerdictIsPerNode) — the neighbouring rule about what + a per-node verdict can and cannot be read to cover. +- [A Census That Counts Must Name](../ACensusThatCountsMustName) — why a count without a + denominator is not a measurement. +- [Operating From The Portal](../OperatingFromThePortal) — where `/health`, `Sample` and the + per-replica readings come from. diff --git a/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs b/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs index 793bb9277d..851cdfda98 100644 --- a/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs +++ b/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs @@ -56,8 +56,31 @@ namespace MeshWeaver.Graph; /// 13.5 s of a 101 s boot); worse, a probe of an adopted-but-not-yet-loadable bake trips the /// loader's corrupt-file self-heal, which DELETES the store's bytes and forces a re-adoption /// (ShippedPrebuiltBundlesTest pins that boot contract). A dynamic type registers the moment an -/// instance hub activates — and a dynamic type with zero instances has no payload carrying its -/// discriminator, so there is nothing to degrade. +/// instance hub activates. +/// +/// 🚨 That last clause used to end "…and a dynamic type with zero instances has no +/// payload carrying its discriminator, so there is nothing to degrade", and the premise is false +/// for a type with FEW instances (Systemorph/MeshWeaver.Plugins#2178, #2180). A per-node hub is +/// a single activation cluster-wide, so a type whose handful of instances all activate on ANOTHER +/// replica registers there and nowhere else — while every other replica reads those nodes through +/// its own stream cache and cannot bind the $type. Measured on two live portals 2026-09-21: +/// 8-11 such types per replica, a different set on each, Hosting/Deployment and +/// Hosting/DeploymentStatus among them, every one of them carrying a usable assembly. The +/// boot scope here is unchanged; what is corrected is the REASON, because the old one reads as +/// "nothing is exposed" and something is. +/// +/// 🚨 Do NOT close it by calling from a degrade seam. +/// Getting a dynamic type's configuration means +/// IMeshNodeHubFactory.ResolveHubConfiguration → NodeTypeEnrichmentHelpers, which +/// TRIGGERS THE COMPILATION CHAIN and can WRITE the shared NodeType record (the stale-Ok self-heal +/// flips it to Pending to force a recompile) and arms the rebind watcher. A read seam that +/// did this would let every non-owning replica independently compile and re-stamp one record that +/// the whole deployment shares — the cross-stamp NodeTypeLiveRecordCensus exists to DETECT. +/// Measured: an adopt-only branch wired that way turned an AdoptedVerified type into +/// Compiled (ANodeTypesSourcesWaitForItsBundleTest). Registration belongs with the +/// component already sanctioned to activate dynamic types on this replica — the pre-warmer, whose +/// AlreadyBaked branch is where the skip happens — not with whoever reads a node. See +/// Doc/Architecture/DynamicContentTypeRegistration. /// public static class ContentTypeRegistration { From 8427c53f90bd24a3102bfe17daf65d296bce9a5e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20B=C3=BCrgi?= <6334612+rbuergi@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:30:15 +0200 Subject: [PATCH 2/5] review(#5079): the /health sentence asserted two causes and the common one was neither MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot findings, both real: - the page's front-matter Description still promised "the on-demand registration that closes it" from a draft that did ship one; the body says the opposite. The Description is what a catalog reader sees INSTEAD of the body, so that is the worst place for it to be stale. Rewritten, and it now says NOT yet closed. - "Systemorph/MeshWeaver.Plugins#2178, #2180" left the second number unqualified. Core has a #2180 too, so the bare form does not fail to resolve — it resolves to the wrong issue. Both fully qualified. And the thing those two issues were actually misdiagnosed BY. ContentDegradationRegistry .Describe told every reader of /health that a type it cannot bind means "the module that declares the type is not loaded here (its prebuilt bundle was declined or its compiled assembly is not on this replica)". Measured on two live portals, neither was happening for any named type, and the sets are disjoint: on memex.systemorph.com the ONE type with no usable assembly is BinaryClickerV2, and the two untypeable ones are Hosting/DeploymentStatus and Store/Plugin; on memex.meshweaver.cloud the three with no usable assembly are all in partition MeshWeaver and none is named. An instrument that asserts two causes as fact sends its reader past the real one — twice, at sev:H. The sentence now names the commonest cause FIRST (a dynamic NodeType this replica ADOPTED rather than compiled is never activated by the bake, so its content type is never registered here), keeps the other two, and names the instrument that decides between them: if bake-report's no-usable-assembly list does not name a type, that type's assembly is present and the first cause is the answer. Pinned by test. Also adds the page to Data/Architecture.md — ArchitectureTopicMapTest caught the omission: a working URL alone does not make a page discoverable. MeshWeaver.Documentation.Test 604/604; the ContentDegradation/UntypedContent seam tests 13/13; Release -warnaserror clean on MeshWeaver.Graph and MeshWeaver.Hosting. Co-Authored-By: Claude Opus 5 (1M context) --- src/MeshWeaver.Documentation/Data/Architecture.md | 1 + .../Architecture/DynamicContentTypeRegistration.md | 2 +- .../ContentTypeRegistrationSweep.cs | 2 +- .../ContentDegradationRegistry.cs | 14 +++++++++++--- .../ContentDegradationRegistryTest.cs | 10 ++++++++++ 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/MeshWeaver.Documentation/Data/Architecture.md b/src/MeshWeaver.Documentation/Data/Architecture.md index 9b81f352fb..6271d867d0 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture.md +++ b/src/MeshWeaver.Documentation/Data/Architecture.md @@ -283,6 +283,7 @@ Each theme starts with its introductory page, followed by related architecture t - [Install-Time Prebuilt Adoption](InstallTimePrebuiltAdoption) — the only lane that serves a package installed AFTER boot, and the four answers its zero must keep apart because a silent non-adoption reads exactly like a successful one - [Adoption and the Sweep Count Different Things](AdoptionAndTheSweepCountDifferentThings) — the cold boot that adopted 78 prebuilt assemblies and then reported 5, with nothing wrong on the share: what each instrument counts, why the sweep could not see its own process's writes, and the node-version ordering that keeps the fix from becoming a stale serve - [A Census That Counts Must Name](ACensusThatCountsMustName) — the one past-RLS census counted a permanently-broken NodeType and dropped its path one call before publication, so its output read as clean; where the identity was lost, what a PUBLIC census may name (the partition, never the node title), and how to tell a fix that is merged from a fix that is running +- [Dynamic Content Type Registration](DynamicContentTypeRegistration) — a dynamic NodeType's content CLR type registers only as a side effect of one of its INSTANCES activating in this process, so a type this replica adopted rather than compiled is untypeable here with a perfectly usable assembly, a clean bake and a clean census; the instance-count discriminator, the disjoint instrument sets that falsify “the bundle was declined”, and why closing it from the degrade seam drives an adopted type into a compile and re-stamps a record the whole deployment shares - [Denied Is Not Absent](DeniedIsNotAbsent) — a Forbidden that a read discards comes back as the thing being MISSING, so the operator announces a node pool, an Ingress or a ConfigMap is absent when it was merely not permitted to look; the three answers a read owes its reader, why fixing the sites a report names does not sweep the defect, and the gate that makes the next one red - [Import Write Ordering](ImportWriteOrdering) — type before instance, and what a foreign type does - [Language Services](LanguageServices) diff --git a/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md index d811e578a8..65b4bab7e3 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md +++ b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md @@ -1,7 +1,7 @@ --- Name: Dynamic Content Type Registration Category: Architecture -Description: A dynamic NodeType's content CLR type registers only as a side effect of one of its instances activating in THIS process — so a type whose few instances live on another replica is untypeable here, with a usable assembly, a clean bake and a clean census. The mechanism, the measurements that separate it from a declined bundle, and the on-demand registration that closes it. +Description: A dynamic NodeType's content CLR type registers only as a side effect of one of its instances activating in THIS process — so a type whose few instances live on another replica is untypeable here, with a usable assembly, a clean bake and a clean census. The mechanism, the measurements that separate it from a declined bundle, and why the obvious on-demand fix must not ship. NOT yet closed. Icon: --- diff --git a/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs b/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs index 851cdfda98..c3d8a6a33b 100644 --- a/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs +++ b/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs @@ -60,7 +60,7 @@ namespace MeshWeaver.Graph; /// /// 🚨 That last clause used to end "…and a dynamic type with zero instances has no /// payload carrying its discriminator, so there is nothing to degrade", and the premise is false -/// for a type with FEW instances (Systemorph/MeshWeaver.Plugins#2178, #2180). A per-node hub is +/// for a type with FEW instances (Systemorph/MeshWeaver.Plugins#2178, Systemorph/MeshWeaver.Plugins#2180). A per-node hub is /// a single activation cluster-wide, so a type whose handful of instances all activate on ANOTHER /// replica registers there and nowhere else — while every other replica reads those nodes through /// its own stream cache and cannot bind the $type. Measured on two live portals 2026-09-21: diff --git a/src/MeshWeaver.Hosting/ContentDegradationRegistry.cs b/src/MeshWeaver.Hosting/ContentDegradationRegistry.cs index b6d783b5e4..6f09f9d592 100644 --- a/src/MeshWeaver.Hosting/ContentDegradationRegistry.cs +++ b/src/MeshWeaver.Hosting/ContentDegradationRegistry.cs @@ -132,9 +132,17 @@ public ImmutableList Snapshot() => public static string Describe(IReadOnlyList degraded) => degraded.Count == 0 ? "every node content read on this replica typed" - : $"{degraded.Count} node type(s) whose content this replica cannot type — the module " - + "that declares the type is not loaded here (its prebuilt bundle was declined or its " - + "compiled assembly is not on this replica), so their pages render empty: " + : $"{degraded.Count} node type(s) whose content this replica cannot type — the CLR type " + + "is not registered in THIS process, so their pages render empty. Commonest cause, and " + + "the one to check first: a dynamic NodeType registers its content type only when one " + + "of its instances activates here, and a type this replica ADOPTED rather than compiled " + + "is never activated by the bake — so a type with few instances, all activated on another " + + "replica, is untypeable here with a perfectly usable assembly " + + "(Doc/Architecture/DynamicContentTypeRegistration). Otherwise the module that declares " + + "the type is not loaded here: its prebuilt bundle was declined, or its compiled assembly " + + "is not on this replica. 🚨 bake-report DECIDES between them — if its no-usable-assembly " + + "list does not name a type below, that type's assembly is present and the first cause is " + + "the answer: " + string.Join("; ", degraded.Select(d => $"{d.NodeType} ×{d.Count} (last {d.LastPath})")); } diff --git a/test/MeshWeaver.Hosting.Test/ContentDegradationRegistryTest.cs b/test/MeshWeaver.Hosting.Test/ContentDegradationRegistryTest.cs index d9b2e6d3e1..66cf94605a 100644 --- a/test/MeshWeaver.Hosting.Test/ContentDegradationRegistryTest.cs +++ b/test/MeshWeaver.Hosting.Test/ContentDegradationRegistryTest.cs @@ -31,6 +31,16 @@ public void RecordsPerNodeType_AndTheSentenceNamesThem() var sentence = ContentDegradationRegistry.Describe(snapshot); sentence.Should().Contain("2 node type(s)").And.Contain("Crm/Client ×2").And.Contain("Store/Install ×1") .And.Contain("not loaded here", "the sentence says WHY, not only what"); + // 🚨 The sentence used to name only two causes — a declined bundle and a missing assembly — + // and BOTH were falsified on two live portals while the third went unnamed: a dynamic + // NodeType this replica ADOPTED is never activated by the bake, so its content type is + // never registered here (Plugins#2178, #2180). An instrument that asserts two causes as + // fact sends its reader past the real one, so the commonest cause is named FIRST and the + // instrument that discriminates is named with it. + sentence.Should().Contain("ADOPTED rather than compiled", + "the commonest cause is the one the sentence used not to mention"); + sentence.Should().Contain("bake-report DECIDES between them", + "naming three causes without naming what tells them apart is a longer way to mislead"); } [Fact] From a519f6407c5cef13a68bbb0af1e3e6d68b6a1183 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20B=C3=BCrgi?= <6334612+rbuergi@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:34:15 +0200 Subject: [PATCH 3/5] docs: retract the claim that this defect explains SelfUpdate's "lists 0" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It does not, and the page asserting it would have sent the next reader down a branch already excluded. Two independent reasons, both measured after the page was written: - SelfUpdateRouting's lookup folds over INDEX ROWS — ResolveRecord → MatchRecord(rows, reference), path/id only. No content is typed, so a degraded content type cannot turn a listed record into a zero. - DeploymentContent is a FRAMEWORK type (core src/MeshWeaver.Deployment.Contract), shipped in the image and reached by reference; Apply reads the record with ContentAs, which deserialises to a statically-known T and never consults the mesh-wide registry. Hosting/PlatformBuildInbox shares six named files from Hosting/Deployment/Source, SelfUpdateRouting among them, and no content declaration. That zero was core#4958 — a zero with two causes treated as one — and the lane now answers RecordVerdict.Unseen at Error instead of concluding absence (Plugins, 2026-09-20 15:18Z). The section is replaced with what the defect DOES reach: every consumer that reads through MeshNodeStreamCache and then asks for the CLR type. And with the boundary that matters more, because it is the one that makes a false attribution easy: ContentAs with a statically-known T is immune, so a silent empty is not by itself evidence of this defect. Retracted on the issue as well. Documentation.Test 604/604. Co-Authored-By: Claude Opus 5 (1M context) --- .../DynamicContentTypeRegistration.md | 34 +++++++++++++------ 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md index 65b4bab7e3..aba56dd67d 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md +++ b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md @@ -91,16 +91,30 @@ falsifies both stated causes and is the cheapest way to recognise this failure control instance's `Sample` action, but it truncates each health body — the untruncated bodies come from calling the public `/health` several times. -## The consequence that is not cosmetic - -The control plane is written in types that live in the mesh. When the replica hosting the -self-update inbox cannot type `Hosting/Deployment`, `SelfUpdateRouting`'s record lookup returns -**0 rows** while the rows are in the table and every other hub lists them. The announcement is -consumed and dropped with no error and no log line naming a cause — the thing that would report -the failure is written in the type that failed. That is -[Plugins#2178](https://github.com/Systemorph/MeshWeaver.Plugins/issues/2178), a 24-hour CD stall; -[Plugins#2180](https://github.com/Systemorph/MeshWeaver.Plugins/issues/2180) is the same shape one -type over. +## The blast radius — and the consumer it does NOT reach + +**Everything that reads a node's content through `MeshNodeStreamCache` and then asks for the CLR +type is affected**: a view renders empty, `Content is X` / `as X` yields null, and a reactive wait +on a typed shape never completes. There is no exception and no log line naming a cause beyond the +degradation warning itself. + +🚨 **It does NOT reach a consumer that uses `ContentAs` with a statically-known `T`, and +getting that wrong is easy.** `ContentAs` deserialises to the type the CALLER names; it never +consults the mesh-wide registry, which is exactly why it is the mandated accessor. So a lane that +reads a record this way is immune to this defect even on a replica that cannot type the same node +for a view. + +That distinction was worth an issue comment to retract. `SelfUpdateRouting` answering *"no +`Hosting/Deployment` record … lists 0"* +([Plugins#2178](https://github.com/Systemorph/MeshWeaver.Plugins/issues/2178), a 24-hour CD stall) +reads exactly like this defect and is NOT it, twice over: its lookup folds over INDEX ROWS +(`MatchRecord` on path/id, no content), and `DeploymentContent` is a framework type shipped in the +image, reached by reference. That zero was core#4958 — a zero with two causes, "none exist" and "I +could not see them", treated as one — and the lane now answers `RecordVerdict.Unseen` at Error +instead of concluding absence. + +The lesson generalises: **a silent empty is not evidence of this defect.** Before attributing one +here, check whether the consumer names its type statically; if it does, look elsewhere. 🚨 **A restart does not fix it and neither does a recycle.** A fresh activation re-reads, but what it *finds* is the same shared store and the same skip: `AlreadyBaked`, never activated, never From 6b82e2eef1577a09532272719c0b77c7daba0b0f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20B=C3=BCrgi?= <6334612+rbuergi@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:37:30 +0200 Subject: [PATCH 4/5] docs: a growing degradation count is the positive reading, and entries do decay MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections to how content-types gets read, both now measured rather than argued. The standing belief is that content-types is a cumulative since-boot counter with no decay, so an entry may be historical. That is half right and the wrong half is load-bearing. ContentTypeHealthCheck calls Unresolved, which re-asks the registry per entry at probe time under BOTH routes, and Clear removes an entry the moment a read of that type succeeds. Measured over 2.7 h on one deployment: the named set CHANGED and members LEFT it — which a counter that never decayed could not do. And the part that is cumulative is an instrument rather than a caveat, if two probes are compared. The count only rises when a read degrades. Measured on memex.meshweaver.cloud, Hosting/DeploymentStatus went x260/x257 to x373/x376 across 2.7 h — those reads are degrading now, not at boot. A count that is static across two probes says the opposite, and is the only case where an entry may be a residue nothing has since asked the registry to clear. That matters because every cheap signal near this defect is an absence, and this is one of the few positive ones available without cluster access. Documentation.Test 604/604. Co-Authored-By: Claude Opus 5 (1M context) --- .../Architecture/DynamicContentTypeRegistration.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md index aba56dd67d..3c6975b866 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md +++ b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md @@ -83,7 +83,15 @@ falsifies both stated causes and is the cheapest way to recognise this failure - **An entry's PRESENCE is a live verdict, not history.** `ContentTypeHealthCheck` calls `ContentDegradationRegistry.Unresolved`, which re-asks the registry per entry at probe time (both routes: the NodeType path and the stored `$type`). A boot-race entry disappears the moment - its type registers. Only the **×count** is cumulative since boot. + its type registers, and `Clear` removes one the moment a read of that type succeeds. Only the + **×count** is cumulative since boot. Measured: over 2.7 h on one deployment the named set + CHANGED and members left it — which a counter that never decayed could not do. +- **A GROWING count is positive evidence, and it is the reading to take.** The ×count only rises + when a read degrades, so comparing two probes turns the weakest part of this instrument into its + strongest: measured on memex.meshweaver.cloud, `Hosting/DeploymentStatus` went ×260 and ×257 to + ×373 and ×376 across 2.7 h. Those reads are degrading NOW. A static count across two probes says + the opposite — nothing has read that type since — and is the case where the entry may be a boot + residue the registry has simply never been asked to clear. - **An entry's ABSENCE is not a clean bill.** A degradation is recorded only when a READ degrades. A type nobody has opened on this replica appears in no list, however unregistered it is. - **Repeated `/health` calls sample different replicas.** Four calls returned four different sets; From afeb162c385d7f87e655686e2548033c6e8b2175 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20B=C3=BCrgi?= <6334612+rbuergi@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:40:32 +0200 Subject: [PATCH 5/5] =?UTF-8?q?docs:=20withdraw=20the=20test=20evidence=20?= =?UTF-8?q?for=20the=20unsafe=20fix=20=E2=80=94=20the=20same=20test=20fail?= =?UTF-8?q?s=20without=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I published a measurement that did not hold, and caught it only by running the control again on the final tree. The claim: with the degrade-seam registrar wired, ANodeTypesSourcesWaitForItsBundleTest failed "Expected AdoptedVerified ... but found Compiled" while clean main ran 861/861 green. That is the exact shape the hazard predicts — an adopted type driven into a compile — so it read as confirmation and I wrote it into the page, the code comment, the PR body and two issue comments. It is not confirmation. The same test failed again on THIS tree, which has the registrar removed and a 862-test assembly identical in count to main's. The diff cannot reach that test. One green run of main is not a control, and two failures against one pass attributes nothing. What survives is the code: ResolveHubConfiguration -> EnrichWithNodeType is handed the compilation service and can write the shared NodeType record (DispatchPending flips a stale-Ok record to Pending to force a recompile) and arms the rebind watcher. A read seam must not do that on a row the whole deployment shares. That argument never needed the test and is now stated as what it is — a reading of the code. The intermittent failure is filed separately rather than explained away: an assertion that an ADOPTED type is never driven through a COMPILE, failing intermittently, is either a test-isolation defect or the race this page is about, and guessing which is how a wrong root cause gets published. Documentation.Test 604/604; Release -warnaserror clean on MeshWeaver.Graph. Co-Authored-By: Claude Opus 5 (1M context) --- .../DynamicContentTypeRegistration.md | 16 ++++++++++++---- .../ContentTypeRegistrationSweep.cs | 4 ++-- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md index 3c6975b866..217e396b37 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md +++ b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md @@ -152,10 +152,18 @@ Getting a dynamic type's hub configuration means `IMeshNodeHubFactory.ResolveHub A NodeType record is ONE row the whole deployment shares. Letting a read seam take that path means every non-owning replica that happens to read a node independently compiles and re-stamps the record — which is precisely the mid-roll cross-stamp `NodeTypeLiveRecordCensus` was built to -DETECT as a fault, and the hazard behind the leaving-hub and per-replica-compile rules. Measured: -with the seam wired that way, `ANodeTypesSourcesWaitForItsBundleTest` failed with *"Expected -AdoptedVerified … but found Compiled"* — a held, correctly-adopted type driven into a compile by -nothing but a read. +DETECT as a fault, and the hazard behind the leaving-hub and per-replica-compile rules. **That +argument is a reading of the code, and it is the whole case.** + +🚨 **It was nearly propped up with a measurement that does not hold, which is worth recording.** +With the seam wired, `ANodeTypesSourcesWaitForItsBundleTest` failed *"Expected AdoptedVerified … +but found Compiled"* while clean `main` ran 861/861 green — the exact shape the hazard predicts, and +it read as proof. It is not: the same test failed again on a tree with the seam REMOVED and the +same 862-test assembly as `main`, so the diff could not reach it. One green control run is not a +control. The test is intermittent, the attribution is withdrawn, and the failure is filed on its +own — an assertion that an ADOPTED type is never driven through a COMPILE, failing intermittently, +is either a test-isolation defect or the very race this page is about, and guessing which is how a +wrong root cause gets published. **So registration belongs with the component already sanctioned to activate dynamic types on this replica: the pre-warmer.** Its `AlreadyBaked` branch is where the skip happens, it runs once per diff --git a/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs b/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs index c3d8a6a33b..594e055a0e 100644 --- a/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs +++ b/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs @@ -76,8 +76,8 @@ namespace MeshWeaver.Graph; /// flips it to Pending to force a recompile) and arms the rebind watcher. A read seam that /// did this would let every non-owning replica independently compile and re-stamp one record that /// the whole deployment shares — the cross-stamp NodeTypeLiveRecordCensus exists to DETECT. -/// Measured: an adopt-only branch wired that way turned an AdoptedVerified type into -/// Compiled (ANodeTypesSourcesWaitForItsBundleTest). Registration belongs with the +/// That is a reading of the code and it is the whole case: a test failure that looked like +/// confirmation was withdrawn when the same test failed with the seam removed. Registration belongs with the /// component already sanctioned to activate dynamic types on this replica — the pre-warmer, whose /// AlreadyBaked branch is where the skip happens — not with whoever reads a node. See /// Doc/Architecture/DynamicContentTypeRegistration.