diff --git a/src/MeshWeaver.Documentation/Data/Architecture.md b/src/MeshWeaver.Documentation/Data/Architecture.md index 9b81f352fb..6271d867d0 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture.md +++ b/src/MeshWeaver.Documentation/Data/Architecture.md @@ -283,6 +283,7 @@ Each theme starts with its introductory page, followed by related architecture t - [Install-Time Prebuilt Adoption](InstallTimePrebuiltAdoption) — the only lane that serves a package installed AFTER boot, and the four answers its zero must keep apart because a silent non-adoption reads exactly like a successful one - [Adoption and the Sweep Count Different Things](AdoptionAndTheSweepCountDifferentThings) — the cold boot that adopted 78 prebuilt assemblies and then reported 5, with nothing wrong on the share: what each instrument counts, why the sweep could not see its own process's writes, and the node-version ordering that keeps the fix from becoming a stale serve - [A Census That Counts Must Name](ACensusThatCountsMustName) — the one past-RLS census counted a permanently-broken NodeType and dropped its path one call before publication, so its output read as clean; where the identity was lost, what a PUBLIC census may name (the partition, never the node title), and how to tell a fix that is merged from a fix that is running +- [Dynamic Content Type Registration](DynamicContentTypeRegistration) — a dynamic NodeType's content CLR type registers only as a side effect of one of its INSTANCES activating in this process, so a type this replica adopted rather than compiled is untypeable here with a perfectly usable assembly, a clean bake and a clean census; the instance-count discriminator, the disjoint instrument sets that falsify “the bundle was declined”, and why closing it from the degrade seam drives an adopted type into a compile and re-stamps a record the whole deployment shares - [Denied Is Not Absent](DeniedIsNotAbsent) — a Forbidden that a read discards comes back as the thing being MISSING, so the operator announces a node pool, an Ingress or a ConfigMap is absent when it was merely not permitted to look; the three answers a read owes its reader, why fixing the sites a report names does not sweep the defect, and the gate that makes the next one red - [Import Write Ordering](ImportWriteOrdering) — type before instance, and what a foreign type does - [Language Services](LanguageServices) diff --git a/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md new file mode 100644 index 0000000000..217e396b37 --- /dev/null +++ b/src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md @@ -0,0 +1,206 @@ +--- +Name: Dynamic Content Type Registration +Category: Architecture +Description: A dynamic NodeType's content CLR type registers only as a side effect of one of its instances activating in THIS process — so a type whose few instances live on another replica is untypeable here, with a usable assembly, a clean bake and a clean census. The mechanism, the measurements that separate it from a declined bundle, and why the obvious on-demand fix must not ship. NOT yet closed. +Icon: +--- + +# Dynamic Content Type Registration + +**A replica can hold a NodeType's assembly, count it baked, report a clean census — and still be +unable to type a single node of it.** Not because the bundle was declined, not because the bytes +are missing, but because nothing in that process ever *registered* the CLR type. This page is +about the third cause, which is the common one, and which no instrument named until now. + +## The mechanism + +A dynamic NodeType — in-mesh `Source/*.cs`, compiled by Roslyn at runtime — declares its content +type inside its own hub configuration. The CLR type enters the process-wide +`IMeshContentTypeRegistry` as a **side effect of that configuration being built** +(`MeshDataSource.WithContentType`), and the configuration is built when a **per-node hub +cold-activates**. Three consequences follow, and none of them is obvious from any one of them: + +1. **A per-node hub is a single activation cluster-wide.** A node's hub lives on exactly one + silo. Building it registers the type *there*. +2. **Every other replica reads the same nodes through its own `MeshNodeStreamCache`**, receives + the payload as JSON, asks its own registry for the `$type`, and is told nothing. The content + stays a bare `JsonElement`. +3. **Nothing at boot closes the gap.** `DynamicTypePreWarmer`'s adopt-only pass *asks, never + builds*, and it is the default path for every deployment. Where the compiling sweep does run, a + type the shared assembly store already holds is reported `AlreadyBaked` and — in the warmer's + own words — *"never activated"*. That is the normal case for **every replica after the first**, + and for every replica of a roll onto an identity that is already baked. + +So the exposure is precisely: **a dynamic NodeType with FEW instances, all of whose instances +happen to activate elsewhere.** A type with many instances gets activated on most replicas sooner +or later and registers itself. A type with zero instances has nothing to read. A type with four — +`Hosting/Deployment`, `Hosting/DeploymentStatus`, `Store/Subscription`, `PG3Reporting/Fund` — is +the one that breaks, and it breaks differently on each replica. + +### The premise that was false + +`ContentTypeRegistrationSweep` registers static definitions at boot and deliberately excludes +dynamic ones, for two good reasons (the per-NodeType boot cost the CI content bake removed, and +the loader's corrupt-file self-heal tripping on an adopted-but-not-yet-loadable bake) and one +that does not hold: + +> A dynamic type registers the moment an instance hub activates — and a dynamic type with **zero +> instances** has no payload carrying its discriminator, so there is nothing to degrade. + +True for zero. False for few. The sentence quietly assumes one process. + +## What it looks like from outside + +It looks like a declined bundle, and it is not one. `content-types` on `/health` says: + +> N node type(s) whose content this replica cannot type — the module that declares the type is not +> loaded here (**its prebuilt bundle was declined or its compiled assembly is not on this +> replica**), so their pages render empty + +Both stated causes are real, and neither was what was measured. The discriminating readings, taken +on two live portals on 2026-09-21: + +| instrument | memex.systemorph.com (core `746b4e48`) | memex.meshweaver.cloud (core `026442ff`) | +|---|---|---| +| `bake-report` | **Healthy** — 229 of 230 types carry a usable assembly | Healthy — 242–302 of 366 with a verdict, all usable | +| the ONE / THREE types with **no** usable assembly | `BinaryClickerV2` (CompileError) | 3 × CompileError, all in partition `MeshWeaver` | +| live record census | **clean** — 0 untyped, 0 foreign, 230 of 230 | clean on each replica sampled | +| `content-types` | `Hosting/DeploymentStatus ×30`, `Store/Plugin ×8` | 8–11 types per replica, **a different set on each**, incl. `Hosting/Deployment` | + +The provenance of a named type closes it: `Store/Plugin` on the control instance reads +`compilationStatus: Ok`, `compiledFrameworkVersion: se271838…` (the live identity) and +**`buildProvenance: AdoptedVerified`** — adopted from the share for exactly this framework, so the +pre-warm reported it `AlreadyBaked` and never activated it. The assembly is verified present and +the type is unregistered: those are not in tension, they are the same sentence read twice. + +🚨 **The two sets are DISJOINT.** The types with no usable assembly are not the types that cannot +be typed, and the types that cannot be typed all have usable assemblies. That single comparison +falsifies both stated causes and is the cheapest way to recognise this failure — it needs one +`/health` body and no cluster access. + +### Reading the instrument correctly + +- **An entry's PRESENCE is a live verdict, not history.** `ContentTypeHealthCheck` calls + `ContentDegradationRegistry.Unresolved`, which re-asks the registry per entry at probe time + (both routes: the NodeType path and the stored `$type`). A boot-race entry disappears the moment + its type registers, and `Clear` removes one the moment a read of that type succeeds. Only the + **×count** is cumulative since boot. Measured: over 2.7 h on one deployment the named set + CHANGED and members left it — which a counter that never decayed could not do. +- **A GROWING count is positive evidence, and it is the reading to take.** The ×count only rises + when a read degrades, so comparing two probes turns the weakest part of this instrument into its + strongest: measured on memex.meshweaver.cloud, `Hosting/DeploymentStatus` went ×260 and ×257 to + ×373 and ×376 across 2.7 h. Those reads are degrading NOW. A static count across two probes says + the opposite — nothing has read that type since — and is the case where the entry may be a boot + residue the registry has simply never been asked to clear. +- **An entry's ABSENCE is not a clean bill.** A degradation is recorded only when a READ degrades. + A type nobody has opened on this replica appears in no list, however unregistered it is. +- **Repeated `/health` calls sample different replicas.** Four calls returned four different sets; + that variation *is* the finding, not noise. The per-replica form with no guesswork is the + control instance's `Sample` action, but it truncates each health body — the untruncated bodies + come from calling the public `/health` several times. + +## The blast radius — and the consumer it does NOT reach + +**Everything that reads a node's content through `MeshNodeStreamCache` and then asks for the CLR +type is affected**: a view renders empty, `Content is X` / `as X` yields null, and a reactive wait +on a typed shape never completes. There is no exception and no log line naming a cause beyond the +degradation warning itself. + +🚨 **It does NOT reach a consumer that uses `ContentAs` with a statically-known `T`, and +getting that wrong is easy.** `ContentAs` deserialises to the type the CALLER names; it never +consults the mesh-wide registry, which is exactly why it is the mandated accessor. So a lane that +reads a record this way is immune to this defect even on a replica that cannot type the same node +for a view. + +That distinction was worth an issue comment to retract. `SelfUpdateRouting` answering *"no +`Hosting/Deployment` record … lists 0"* +([Plugins#2178](https://github.com/Systemorph/MeshWeaver.Plugins/issues/2178), a 24-hour CD stall) +reads exactly like this defect and is NOT it, twice over: its lookup folds over INDEX ROWS +(`MatchRecord` on path/id, no content), and `DeploymentContent` is a framework type shipped in the +image, reached by reference. That zero was core#4958 — a zero with two causes, "none exist" and "I +could not see them", treated as one — and the lane now answers `RecordVerdict.Unseen` at Error +instead of concluding absence. + +The lesson generalises: **a silent empty is not evidence of this defect.** Before attributing one +here, check whether the consumer names its type statically; if it does, look elsewhere. + +🚨 **A restart does not fix it and neither does a recycle.** A fresh activation re-reads, but what +it *finds* is the same shared store and the same skip: `AlreadyBaked`, never activated, never +registered. Rolling forward "fixes" it only by accident — a new framework identity makes the +prebuilts stale, which forces a local compile, and a local compile activates the hub and registers +the type. That is why the remedy looked like a build-artifact remedy and why it kept coming back. +See [Stale State Until Recycle](../StaleStateUntilRecycle) for the general rule this is an +instance of, and for why a second recycle proves nothing the first did not. + +## Closing it — and the fix that must NOT be shipped + +🚨 **The obvious fix is unsafe, and this is the most useful thing on this page.** The tempting +move is to call the existing `ContentTypeRegistration.ProbeRegister` from the degrade seam: the +place that already *reports* the gap would *close* it, on demand, once per type, at zero boot cost. +That was built and it works — and it must not ship, because of what obtaining the configuration +costs. + +Getting a dynamic type's hub configuration means `IMeshNodeHubFactory.ResolveHubConfiguration` → +`NodeTypeEnrichmentHelpers.EnrichWithNodeType`, and that path is **not a read**: + +- it is handed the `IMeshNodeCompilationService` and **triggers the compilation chain** when the + type's sources are ahead of its build; +- it **writes the shared NodeType record** — the stale-`Ok`-with-null-assembly self-heal flips the + record to `Pending` to force a recompile; +- it **arms the rebind watcher** for that address. + +A NodeType record is ONE row the whole deployment shares. Letting a read seam take that path means +every non-owning replica that happens to read a node independently compiles and re-stamps the +record — which is precisely the mid-roll cross-stamp `NodeTypeLiveRecordCensus` was built to +DETECT as a fault, and the hazard behind the leaving-hub and per-replica-compile rules. **That +argument is a reading of the code, and it is the whole case.** + +🚨 **It was nearly propped up with a measurement that does not hold, which is worth recording.** +With the seam wired, `ANodeTypesSourcesWaitForItsBundleTest` failed *"Expected AdoptedVerified … +but found Compiled"* while clean `main` ran 861/861 green — the exact shape the hazard predicts, and +it read as proof. It is not: the same test failed again on a tree with the seam REMOVED and the +same 862-test assembly as `main`, so the diff could not reach it. One green control run is not a +control. The test is intermittent, the attribution is withdrawn, and the failure is filed on its +own — an assertion that an ADOPTED type is never driven through a COMPILE, failing intermittently, +is either a test-isolation defect or the very race this page is about, and guessing which is how a +wrong root cause gets published. + +**So registration belongs with the component already sanctioned to activate dynamic types on this +replica: the pre-warmer.** Its `AlreadyBaked` branch is where the skip happens, it runs once per +process, it is sequenced after bundle seeding, and it already holds the bake's verdict that the +type has usable bytes here. The shape that fits the two original objections is a **registration-only +pass over the `AlreadyBaked` types, as a paced background trickle AFTER the sweep settles** — off +the readiness path, so the per-NodeType boot cost the content bake removed stays removed, and after +the bake has settled, so no probe meets an adopted-but-not-yet-loadable bundle. That pass covers +every adopted type rather than only the ones somebody read, which also closes the blind spot below. + +The cost is real and is a judgement call, not a detail: it is the ~13.5 s of assembly opening that +#1660 removed from boot, moved to a background trickle. That trade wants the maintainer's decision, +which is why this page describes the shape rather than shipping it. + +## What this does not establish + +- **No fix is shipped and neither issue is closed.** What is established is the mechanism, the + measurements that separate it from a declined bundle, and that the cheap fix is unsafe. +- **No production verification.** The acceptance measurement for any fix is positive, not an + absence: on a replica reporting a type under `content-types`, that entry disappears from the next + `/health` probe while `bake-report` is unchanged, and the type's record still reads + `buildProvenance: AdoptedVerified` afterwards — a fix that heals by compiling has re-stamped the + shared record and is the unsafe shape wearing a green tick. +- **The residual population is not bounded.** Every dynamic NodeType this replica adopted rather + than compiled is unregistered here until something reads one; the measurement can name only the + ones that HAVE been read. On the control instance that is 229 of 230 types adopted, and 2 named. +- **Whether a type also fails to register after a genuine local compile** was not tested; every + case measured was an adoption. + +## See also + +- [NodeType Compilation](../NodeTypeCompilation) — how an in-mesh type becomes an assembly. +- [Module Build Architecture](../ModuleBuildArchitecture) — the bake, the share, and why a second + replica finds the first one's work. +- [A Content Verdict Is Per Node](../AContentVerdictIsPerNode) — the neighbouring rule about what + a per-node verdict can and cannot be read to cover. +- [A Census That Counts Must Name](../ACensusThatCountsMustName) — why a count without a + denominator is not a measurement. +- [Operating From The Portal](../OperatingFromThePortal) — where `/health`, `Sample` and the + per-replica readings come from. diff --git a/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs b/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs index 793bb9277d..594e055a0e 100644 --- a/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs +++ b/src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs @@ -56,8 +56,31 @@ namespace MeshWeaver.Graph; /// 13.5 s of a 101 s boot); worse, a probe of an adopted-but-not-yet-loadable bake trips the /// loader's corrupt-file self-heal, which DELETES the store's bytes and forces a re-adoption /// (ShippedPrebuiltBundlesTest pins that boot contract). A dynamic type registers the moment an -/// instance hub activates — and a dynamic type with zero instances has no payload carrying its -/// discriminator, so there is nothing to degrade. +/// instance hub activates. +/// +/// 🚨 That last clause used to end "…and a dynamic type with zero instances has no +/// payload carrying its discriminator, so there is nothing to degrade", and the premise is false +/// for a type with FEW instances (Systemorph/MeshWeaver.Plugins#2178, Systemorph/MeshWeaver.Plugins#2180). A per-node hub is +/// a single activation cluster-wide, so a type whose handful of instances all activate on ANOTHER +/// replica registers there and nowhere else — while every other replica reads those nodes through +/// its own stream cache and cannot bind the $type. Measured on two live portals 2026-09-21: +/// 8-11 such types per replica, a different set on each, Hosting/Deployment and +/// Hosting/DeploymentStatus among them, every one of them carrying a usable assembly. The +/// boot scope here is unchanged; what is corrected is the REASON, because the old one reads as +/// "nothing is exposed" and something is. +/// +/// 🚨 Do NOT close it by calling from a degrade seam. +/// Getting a dynamic type's configuration means +/// IMeshNodeHubFactory.ResolveHubConfiguration → NodeTypeEnrichmentHelpers, which +/// TRIGGERS THE COMPILATION CHAIN and can WRITE the shared NodeType record (the stale-Ok self-heal +/// flips it to Pending to force a recompile) and arms the rebind watcher. A read seam that +/// did this would let every non-owning replica independently compile and re-stamp one record that +/// the whole deployment shares — the cross-stamp NodeTypeLiveRecordCensus exists to DETECT. +/// That is a reading of the code and it is the whole case: a test failure that looked like +/// confirmation was withdrawn when the same test failed with the seam removed. Registration belongs with the +/// component already sanctioned to activate dynamic types on this replica — the pre-warmer, whose +/// AlreadyBaked branch is where the skip happens — not with whoever reads a node. See +/// Doc/Architecture/DynamicContentTypeRegistration. /// public static class ContentTypeRegistration { diff --git a/src/MeshWeaver.Hosting/ContentDegradationRegistry.cs b/src/MeshWeaver.Hosting/ContentDegradationRegistry.cs index b6d783b5e4..6f09f9d592 100644 --- a/src/MeshWeaver.Hosting/ContentDegradationRegistry.cs +++ b/src/MeshWeaver.Hosting/ContentDegradationRegistry.cs @@ -132,9 +132,17 @@ public ImmutableList Snapshot() => public static string Describe(IReadOnlyList degraded) => degraded.Count == 0 ? "every node content read on this replica typed" - : $"{degraded.Count} node type(s) whose content this replica cannot type — the module " - + "that declares the type is not loaded here (its prebuilt bundle was declined or its " - + "compiled assembly is not on this replica), so their pages render empty: " + : $"{degraded.Count} node type(s) whose content this replica cannot type — the CLR type " + + "is not registered in THIS process, so their pages render empty. Commonest cause, and " + + "the one to check first: a dynamic NodeType registers its content type only when one " + + "of its instances activates here, and a type this replica ADOPTED rather than compiled " + + "is never activated by the bake — so a type with few instances, all activated on another " + + "replica, is untypeable here with a perfectly usable assembly " + + "(Doc/Architecture/DynamicContentTypeRegistration). Otherwise the module that declares " + + "the type is not loaded here: its prebuilt bundle was declined, or its compiled assembly " + + "is not on this replica. 🚨 bake-report DECIDES between them — if its no-usable-assembly " + + "list does not name a type below, that type's assembly is present and the first cause is " + + "the answer: " + string.Join("; ", degraded.Select(d => $"{d.NodeType} ×{d.Count} (last {d.LastPath})")); } diff --git a/test/MeshWeaver.Hosting.Test/ContentDegradationRegistryTest.cs b/test/MeshWeaver.Hosting.Test/ContentDegradationRegistryTest.cs index d9b2e6d3e1..66cf94605a 100644 --- a/test/MeshWeaver.Hosting.Test/ContentDegradationRegistryTest.cs +++ b/test/MeshWeaver.Hosting.Test/ContentDegradationRegistryTest.cs @@ -31,6 +31,16 @@ public void RecordsPerNodeType_AndTheSentenceNamesThem() var sentence = ContentDegradationRegistry.Describe(snapshot); sentence.Should().Contain("2 node type(s)").And.Contain("Crm/Client ×2").And.Contain("Store/Install ×1") .And.Contain("not loaded here", "the sentence says WHY, not only what"); + // 🚨 The sentence used to name only two causes — a declined bundle and a missing assembly — + // and BOTH were falsified on two live portals while the third went unnamed: a dynamic + // NodeType this replica ADOPTED is never activated by the bake, so its content type is + // never registered here (Plugins#2178, #2180). An instrument that asserts two causes as + // fact sends its reader past the real one, so the commonest cause is named FIRST and the + // instrument that discriminates is named with it. + sentence.Should().Contain("ADOPTED rather than compiled", + "the commonest cause is the one the sentence used not to mention"); + sentence.Should().Contain("bake-report DECIDES between them", + "naming three causes without naming what tells them apart is a longer way to mislead"); } [Fact]