Repository navigation
docs: a dynamic NodeType registers its content type only where an instance activates (Plugins#2178, #2180) - #5079
Merged
meshweaver-cloud[bot] merged 5 commits intoSep 21, 2026
Conversation
…tance activates Plugins#2178 and #2180 both report a Hosting type that one replica cannot type. Both attribute it to a declined prebuilt bundle. Measured on two live portals, that is not what is happening, and the difference decides the fix. A dynamic NodeType's content CLR type enters IMeshContentTypeRegistry as a side effect of its hub configuration being built, which happens when a per-node hub cold-activates. A per-node hub is a single activation cluster-wide. DynamicTypePreWarmer's adopt-only pass ASKS, NEVER BUILDS (the default for every deployment), and its compiling sweep reports a type the shared store already holds as AlreadyBaked and "never activated". So on every replica after the first, a dynamic type whose few instances activate elsewhere has a usable assembly and no registration — and every read of one degrades. ContentTypeRegistrationSweep's own reason for skipping dynamic types carried the false half of this: "a dynamic type with zero instances has no payload carrying its discriminator, so there is nothing to degrade". True for zero, false for few. Corrected in place, with the measurements. Measured 2026-09-21, read-only: - memex.systemorph.com: bake-report Healthy, 229/230 usable, live record census clean (0 untyped, 0 foreign) — while content-types named Hosting/DeploymentStatus and Store/Plugin as untypeable on that replica. - memex.meshweaver.cloud: 8-11 untypeable types per replica, a DIFFERENT set on each, Hosting/Deployment among them; the 3 types with no usable assembly are all in partition MeshWeaver, disjoint from every named type. - The discriminator is instance count, not bundle state: Hosting/InstanceAction has >=200 instances and is untypeable nowhere; Hosting/DeploymentStatus has 5 and Hosting/Deployment has 4, and both are untypeable now. - Store/Plugin reads buildProvenance: AdoptedVerified on the live framework identity — adopted, never compiled, never activated. Also records the fix that must NOT ship, because it was built and measured: calling ProbeRegister from the degrade seam obtains the configuration through EnrichWithNodeType, which triggers the compilation chain and WRITES the shared NodeType record (the stale-Ok self-heal flips it to Pending). A read seam doing that lets every non-owning replica compile and re-stamp one row the whole deployment shares — the cross-stamp NodeTypeLiveRecordCensus exists to detect. Control on each side, full MeshWeaver.Hosting.Test suite: clean main 861/861 green; with the seam wired, ANodeTypesSourcesWaitForItsBundleTest failed "Expected AdoptedVerified ... but found Compiled". No behaviour change here. Registration belongs with the pre-warmer's AlreadyBaked branch as a post-sweep background trickle; that trade (the ~13.5 s of assembly opening #1660 removed from boot) is a maintainer decision. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The new doc page’s front-matter description claims it includes “on-demand registration that closes it” while the page content explicitly says no fix ships and that the on-demand degrade-seam fix is unsafe (plus a minor ambiguous issue reference in XML docs).
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
Documents and clarifies a production failure mode where dynamic (runtime-compiled) NodeType content types are only registered on the replica that activates an instance hub, leaving other replicas able to load assemblies yet still unable to type node content.
Changes:
- Adds a new architecture page explaining the registration mechanism, observed measurements, and why the “register-on-degrade” fix is unsafe.
- Updates
ContentTypeRegistrationSweepXML docs to correct the rationale for excluding dynamic types and to point readers to the new doc page.
| File | Description |
|---|---|
| src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs | Refines XML documentation around dynamic type registration behavior and documents the unsafe “probe from degrade seam” approach. |
| src/MeshWeaver.Documentation/Data/Architecture/DynamicContentTypeRegistration.md | New architecture doc capturing the mechanism, diagnostics, and acceptance criteria for any future fix. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| --- | ||
| Name: Dynamic Content Type Registration | ||
| Category: Architecture | ||
| Description: A dynamic NodeType's content CLR type registers only as a side effect of one of its instances activating in THIS process — so a type whose few instances live on another replica is untypeable here, with a usable assembly, a clean bake and a clean census. The mechanism, the measurements that separate it from a declined bundle, and the on-demand registration that closes it. |
Contributor
Author
There was a problem hiding this comment.
Correct, and it is the worst place for this page to be stale: the Description is what a catalog reader sees INSTEAD of the body. It is left over from an earlier draft that did ship a degrade-seam registrar; the body now records that that fix was built, measured, and must NOT ship (it drives an AdoptedVerified type into a compile and re-stamps a record the whole deployment shares).
Rewritten to say what the page actually delivers: the mechanism, the measurements that separate it from a declined bundle, and why the obvious fix is unsafe — no claim that anything is closed.
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
…n one was neither Copilot findings, both real: - the page's front-matter Description still promised "the on-demand registration that closes it" from a draft that did ship one; the body says the opposite. The Description is what a catalog reader sees INSTEAD of the body, so that is the worst place for it to be stale. Rewritten, and it now says NOT yet closed. - "Systemorph/MeshWeaver.Plugins#2178, #2180" left the second number unqualified. Core has a #2180 too, so the bare form does not fail to resolve — it resolves to the wrong issue. Both fully qualified. And the thing those two issues were actually misdiagnosed BY. ContentDegradationRegistry .Describe told every reader of /health that a type it cannot bind means "the module that declares the type is not loaded here (its prebuilt bundle was declined or its compiled assembly is not on this replica)". Measured on two live portals, neither was happening for any named type, and the sets are disjoint: on memex.systemorph.com the ONE type with no usable assembly is BinaryClickerV2, and the two untypeable ones are Hosting/DeploymentStatus and Store/Plugin; on memex.meshweaver.cloud the three with no usable assembly are all in partition MeshWeaver and none is named. An instrument that asserts two causes as fact sends its reader past the real one — twice, at sev:H. The sentence now names the commonest cause FIRST (a dynamic NodeType this replica ADOPTED rather than compiled is never activated by the bake, so its content type is never registered here), keeps the other two, and names the instrument that decides between them: if bake-report's no-usable-assembly list does not name a type, that type's assembly is present and the first cause is the answer. Pinned by test. Also adds the page to Data/Architecture.md — ArchitectureTopicMapTest caught the omission: a working URL alone does not make a page discoverable. MeshWeaver.Documentation.Test 604/604; the ContentDegradation/UntypedContent seam tests 13/13; Release -warnaserror clean on MeshWeaver.Graph and MeshWeaver.Hosting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It does not, and the page asserting it would have sent the next reader down a branch already excluded. Two independent reasons, both measured after the page was written: - SelfUpdateRouting's lookup folds over INDEX ROWS — ResolveRecord → MatchRecord(rows, reference), path/id only. No content is typed, so a degraded content type cannot turn a listed record into a zero. - DeploymentContent is a FRAMEWORK type (core src/MeshWeaver.Deployment.Contract), shipped in the image and reached by reference; Apply reads the record with ContentAs<DeploymentContent>, which deserialises to a statically-known T and never consults the mesh-wide registry. Hosting/PlatformBuildInbox shares six named files from Hosting/Deployment/Source, SelfUpdateRouting among them, and no content declaration. That zero was core#4958 — a zero with two causes treated as one — and the lane now answers RecordVerdict.Unseen at Error instead of concluding absence (Plugins, 2026-09-20 15:18Z). The section is replaced with what the defect DOES reach: every consumer that reads through MeshNodeStreamCache and then asks for the CLR type. And with the boundary that matters more, because it is the one that makes a false attribution easy: ContentAs<T> with a statically-known T is immune, so a silent empty is not by itself evidence of this defect. Retracted on the issue as well. Documentation.Test 604/604. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s do decay Two corrections to how content-types gets read, both now measured rather than argued. The standing belief is that content-types is a cumulative since-boot counter with no decay, so an entry may be historical. That is half right and the wrong half is load-bearing. ContentTypeHealthCheck calls Unresolved, which re-asks the registry per entry at probe time under BOTH routes, and Clear removes an entry the moment a read of that type succeeds. Measured over 2.7 h on one deployment: the named set CHANGED and members LEFT it — which a counter that never decayed could not do. And the part that is cumulative is an instrument rather than a caveat, if two probes are compared. The count only rises when a read degrades. Measured on memex.meshweaver.cloud, Hosting/DeploymentStatus went x260/x257 to x373/x376 across 2.7 h — those reads are degrading now, not at boot. A count that is static across two probes says the opposite, and is the only case where an entry may be a residue nothing has since asked the registry to clear. That matters because every cheap signal near this defect is an absence, and this is one of the few positive ones available without cluster access. Documentation.Test 604/604. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ails without it I published a measurement that did not hold, and caught it only by running the control again on the final tree. The claim: with the degrade-seam registrar wired, ANodeTypesSourcesWaitForItsBundleTest failed "Expected AdoptedVerified ... but found Compiled" while clean main ran 861/861 green. That is the exact shape the hazard predicts — an adopted type driven into a compile — so it read as confirmation and I wrote it into the page, the code comment, the PR body and two issue comments. It is not confirmation. The same test failed again on THIS tree, which has the registrar removed and a 862-test assembly identical in count to main's. The diff cannot reach that test. One green run of main is not a control, and two failures against one pass attributes nothing. What survives is the code: ResolveHubConfiguration -> EnrichWithNodeType is handed the compilation service and can write the shared NodeType record (DispatchPending flips a stale-Ok record to Pending to force a recompile) and arms the rebind watcher. A read seam must not do that on a row the whole deployment shares. That argument never needed the test and is now stated as what it is — a reading of the code. The intermittent failure is filed separately rather than explained away: an assertion that an ADOPTED type is never driven through a COMPILE, failing intermittently, is either a test-isolation defect or the race this page is about, and guessing which is how a wrong root cause gets published. Documentation.Test 604/604; Release -warnaserror clean on MeshWeaver.Graph. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Plugins#2178 (CD stalled 24 h) and Plugins#2180 (
Hosting/InstanceActionuntypeable on one replica) both say a Hosting type is untypeable because its prebuilt bundle was declined. Measured read-only on both live portals today, that is not the mechanism — and the difference decides where the fix goes.Root cause
A dynamic (in-mesh, Roslyn-compiled) NodeType's content CLR type enters
IMeshContentTypeRegistryas a side effect of its hub configuration being built, which happens at a per-node hub's cold activation. A per-node hub is a single activation cluster-wide, and nothing at boot closes the gap:DynamicTypePreWarmer's adopt-only pass "ASKS, NEVER BUILDS" — and it is the default path for every deployment;AlreadyBakedand, in the warmer's own words, "never activated" — the normal case for every replica after the first;ContentTypeRegistrationSweepcovers static definitions only.So a dynamic type whose few instances all activate on another replica has a usable assembly and no registration here, and every read of one degrades to a bare
JsonElement: pages render empty,ContentAs<T>yields null, andSelfUpdateRoutingasking "is there aHosting/Deploymentrecord" is answered 0 while the rows are in the table.A declined prebuilt cannot be the cause: the decline path logs "DECLINED … — compiling instead", and a local compile activates the hub and registers the type. The decline was correlated, not causal.
Measurements (2026-09-21, read-only, both portals)
746b4e48)026442ff)bake-reportBinaryClickerV2MeshWeavercontent-typesHosting/DeploymentStatus ×30,Store/Plugin ×8Hosting/Deployment🚨 The two sets are DISJOINT — the types with no usable assembly are not the untypeable ones, and vice versa. The discriminator is instance count:
Hosting/InstanceActionhas ≥200 instances and is untypeable on no replica sampled;Hosting/DeploymentStatushas 5 andHosting/Deploymenthas 4, and both are untypeable right now.Store/PluginreadsbuildProvenance: AdoptedVerifiedon the live framework identity — adopted, never compiled, never activated.Also corrected here:
content-typesentry presence is a live verdict, not a since-boot residue —ContentTypeHealthCheckcallsContentDegradationRegistry.Unresolved, which re-asks the registry per entry at probe time. Only the ×count is cumulative.The fix that must NOT ship — measured, with a control on each side
Calling
ContentTypeRegistration.ProbeRegisterfrom the degrade seam is the obvious fix. It was built, it works, and it is unsafe: obtaining a dynamic type's configuration goes throughEnrichWithNodeType, which triggers the compilation chain and writes the shared NodeType record (the stale-Okself-heal flips it toPending). A read seam doing that lets every non-owning replica independently compile and re-stamp one row the whole deployment shares — the cross-stampNodeTypeLiveRecordCensusexists to DETECT.🚨 A measurement that looked like proof of this has been WITHDRAWN, and the withdrawal is in the diff. With the seam wired,
ANodeTypesSourcesWaitForItsBundleTestfailed "ExpectedAdoptedVerified… but foundCompiled" while cleanmainran 861/861 — the exact shape the hazard predicts. Re-running the control on the FINAL tree, with the seam removed and a 862-test assembly identical in count tomain's, the same test failed again. The diff cannot reach it; one green run ofmainis not a control. The hazard argument above never needed the test and stands on the code alone; the intermittent failure is filed separately rather than explained away.The behavioural change is not in this PR. Registration belongs with the component already sanctioned to activate dynamic types on this replica — the pre-warmer's
AlreadyBakedbranch — as a registration-only pass on a paced background trickle after the sweep settles. That trade (the ~13.5 s of assembly opening #1660 removed from boot) is a maintainer decision, so the page describes the shape rather than shipping it.What is in the diff
Doc/Architecture/DynamicContentTypeRegistration): the mechanism, the measurements, how to readcontent-types, the unsafe fix and why, and the acceptance test any fix must pass. Listed in the Architecture topic map —ArchitectureTopicMapTestcaught the omission.ContentTypeRegistrationSweep's stated reason for skipping dynamic types, corrected in place. The old sentence ("a dynamic type with zero instances has no payload … so there is nothing to degrade") reads as "nothing is exposed", and something is. It now also says, at the one place someone would go to write the fix, why the degrade-seam version must not be it./healthsentence itself — the instrument that misdiagnosed both issues.ContentDegradationRegistry.Describeasserted two causes as fact ("its prebuilt bundle was declined or its compiled assembly is not on this replica") and the commonest one was neither. It now names that one FIRST, keeps the other two, and names what decides between them: ifbake-report's no-usable-assembly list does not name a type, that type's assembly is present. Pinned by test.SelfUpdate's "lists 0". It does not — that lookup folds over index ROWS andDeploymentContentis a framework type reached by reference, soContentAs<T>never asks the registry. That zero was core#4958, since fixed. The section now carries the boundary instead, because it is what makes a false attribution easy.The only runtime change is the text of one
/healthdescription string.dotnet build -c Release -warnaserrorclean onMeshWeaver.GraphandMeshWeaver.Hosting;MeshWeaver.Documentation.Test604/604; theContentDegradation/UntypedContentseam tests 13/13.What this does not establish
No production verification, and neither issue is closed. The acceptance measurement for any fix is positive, never an absence: on a replica reporting a type under
content-types, that entry disappears from the next/healthprobe whilebake-reportis unchanged and the record still readsbuildProvenance: AdoptedVerified— a fix that heals by compiling has re-stamped the shared record and is the unsafe shape wearing a green tick.🤖 Generated with Claude Code