fix(#3472): a process refuses a build keyed to a framework identity that is not its own, and stops reporting Ok for it by rbuergi · Pull Request #3509 · Systemorph/MeshWeaver

rbuergi · 2026-09-06T23:58:58Z

"a NodeType's self-report is not proof" — AGENTS.md, written before the day it was proved.

#3491 (MeshPublicationGate, d54cf3d80) is the membership half of 2026-09-06: a pod its
readiness gate had refused kept running and kept publishing. This is the adoption half — what a
healthy, admitted process must refuse to load, and what it may honestly report about a build it
cannot load. It composes with that gate rather than duplicating it: that one decides whether
this process may publish; this one adds nothing to publish.

Criterion 2 first, because it is the one that needed a design

🚨 CompilationStatus.Ok is a claim SCOPED to CompiledFrameworkVersion, persisted as though it
were absolute. "The last compile succeeded" is true and process-independent. "These bytes can be
loaded" is only ever answerable relative to a process. The record has always carried both halves
— the verdict and its scope — in two separate fields, and every instrument read the first one.

That is why no writer-side rule fixes it alone: the pod that wrote Ok could load what it had
just built. It was telling the truth about itself. So the answer is split, and each half is necessary.

Half one — the writer, and a live defect that needs no second image

NodeTypeContractHandler.ResolvedStoreVersion already argues the rule, for the store-key version:

The path and the version are ONE reference, so they must come from ONE source.

The framework identity is the fourth member of that reference and was left out of it.
ApplyResolvedSuccess decided the assembly path and the store key by "did the response carry a
reference" and decided CompiledFrameworkVersion by a different predicate (freshCompile). On the
HYDRATE path those disagree — and a hydrate only succeeds when the assembly store resolved
bytes, under a key whose framework tag is this process's own. The record then reads:

compilationStatus        Ok
latestAssemblyPath       <bytes the store just handed us>
compiledFrameworkVersion <somebody ELSE's identity>

Ok, naming loadable bytes, declaring them unloadable. HasUsableBuild is false forever; every
per-instance activation takes the ABI-stale recompile path; after MaxRecompileAttempts the instance
binds the fallback configuration — and a hub resolves its configuration exactly once, so that is
"Area not found" for the grain's whole lifetime, under a green tick. The handler also races the
activity write-back, so it re-imposes the foreign stamp over each correct one and the type never
converges. One extracted predicate (StoreVersionFromResponse / ReferenceCameFromResponse) now
decides all four fields, so they cannot drift. The retained branch still retains — that is the
only case where an ABI-staleness marker is real, and stamping live over retained coordinates would be
the same #1368 defect reached the other way round.

Half two — the reader: `CompilationStatus.Foreign`, derived and never persisted

Foreign-Ok records still arrive, and no writer rule can stop them: a peer replica on another
image (#3395), a node repo that COMMITS a record (Plugins ships Store/Catalog with a July hash), a
prebuilt bundle. NodeTypeBuildIdentity.ReportedStatus folds the pair.

Why a fourth state rather than a redefinition — each candidate carries a different, wrong remedy:

state	says	why not
`Error`	correct the code	the code is fine — #641's defect in the other direction
`Unavailable`	retry or wait	the state is fully determined; waiting is what does not help
`Ok`	it is fine	the lie

Foreign says "recompile here", which is the remedy, and which the compile watcher already
drives. 🚨 Nothing writes it. Writing a reader-relative verdict into a shared record is exactly
how #3395's ping-pong was made — two replicas, two identities, each correctly overwriting the other's
answer forever. The record keeps saying what the compiler did; only the report is scoped.

Reported through: get_diagnostics (status: "Foreign", both identities in error), the
compile-progress overlay (which used to read the green status and redirect the viewer back to the
page that cannot render, which bounced them back — now it holds and explains, localized), and the
NodeType overview progress line (which showed ✓ Compiled while printing the foreign hash beside
it as decoration).

Criterion 1 — the refusal, at seven load paths instead of one

HasUsableBuild has carried the framework equality since #464 — on one of the seven paths that
load a NodeType's assembly. The other six gated on the two coordinate fields being populated, or on
CompilationStatus == Ok alone: both pinned-release branches (NodeTypeEnrichmentHelpers,
NodeTypeContractHandler — NodeTypeRelease.FrameworkVersion exists and nothing read it), the
published-release hydrate, MeshDataSource.HandleNodeTypeSchemaRequest,
NodeTypeDataModelAreas.ResolveInstanceHubConfig, MeshOperations.ResolveHubConfigForSchema, and
CellSurfaceAssemblyProvider (which takes a lifetime lease).

🚨 What stood between those and a foreign load was an eight-character substring inside one store
implementation's glob — FileSystemAssemblyStore's v{version}-{FrameworkTag}-*.dll. Nothing
asserted that property, and the other implementation (BlobAssemblyStore) ships in a module this
repository cannot compile. An implicit guard is not a guard.

Refusing is not "the type is dead". Every site takes exactly the branch a store miss already
takes — a local compile, a fallback configuration, or a no-answer — so behaviour on a tag-carrying
store is unchanged, and the operator gains a line naming both identities. Naming only one is how
the bake gate's "regressed on this image" sent three investigations to the wrong repository that
same night.

Falsification — three arms, with numbers

Two pure arms on the functions that decide it, one end-to-end on a real mesh (MonolithMeshTestBase,
a real Roslyn compile, no mocking).

arm	result
Fix present	`AdoptTimeFrameworkIdentityTest` 17/17, `ForeignFrameworkBuildIsRefusedTest` 1/1, and `HydrateIsNotACompileTest` (#2895's no-op pins) 6/6 unchanged
Writer REVERTED (`CompiledFrameworkVersion = freshCompile ? live : def.…`)	1 failed / 23 — `ResolvedSuccessNeverLeavesOkStandingOverAForeignIdentity`: the record kept `sc273ee39fdccbfc088f9aaf1fc548a9a`, the incident's own identity, where the live one belonged. The other 22, including all six #2895 no-op pins, stayed green — the change does not touch the hydrate no-op
Reader REVERTED (`ReportedStatus => definition?.CompilationStatus`)	1 failed / 17 — `AForeignOkNeverReportsOk`. The other 16, including the `Ok`-control and the five pass-through statuses, stayed green
Cell-surface gate REVERTED (real mesh)	1 failed / 1 — "Expected value to be false … but found True": the foreign-identity assembly was joined into the kernel session's cell surface and leased
Controls, always asserted first	this process's own build is not refused and does report `Ok`; the pack is on the cell surface with an honest build. Without them every assertion would pass against a gate that refused everything

The mesh test asserts the premise after the measurement — it re-reads the record and requires it
to still be foreign before judging the resolution — so a healed record makes the test say so
instead of passing. Not a retry: the framework-stale kickoff's store probe answers Skip here
(bytes exist under the live tag), which is exactly why the state is durable.

Local verification (Release, -warnaserror, one project per invocation): MeshWeaver.Mesh.Contract,
MeshWeaver.Compiler.Pipeline, MeshWeaver.Graph, MeshWeaver.Mesh.Operations,
MeshWeaver.Messaging.Hub, MeshWeaver.Hosting, MeshWeaver.Documentation, Memex.Portal.Shared —
0 warnings, 0 errors each. MeshWeaver.Compiler.Pipeline.Test 673/673, MeshWeaver.Graph.Test
1382/1382, MeshWeaver.Hosting.Test 404/404, MeshWeaver.Documentation.Test 304/304
(link integrity, What's New integrity, every localization guard), OrleansCompileActivityAccessTest
8/8 — including FrameworkStaleAssembly_SelfHealsOnInstanceActivation, the test the
ResolveStaleBuildAction invariant note says goes red when this rule is relaxed.

🚨 Two findings that falsify a premise, both measured while building this

#3472's own hypotheses. (a) a pod on a different platform build compiled these types was
confirmed on the issue and is the incident's mechanism. (b) the watcher adopted a prebuilt published
under a foreign identity was not needed to explain anything observed — and this change found the
third route the issue did not name: the writer can mint the foreign-Ok record itself, with one
image, no bundle and no peer, through ApplyResolvedSuccess's hydrate path.

🚨 search 'nodeType:NodeType compilationStatus:Error' — AGENTS.md's stated sweep instrument —
cannot see this, and on some hosts cannot see anything. Not changed here (both fixes have blast
radius far beyond this issue), documented instead:

The sweep never returns the value. MeshOperations' search envelope projects
Path/Name/NodeType/Version/LastModified only; compilationStatus is a WHERE clause and nothing
more.
🚨 On an in-memory or FileSystem-backed host the filter matches NOTHING.
QueryEvaluator.GetDirectPropertyValue resolves a selector by reflection against MeshNode,
which has no compilationStatus property and no Content fallback — so the comparison is
null == "Error" and the sweep returns count: 0 for a mesh full of broken types. The
translation that makes it work is in PostgreSqlSqlGenerator, which is not in this repository.
A green sweep on a dev Monolith is evidence of nothing. The instrument that is
reader-relative is get_diagnostics, which now answers Foreign.

Also filed for the record in the doc page: MeshDataSource.HandleNodeTypeSchemaRequest and
MeshOperations.ResolveHubConfigForSchema were reading node.Content is NodeTypeDefinition /
(NodeTypeDefinition)node.Content — the forbidden payload cast. Both are now ContentAs.

Scope boundaries (deliberate)

It does not stop two images sharing one mesh — that is fix(#3478): a process its readiness validation refuses no longer JOINS the mesh #3491 (membership) and Roll selection must pick the latest platform version that ships ALL of an environment's plugins — completeness is a selection criterion, not a post-hoc verdict #3479 (roll
selection).
It does not bind a record to bytes it does not name. DecideStaleBuildAction still answers
Skip when the framework moved and the store holds live-tag bytes; nothing has hashed those bytes,
and restamping would assert validity for a build the lane never examined. That boundary is
unchanged — and it is why a record can sit at Ok with a foreign identity indefinitely, which is
what makes half two necessary rather than cosmetic.
It does not make a foreign record heal without an activation. The instance-activation self-heal
is still the path that corrects it.

Also in this change

Doc/Architecture/BuildIdentityAdmission — the design, the seven load paths, the fourth-state
argument, and the two sweep findings. Cross-linked from MeshAdmission, which now names its
counterpart.
A What's New entry (Category: Fix).
Two localization keys (ui.compileForeignFramework, …Body) in both catalogs. 🚨 The Plugins
mirror needs the hand-over after this merges: npm run sync:i18n -- --ref <merged core sha>. It
reds nothing today (the guard compares against a pinned core commit), which is exactly why it has
to be said here.

No public type or member is removed, and no interface gains a member, so neither cross-repo gate
applies.

🤖 Generated with Claude Code

…hat is not its own, and stops reporting Ok for it Closes #3472 The adoption half of the 2026-09-06 outage; #3491 (`MeshPublicationGate`) was the membership half. `Crm/Offer` and `Crm/Opportunity` reported `compilationStatus: Ok` for two and a half hours while every one of their per-instance hubs was dead. `CompilationStatus.Ok` is a claim SCOPED to `CompiledFrameworkVersion`, persisted as though it were absolute. The record has always carried both halves — the verdict and its scope — in two separate fields, and every instrument read the first one. That is why no writer-side rule fixes it alone: the pod that wrote `Ok` could load what it had just built. THE WRITER HALF — the identity travels with the coordinates. `NodeTypeContractHandler.ApplyResolvedSuccess` decided the assembly path and the store key by "did the response carry a reference" and decided `CompiledFrameworkVersion` by a different predicate (`freshCompile`). On the HYDRATE path those disagree, and a hydrate only succeeds when the store RESOLVED bytes — under a key whose framework tag is this process's own. The record then said `Ok`, named loadable bytes, and declared them unloadable; `HasUsableBuild` is false forever, and after `MaxRecompileAttempts` the instance binds the fallback configuration for the grain's whole life ("Area not found", under a green tick). One extracted predicate now decides all four fields. The retained branch still retains — that is the only case where an ABI-staleness marker is real. THE READER HALF — `NodeTypeBuildIdentity`, one pure predicate with two consumers, deliberately the shape `MeshPublicationGate` uses because the defect is a divergence. `RefusalReason` gates the six load paths that never consulted `HasUsableBuild`; each takes exactly the branch a store MISS already takes, so behaviour on a tag-carrying store is unchanged and the operator gains a line naming BOTH identities. `ReportedStatus` folds the pair into `CompilationStatus.Foreign`. `Foreign` is a fourth state because each existing one carries a different, wrong remedy: `Error` says "correct the code" (#641's defect in the other direction), `Unavailable` says "retry or wait" (the state is determined, and waiting does not help), `Ok` is the lie. It is DERIVED and never persisted — writing a reader-relative verdict into a shared record is exactly how #3395's ping-pong was made. Log level: the refusals are `LogError`, matching the sibling `NodeTypeExecutionGate` refusal in `CellSurfaceAssemblyProvider`. This is a verdict about the bytes that only a rebuild or a rebake changes, not an availability problem that resolves itself — the 2026-09-06 cost of it being unsaid was two and a half hours of a client portal, so the Loki volume is bought deliberately. It fires once per refused load site, not per request. Doc: `Doc/Architecture/BuildIdentityAdmission`, cross-linked from `MeshAdmission`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

🟡 Changes recommended

There are two confirmed correctness/i18n issues in the updated code paths (stored as PR comments) that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR tightens NodeType build adoption by enforcing a framework-build-identity match at load time, and by reporting a derived non-persisted CompilationStatus.Foreign when a record says Ok but is not loadable by the current process—addressing the outage class described in #3472.

Changes:

Introduces CompilationStatus.Foreign (derived at read time) and NodeTypeBuildIdentity to centralize “refuse vs report” decisions.
Enforces the identity refusal across previously unguarded NodeType assembly load paths (schema probes, pinned releases, cell surface, diagnostics reporting).
Adds end-to-end + unit tests, localized UI copy for the “foreign framework” overlay, and architecture/What’s New documentation.

File summaries

File	Description
test/MeshWeaver.Compiler.Pipeline.Test/ForeignFrameworkBuildIsRefusedTest.cs	End-to-end mesh test asserting foreign-identity builds are not served and do not report `Ok`.
test/MeshWeaver.Compiler.Pipeline.Test/AdoptTimeFrameworkIdentityTest.cs	Unit tests pinning refusal and reported-status behavior, plus the writer-side hydrate stamping invariant.
src/MeshWeaver.Messaging.Hub/Localization/strings.en.json	Adds localized keys for the foreign-framework compile overlay (EN).
src/MeshWeaver.Messaging.Hub/Localization/strings.de.json	Adds localized keys for the foreign-framework compile overlay (DE).
src/MeshWeaver.Mesh.Operations/MeshOperations.cs	Uses scoped/derived status for diagnostics and refuses foreign builds in schema hub-config recovery.
src/MeshWeaver.Mesh.Contract/Services/CompilationStatus.cs	Appends `Foreign` enum member with documentation stating it is derived and not persisted.
src/MeshWeaver.Graph/NodeTypeLayoutAreas.cs	Uses `ReportedStatus` to avoid redirect loops and renders localized foreign-framework messaging.
src/MeshWeaver.Graph/NodeTypeDataModelAreas.cs	Refuses foreign builds when resolving instance hub configuration, falling back to node config.
src/MeshWeaver.Graph/MeshDataSource.cs	Uses `ContentAs` and refuses foreign builds in the NodeType schema probe path.
src/MeshWeaver.Graph/Configuration/NodeTypeEnrichmentHelpers.cs	Refuses pinned releases built for a different framework identity, returning the existing “assembly unavailable” overlay path.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-07-a-page-no-longer-reports-ok-when-it-cannot-open.md	What’s New entry describing the user-visible behavioral change.
src/MeshWeaver.Documentation/Data/Architecture/MeshAdmission.md	Cross-links the membership/admission doc to the new adoption/identity doc.
src/MeshWeaver.Documentation/Data/Architecture/BuildIdentityAdmission.md	New architecture doc detailing the design, enforcement sites, and scope boundaries.
src/MeshWeaver.Compiler.Pipeline/NodeTypeContractHandler.cs	Refuses foreign-build hydration, adds pinned-release framework check, and fixes hydrate stamping so identity follows resolved coordinates.
src/MeshWeaver.Compiler.Pipeline/NodeTypeBuildIdentity.cs	New centralized predicate for refusal + derived reported status for foreign-identity `Ok`.
src/MeshWeaver.Compiler.Pipeline/CellSurfaceAssemblyProvider.cs	Refuses joining foreign-identity assemblies into the kernel session cell surface.

Review details

Files reviewed: 16/16 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+                // ContentAs, never a cast: the Where above already accepted this node, but the
+                // value can still arrive as raw JSON on a hub that did not register the type.
+                var def = node.ContentAs<NodeTypeDefinition>(hub.JsonSerializerOptions);
+                if (def is null)
+                    return Observable.Return<NodeCompilationResult?>(null);


+    private static string Short(string? identity)
+        => string.IsNullOrEmpty(identity)
+            ? "(none)"
+            : identity[..Math.Min(8, identity.Length)];


github-actions · 2026-09-07T00:03:06Z