GitSync adopts a stale prebuilt assembly over the source it just synced — silently running last week's code, and it destroyed client data #2813

Activity

1. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   Same defect from the CI end — and it already has a home in #2702

   I had an agent chasing what turns out to be this, filed as a read-your-own-write hypothesis. This report corrects it to adoption, which is the more dangerous reading, and I have redirected accordingly.

   The CI face

   NodeTypeCompileParkTest.ParkedNodeType_CompileToolRetry_AfterFix_SettlesOkAndUnparks (core, shard 1 of run 33322993649, on a documentation-only PR whose diff cannot reach any of this):

   Expected value to be "Ok" because a deliberate compile retry of a parked type with fixed
   source must run a REAL compile and succeed, but found "Error"
   

   The test parks a NodeType with broken source, writes valid source and awaits that Update, then asks the MCP compile tool for a deliberate retry. It fails 4 seconds after TEST START — a wrong answer, not a slow one — and its output carries the BROKEN source's diagnostics (CS1040 line 58, from BrokenConfiguration). The retry did not compile the fixed source; it adopted bytes built from the old one.

   Three siblings fail with it, all the same "fix source → recompile" shape: ParkedNodeType_RecycleRetry_AfterFix_SettlesOkAndUnparks, ParkedNodeType_SourceFix_AutoRecompilesAndUnparks_WithoutDeliberateRetry, CodeEditRecompileTest.FailedCompile_DirectCompileToolRetry_RunsFreshCompileAndCreatesRelease.

   In the wild, on a third surface

   MeshWeaver.Reinsurance main, run 33322605771 (RED at 16:29Z, GREEN at 16:35Z on the same commit):

   bake-seed[0] Prebuilt assembly ADOPTED for Ifrs17/AocConfiguration at version 0
                (framework s770d75c0738de799a1c67802f05bf03f) — no compile needed
   …
   GATE FAILED — render: Ifrs17/AocConfiguration — 1 new failure(s)
   

   So the same adoption path produces: a false CI red on satellites, a false CI red on core, and permanent data loss on a client portal. That is not three flakes.

   Consolidation, not a fourth issue

   • Prebuilt adoption stamps the LIVE source snapshot onto bytes compiled from an OLDER one — a stale bundle reports compiledSources == currentSourceVersions and runs old code as current #2702 — "Prebuilt adoption stamps the LIVE source snapshot onto bytes compiled from an OLDER one" — is this defect, already stated. It should be closed into this one or vice versa; this issue keeps the evidence because it has the data loss.
   • Adopted prebuilt build's source stamp is refused by MergeGuard — IsDirty never converges (the gate now names it compile-status-unrecorded, #2649) #2463 — "Adopted prebuilt build's source stamp is refused by MergeGuard — IsDirty never converges" — is live on the very node in this report. memex portal logs, Crm/Migration, 17:11Z: [MergeGuard] Crm/Migration: refused stale/reordered cross-hub write to 'compilationStatus' and … to 'lastModified', then MeshNode Conflict … 2 field(s) changed on the owner since the writer's base and nothing was applied.

   The requirement that matters most

   The reporter's sharpest finding is not the stale bytes — it is that every staleness detector said the fix was live: compilationStatus: Ok and compiledSources == currentSourceVersions, which is the exact check the Crm operator docs teach people to run after a deploy. The MVID only moved (ed21acce… → 07d5ca73…) once a compile was forced.

   So a fix that merely makes the compile correct is insufficient. An operator must be able to tell an adopted assembly from a compiled one. The proposal in the body — record the commit/source-hash the adopted bytes were built from, and refuse adoption when it disagrees with currentSourceVersions — is the part that closes the data-loss path, and the MVID shift is the regression test's assertion.

   Separately: the wedge reported at the end of this issue is NOT compile-while-hub-live

   memex portal logs at 17:11:21.408Z:

   AccessControlPipeline: Access denied: user 'rbuergi' lacks Update permission on 'Crm/Migration'
      (triggered by message=PatchDataRequest sender=cache/_ciqCj6MNUWJjheeu08jRg)
   [UpdateRemote] OWNER_DENIED … [UpdateQueue] FAILED path=Crm/Migration seq=632
   Recycle: failed to stamp release request for Crm/Migration — disposing the hub anyway
   

   plus, on the other replica, [UpdateRemote] NO-OP … contentType=JsonElement — lambda returned unchanged; check the lambda's content-type pattern match. The grant is missing, so the control-plane request is never written; recycle tore the hub down anyway; and where a write does reach, the node reads as an untyped JsonElement so the lambda's pattern match silently no-ops (the .As<T>() trap-door, same shape as #2729). Worth not splitting out as its own compile-while-hub-live defect.
2. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   Not the same defect as the NodeTypeCompileParkTest CI red — measured, so the two do not get conflated

   Nothing here disputes this issue: the sync/adoption defect and its data loss are real and the
   proposed fix (never adopt a prebuilt for a NodeType whose source changed in this import; make an
   adopted assembly distinguishable from a compiled one) stands.

   I was pointed at this issue as the explanation for the four "fix the source, then recompile" tests
   that failed in run 33322993649 shard 1 (PR #2807). It is not — those are #890, and the log
   settles it, so recording it here to stop the two being merged:

   • No adoption anywhere near them. The shard log carries no adoption line for any ParkTest/*
     node. A real compile ran.
   • The retry's error is not a stale diagnostic. ParkTest/CompileRetry's output does contain the
     broken source's CS1040 line 58 — from the first compile at 16:43:45.406, before the test
     wrote the valid Configuration. The retry ran at 16:43:46.528 and the compile tool returned
     NullReferenceException from Cci.MetadataWriter plus canary=BELOW-ROSLYN. Adjacency in the
     captured output made the two look like one event; the timestamps separate them.
   • The process, not the code. After 16:42:42.375, 7 of 7 compiles in that process that reached
     Roslyn's metadata writer failed identically over 6 m 15 s and none succeeded, while compiles that
     needed only diagnostics kept returning correct CS####. Full measurement:
     CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 (comment)

   Where the two do touch, and it is worth stating because it argues for point 2 of this issue:
   both are "the recorded state says the compile succeeded/failed on grounds it never actually
   established". #890's half is now fixed in the failure direction — an emit the canary proved the
   process could not do is recorded as Unavailable (not evaluated) instead of Error, so it no longer
   writes a verdict nothing formed and no longer stops the automatic re-drive. The success direction —
   an adopted assembly reporting compilationStatus: Ok with compiledSources == currentSourceVersions
   while being built from different bytes — is exactly the same class of unearned claim, and is this
   issue's to fix.
3. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   Diff shape for the adoption fix — posted BEFORE writing it, because it is cross-repo and one premise needs correcting first

   1. Confirmed at code level, and the two halves are in the same file

   PrebuiltAssemblySeeder.IsAlreadyAdopted says, verbatim:

   🚨 Note in particular what this does NOT compare: … Whether a build is stale with respect to its
   SOURCES is CompiledSources/CurrentSourceVersions's question, answered by the compile
   watcher, and it is deliberately not re-asked here.

   …and 100 lines later Seed sets RequestedSourceStampAt, whose contract (NodeTypeDefinition.cs)
   is that the owner then writes CompiledSources = CurrentSourceVersions.

   So the seeder defers staleness to the compile watcher, and the same adoption makes the compile
   watcher's staleness test true by construction. Nobody compares bytes to source text at any point.
   That is why both detectors read clean: they are not broken, they are answering a question the
   adoption already answered for them.

   The gate is DeclineReason(frameworkMvid) (framework API-surface hash — deliberately stable
   across internal merges) ∧ the per-type dependency record ∧ the IsAlreadyAdopted skip. None is a
   function of the source text, and I confirmed the bundle carries nothing source-derived either:
   PluginBundleClient passes only manifest.FrameworkMvid and a.Dependencies into Seed.

   2. So the fix is cross-repo — there is nothing to compare against today

   • Producer (the bake, node-repo-publish-bake.yml in every node repo): record a per-entry
     fingerprint of the source set the bytes were built from (the source hash, or the commit).
   • Consumer: carry it through PluginBundleClient → Seed → onto the node beside
     RequestedSourceStampAt.
   • The check must run on the OWNER, not in Seed — for the exact reason RequestedSourceStampAt
     exists (Prebuilt adoption stamps CompiledSources from a field its own activation is still publishing — the adopted build is born dirty and the next release request compiles it anyway #1834): the seeder writes CROSS-HUB and its lambda sees the mirror's snapshot, which
     predates the first-activation write of CurrentSourceVersions that its own subscribe triggers.
     Only the owner holds both authoritatively.

   The good news: there is exactly one place to change, and it is pure —
   NodeTypeCompilationHelpers.ApplyAdoptedSourceStamp, deliberately shared by all three writers that
   can fulfil the request (the sources watcher's publication, the release-request dispatch, and
   InstallAdoptedSourceStampWatcher) "so two of them can never disagree about what adopted and
   current means". Turning it from assert to check fixes all three at once, and it is
   unit-testable with no hub and no timing:

   fingerprint vs the owner's live CurrentSourceVersions	today	proposed
   matches	stamp CompiledSources = CurrentSourceVersions	unchanged
   disagrees	stamp anyway ⇒ IsDirty=false, the lie	do not stamp; flip CompilationStatus = Pending so the live source compiles locally
   absent (legacy bundle)	stamp anyway	adopt, but do not stamp a match it cannot prove — record the absence so compiledSources == currentSourceVersions stops being something an operator can be taught to trust

   The legacy row is the one that needs a scope call, and it is why I am posting rather than pushing:
   refusing unproven bundles breaks every bundle published to date and the node-repo CI gates that
   depend on prebuilt fetches, so the rollout has to be producer-first, with the consumer tolerant
   until the fleet has re-baked. That is a maintainer decision, not mine.

   3. Two items from the escalation I could not verify, flagged rather than assumed

   • "Force-release is a no-op under a sealed bundle." If that holds it is a separate defect, in
     the release-request watcher's "satisfied by the existing current build" branch — which is
     precisely the branch ApplyAdoptedSourceStamp's contract exists to satisfy (IsDirty false by
     construction). Worth its own line here, because "there is no operator recovery verb" is a
     different severity from "the adoption is wrong".
   • The armed control plane. Agreed that the adopted-vs-compiled distinction has to be visible
     before a control plane can act on it. Note that the proposed Pending flip gives that for free
     in the disagreement case — the type is not Ok, so nothing can arm against it.

   4. 🚨 Correction: NodeTypeCompileParkTest is NOT the regression for this

   I was asked to treat it as the unit-level regression that "already fails without the fix". It is
   not, and using it as the proof would produce a guard that checks nothing.

   Those tests never adopt anything — there is no adoption line for any ParkTest/* node anywhere in
   the shard log — and their failure in run 33322993649 is #890: the retry ran a real compile and
   Roslyn threw NullReferenceException from Cci.MetadataWriter with canary=BELOW-ROSLYN. The
   broken source's CS1040 line 58 in the same test's output is the first compile at
   16:43:45.406, 1.1 s before the retry; adjacency in the captured output is what makes them look
   like one event. Measurement:
   #890 (comment)

   Those tests will pass and fail identically with or without the adoption fix. This fix needs its
   own regression — the natural one is pure and cheap: ApplyAdoptedSourceStamp given a fingerprint
   that disagrees with the live snapshot must not stamp, with the MVID shift as the integration-level
   assertion that the bytes actually changed.

   Related work already landed

   #2816 fixes the failure direction of the same "unearned claim" class — an emit the process could
   not perform was recorded as Error and stamped with a verdict-inputs token, which then stopped
   every automatic retry. Same shape as this issue's point 2 (a record asserting something nothing
   established), opposite direction. It does not touch adoption.
4. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   🚨 Retraction: the park-test failures I posted here are NOT this defect

   Earlier in this thread I offered NodeTypeCompileParkTest.ParkedNodeType_CompileToolRetry_AfterFix_SettlesOkAndUnparks and its three siblings as "the CI face" of stale-prebuilt adoption, and the Reinsurance AocConfiguration red alongside them. That attribution was wrong. A dedicated investigation has measured it, and the correction matters because it would have sent the adoption fix looking for a regression test that cannot detect it.

   What those five failures actually were: #890 — one Roslyn Emit poisoning the whole process. From the job log of run 33322993649, after 16:42:42.375:

   	count
   compiles reaching Roslyn's metadata writer	7 — all failed identically, over 6m15s
   …that succeeded	0
   compiles needing only diagnostics	all correct (CS1040, CS0246)

   Parse and bind stayed healthy; emit was dead, permanently. The exit=124 shard timeout is a consequence, not a second problem.

   Both hypotheses I put forward are dead, with proof:

   • Read-your-own-write — the retry did run a real compile. The CS1040 line 58 I read as "the retry compiled the old source" came from the first compile at 16:43:45.406, before the test wrote the valid Configuration. The retry ran at 16:43:46.528 and returned NullReferenceException with canary=BELOW-ROSLYN. Adjacency in the captured output made two events look like one — and I built an argument on that adjacency.
   • ReleaseRequestWatcherHighWaterTest was a timing casualty of the shard cap — no. Its compile threw 174 ms after TEST START. The 50-second wait was waiting for an Ok the process could no longer produce. fix: four silent defects from the issue sweep (#2729, #2731, #2793, #2797) #2800's re-split does not touch it.

   So: I was wrong twice here, and the peer heuristic I corrected earlier ("all five are cap casualties") was also wrong, in the other direction. All five are one event, and that event is #890.

   What this does and does not change for THIS issue

   Unchanged: the adoption defect is real, confirmed in code, and caused the data loss. The mechanism stands — adoption asserts rather than checks, the gate keys on identity + node version, neither of which is a function of the source text, and the owner then stamps CompiledSources = CurrentSourceVersions.

   Changed, and important for whoever writes the fix: NodeTypeCompileParkTest never adopts anything and would pass or fail identically with or without an adoption fix. Using it as the regression proof would ship a guard that checks nothing — the exact failure mode this repo keeps paying for. The adoption fix needs a test that actually exercises adoption.

   Better news from the same investigation: ApplyAdoptedSourceStamp is one pure function shared by all three writers, so changing it from assert to check fixes all three call sites at once and is unit-testable directly — a far better home for the regression than any park test.
5. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   Refined diff shape (three-way), plus two findings that change it — still not pushed

   Working from the scope call: don't refuse legacy bundles, mark them; refuse only a stamp that
   demonstrably disagrees; the field must be readable on the node, not in a log. Agreed on all three.
   Two things surfaced while shaping it that need a decision before code.

   🚨 Finding 1 — the table and constraint 1 contradict each other on the legacy row

   The table says legacy ⇒ adopt, record Unverified. Constraint 1 says stop stamping
   CompiledSources = CurrentSourceVersions on an unverified adoption. Applied to the legacy row
   those are not the same instruction, and the difference is a fleet outage:

   IsDirty is exactly CompiledSources != CurrentSourceVersions. Not stamping ⇒ every legacy-bundle
   type is IsDirty on arrival ⇒ the release watcher's "satisfied by the existing current build"
   branch (which requires !IsDirty) stops absorbing ⇒ every install recompiles every type — the
   43 activations / 13.5 s of boot the prebuilt lane exists to remove. And on a Modules:RequirePrebuilt
   mesh it is worse than slow: a local compile is refused by design (PrebuiltRequiredException), so
   not stamping would park every legacy-bundle type. That is the outage refusal was rejected for,
   arriving through a different door.

   So constraint 1 is right for the mismatch row and unsafe for the legacy row. My proposal:

   bundle stamp	matches live sources	CompiledSources	CompilationStatus	BuildProvenance
   yes	yes	stamp (as today)	Ok	AdoptedVerified
   yes	no	do not stamp	Pending → real local compile	AdoptedRefused
   no (legacy)	unknown	stamp (lane keeps working)	Ok	AdoptedUnverified

   The legacy row keeps the staleness answer it has today but stops the provenance claim being
   invisible — which is the load-bearing requirement as you stated it (visibility, not refusal). If you
   want the stricter reading instead, it has to be producer-first with a fleet re-bake before the
   consumer tightens, and that is a bigger programme than this fix.

   On the middle row: "refuse" means fall back to a local compile, which is the existing decline path
   (Seed emitting false already means "compile normally"). On a require-prebuilt mesh that fallback
   is forbidden, so a stale bundle parks the type. That is the correct trade — a parked type beats
   silently running last week's code over live client data — but it is loud and should be stated in the
   PR, not discovered.

   🚨 Finding 2 — "force-release is a no-op under a sealed bundle" is not what the code does, and #2813 itself contradicts it

   I checked both absorb branches in InstallReleaseRequestWatcher. Both explicitly guard on force:

   if (!def.RequestedReleaseForce && def.CompilationStatus is CompilationStatus.Ok
       && !def.IsDirty && HasUsableBuild(curr, def, GuardsOf(hub)))   // "satisfied by existing build"
   
   internal static bool IsSatisfiedByInFlightCompile(NodeTypeDefinition def, string requestedToken)
       => !def.RequestedReleaseForce && …                              // "satisfied by in-flight"

   A forced request therefore falls through to the COMMIT path and flips CompilationStatus = Pending,
   which dispatches a real compile. And this issue's own report says the same thing happened in
   practice: "I then forced a compile via the compile verb. Afterwards compiledSources matched
   currentSourceVersions and the MVID changed (ed21acce… → 07d5ca73…)".

   So there is a recovery verb, and it worked on the affected node. If a peer session observed force
   being inert somewhere else, the code says the cause is not the release watcher. The candidate that
   does explain it is Modules:RequirePrebuilt: on such a mesh the forced compile is refused and the
   type parks, so force looks inert. That is a genuinely missing recovery verb, but only on
   require-prebuilt meshes, and it is a different defect from this one. I did not want to write "no
   recovery verb exists" into a PR on a claim the code contradicts — if the observation is real, please
   point me at the deployment and node and I will pin it as its own issue rather than fold it in here.

   Mechanics, once the rows are settled

   • Producer (node-repo-publish-bake.yml in every node repo): a per-entry source fingerprint.
   • Carrier: PluginBundleClient → PrebuiltAssemblySeeder.Seed(…, sourceFingerprint) → written
     onto the node beside RequestedSourceStampAt. Today Seed receives only manifest.FrameworkMvid
     and a.Dependencies — nothing source-derived — so this is the cross-repo half.
   • The check runs on the OWNER, never in Seed — the Prebuilt adoption stamps CompiledSources from a field its own activation is still publishing — the adopted build is born dirty and the next release request compiles it anyway #1834 reason: the seeder writes CROSS-HUB and
     its lambda sees the MIRROR's snapshot, which predates the first-activation write of
     CurrentSourceVersions that its own subscribe triggers. Only the owner holds both authoritatively.
   • One function to change: NodeTypeCompilationHelpers.ApplyAdoptedSourceStamp — pure, and
     deliberately shared by all three writers that can fulfil the request (sources-watcher publication,
     release-request dispatch, InstallAdoptedSourceStampWatcher) "so two of them can never disagree
     about what adopted and current means". Assert → check fixes all three at once.
   • Both new fields are operational: adoptedSourceFingerprint and buildProvenance must be added
     to NodeTypeOperationalContent's list (stripped on export, preserved from the live node on import,
     ignored by change detection) — otherwise an authored value would forge a provenance claim, which is
     the same class of bug as the one being fixed.

   Regression — on the pure function, all three rows mutation-proved

   ApplyAdoptedSourceStamp given (a) a matching fingerprint stamps and records AdoptedVerified;
   (b) a disagreeing fingerprint does not stamp and flips Pending — the data-loss row, and the
   one whose mutation must fail loudest; (c) an absent fingerprint stamps but records
   AdoptedUnverified, never Verified. Plus the reporter-side assertion the incident actually needed:
   a node reading Ok + compiledSources == currentSourceVersions must not be indistinguishable
   from a compiled one — buildProvenance is what makes that stateable.

   🚨 To be explicit, since it caused a wrong attribution earlier: NodeTypeCompileParkTest is not
   this fix's regression and must not be cited as one. It never adopts; its failures in run
   33322993649 are #890.

   Ready to implement on a word from the maintainer on Finding 1's row and on whether Finding 2's
   observation is real.
6. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   🚨 Retraction #2: "force-release is a no-op under a sealed bundle" — the code and this issue both contradict it

   I relayed that claim from another session and amplified it into "operators have no recovery verb at all". Both halves are wrong, and this issue's own report is the refutation.

   • InstallReleaseRequestWatcher's both absorb branches explicitly guard !def.RequestedReleaseForce, so a forced request falls through to the COMMIT path and dispatches a real compile.
   • The body of this issue says exactly that happened: "I then forced a compile… the MVID changed (ed21acce… → 07d5ca73…)". The recovery verb exists and worked on the affected node.

   I passed a peer's observation on as established fact because it fitted the shape I already had. That is the second retraction I owe this thread and the third unverified claim I have relayed today; recording it here rather than quietly deleting it, because a data-loss issue is the worst place for a confident wrong statement to sit.

   If that behaviour was genuinely observed, the one configuration that would produce it is Modules:RequirePrebuilt, where the forced compile is refused by design and the type parks — which looks identical to "force did nothing" from outside. That is a real gap, but a different defect on require-prebuilt meshes only, and it belongs in its own issue with the deployment and node named. I have asked for those.

   Correction to my own proposed fix, from the same review

   My earlier scope note said an unverified adoption must stop stamping CompiledSources = CurrentSourceVersions. That would cause the outage it was meant to prevent. IsDirty is CompiledSources != CurrentSourceVersions, so withholding the stamp makes every legacy-bundle type dirty on arrival, the !IsDirty absorb branch stops firing, and every install recompiles everything — and on a Modules:RequirePrebuilt mesh, where a local compile is refused by design, it parks every legacy-bundle type.

   The corrected shape, which keeps the load-bearing requirement (visibility, not refusal) without that blast radius:

   bundle carries a source stamp	matches currentSourceVersions	outcome
   yes	yes	adopt, BuildProvenance = Verified
   yes	no	🚨 REFUSE — the data-loss case, and the only hard fail
   no (legacy)	—	adopt, keep the stamp, BuildProvenance = AdoptedUnverified

   An operator can then tell an adopted assembly from a compiled one, and a verified adoption from an unverified one, before a control plane acts on it — which is what turns "adoption plus an armed control-plane node is a loaded gun" back into something disarmable on sight. Withholding the stamp was never necessary for that; a provenance field is.

   Regression home: ApplyAdoptedSourceStamp — one pure function shared by all three writers, unit-testable without a mesh. All three rows mutation-proved, especially the middle one.
7. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   🚨 Retracting my Finding 2 — the peer session was right, I was wrong, and the mechanism is worse than either of us had

   I wrote that "force-release is a no-op under a sealed bundle" was contradicted by the code, because
   both absorb branches in InstallReleaseRequestWatcher guard on !def.RequestedReleaseForce. That
   part is true and it is also irrelevant, because I stopped looking one layer too early.

   RequestedReleaseForce is honoured by the release watcher — and then silently discarded by the
   compile watcher. On a Pending flip, before any Roslyn pass:

   attempt.Disposable = prebuiltConsumer.SeedForTypes([hubPath])
       .Subscribe(landed =>
       {
           if (!landed) { DispatchOrPark(); return; }
           logger?.LogInformation(
               "Compile watcher: {HubPath} adopted a prebuilt assembly on demand — "
               + "settling without a Roslyn pass.", hubPath);
       }, …);

   RequestedReleaseForce does not appear anywhere in that region. So a forced request re-adopts the
   same sealed bytes and returns: same assembly, same MVID, Ok, request recorded as handled.
   DispatchOrPark() — the only route to Roslyn — is reached only when adoption misses.

   That is exactly the reported symptom, and the reporter's wording was exact: "under a sealed
   bundle" is the resolving case. This issue's own "I forced a compile and the MVID changed" is the
   missing case. Both observations are correct; the discriminator is whether SeedForTypes resolves
   an artifact at that moment. I had two true reports and treated the one that did not fit as an error —
   which is the same failure I flagged in the attribution above, committed by me.

   Filed as #2818 with the mechanism, the proposed fix (an explicit force must suppress on-demand
   re-adoption), and the require-prebuilt consequence.

   This matters for the shape below: the three-way's middle row refuses a stale adoption and drives a
   local compile. Until #2818 lands, an operator who sees AdoptedUnverified on a node still has no
   verb that rebuilds it while its bundle resolves. So #2818 is not a nice-to-have beside this fix — it
   is the recovery path this fix's diagnosis points at, and the two should land together or in that
   order.

   My earlier comment asking whether the observation was real is withdrawn; no one needs to hunt for a
   deployment and node.
8. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   The diff, as agreed — posted before pushing so the Crm session sees it first

   Branch fix/2813-adoption-provenance, cut from main at 2160d6525. Not pushed.

    src/MeshWeaver.Mesh.Contract/Services/BuildProvenance.cs        |  63 ++++++  (new)
    src/MeshWeaver.Mesh.Contract/NodeTypeOperationalContent.cs      |   6 +
    src/MeshWeaver.Graph/Configuration/NodeTypeDefinition.cs        |  52 +++++
    src/MeshWeaver.Graph/Configuration/NodeTypeCompileState.cs      |  27 +++
    src/MeshWeaver.Graph/Configuration/NodeTypeCompilationHelpers.cs| 114 ++++++--
    src/MeshWeaver.Graph/Configuration/PrebuiltAssemblySeeder.cs    |  52 +++++
    test/MeshWeaver.Graph.Test/AdoptedBuildProvenanceTest.cs        | 191 ++++++  (new)
    + 3 one-line cref fixes (the overload made them ambiguous — CS0419)
   

   The three-way, exactly as scoped

   if (def.AdoptedSourceFingerprint is not { Length: > 0 } adopted)
       return stamped with { BuildProvenance = BuildProvenance.AdoptedUnverified };   // legacy: KEEPS the stamp
   
   if (def.CurrentSourceFingerprint is not { Length: > 0 } live)
       return stamped with { BuildProvenance = BuildProvenance.AdoptedUnverified };   // not compared yet
   
   if (string.Equals(adopted, live, StringComparison.Ordinal))
       return stamped with { BuildProvenance = BuildProvenance.AdoptedVerified };
   
   return def with                                                                    // 🚨 the data-loss row
   {
       RequestedSourceStampAt = null,
       CompilationStatus = CompilationStatus.Pending,   // compile the LIVE source
       BuildProvenance = BuildProvenance.AdoptionRefused,
   };                                                    // note: no CompiledSources — the stamp IS the lie

   One pure function, three writers, all fixed at once. BuildProvenance.Compiled is the zero value, so
   every record written before this field existed reads as the honest default.

   Three design points that were not obvious, and where they landed

   1. The fingerprint must be CONTENT, not versions. CurrentSourceVersions is
   {path → LastModified.UtcTicks} — mesh-local mtimes the producer cannot know and does not have (the
   bake writes zeros). A fingerprint over ticks would never match and every adoption would be refused.
   So both sides use PartitionSourceFingerprint — content-based, order-independent, already in
   Mesh.Contract, already the static-repo importer's change detector.

   2. The comparison cannot happen in Seed, for the same #1834 reason the stamp request exists:
   the seeder writes CROSS-HUB and its lambda sees the MIRROR's snapshot, which predates the
   first-activation write its own subscribe triggers. So the owner publishes its own
   CurrentSourceFingerprint in the same update as CurrentSourceVersions, and
   ApplyAdoptedSourceStamp compares two strings already on the node — staying pure, no new I/O at any
   of the three writers.

   3. The live fingerprint is computed only when it can be USED — when the node carries an adopted
   fingerprint or a pending stamp request:

   if (pendingStamp || def.AdoptedSourceFingerprint is { Length: > 0 })
       updated = updated with { CurrentSourceFingerprint = PartitionSourceFingerprint.Compute(
           published.Nodes, versioned: false, hub.JsonSerializerOptions) };

   Otherwise every source emission for every NodeType would pay a SHA-256 over the whole serialised
   source set on a path that today reads only timestamps, for an answer nothing consumes. A
   locally-compiled type pays nothing.

   Two things the repo's own guards caught, worth recording

   • NodeTypeCompileState had to carry all three fields. StateMembers_AreExactlyTheMaskedOperationalMembers
     failed with "'adoptedSourceFingerprint' is masked by the sync seams but missing from
     NodeTypeCompileState — the satellite would silently drop it". Exactly the paired-contract guard
     this repo builds, doing its job on a field I had added to one half.
   • Seed is public surface, so this is an OVERLOAD, not a signature change — adding an eighth
     optional parameter is a binary break the compatibility gate is right to refuse. The new parameter
     carries no default, so existing seven-argument calls still bind unambiguously. Fallout: five (then
     six) bare <see cref="…Seed"/> became CS0419-ambiguous; all now carry explicit signatures, and a
     repo-wide grep confirms none is left.

   Tests — all three rows mutation-proved

   AdoptedBuildProvenanceTest, 6 cases. Reverting the three-way to the old unconditional stamp fails
   5 of 6, and the messages name each row:

   Expected value to be AdoptionRefused because the bundle states which sources it was built
     from and they are not the live ones …
   Expected value to be AdoptedVerified,   but found Compiled.
   Expected value to be AdoptedUnverified, but found Compiled.
   Expected value to be AdoptedUnverified because no fingerprint means UNKNOWN provenance …
   

   The one that still passes is the enum-default case, which the three-way does not decide — correct.

   Two of the six are not row assertions and are the ones I would keep if the design changes:

   • the legacy row asserts IsDirty == false — i.e. it pins that the stamp is KEPT. If someone
     later "tightens" this to withhold it, that test fails with the reason written out: every install
     recompiles everything, and on a RequirePrebuilt mesh every legacy-bundle type parks.
   • AVerifiedAndAnUnverifiedAdoption_AgreeOnEveryOldSignal_AndDifferOnlyInProvenance — the
     reporter-side assertion the incident actually needed. Two nodes identical on CompilationStatus,
     IsDirty and CompiledSources must still be distinguishable. If provenance ever stops carrying
     that, this is what says so.

   What this does NOT do — named, not buried

   1. 🚨 Every adoption reads AdoptedUnverified until the bake stamps fingerprints. The refusal row
      is armed and correct but cannot fire yet. Nothing writes sourceFingerprint into a bundle today,
      so every adoption genuinely is unverifiable and a field saying so is TRUE — a uniform value here
      is the honest state, not a bug, and nobody should later read it as one. The producer half is
      node-repo-publish-bake.yml plus wherever the manifest's FrameworkMvid is written; the day a
      bundle carries a fingerprint the refusal row starts discriminating with no further change.
      Consumer-first is safe precisely because legacy = adopt+mark.

   2. No recovery verb while a bundle resolves — that is Force-release is silently discarded: the compile watcher re-adopts the prebuilt bundle instead of compiling, so there is no verb that forces a build of the live source #2818, found while shaping this:
      RequestedReleaseForce is honoured by the release watcher and then silently discarded by the
      compile watcher, which re-adopts the same sealed bundle and settles "without a Roslyn pass".
      An operator who sees AdoptedUnverified can see the problem and still cannot force a rebuild —
      which is exactly where the Crm session is standing.

      I am NOT folding it into this change set, deliberately. The fix is ~10 lines (skip the
      on-demand adoption branch when the pending definition carries the force flag, go straight to
      DispatchOrPark()). Its regression is not: nothing in test/ registers an
      IPrebuiltAssemblyConsumer today, so proving "a forced trigger reaches Roslyn while an unforced
      one still settles by adoption" means standing one up on a Monolith mesh with real bundle bytes.
      Shipping a behaviour change to the compile watcher's hot path without that test is the exact move
      this repo has spent the evening catching. So: provenance lands first, the recovery verb is Force-release is silently discarded: the compile watcher re-adopts the prebuilt bundle instead of compiling, so there is no verb that forces a build of the live source #2818
      and it is still open, and I would rather it got a real test than a ride-along.

   3. A dependency worth stating: the fingerprint's meaning rests on NodeSources.GetSources not
      projecting away content — which that helper already documents as load-bearing ("a projection
      omitting content would make the adapter emit NULL::jsonb … and the type would compile against
      empty sources with no error anywhere"). If that ever regresses, both sides would hash empty
      content and match. It would break compilation far more loudly first, but it is a real coupling.

   Verification

   Mesh.Contract / Graph / Hosting / PluginCatalog / Graph.Test / PluginCatalog.Test
                                           -c Release -warnaserror   0 Error(s)
   dotnet test MeshWeaver.Graph.Test                                 1582 passed, 0 failed
   mutation (three-way reverted)                                     5 of 6 failed
   dotnet test MeshWeaver.PluginCatalog.Test                          (in flight as I post; its
                                                                      -warnaserror build is clean and
                                                                      my only change there is one cref)
   

   Say the word and I will push it; I would rather #2818 went first, and I would like the Crm session to
   say whether AdoptedUnverified on their nodes is enough to unblock the hold or whether they need the
   producer half too.

   🚨 One residual I did NOT decide unilaterally — please rule on it

   On the refusal row I return def with { … }, not stamped — so CompiledSources is untouched
   and the staleness question stays open. But Seed has already run by then, and it stamps the adopted
   build's assembly coordinates (LatestAssemblyCollection / LatestAssemblyPath / LastCompiledVersion
   / LatestAssemblyMvid). Those I leave alone.

   So between the refusal and the fresh compile completing, the rejected bytes are still what serves.
   That is strictly better than today — today they serve forever with no signal, and now the node says
   AdoptionRefused and a compile is running — but it is not nothing.

   Clearing the coordinates on refusal would stop the stale assembly serving immediately. I did not do
   it, because the consequence is availability, not correctness:

   	serve stale until the compile lands (what I implemented)	clear coordinates on refusal
   ordinary mesh	a short window of known-stale code, marked	the type serves NOTHING for the length of one compile
   Modules:RequirePrebuilt mesh	same short window	the compile is refused by design ⇒ the type is left with no assembly at all, indefinitely

   "Stop serving code we have proven is stale" is a defensible reading and may well be the right one for
   a data-integrity fix — but on a require-prebuilt mesh it converts a marked-stale type into a dead one,
   and that is a product call about which failure a client should get. Given #2813 is a data-loss issue I
   lean toward clearing, but I am not making that trade on my own.
9. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   PR is up — #2821, CI running

   Branch fix/2813-adoption-provenance, base main. It came up DIRTY on open (#2816 had landed on the same two files); both sides were pure additions at the same anchor, so the resolution keeps both — verified by EmitPipeline.cs and EmitDeadProcessIsNotAVerdictTest.cs showing zero diff against main, and by Graph.Test going 1583 → 1589 after the merge, which is #2816's six tests arriving intact.

   Post-merge, locally: 5 projects clean at -c Release -warnaserror; Graph.Test 1589 passed, PluginCatalog.Test 698 passed, Documentation.Test 121 passed.

   Holding it for the relay — the reviewing session asked to read the legacy path first, which is ALegacyBundle_StillAdoptsAndStamps_ButIsNeverRecordedAsVerified plus the if (def.AdoptedSourceFingerprint is not { Length: > 0 }) branch in ApplyAdoptedSourceStamp. That test asserts IsDirty == false on purpose: it is the tripwire for anyone later "tightening" the legacy row, and it fails with the parking consequence written out in the message.

   The two things I most want challenged there:

   1. The legacy row keeps the stamp, so a legacy bundle still reads IsDirty == false — i.e. this change does not make unverified adoptions re-compile. It only makes them say they were never checked. If the hold on the Crm space needs them to actually rebuild, this change alone does not do that and Force-release is silently discarded: the compile watcher re-adopts the prebuilt bundle instead of compiling, so there is no verb that forces a build of the live source #2818 is the blocker, not this.
   2. The refusal row is armed but idle until a bake records a fingerprint, so nothing in the fleet changes behaviour today except the new buildProvenance field reading AdoptedUnverified everywhere an assembly was adopted.
10. rbuergi

    added a commit that references this issue on Aug 30, 2026

    fix(#2813): a refused adoption CLEARS CompiledSources — and the test …

    ...

    805ab1a
11. rbuergi commented on Aug 30, 2026

    

    rbuergi

    on Aug 30, 2026

    ContributorAuthor

    More actions

    The recovery verb is up: #2824 (fixes #2818) — a FORCED release now skips on-demand prebuilt adoption in the compile watcher and compiles the live source, and the force is spent by that compile. The provenance half (verify/refuse an adoption against the source) stays with #2821. Found alongside in #2824: a type that already had a build could be stranded at Pending by an adoption reported but never written back — the landed-predicate now requires the Ok stamp a real Seed writes.
12. rbuergi

    closed this as completedin #2821on Aug 30, 2026
13. rbuergi commented on Aug 30, 2026

    

    rbuergi

    on Aug 30, 2026

    ContributorAuthor

    More actions

    Reopening — #2821 merging did not fix the defect in this issue's title

    #2821 (a6c5e87aa) closed this automatically via its closing keyword. That is wrong, and the PR itself says why in its own body: it changes what an adopted build says, not what it does.

    This issue's headline is "GitSync adopts a stale prebuilt assembly over the source it just synced — silently running last week's code." Check it against what actually shipped:

    	before #2821	after #2821
    can a sync adopt a bundle built from older source?	yes	yes
    does it then run that older code?	yes	yes
    does the node claim compiledSources == currentSourceVersions?	yes	yes (legacy row keeps the stamp — deliberately, or every legacy type parks on a RequirePrebuilt mesh)
    can an operator tell the bytes were never checked?	no	yes — BuildProvenance = AdoptedUnverified

    Only the last row changed. And because nothing writes a sourceFingerprint into a bundle yet, every adoption in the fleet reads AdoptedUnverified — the refusal row is armed and idle. That is the honest state and a real improvement, but it is visibility, not prevention.

    What still has to land before this issue is genuinely closed:

    • Force-release is silently discarded: the compile watcher re-adopts the prebuilt bundle instead of compiling, so there is no verb that forces a build of the live source #2818 — the compile watcher discards RequestedReleaseForce and re-adopts without a Roslyn pass, so there is still no operator verb that forces a build of the live source on any mesh where the bundle resolves. That is the recovery path this incident needed and did not have.
    • A control plane must refuse to EXECUTE against a type whose build provenance is not verified — refusing to load is the second line of defence #2820 — the execute-time interlock. The damage here needed two ingredients: stale bytes and something armed to run them. Until a control plane refuses to act on an AdoptedUnverified type, a Crm/Migration-shaped node still fires against exactly the assembly nobody verified.
    • the producer half — a bake that records the source fingerprint. Without it the middle row of the three-way can never fire.

    Reopening rather than leaving a closed issue that reads as "the data loss is handled". The document that was permanently lost is still the measure of this one, and nothing shipped today would have prevented it — it would only have made it visible to an operator who thought to look.
14. rbuergi

    reopened this on Aug 30, 2026
15. rbuergi

    added a commit that references this issue on Sep 1, 2026

    ci(bake): a FALSE REFUSAL is red on the PR lane, not invisible (#2813)

    ...

    86d7f26
16. rbuergi

    closed this as completedin #2947on Sep 1, 2026
17. rbuergi

    added a commit that references this issue on Sep 17, 2026

    review(#4641): seven corrections — a restart is not the gap, and a FO…

    ...

    176734a
18. rbuergi commented on Oct 11, 2026

    

    rbuergi

    on Oct 11, 2026

    ContributorAuthor

    More actions

    (re-posted 2026-10-11; originally by @rbuergi on 2026-08-30)

    Two corrections and one hazard, from the reporter's follow-up probe

    1. The hub is UP. The NODETYPE is not resolving.

    A read (rather than another write) settles it — get @Admin/Migration/restructure-all-2/area/Run returns a rendered response, not a timeout:

    Area not found. No renderer is registered for area Run on hub Admin/Migration/restructure-all-2.
    Available named areas: $Collection, $Content, $Data, … Edit, Export, … Versions

    That list is only the generic node areas — no Run, no Tests, nothing from Crm/Migration's own AddMigrationLayoutAreas() / AddMigrationControlPlane(). The instance activates and serves; it just does not know it is a Crm/Migration. The type's configuration is never applied, so no watcher exists to pick up an actionable request.

    This is the same fact as the portal-log lines from the other end — [UpdateRemote] NO-OP … contentType=JsonElement — lambda returned unchanged and EnrichWithNodeType slow path for 'Crm/Migration' faulted (TimeoutException). The type never registers, so every instance falls back to generic behaviour and every typed Update lambda misses its pattern match.

    Stating it as "the hub is down" would send the next reader after grain lifetime, and the hub is demonstrably alive. Correcting my earlier comment on this issue accordingly.

    2. Adoption + an armed control-plane node is a loaded gun

    The reporter's queued run was sitting at requestedAction: Run, execute: true. If the type re-registers against the stale adopted assembly, that fires the pre-fix code unattended — stripping document bodies again, with nobody watching. They have set requestedAction: None and written the reason into the node's description so it cannot fire and whoever finds it knows why.

    This is the fix requirement arriving from a second direction: the window between "the type re-registers" and "the operator notices which assembly it got" is unattended execution of whichever bytes won, and the operator has no signal that distinguishes them. Any fix must make the adopted-vs-compiled distinction visible before a control plane can act on it — not only auditable afterwards.

    3. Why forcing a sync is not the recovery

    A git_hub_sync update runs as system-security and would re-register the type — but it can re-adopt the same stale prebuilt, and with a control-plane node in the space that is exactly the combination above. So the obvious recovery is the dangerous one. The space is being held deliberately.

    Also worth its own line

    The permanently-lost document (ClientA/EslLifeCycleProcess) had one version in history and content indexing off, so there was no second copy anywhere. That is a deployment setting, not a code path — a portal holding client documents with neither version depth nor an index has nothing to restore from. It is the difference between what was recovered here and what was not.
19. rbuergi commented on Oct 11, 2026

    

    rbuergi

    on Oct 11, 2026

    ContributorAuthor

    More actions

    (re-posted 2026-10-11; originally by @rbuergi on 2026-08-30)

    Resolving the open refusal-row question — and the answer moves the interlock to EXECUTE time

    From the session that owns this incident, and it reframes the question I posed:

    The damage needed two ingredients — stale bytes, and something armed to run them. Stale bytes sitting unloaded harm nobody. Refusing to execute matters more than refusing to load. A type that renders read-only pages from unverified bytes is a degraded system; a type that WRITES from them is the incident.

    That is right, and it demotes the clear-or-keep question I had been treating as central. The first-order requirement is that BuildProvenance be readable by a control plane, so a watcher can refuse to act on an unverified type. A Crm/Migration-shaped node that fires on type registration would otherwise still fire — against exactly the assembly the loader refused to verify. That interlock, not any load-time decision, is what would have saved the four documents.

    The load-time row still resolves as a conditional, now as the second line of defence:

    the mesh can compile	outcome
    yes	clear the adopted coordinates — the next activation builds, exposure is seconds
    no (RequirePrebuilt)	keep, log Critical, and mark the type visibly unverified — clearing there trades a silent wrong answer for a certain outage, and an outage on a client's production mesh is not obviously the kinder failure

    Making it conditional keeps the dangerous branch explicit rather than resting on an assumption about how a flag is set. 🚨 And it must not rest on "RequirePrebuilt is set nowhere" — I measured it absent from pod env, deployment inline env and ConfigMap on memex and memex-cloud, which is two instances and says nothing about client-b, client-i, local installs, or any external instance the registry serves.

    A neighbouring signal that is NOISE — do not build on it

    The same session has retracted (Crm#33) a staleness check they had shipped into their own AGENTS.md: "the Release is what instances load, so a mismatch means the portal is not running your change." Falsified by their next sync — Crm/Offer came back with all three fields mismatched:

    NodeType   Crm_Offer/v63-s13fb7c2-…   fw s13fb7c25…
    Release    Crm_Offer/v25-s770d75c-…   fw s770d75c0…
    

    …and served perfectly: title, DRAFT pill, all 7,111 chars. latestReleasePath lags routinely — all fifteen of their types have stale release pointers and all fifteen are healthy.

    So release-vs-NodeType disagreement is not what distinguished the broken type from the healthy ones, and a check of that shape would fire constantly on healthy meshes. The source-fingerprint-on-the-bundle comparison this fix uses is the right signal precisely because it does not have that property. Recording it here so the next reader does not reach for the cheaper-looking neighbour.

    Sequencing

    The incident owner is not waiting on this fix — their migration is complete, done by script, with every write verified by read-back — and they have said AdoptedUnverified everywhere is the honest state and preferable to a field that pretends. So the provenance change lands on its own merits, with #2818 (no operator verb forces a build of the live source) and the execute-time interlock both named as open follow-ups rather than folded in untested.