Repository navigation
Content key over the fully generated compile input (#1707 slice 4) — and why the full-MVID rule cannot go with it - #1994
Conversation
…slice 4) Every invalidation signal for a compiled NodeType has so far been a PROXY. `FrameworkBuildIdentity.FullMvidAssemblies` folds the toolchain's whole implementation MVID into the framework identity precisely because a generator body change reshapes what Roslyn is fed with no API change — surface hashing cannot see it. Correct, and unbelievably coarse: a body-only commit to any member of that closure (16 assemblies, 383 commits/30d — #1976) mints a new global identity, empties the assembly share's key-space and rebakes every NodeType on every deployment, including the ones whose generated input is byte-for-byte what it was. This is the direct observation the proxy stands in for. `GeneratedInputIdentity` hashes the FULLY GENERATED compilation input — the exact text handed to Roslyn after skeleton generation, source aggregation, `@@`-include expansion and the `#r` strip — together with everything else that decides the bytes given that text: the assembly name, the reflected parse/compilation/emit option set, the Roslyn build, the source generators that run, and the pruned reference-surface set. Two compiles whose keys are equal produce interchangeable bytes, whoever built them and whenever. Two stages, because the halves are known at different moments: `OfGeneratedInput` runs before Roslyn where the text exists; `Combine` folds in the reference surfaces read off the EMITTED assembly afterwards. Hashing the pruned set rather than the candidate set is what keeps the key per-type. The result lands in the per-type dependency record under the reserved `!input` key, folded over the record's ASSEMBLY entries only — folding the toolchain id in would reproduce the proxy it replaces. Determinism is the whole product, so exactly three normalisations, each a deliberate trade documented on the type: a leading BOM is dropped; line endings fold to `\n` (the skeleton is built with `AppendLine`, so the same node generates CRLF on Windows and LF in a pod — without the fold nothing could ever share a build across hosts); and the generator's `// Generated at: <UtcNow>` header becomes a fixed marker (a wall clock in the generated text means the input is never twice the same, and no key over it could ever hit — it is also why the store's digest-of-the-emitted-bytes can never dedupe two compiles of identical content). Nothing else is touched. The option set is REFLECTED rather than hand-listed: a hand-list is how `NuGetDirectiveParser` sat outside the identity boundary while shaping compile input, and an option the toolchain starts setting must not be able to sit silently outside the key. `FindMismatch` gains an optional live content key. A caller that REGENERATED the input gets an exact verdict; a metadata-only caller skips the entry, and the toolchain entry still governs — see the next commit's note and the pinned `DeletingTheFullMvidRule_WouldLeaveNothingWatchingTheToolchain`. Tests pin all four discriminations plus golden vectors that a different process on a different architecture has to reproduce. Each was verified non-vacuous by breaking the property and watching it go red. Internal only: no user-visible effect, so no What's New entry. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The key from the previous commit is only worth having if PRODUCERS record it. Both paths that emit a dynamic NodeType now compute it at the one moment the fully generated input exists as text — after the `#r` strip, before a single Roslyn call — and complete it at write-back with the pruned reference surfaces Roslyn proved the bytes actually bind: - `NodeSetCompiler.Compile` (the build-process bake, #1763) — it already had `inputs.GeneratedSource` in hand; - `MeshNodeCompilationService.CompileAsyncCore` (the runtime) — the digest rides out on a new `CompileEmit` through `CompileAttempt` to `CompileResultFromAssembly`, beside the assembly it describes. The two paths differ only in where the nodes came from, so their keys must agree for identical content — that is what makes a CI bake's bytes adoptable by a portal, and `GeneratorPipeline.EffectiveGeneratorPaths` is shared by the hash and the generator run so a key can never describe a compile that never happened. Paths where no compile ran — a disk-cache hit, the assembly-hydration shortcut — pass null and the record simply carries no content key, exactly as before it existed. Nothing yet CONSUMES the key: no metadata-only caller can compute a live one, which is the whole reason `FullMvidAssemblies` is still here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds and stamps a deterministic “content key” for compiled NodeTypes based on the fully generated Roslyn input (stage-1 digest) plus the pruned reference-surface set (stage-2), so CI-baked artifacts and runtime compiles can be compared/adopted with a stable, input-derived identity rather than relying solely on coarse toolchain MVID proxies.
Changes:
- Introduces
GeneratedInputIdentityand a pinned test suite to guarantee cross-process/host determinism (including normalizations for line endings, BOM, and the generator wall-clock header). - Stamps the stage-1 generated-input digest during runtime compilation (
MeshNodeCompilationService) and during bakes (NodeSetCompiler), and threads it intoCompiledDependenciesas!input. - Factors generator-candidate path computation into
GeneratorPipeline.EffectiveGeneratorPathsso the content key hashes the same generator set that actually runs.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/MeshWeaver.Graph.Test/GeneratedInputIdentityTest.cs | New tests pinning the content-key algorithm and its determinism/normalization claims. |
| src/MeshWeaver.Graph/Configuration/MeshNodeCompilationService.cs | Computes and propagates the stage-1 generated-input digest during runtime compiles; folds it into the dependency record. |
| src/MeshWeaver.Compiler/NodeSetCompiler.cs | Computes the same stage-1 digest during bake compilation and stamps it into CompiledDependencies. |
| src/MeshWeaver.Compiler/GeneratorPipeline.cs | Extracts EffectiveGeneratorPaths so generator identity hashing matches the actual generator load set. |
| src/MeshWeaver.Compiler/GeneratedInputIdentity.cs | New implementation of the stage-1/2 content key and deterministic option/value rendering. |
| src/MeshWeaver.Compiler/EmitPipeline.cs | Adds a process-constant OptionsFingerprint used by the content key to capture compilation option shaping. |
| src/MeshWeaver.Compiler/CompiledDependencies.cs | Adds reserved !input entry and updates mismatch logic to optionally validate it when a live key is available. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Test Results (shard 0)1 229 tests +589 1 225 ✅ +688 11m 28s ⏱️ -7s Results for commit 4834705. ± Comparison against base commit a43f591. This pull request removes 223 and adds 812 tests. Note that renamed tests count towards both.This pull request removes 103 skipped tests and adds 4 skipped tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Test Results (shard 3) 10 files ± 0 10 suites ±0 4m 40s ⏱️ -31s Results for commit 4834705. ± Comparison against base commit a43f591. This pull request removes 762 and adds 226 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Test Results (shard 5)1 215 tests - 476 1 214 ✅ - 475 7m 7s ⏱️ +14s Results for commit 4834705. ± Comparison against base commit a43f591. This pull request removes 547 and adds 71 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Test Results (shard 4)1 790 tests +231 1 682 ✅ +127 8m 46s ⏱️ -21s Results for commit 4834705. ± Comparison against base commit a43f591. This pull request removes 107 and adds 338 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Test Results (shard 1)2 039 tests +430 1 846 ✅ +241 8m 54s ⏱️ +12s Results for commit 4834705. ± Comparison against base commit a43f591. This pull request removes 162 and adds 592 tests. Note that renamed tests count towards both.This pull request removes 4 skipped tests and adds 193 skipped tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Test Results (shard 2)3 027 tests - 186 3 027 ✅ + 6 8m 0s ⏱️ +24s Results for commit 4834705. ± Comparison against base commit a43f591. This pull request removes 666 and adds 480 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Test Results 59 files - 2 59 suites - 2 48m 59s ⏱️ -8s Results for commit 4834705. ± Comparison against base commit a43f591. This pull request removes 65 and adds 117 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
…slice 4)
`BakeEquivalenceTest` went red on the content key and the divergence was real:
the compiler-driven bake produced the same key on three consecutive runs while
the mesh-driven bake produced a different one every time.
run 1 mesh ic920384a… tree i3494e62e…
run 2 mesh iead1b585… tree i3494e62e…
run 3 mesh i70db762d… tree i3494e62e…
`CollectCompileSources` kept "the order the snapshot delivered them", and on the
mesh path that delivery is `map.Values.ToList()` over an `ImmutableDictionary`
(MeshNodeCompilationService's source probe) — hash-bucket order over string
hashes that .NET randomises PER PROCESS. The join order is part of the emitted
bytes, so the mesh compiled identical content into a different assembly on every
process: not reproducible against the compiler-driven bake, and not reproducible
against ITSELF. That is why the assembly store's digest of the emitted bytes
could never dedupe, and it would have made the content key move on every compile
— an identity that invalidates everything on every build, which is strictly
worse than the proxy it replaces.
So the order becomes a function of the SOURCE SET: ordinal by path, inside the
toolchain function that already owns the join order, so both producers agree by
construction rather than by convention. C# does not care — top-level
declarations in a concatenated unit are order-independent, which is exactly what
BakeEquivalenceTest's type-and-member SURFACE comparison proves, and it is what
made the old non-determinism invisible.
One-time effect: every NodeType's bytes change once, as they would on any
identity move.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With the source order fixed the two bakes still disagreed, and the mesh key was still different on every run (iaa2ed0ef…, ib704fb17… against a stable i770851b4…). The remaining wall clock is not in the generator: the skeleton emits the NodeType node's own `LastModified`, and `PackageInstaller.BulkSave` stamps that field with `DateTimeOffset.UtcNow` on EVERY import (PackageInstaller.cs:2158). Same repo content, imported twice, generates different text. That is fatal to the key rather than inconvenient. A CI bake and a portal import the same commit at different moments BY CONSTRUCTION, so a key that discriminated on the stamp could never match a published bundle against the input a portal regenerates — the adopt case the key exists to serve. Unlike the other three normalisations this one is a real trade and is documented as such: the value IS a string literal in the emitted IL, so two assemblies differing only in it hash equal. What differs is one provenance timestamp on the provider's node — not a type, a member, a signature or a reference — against the alternative of a key that cannot work at all. The match is anchored to the generator's exact emitted shape, so the only false positive available is a same-shaped line in user code, whose only consequence is that one timestamp literal stops discriminating; a differently-shaped mention still moves the key, which the test pins. With both fixes the mesh-driven and compiler-driven bakes agree, and agree on three consecutive runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Update — CI was red, and the failing test caught something realTwo shards were red. One was mine, one was not, and the classification is evidence-backed rather than by name. Shard 3 — MINE.
|
| run | MESH | TREE |
|---|---|---|
| 1 | ic920384a… |
i3494e62e073f2a551796fa08440330ed |
| 2 | iead1b585… |
i3494e62e073f2a551796fa08440330ed |
| 3 | i70db762d… |
i3494e62e073f2a551796fa08440330ed |
The compiler bake is stable across processes; the mesh bake is different every run. Two root causes, both fixed at the level they occur, neither by weakening the assertion:
-
The source join order was the delivery's, not the set's (
c9eccc76a).CollectCompileSourceskept "the order the snapshot delivered them", and on the mesh path that delivery ismap.Values.ToList()over anImmutableDictionary— hash-bucket order over per-process randomised string hashes. The join order is in the emitted bytes, so the mesh compiled identical content into a different assembly on every process: not reproducible against the other producer, and not reproducible against itself. This is independently why the store's digest-of-emitted-bytes can never dedupe. Now ordinal by path, inside the toolchain function that already owns the join order, so both producers agree by construction. -
The importer's node timestamp (
3bf42ed6d). With the order fixed the mesh key was still per-run unique. The remaining wall clock is not in the generator: the skeleton emits the NodeType node's ownLastModified, andPackageInstaller.BulkSavestamps that withDateTimeOffset.UtcNowon every import (PackageInstaller.cs:2158). A CI bake and a portal import the same commit at different moments by construction, so a key discriminating on it could never match a bundle against the input a portal regenerates — the adopt case the key exists for. Normalised, and unlike the other three normalisations this one is a stated trade (the value is IL): what stops discriminating is one provenance timestamp on the provider node, against the alternative of a key that cannot work at all.
BakeEquivalenceTest now passes on three consecutive runs, and both fixes carry cheap unit regressions (CompileSourceOrderTest, TheNodeTimestampStamp_IsNormalizedOut_ButOnlyInItsGeneratedShape), each verified non-vacuous by reverting the fix.
This changes the earlier risk note: the entry is in the artifact, and something does compare artifacts. The consumer claim still holds (no consumer reads
!input), but the equivalence claim did not — and the test was right.
Shard 2 — NOT mine. #890's population, and not #2001.
Shard 2's 11 failures are all MeshWeaver.Hosting.Monolith.Test, all the same frame, terminating exit=124:
NullReferenceException at Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol
.Microsoft.Cci.ITypeDefinitionMember.get_ContainingTypeDefinition()
at Microsoft.Cci.MetadataWriter.GetConsolidatedTypeParameters …
[CI] MeshWeaver.Hosting.Monolith.Test (part 2/2) exit=124 TIMEOUT
That is #890 verbatim — "one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124". The repo's own emit canary adjudicated it in the log:
canary=BELOW-ROSLYN… a trivial compilation with BRAND-NEW references and freshly parsed source cannot emit either, and BOTH legs died in the same frame ⇒ nothing about the reference set explains this; the broken state is below Roslyn (CLR heap / JIT / GC), so no reference-set change can fix it
My diff computes a SHA-256 over strings before Roslyn is invoked. It cannot poison the CLR heap, and the canary's pristine leg — a minimal compilation with brand-new references and no relation to any input I touch — failed identically, which excludes the compile inputs entirely. Corroboration: main's own most recent failed run (a43f5912, 25 min earlier) died in the same shard with MeshWeaver.FutuRe.Test exit=139 SIGNAL SIGSEGV — same family, on code this branch never touches. And all 21 of those monolith compile tests, including the named MeshNodeCompilationIntegrationTest cases, pass locally on this branch.
It is not #2001. That issue is MeshWeaver.AI.Test.ThreadTokenUsageTest — a wall-clocked satellite wait that "emitted nothing at all" while missing from ConcurrencyStressCollection. Nothing in shard 2 matches that: different assembly, different test, an exception rather than a timeout, and a named CLR frame. Attributing it there would file a real process-poisoning recurrence under a scheduling bug.
#1976 / #1977 — unchanged, and #1977 now merged
The recommendation stands and has been actioned: #1977's pin is more necessary now, guarding a boundary this PR documents as un-deletable-for-now. #1976 stays open, retargeted at the re-evaluation lane.
Copilot review on #1994. `AssemblyFileIdentities` keyed by file name and assigned, so two DIFFERENT assemblies resolving to the same file name — two NuGet packages, or two target-framework folders, both of which `RunSourceGenerators` loads — silently dropped one from the key. Two things wrong with that, and the second is the reason it matters here. The key would claim "same input" for a genuinely different effective generator set, which is the UNDER-invalidating direction: stale bytes, the outcome this whole slice is written to avoid. And which of the two survived depended on enumeration order, reintroducing exactly the order-dependence the type exists to remove — the same defect as the source-join order two commits ago, one level down. The value is now the ordinal-sorted, de-duplicated SET of identities for that file name. The same assembly reached by two paths still collapses to one id, because it IS one input. Keyed by file name rather than path is unchanged and deliberate: a host path could never match between a bake container and a portal pod. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Merging this whole, and closing #2007 — which was a cherry-pick of Carrying #2007's analysis here, because it belongs on the commit that actually lands: The join-order fix is bigger than the content key that found it. What that independently explains:
Deploy note: emitted bytes change for every NodeType with no identity change. Safe — existing cached DLLs stay valid for their content, and recompiles simply write new content-hashed names. |
Ships the generated-input content hash of #1707 slice 4. It does not delete the full-MVID rule, and the reason is the point of this PR — see "Why the deletion is blocked".
What the generated-input pipeline actually is
Established by reading
MeshWeaver.Compiler, not from the issue's list. Both compile paths run the same eight stages (MeshNodeCompilationService.CompileCore→CompileAsyncCore;NodeSetCompiler.ResolveInputs→Compile):CodeQueryResolver.ExpandAlloverSources∪Tests, raced/established as aSourceSnapshot@@-include expansion —NodeCompileShaping.ResolveCodeIncludes+AnchorIncludePathCollectCompileSourcesCombineSources(joined"\n\n", snapshot order)DynamicMeshNodeAttributeGenerator.GenerateAttributeSource(standard usings, hoisted user usings, assembly attribute, user code, the provider class carrying theconfigurationlambda and content collections)#rstrip —NuGetDirectiveParser.Extract+GeneratorPipeline.StripBuiltInScopeGeneratorRef, then NuGet resolutionEmitPipeline.CreateEmitCompilationwith the canonical parse + compilation optionsGeneratorPipeline.RunSourceGeneratorsover the compilation, then emitTwo stages the issue does not name belong in the key and are in it: the option set (7) — a body-only edit to
CreateCompilationOptionsrewrites every assembly while the text is identical — and the source generators (8), which add source the caller never sees. So does Roslyn's own build:ComputeToolchainClosurewalksMeshWeaver.*refs only, soMicrosoft.CodeAnalysis.CSharphas never been in the identity directly — it rode in onMeshWeaver.Compiler's MVID, which a package bump moves. A key replacing that MVID must carry the compiler explicitly or it under-invalidates on the change that rewrites everything.What is hashed, and what is normalised
Two stages, because the halves are known at different moments.
OfGeneratedInput(prefixg) runs before Roslyn: assembly name, generated text, reflected option fingerprint, Roslyn identity, generator-assembly MVIDs.Combine(prefixi) folds in the pruned reference surfaces read off the emitted assembly — pruned, not candidate, is what keeps the key per-type. The result lands in the per-type dependency record as!input, folded over the record's assembly entries only; folding the toolchain id in would reproduce the proxy it replaces.The option set is reflected, never hand-listed. A hand-list is how
NuGetDirectiveParsersat outside the identity boundary while shaping compile input; an option the toolchain starts setting must not be able to sit silently outside the key.Exactly three normalisations, each a deliberate trade:
\nAppendLine, i.e.Environment.NewLine, so the SAME node generates CRLF on a Windows dev box and LF in a pod — unfolded, nothing could ever share a build across hosts// Generated at: <UtcNow>→ a fixed markerGenerateAttributeSourcestamps a wall clock into the generated text, so the input is never twice the same and no key over it could ever hitNothing else is touched: no trimming, no whitespace collapsing, no comment stripping. Everything enumerable is ordinal-sorted; every value renders invariant-culture.
The four discriminations, and their non-vacuity
Each property was verified by breaking the implementation and watching the test go red, then restoring.
TheKey_IsAPinnedFunctionOfItsInputs_AcrossProcessesAndHosts,IdenticalGeneratedInput_HashesIdentically_WithinOneProcess,LineEndings_AreNormalized_…,TheGeneratorsWallClockHeader_IsNormalizedOutAGeneratorBodyChange_ThatAltersTheGeneratedText_MovesTheKey,AGeneratorAssemblyBodyChange_MovesTheKey_EvenWhenTheTextIsIdentical,AnOptionChange_MovesTheKeyAToolchainBodyChange_ThatDoesNotAlterTheGeneratedInput_LeavesTheKeyPut,TheRecordsContentKey_IsFoldedOverTheAssemblyEntriesOnlyAReferenceSurfaceChange_MovesTheKeyCombineTwo processes, and a hostile one. The cross-process claim is expressed as golden vectors — literals a different process on a different architecture must reproduce, which every CI run re-verifies. Locally, a second process under Turkish culture (the classic ambient-culture trap):
Final state:
GeneratedInputIdentityTest18/18; the compile/identity suite 138/138;MeshWeaver.Hosting.Monolith.Testcompile tests 13/13; Release +-warnaserrorclean onMeshWeaver.Compiler,.Graph,.Hosting,.Plugin.Build,PluginTesterand the five affected test projects — 0 Warning(s), 0 Error(s).🚨 Why the deletion is blocked — and what unblocks it
The instruction was explicit: if the hash turns out not to capture something the MVID did, stop and report rather than deleting. It does not capture one thing, and it is structural rather than fixable by a better key.
FullMvidAssembliesis read in exactly two places:ComputeSurfaceIdentity(→ the globalFrameworkVersion) andComputeToolchainId(→ the record's!toolchain). Empty the set and both degenerate. Then trace a body-only toolchain commit that DOES change generated input — the #1802 global-usings fix and theAnchorIncludePathfix are both exactly this shape — throughHasStaleFrameworkBuild, the only automatic re-drive:CompiledFrameworkVersion == FrameworkVersion— unchanged, no API moved;!toolchain— a constant;!input— skipped, becauseFindMismatchhas no live value to compare it against.⇒
HasStaleFrameworkBuild= false. Nothing recompiles, the stale bytes are served, and the failure surfaces as aTypeLoadExceptioninside an ALC at activation with no diagnostic.The gap is not in the key. The key has no cheap live counterpart by construction: it is a function of the toolchain and the source text, so evaluating it means regenerating the input (discovery, includes, skeleton), and every consumer that decides "rebuild or not" today is deliberately metadata-only (
HasUsableBuild: "noIAssemblyStoreprobe, noFile.Exists"). The toolchain MVID is therefore the only cheap witness that the toolchain moved at all.This is pinned as a test rather than left as prose —
DeletingTheFullMvidRule_WouldLeaveNothingWatchingTheToolchaindemonstrates the identity collapsing to one value with the rule emptied, and the record validating forever across the change.What unblocks it is a re-evaluation lane, not a better key. On a toolchain change, regenerate the input and compare the key: equal ⇒ keep the bytes and restamp; different ⇒ compile. The MVID then stops being the invalidation unit and becomes the trigger for that comparison — which is the demotion #1976 actually wants, at a cost of a generation pass instead of a Roslyn compile per type. This PR is its precondition: the key exists, discriminates, and is stamped by both producers.
#1976 / #1977
Neither can be closed by this PR. #1976's complaint — the closure moves more often than the single root it replaced, and any member's commit empties the share's key-space — is untouched while the rule stands. #1977's pin stays necessary, and more so: it is now the guard on a boundary we have documented as un-deletable-for-now, so a silent widening would be a regression against a known-load-bearing set. Recommend merging #1977 and keeping #1976 open, retargeted at the re-evaluation lane.
Risk
The record gains one optional entry; no consumer reads it yet (
FindMismatch's new parameter defaults to null, soHasUsableBuild,HasStaleFrameworkBuild,NodeTypeBakeStatus.ClassifyandPrebuiltAssemblySeederbehave exactly as before). Existing records without the entry validate unchanged. No identity, filename tag or store key changes, so no rebake.Internal only — no user-visible effect, so no What's New entry.
🤖 Generated with Claude Code