Repository navigation
π¨ A platform build published tonight removes AI types from the in-mesh compile surface β 12 Reinsurance NodeTypes broke on an UNCHANGED commitΒ #2561
Description
The measurement
MeshWeaver.Reinsurance main, the same commit, one hour apart:
| run | framework | source | result |
|---|---|---|---|
33114326364 20:38 |
s81a9b60ed905b4f5263f55ff8369976d |
7f454af0 |
179/179 compiled, 0 new failures |
33119003918 21:37 |
s96fb96913046dff7a1d6773286ad3fc8 |
7f454af0 |
167/179 β 12 NEW failures |
Identical content. Different platform build. A framework change published between those two runs
removed a set of AI types from the surface in-mesh source compiles against.
What went missing
Not one symbol β a cluster, which is the signature of an assembly leaving the compilation surface
rather than a deletion:
CS0246 The type or namespace name 'ThreadMessage' could not be found (Γ4)
CS1061 'IMessageHub' does not contain a definition for 'StartThread'
and no accessible extension method 'StartThread' accepting a first argumentβ¦
CS0246 The type or namespace name 'ChatClientCredentialResolver' could not be found
CS0246 The type or namespace name 'AiSettings' could not be found (Γmany)
CS0103 The name 'AiSettingsNodeType' does not exist in the current context (Γmany)
StartThread is the canonical thread-submission extension AGENTS.md documents as the entry
point. AiSettingsNodeType is the same type whose AddSkillSource was deleted in #683 and put
nine plugin partitions on the compilation-error overlay in production β the precedent is in the
repo's own memory.
Why nothing went red until now
This is the case AGENTS.md opens with:
π¨π¨π¨ Green CI does NOT mean the mesh compiles β in-mesh source is invisible to
dotnet build.
Deleting or renaming ANY public framework surface is a breaking change to code the compiler cannot
see⦠A clean-c Release -warnaserrorbuild proves nothing here.
Every core PR tonight was green. The 12 failures live in Reinsurance's in-mesh Source/*.cs, which
only the bake gate compiles β and only against whichever platform image it pulls.
Why it is urgent rather than merely broken
Per AGENTS.md: a NodeType left at CompileError refuses portal readiness and parks every
instance hub for the full 60 s activation budget β hung pages, failed liveness probes, dropped
silos. And a framework-version bump recompiles every dynamic NodeType, so this does not trickle in;
it lands wholesale on the next roll.
Mitigating, for the moment only: memex has not self-updated since 14:27 (#2494/#2553), so the
running fleet is still on ci.6105 and has not met this framework. The stalled roller is the only
thing standing between this and the overlay. When #2494 is fixed, this ships.
Where to look
Between the two runs, main merged the AI-exit and extraction work β #2537 ("AI tools: 22 Task
bridges gone"), #2547 ("Speech.Contract and the log watcher leave core (#2276)"), alongside
#2540/#2542/#2549. Earlier, 16497893b moved the AI engine behind an assembly attribute.
The shape of the breakage β a whole namespace's worth of types vanishing at once, not one renamed
member β points at MeshWeaver.AI no longer being among the references in-mesh NodeType
compilation sees, rather than at any individual deletion. That is a one-line difference in whatever
assembles the compilation's reference set, and it is worth confirming before anything is reverted.
What I did not do
I did not revert anything or add allow-list entries. 0 known-debt failure(s) allowed is the right
setting and the 12 should not be papered into it β AGENTS.md is explicit that an allow-list entry
here is the band-aid, and the sweep protocol (Search('nodeType:NodeType') β LspDiagnosticsForNode
per type β fix roots first) is the actual remedy.
Check the other node repos before assuming this is Reinsurance's alone β SocialMedia, Education
and Manufacturing consume the same platform image and their bakes have not all run against
s96fb969β¦ yet.
Activity
rbuergi commentedon Aug 27, 2026
Narrowing it β three candidates excluded by measurement, so nobody retraces them.
Not a deletion: every named type still exists on
mainSo this is not the #683 shape (a public surface deleted with in-mesh callers). The source is intact; the types are simply not visible where the bake compiles.
Not #2541, despite the name
feat/2276-gate-loads-external-modulestouchesPluginGateRunner.cs, which is exactly the right neighbourhood β but the diff is a pure refactor of that block.MeshWeaver.AI.dllwas alreadyModules:Assemblies:0andModules:Required:0before it; the change only lifts the dictionary into a variable so extra--modulepaths can be appended. Behaviour for the AI module is unchanged.Not a failure to resolve the module
The gate fails RED with an explicit message when a required module does not resolve beside the binary ("the gate's required module(s) did not resolveβ¦ A run whose modules are missing cannot gate"). That did not happen β the run proceeded to compile and produced 12
CS0246/CS1061/CS0103. SoMeshWeaver.AI.dllwas found and activated.Which leaves the interesting question
The module loaded, the types exist in it, and in-mesh source still cannot name them. That is a registration / compile-reference question, not a packaging one: either the AI module's assembly stopped being contributed to the reference set that NodeType compilation sees, or its types stopped being registered in a way
includeGlobalTypespicks up.Worth checking against tonight's two changes that touch exactly that seam:
MeshHostApplicationBuildernow takes the configuration at construction, soFeatures:StaticRepoSync:Partitionsis honoured where it previously read as absent (that was the whole defect). The gate setsFeatures:StaticRepoSync:Partitions:0 = Agent. A setting that was silently ignored and is now obeyed is a behaviour change on the module path by construction β and this is the first bake to run with it.Task-bridge purge, which is the only other change in the window touchingMeshWeaver.AIitself.I have not confirmed either; both are hypotheses with a reason, not findings. The decisive experiment is cheap: bake the same Reinsurance commit against the last-green framework
s81a9b60ed905b4f5263f55ff8369976dand the reds96fb96913046dff7a1d6773286ad3fc8, and diff what the compilation's reference set contains.rbuergi commentedon Aug 27, 2026
Retracting the #2534 hypothesis from my previous comment β I checked it and it does not hold.
I suggested #2534 was the likely cause on the reasoning that
Features:StaticRepoSync:Partitionshad been silently ignored and is now honoured, so the gate would newly serveAgentfrom an empty in-memory partition. That reasoning requires the gate'sbuilder.Configurationto have been null before. It never was:InstallConfiguredModulesis precisely the helper that did hand the configuration over β that is what #2534's whole finding turned on (the portal composes its own module union and callsInstallAssembliesdirectly; the gate does not). So the gate has always read that setting, and #2534 changed the portal, not this path.Exclusions so far, so the search space is small
main:Thread.cs:447(ThreadMessage),AiSettings.cs:15,AiSettingsNodeType.cs,ChatClientCredentialResolver.cs,HubThreadExtensions.cs(StartThread)src/MeshWeaver.AI/gate-loads-external-modules)MeshWeaver.AI.dllwas alreadyModules:Assemblies:0andModules:Required:0InstallConfiguredModules, which always passed configurationWhat remains is a change inside itself or in how its types reach the compile surface, between framework (green, 20:38) and (red, 21:37). The only merge in that window touching the assembly is #2537 (the AI -bridge purge). That is where I would look next, and I have not verified it.
The decisive experiment is unchanged and cheap: bake the same Reinsurance commit () against both framework builds and diff what the compilation's reference set contains. That answers it without guessing.
rbuergi commentedon Aug 27, 2026
What remains is a change inside
MeshWeaver.AIitself, or in how its types reach the compilesurface, between framework
s81a9b60ed905b4f5263f55ff8369976d(green, 20:38) ands96fb96913046dff7a1d6773286ad3fc8(red, 21:37). The only merge in that window touching theassembly is #2537 (the AI
Task-bridge purge). That is where I would look next, and I havenot verified it.
The decisive experiment is unchanged and cheap: bake the same Reinsurance commit (
7f454af0)against both framework builds and diff what the compilation's reference set contains. That answers
it without guessing.
(Reposting the closing paragraphs β the previous comment lost four code spans to shell expansion,
so it read as if the sentences had no subjects. The exclusion table above it is intact.)
rbuergi commentedon Aug 27, 2026
Attributed:
35a9c95c4, and the subject line says it20:58 sits exactly between the green bake (20:38) and the red one (21:37), and its diff is four
files:
FrameworkBuildIdentity.csis what produces the identity that changed froms81a9b60ed905b4f5263f55ff8369976dtos96fb96913046dff7a1d6773286ad3fc8. So this commit is theframework change, not merely a commit inside its window.
The mechanism, and it is one line of composition
In-mesh NodeType compilation builds its reference set at
MeshNodeCompilationService.cs:183-185:β a base set plus the assemblies of installed modules, registered one-by-one at
MeshBuilder.cs:231(services.AddSingleton(new InstalledModuleAssembly(assembly))).So AI types reach in-mesh source either because the AI engine is part of the content surface,
or because
MeshWeaver.AIis registered as anInstalledModuleAssembly. This commit moves itfrom the first to the second. The gate's required-module check still passes β the module resolves and
loads, which is why nothing failed RED β but
AiSettings,ThreadMessage,AiSettingsNodeType,ChatClientCredentialResolverand theStartThreadextension are no longer where Reinsurance'sin-mesh source was resolving them from.
That matches every symptom precisely: a cluster of types from one assembly disappearing at once,
with the assembly itself present, no deletion, and no gate-level error.
This supersedes both of my earlier guesses
#2534 was retracted above. #2537 was also wrong β its diff touches only
MeshPlugin.csandPlugins/*(tool implementations), and no module registration,AiMeshModuleAttribute, global-typeor NodeType-registration code. It cannot have moved the compile surface.
What this does not settle
Whether Reinsurance's in-mesh source should be following the AI engine, or whether the module's
assembly should be reaching the compile surface and is not. Those are opposite fixes:
InstalledModuleAssemblyregistration is not reaching this mesh's service tree in the bake, andthat is the bug;
NodeTypes need porting, and this is expected fallout of MeshWeaver.AI leaves the platform: carve out the three non-AI surfaces, then move the engine to MeshWeaver.PluginsΒ #2276 that simply had no gate to announce
it.
The commit's own message ("leaves the content surface") reads like the second. If so, the
right follow-up is not a revert but a port plus a heads-up to every node repo β and the fact that
this surfaced only because a bake happened to re-run on an unchanged commit is itself worth fixing:
nothing announced a content-surface removal to the repos that consume it.
π¨ Either way the ordering against #2494 stands. The fleet has not rolled since 14:27, so no
instance has met this framework. Fixing the roll before this is resolved delivers 12
CompileErrorNodeTypes to production, which refuse portal readiness.
rbuergi commentedon Aug 27, 2026
Blast radius: staggered, and the most exposed repo has not met it yet
Each node repo pulls its own platform image, so they encounter the new framework at different times.
Measured just now:
MeshWeaver.Reinsurances96fb96913046dff7a1d6773286ad3fc8MeshWeaver.SocialMediasae3a9fec4bd24c473d387132a4d86ba8MeshWeaver.ManufacturingMeshWeaver.EducationTwo things follow.
SocialMedia's green is not evidence that the change is safe. It baked against a third framework
(
sae3a9fecβ¦, neither the green nor the red Reinsurance one) and its 43 NodeTypes do not referencethe AI engine. It discriminates nothing about this defect.
π¨ Education is the repo to watch, and it has not baked against the new framework at all. Its
last main bake is 19:07 β before
35a9c95c4(20:58) β and it is red for the separate view-packreason (#2548). It also carries by far the most AI-dependent content:
AgenticBusiness,AgenticOffice,AgenticPrimer,AgenticPrimerDe,AgenticEngineering. If in-mesh source therenames
AiSettings,ThreadMessageorStartThreadthe way Reinsurance's does, it meets the samewall on its next bake β and it will look like a new failure on whatever change happens to trigger
that run, not like fallout from this one.
Worth doing before that happens: grep the node repos' in-mesh source for the five names, so the
port list is known in advance rather than discovered one red bake at a time.
That is the announcement this change did not make.
rbuergi commentedon Aug 27, 2026
Correction: the 12 are not Reinsurance's content. They span THREE packages, and five are
Store.I said above that Education "has not met it yet" and was "the repo to watch". Both wrong. The bake
names every failing type, and I should have read that before reasoning about which repo owns them:
Five of the twelve are
Store/*β the plugins repo's own Store package, which every instanceruns.
Store/CatalogandStore/Pluginare what render the store front and its package pages.That is a materially bigger surface than a single vertical's content, and it is the part that should
drive the decision.
One is
Edu/PromptCellβ so Education's content is already breaking. It simply surfacedthrough Reinsurance's gate, because that gate installs the Edu package as a dependency. My "not yet
affected" was an artefact of looking at which repo's CI happened to run, not at what the run
compiled.
Six are
UWDeepfield/*, Reinsurance's own β and a grep of that repo for the five names findsexactly two files, both using only
.StartThread(:So the other four UWDeepfield types fail through shared source or a dependency, not their own direct
use β which means a port list built by grepping each repo would have missed a third of them. The
bake's own RED list is the authority; a text search is not.
What this changes
Store/*is platform-adjacent content thatevery portal serves, so "content stops depending on the AI engine" means changing the Store
package too β and that is a much bigger call than MeshWeaver.AI leaves the platform: carve out the three non-AI surfaces, then move the engine to MeshWeaver.PluginsΒ #2276's commit message implies on its face.
Store/Catalogappearing here is worth cross-checking againstMeshWeaver.Plugins#776, which reports
Store/Catalogintermittently failing the mesh gate("did not answer within 15s"). Different symptom, plausibly the same root; worth confirming before
they are treated as two problems.
having baked yet. They are exposed the moment any bake anywhere installs these packages.
rbuergi commentedon Aug 28, 2026
Evidence now leans toward "the module's assembly should reach the compile surface and does not"
Stating this as a weighting, not a conclusion β I have been wrong three times on this issue tonight
by publishing an inference before checking it, so here is what is measured and what is not.
Measured.
MeshBuilder.cs:218-233composes the compile reference set as TPA + installedmodules, and its comment states the transition exactly:
So before
35a9c95c4,MeshWeaver.AIwas in the publish closure β in TPA β visible to in-meshcompilation unconditionally. After it, the only route is the
InstalledModuleAssemblysingletonregistered a few lines below.
Measured. That registration is deliberately skipped for a skewed module:
That would explain the symptom exactly β module loads, required-module check passes, types absent.
Measured, and it argues against that explanation: the bake log contains no skew or
incompatible-module report (
grep -i "incompatible|skew|MissingMethod|TypeLoad"β nothing). So theAI module was not classified incompatible; it installed normally and should be in the list.
Not measured, and this is the gap: whether the
InstalledModuleAssemblysingletons actuallyreach the service provider the compile service resolves from, in the bake's mesh specifically.
MeshNodeCompilationServicetakes them fromhub.ServiceProviderthrough aLazywhose commentasserts "modules install at boot, before any compile runs" β an ordering claim I have not verified
holds for the gate's composition.
Why this matters for the decision
It shifts the likely fix from "port 12 NodeTypes across three packages, including the Store" β
which would be a substantial content change touching what every instance serves β toward "the
transition dropped one registration path and needs it restored", which is small and does not touch
content at all.
I would not act on that yet. The one experiment that settles it is still the cheap one: bake
7f454af0against both frameworks and dump whatCompileReferences.ComposeWithModulesactuallyreturns in each. That answers "is
MeshWeaver.AIin the reference set?" directly, instead ofreasoning about whether it ought to be.
rbuergi commentedon Aug 28, 2026
The change knew it was removing a surface, restored four assemblies, and missed the fifth β the engine itself
35a9c95c4's diff totools/MeshWeaver.PluginTester/MeshWeaver.PluginTester.csprojis the wholestory, and it is worth quoting rather than paraphrasing:
The author identified the exact hazard and handled four fifths of it. Removing the reference
drops everything it used to supply transitively; four such assemblies were named and restored
explicitly, with the reason written down.
What was not accounted for is
MeshWeaver.AI's own types. The change treats the engine assomething in-mesh source reaches through rather than something it compiles against. The 12 RED
NodeTypes say otherwise β
Edu/PromptCell, fiveStore/*, sixUWDeepfield/*nameAiSettings,ThreadMessage,AiSettingsNodeType,ChatClientCredentialResolverandStartThreaddirectly.So the earlier framing was wrong in both directions
Not "the registration path broke" (it is working as designed β the module lane is deliberate), and
not simply "port the content" (nobody decided that; the inventory was incomplete). It is:
Which makes the decision specific, and the constraint is already documented
The comment rules out the obvious fix in advance, correctly:
So restoring the
ProjectReferencetrades this defect for a different one. The real options are:InstalledModuleAssemblyroutegenuinely reaching
CompileReferences.ComposeWithModulesin the bake. If that already works forother modules, this is a gap in the AI module's path specifically and is small.
MeshWeaver.AI.Contract-shaped assembly carrying the types contentcompiles against (
AiSettings,ThreadMessage,StartThread, β¦) stays in the closure; theengine itself rides the module lane. This is the shape the four restored references already
imply.
Store/*, which everyinstance serves.
(2) looks closest to the intent already expressed in that comment, but this is a design call about
what "content surface" means, and it is yours rather than mine.
Nothing here changes the ordering against #2494. The fleet has not rolled since 14:27, so no
instance has met this framework; fixing the roll first ships 12
CompileErrorNodeTypes.rbuergi commentedon Aug 28, 2026
Same cause as #2563, two different consequences β worth reading together
#2563 (filed 00:57 on memex-cloud) and this issue are the bundle half and the compile half of
one change: the AI engine leaving
ContentSurfaceAssemblies(#2524 /35a9c95c4) to ship as aregistry module.
ref:<hash>, a module presentsmvid:<β¦>InstalledModuleAssemblyCompileErrorβEdu/PromptCell, 5ΓStore/*, 6ΓUWDeepfield/*#2563 names the mechanism more precisely than I did and is worth quoting here, because it explains
why nothing failed loudly:
ref:andmvid:are not comparable, so the gate declines correctly.A gate doing its job produced a silent, fleet-wide recompile.
Two things this changes about what I wrote above
1. memex-cloud has ALREADY met the change. I said no instance had, on the strength of
memexnot having rolled since 14:27 (#2494/#2553). That is true of
memexand false ofmemex-cloud,which rolls on a different path. So the "stalled roller is the only thing protecting production" line
holds for one instance, not the fleet.
2. The observed severity there is degradation, not breakage β 218 first-access compiles, pages
working. That is the bundle half landing alone. Whether
memex-cloudalso carries the twelveCompileErrortypes depends on which of those packages it has installed;Store/*is on everyinstance, so that is worth checking directly rather than inferring.
Both point at the same decision, which is the one this issue ends on: whether the engine's public
types are content surface (β they must remain reachable, by a contract assembly or by the module lane
genuinely supplying the compile references) or whether content ports off them (β 12 NodeTypes,
including the Store package every instance serves).
Fixing either half alone leaves the other. And #2563's own framing β "correctly, by a gate doing its
job" β is the right way to hold it: nothing here is misbehaving; a surface moved and two consumers
of that surface were not moved with it.
rbuergi commentedon Aug 28, 2026
The
Store/*half was already analysed, five days before it broke: Systemorph/MeshWeaver.Plugins#764.Filed 08-27 off a Copilot review on the AI registry cutover, it names the mechanism precisely:
That is this issue's mechanism, written down before the failure.
But it contains one assumption the measurement falsifies, and it is the load-bearing one:
The composed state is exactly what failed. The gate had
Modules:Assemblies:0andModules:Required:0set toMeshWeaver.AI.dllthroughout, and it fails RED naming the module when one does not resolve β it did not. The engine landed, and fiveStore/*types still failedCS0246 'AiSettings'.So the broken case is not only "a deployment that removes the AI module", which is how #764 scoped its severity. It is the composed state too β module present, required, and its types still absent from the reference set.
Two independent readings now converge on the same fix. #764's first option:
is option (2) in my comment above, reached separately. Its second option β skip skill-source merging when the AI module is absent β does not cover the measured case, because nothing is absent.
And #764 names a consequence I had not:
Store/Installeris among the fiveCompileErrortypes, and the installer is the only route to reinstall the engine. That bootstrap deadlock was written up as hypothetical; it is the state today on any mesh baking againsts96fb969β¦. Whichever fix lands has to leave the installer compilable without the engine, or recovery has no entry point.rbuergi commentedon Aug 30, 2026
Fixed for the compile surface: since #2630/#2707 the gates compose
MeshWeaver.AIfrom the sealed plugins publication, and Reinsurance compiles 179/179 on today's image (run 33295773044). The bundle half remains #2563.