Repository navigation
DESIGN: the modules/ publish layout — cutting the ships-the-bits ProjectReferences #1644
Description
The module lane is complete on the ACTIVATION side (every pack self-registers via MeshNodeProviderAttribute; Modules:Assemblies is the on/off switch), but every module still rides a ProjectReference from Memex.Portal.Shared whose only purpose is landing the DLL in the publish output. This design cuts those references — the step the extraction queue called "modules-folder publish" and the precondition for #1630's bundles becoming the distribution channel for first-party packs too.
Goal
A deployment's image = the base portal + a modules/ folder; which modules EXIST is a publish/composition decision, which ACTIVATE stays Modules:Assemblies. ReferenceOutputAssembly=false was already proven NOT to work (drops the DLLs) — the publish target is the missing piece.
Design
1. Publish layout. A PublishModules MSBuild target on each host publishes every module project into modules/<ModuleName>/ beside the app: the module DLL + the part of its transitive closure NOT in the app's own closure (dotnet publish per module, then prune assemblies whose identity exists in the app output — same-identity duplication is the documented trap-door class, so pruning is a MUST, and the target FAILS if a listed module is still ProjectReferenced by the host).
2. Loading. Modules:Assemblies entries resolve BaseDirectory-relative today; extend the resolution to probe modules/<name>/<name>.dll. Assembly.LoadFrom (Default ALC) probes the module's own folder for its private deps — no ALC work needed.
3. In-mesh compile surface. Module DLLs leave the publish closure ⇒ leave TRUSTED_PLATFORM_ASSEMBLIES ⇒ in-mesh consumers (e.g. Cornerstone's MapControl) lose them. Fix: InstallAssemblies records each loaded module (an InstalledModuleAssembly enumerable singleton), and MeshNodeCompilationService composes its reference list as the static TPA baseline + the mesh's installed modules. (Deliberately NOT done while modules ride the image — it becomes necessary exactly here.)
4. Bake fingerprint. CompiledFrameworkVersion stays the Graph MVID (rule 3 untouched). A NEW CompiledModulesHash — hash of the sorted installed-module MVIDs — joins HasUsableBuild: a module upgrade invalidates baked builds that could reference it. Cost: a module bump recompiles the backlog like a framework bump does — bounded by the batch bake (~0.3s/type Roslyn; the cost is activations + pacing).
5. Plugin-gate parity. compile-check.py --refs today mirrors the app closure; the exported ref set gains the module set of the deployment shape being gated, or the gate silently diverges from prod (the #683 class).
6. RCL static assets. Runtime-loaded view packs bring no build-time staticwebassets manifest (the documented wall). Publish copies each module's wwwroot into modules/<name>/wwwroot; a ModuleStaticFileProvider maps _content/<ModuleName>/ there at startup (modules load before endpoint mapping, so ordering works). Scoped-CSS bundles (*.styles.css) remain build-time-only — view packs must keep styling via inline styles or plain CSS files (they already do).
Migration order (each step shippable alone)
- Additive: publish target + probing + compile-surface + fingerprint + gate parity — modules still ALSO referenced (double-shipped briefly; pruning tolerates it via the fail-if-referenced check being flipped LAST per module).
- Flip references one module at a time: storage packs first (no assets, no in-mesh consumers), then export/observability/speech/indexing, then map providers + GUI packs (assets),
MeshWeaver.MapsLAST (in-mesh consumers exercise step 3+4 for real). - Then feat: serve prebuilt plugin assemblies as bundles, and adopt them on install #1630 bundles can deliver first-party modules externally — the
modules/folder is the shared local layout both lanes target.
Risks
- Same-identity double-load (module in modules/ AND app closure) — the trap-door class; the publish target's fail-if-referenced check is the guard.
- Version skew between a module and the app it lands beside — the framework-MVID gate from feat: serve prebuilt plugin assemblies as bundles, and adopt them on install #1630's bundle index is the precedent; the loader should log module + framework identity at install.
- Bake storms on module updates — bounded by batch bake; the fingerprint MUST include only modules, never content, or every plugin install would invalidate all builds.
Review wanted before any code — sequencing, the fingerprint extension (rule-3-adjacent!), and whether the gate-parity work should land in the Plugins repo first.
Activity
rbuergi commentedon Aug 16, 2026
Step 1 SHIPPED as #1653 (riding CI): publish target (thin layout — a full per-module publish drags ~1400 shared-framework assemblies a filename prune can't see, so closure layout is per-flip work),
ResolveModulePathprobing,InstalledModuleAssembly→ compile-surface composition,CompiledModulesHashstamped write-side (decisive at step 2, null grandfathered), and the Architecture/Modules reference page.Step-2 flip order (Wave 6), one module per PR, each upgrading its targets entry to a module-correct closure layout + verifying by real publish and the feature's own e2e:
MeshWeaver.Markdown.Export— smallest closure, no assets, no in-mesh consumers. This flip also lands themodules/sweep in the Plugins repo'sfetch-refs.py(gate parity, design §5).MeshWeaver.Observability,MeshWeaver.ContentCollections.Indexing.PostgreSql— service-only.MeshWeaver.Speech— native runtime assets (the known closure exception).MeshWeaver.OgCard, then the AI provider packs.Blazor.Radzen/Analysis/GoogleMaps) LAST — they need the §6 static-asset story (ModuleStaticFileProvider) first, and the GoogleMaps flip doubles as the falsification check for the old queue item "move GoogleMapControl into MeshWeaver.Layout because TPA freeze breaks in-mesh consumers": feat(modules): the modules/ publish layout, step 1 — probing, compile surface, fingerprint (#1644) #1653's compile-surface composition should make in-meshMapControlconsumers compile with the pack outside the app closure — verify on the flipped deployment shape, then close that queue item as obsolete.Also filed: #1655 (endpoint-contribution hook design) — the seam that would let a flipped module carry its HTTP half (
/api/og,/api/speech/transcribe,/api/log-incidents) instead of leaving it compiled with an optional-service 503.Merge pull request #1653 from Systemorph/feat/modules-folder-publish
Merge pull request #1666 from Systemorph/feat/modules-flip-markdown-e…
Merge pull request #1681 from Systemorph/feat/flip-all-service-modules
rbuergi commentedon Aug 21, 2026
Closing: the design is implemented and in production; all six parts shipped, several of them hardened past what was specified. What is left is execution of step 2 for five remaining modules, carried to #1974 (the three view packs, plus AppleMaps/OpenStreetMap) — the other two have their own, unrelated blockers, stated below.
Audited against
origin/main@ 3d85424.1. Publish layout — DONE.
memex/MeshModulesPublish.targets, targetPublishMeshModules(:145) + the shared workerLayoutMeshModuleClosures(:218) + a dev-run hookLayoutMeshModuleClosuresAfterBuild(:207) the design did not call for anddotnet runneeds. Two lanes as designed: thin@(MeshModule)copies own outputs (:163-174), closure@(MeshModuleClosure)does a real per-modulePublishintomodules/<Name>/(:279-283).Pruning is stricter than the design asked: the closure lane compares by SHA-256 bytes, not name (
:326-350), after a same-name-different-version prune took production down on 2026-08-19 (Could not load file or assembly 'Microsoft.Extensions.AI.OpenAI, Version=10.8.0.0'— the reasoning is at:304-325, and the keep-branch says so out loud). Two prunes the design did not name were added: shared-framework assemblies globbed from the targeting packs, with a hardErrorif the glob comes back empty (:358-374), and the RID tree — which has since been narrowed to keep the loadable natives, because two modules on this lane genuinely need them (#1728, #1972).Fail-if-referenced — DONE (
:238-239), default-on (:142), host-scoped escape only (Memex.Portal.Monolith.csproj:22, for the Northwind sample's transitiveMeshWeaver.Import). A subset naming nothing fails RED (:232-233) rather than laying out nothing and reporting success.2. Loading — DONE, and extended:
MeshBuilder.ResolveModulePathprobes landed → image → app closure (src/MeshWeaver.Mesh.Contract/MeshBuilder.cs:59-77), the landed root being the writable, pod-shared one a registry install writes to.3. In-mesh compile surface — DONE.
InstallAssembliesregisters anInstalledModuleAssemblyper module (MeshBuilder.cs:95-100);CompileReferences.ComposeWithModules(src/MeshWeaver.Compiler/CompileReferences.cs:102-116) is TPA baseline ∪ modules, consumed byMeshNodeCompilationService(:183-185),MeshScriptEnvironment, the endpoint hook and Grpc hosting.4. Bake fingerprint — DONE and decisive.
CompiledModulesHashis stamped (NodeTypeCompilationHelpers.cs:1709) and joined into the decision viaDependenciesValid(:1452-1460), with the twin rule on the rebuild-kickoff side (:1543-1560) resolved through oneGuardsOfso the two cannot fork. This closes what #1664 step 11 recorded as half-built. It has since been superseded upward: when a per-typeCompiledDependenciesrecord is available it decides, and the whole-set hash is the fallback — which is the finer-grained invalidation rule 4 was reaching for.5. Plugin-gate parity — DONE, both ways in. CI copies
/app/.wholesale andcompile-check.pysweeps a flat ref dir recursively; locallyMeshWeaver.Plugins/scripts/fetch-refs.py:130-143liftsmodules/*/*.dllinto the ref set by name and citing this design, one level deep on purpose soruntimes/natives andwwwroot/are never handed to the compiler.6. RCL static assets — DONE, as #1724 (
MeshModuleStaticAssetExtensions, #1775): path-convention mount, dependency namespaces, the scoped-CSS bundle surfaced forApp.razorto link, and precompressed negotiation. Fingerprinting is the one unmet piece; carried to #1974.Step 2 — the flip ledger, as it stands
Flipped (closure lane, 5):
MeshWeaver.Social,MeshWeaver.OgCard,MeshWeaver.Import,MeshWeaver.Hosting.Cosmos,MeshWeaver.Hosting.Snowflake— the last two born flipped.Not flipped (thin lane, 5):
Blazor.Radzen,Blazor.GoogleMaps,Blazor.Analysis— were blocked on static assets; that blocker is gone. → Flip the Blazor view packs onto modules/ — the static-asset provider shipped and has no consumer #1974.MeshWeaver.Speech— compile-time coupling:ISpeechTranscriberlives in the module with no contract split (targets:60-66).MeshWeaver.Hosting.Grpc—UseMeshWeaverGrpcWebWhenInstalledis middleware-order-coupled and cannot ride the endpoint hook.Two things the ledger no longer captures
0b3b4266f("Fourteen modules leave the platform", 2026-08-19) deleted Markdown.Export — the first flip, and this issue's worked example — along with Observability, the four AI packs, Teams, Courses, Mcp, AI.WebSearch and others fromsrc/. They ship as registry Store bundles landed intomodules/<Name>@<gen>/on the writable root. So the design's step 3 ("then feat: serve prebuilt plugin assemblies as bundles, and adopt them on install #1630 bundles can deliver first-party modules externally") happened, and for those modules the ProjectReference question is moot. Several "deliberately NOT referenced" comments still standing inMemex.Portal.Shared.csprojdescribe modules that are no longer in this repo at all.ImportModuleLayoutTestis named in two csprojs and does not exist (Memex.Portal.Shared.Test.csproj:3,Memex.Portal.Distributed.csproj:88).MeshWeaver.ImportIS laid into the test bin viaMeshModulesClosureSubset, but onlyStorageModuleLayoutTestsurvives and it covers Cosmos and Snowflake only — so Import's shipped layout is asserted by nothing. Carried to Flip the Blazor view packs onto modules/ — the static-asset provider shipped and has no consumer #1974.Audited while sweeping the #1660 programme.
Anchor the view-pack test on Analysis directly