Move Cosmos + Snowflake (source, tests, releases) out of the platform: ship released binaries, run their tests only when they change #1752

Activity

1. rbuergi commented on Aug 17, 2026

   

   rbuergi

   on Aug 17, 2026

   ContributorAuthor

   More actions

   Worked this end to end. The gate is shipped (#1759). The move is blocked, and I did not tag to get around it. Also: the CI prize is far smaller than it looks, and I think that changes the decision.

   ---

   1. The measured CI delta — it is 0.4 seconds

   Measured on green main run 32031030133 (2026-08-17), not estimated.

   	measured
   Hosting.Cosmos.Test	72 tests, 25.2 s, on shard 3 — which finished at 12:55:08 while the long-pole shard 1 finished at 12:58:49. 3 min 41 s of slack.
   Hosting.Snowflake.Test	140 tests of which 103 skip, 0.4 s, at the tail of the long-pole shard
   wall-clock saved by dropping both suites	~0.4 s

   Of Cosmos's 25.2 s, 16 s is pulling and creating the emulator container and ~7 s is readiness — the assertions are ~2 s. And Snowflake's emulator classes skip wholesale in CI (no SNOWFLAKE_CONNECTION / LOCALSTACK_AUTH_TOKEN), so 74% of that suite has never run here.

   What a move would genuinely free, honestly stated:

   • 18,976 LOC (12,631 src + 6,345 test) of 674,300 — 2.8%.
   • An exclusive dependency closure. Microsoft.Azure.Cosmos and Snowflake.Data are each referenced by exactly one project, and Apache.Arrow / AWSSDK.Core / AWSSDK.S3 / Google.Cloud.Storage.V1 appear in no csproj at all (transitive from Snowflake.Data only). ~13 MB of nupkg restore, ~25 MB of publish output, plus 4 assemblies off the critical-path Build solution (once) job.
   • One of two Testcontainers dependencies (PostgreSql.Test keeps the other, so Docker stays a CI requirement regardless).

   So the prize is code volume and dependency closure. It is not CI minutes on the critical path, and I would not spend an outage-class risk to buy 0.4 s.

   2. Repo choice: MeshWeaver.Plugins — if the move happens

   Not because it is comfortable, but because three things it needs already exist there and nowhere else:

   1. The only live compiled-module precedent WITH a test job. src/MeshWeaver.Approvals + .Test build -c Release -warnaserror and run dotnet test --logger trx. (SocialMedia has the module job but no tests at all; Reinsurance/Manufacturing are content-only — Reinsurance's legacy/*.csproj explicitly does not compile.)
   2. Docker is already first-class in its CI (login/pull/create/cp/run on ubuntu-latest), so the Cosmos emulator fixture ports as-is — it already uses the generic Testcontainers builder with the arm64 vnext-preview image, not Testcontainers.CosmosDb.
   3. repository_dispatch: [meshweaver-framework-released] is already wired. That is literally your "or update the platform" trigger, and a new repo would have to re-create it.

   What it costs, stated plainly: it is a repo whose README says "no rebuild, no NuGet — the node is the manifest", so compiled infrastructure sits against its grain. It needs a top-level test/ tree, new .gitignore rules, and central package management (it has none — Approvals hard-pins versions inline). And the MW_PLATFORM_REF pin, today covering one platform project for Approvals, would cover eight for Cosmos.Test — the staleness hazard its own CI header documents at length grows with that surface.

   A dedicated repo is cleaner in isolation and worse in total: it re-creates preflight, ACR creds, the dispatch wiring and a second pin to forget, and it does not solve §3.

   3. The blocker: there is no channel that can deliver a bootstrap-tier module

   The chain is short and every link is checked:

   • Bootstrap tier ⇒ the bits must be in the image.
   • The portal image is a pure SDK container build — there is no portal Dockerfile. So the bits must be on disk at publish time, as an MSBuild item.
   • MeshModulesPublish.targets has no non-ProjectReference lane. Build-time is ProjectReference only; the runtime writer (ModuleLandingService) is the Store lane, which bootstrap tier forbids by definition. A third lane would have to be built.
   • To feed it: nuget.org needs NUGET_PAT + a version ⇒ a release ⇒ held; GitHub Packages is worse than held — the org is at its billing limit, which is why plugin-publish.yml is called with dry-run: true (a push returns permission_denied/403 and reds main), and the source was deliberately removed from nuget.config; a GitHub Release asset needs a tag ⇒ held; cross-run actions/download-artifact was explicitly ripped out of main-cd.yml in Platform content bake publishes under a DIFFERENT framework identity than the runtime image resolves — no pod can ever adopt it #1725 and artifacts expire — not "pinned released".

   Worth knowing regardless of the hold: the existing satellite module lane also stops at a run artifact today — "runtime distribution ... task #74 wires satellite-built bundles into that funnel" — and it is the Store lane anyway.

   So: every clean mechanism needs a version tag or a NuGet publish. Per the hold I stopped rather than tagged. Un-blocking it is a scope call: either lift the hold for a satellite v* tag, or fund the third lane (my pick if it goes ahead: a MeshWeaver.* package on nuget.org, consumed via a PackageReference + a package-sourced row in the closure lane — nuget.config's own comment pre-authorizes re-adding a Systemorph source with a matching packageSourceMapping entry, and nuget.org needs no read credential, which GitHub Packages does).

   4. What shipped instead — #1759

   The contract gate this issue asks for, which turns out to be needed today, not only after a move. #1727 landed the closure lane ~20 minutes before this issue was filed, and nothing verifies the folder it lays out is loadable — nor would anything, since neither backend has a compiled reference in the tree.

   Two things that look like coverage are not: the compiler proves the source binds but says nothing about the publish layout (where these two are fragile — their drivers exist nowhere else in the image, so a prune that took one away lands exactly the failure the targets file warns about), and the emulator suites green-SKIP, so they can pass by not running.

   StorageModuleLayoutTest walks the seam a portal walks: ResolveModulePath lands inside modules/<Name>/ rather than the app-folder fallback, the private driver survived the prune and loads, InstallAssemblies folds the MeshNodeProviderAttribute, and the keyed IStorageAdapterFactory comes from that DLL. No emulator, no network, ~40 ms.

   I verified it can fail, twice: deleting Microsoft.Azure.Cosmos.Client.dll reds both Cosmos tests (the seam one surfacing the real FileNotFoundException from JITting Create); deleting modules/ reds all four. Host publish is unchanged — 6 thin + 14 closure, 99 files, both storage folders intact.

   And it is exactly the gate a move would need: point it at a pinned released binary instead of the in-tree build and it answers the question a moved backend raises. Whatever is decided in §3, this had to exist first.

   5. Deferred

   The move itself — source, tests, and the release lane — pending the §3 scope call. Nothing in #1759 presumes either outcome. Systemorph/MeshWeaver.Plugins#1140 (Cosmos has no access control in its query path) is untouched and still gates production readiness independently.
2. rbuergi commented on Aug 21, 2026

   

   rbuergi

   on Aug 21, 2026

   ContributorAuthor

   More actions

   Closing as superseded — not because the move happened, but because the thing this called "the actual prize" was measured and priced by someone else after this was filed, and it is 31 seconds.

   The prize, measured

   "Platform CI stops running the Cosmos/Snowflake suites. … This is the actual prize — MeshWeaver.Hosting.Cosmos.Test needs the Cosmos emulator via Testcontainers, which is expensive to stand up on every platform commit for code that changes rarely."

   .github/scripts/shard-assign.sh weights every test project in seconds, heaviest first:

   345 MeshWeaver.Hosting.Monolith.Test
   263 MeshWeaver.AI.Test
   239 MeshWeaver.Hosting.Orleans.Test
   …
    31 MeshWeaver.Graph.Test
    30 MeshWeaver.Hosting.Cosmos.Test      ← the emulator suite
   …
     1 MeshWeaver.Hosting.Snowflake.Test   ← green-skips: needs SNOWFLAKE_CONNECTION
   

   30 s + 1 s, LPT-balanced across shards. And the "expensive to stand up" premise was investigated and fixed head-on — the same file records the decision, taken after this issue:

   "Cosmos was excluded until the emulator was verified on a runner: the CLASSIC azure-cosmos-emulator image really is too heavy (multi-GB, minutes to start, HTTPS-only, and it publishes a linux/amd64 manifest ONLY…). The vnext-preview image CosmosFixture pins is ~0.68 GB, reports ready in ~10 s over plain HTTP, and ships arm64 — so the project now runs like any other, and green-SKIPS (never reds) when Docker is unavailable."

   So the suite this proposes to stop running costs half a minute and, unlike a moved suite, actually exercises the backend against the platform it ships with. # EVERY test project runs — there is no exclusion list. is a deliberate policy, not an oversight.

   What DID land, and it is the part that mattered

   #1759 shipped the contract gate this issue said must be paired with the move — test/Memex.Portal.Shared.Test/StorageModuleLayoutTest.cs. With no emulator, no network, in milliseconds, it asserts that ResolveModulePath finds each backend under modules/, that its private driver survived the closure prune, that InstallAssemblies folds the assembly's MeshNodeProviderAttribute, and that the keyed IStorageAdapterFactory which Graph:Storage:Type resolves comes from that DLL — from a project holding no compiled reference to either backend. Its own header names it as the prerequisite for a satellite move:

   "point it at a pinned released binary instead of the in-tree build and it answers the question a moved backend raises — does a module built against platform N−1 still bind in N."

   That gate is the durable value here, and it stands whether or not the source ever moves. (While auditing I also found the closure lane was deleting both backends' native payloads — libsf_mini_core.*, libMono.Unix.*, Cosmos' ServiceInterop — so Graph:Storage:Type = Snowflake on a shipped image would have failed at its first native call; fixed in #1972, with the layout now asserted by that same test.)

   Where the rest of the shape stands

   step	state
   1. source + tests move to a plugin repo	not done — neither backend is in MeshWeaver.Plugins/src/ (33 dirs) nor in its 16-entry module matrix
   2. that repo publishes versioned releases with a MinMeshVersion floor	not done
   3. the image consumes a pinned released artifact	not done — MeshModulesPublish.targets:104-105 names the in-tree csprojs
   4. platform CI stops running the suites	not done — and now priced at 31 s
   5. contract gate	done (#1759)

   Steps 1–3 are still a legitimate ownership preference, and they remain available: the bits are already module-only (no host ProjectReference anywhere; only the two test csprojs and the closure lane reference them), so the move is a relocation, not a decoupling.

   But they can never be Store packages — these are bootstrap tier, as StorageModuleLayoutTest.cs:15-18 puts it: "the mesh cannot read itself without a storage backend, so they can never be Store-installed (the Store's catalog lives behind the very storage you would be installing)". Delivery would stay "pinned released binary baked into the image". And the cost this issue itself names is unchanged: a framework change that breaks IStorageAdapterFactory becomes invisible until a portal fails to boot — traded now for 31 seconds rather than for the minutes the argument assumed.

   Reopen if the ownership half is still wanted on its own merits. The CI-minutes argument is spent.

   _{Measured 2026-08-21 against origin/main @ 3d85424 while sweeping the #1660 programme.}
3. rbuergi

   closed this as not plannedon Aug 21, 2026