The combo gate exists and is never called — nothing verifies an image against the modules an instance actually runs

Third instance today of the same shape: a mechanism that is fully built, documented and tested, and invoked by nothing. This one is the CD gate that would have prevented tonight's production trap.

The gap

InstanceComboVerifier (src/MeshWeaver.PluginCatalog/InstanceComboVerifier.cs) does exactly the right thing — verifies a candidate image against the combo an instance actually runs, folding the evidence into one verdict with three carefully-distinguished outcomes (Green / Red / NotVerifiable, explicitly never conflated). ComboVerification's own doc says the verdict is stored on Admin/UpdatePolicy → UpdatePolicyContent.ComboVerifications.

Measured:

grep -rn 'InstanceComboVerifier|\.Verify\(' src/ | grep -v 'InstanceComboVerifier.cs|\.Test/'
   → nothing (the only .Verify( hits are token verification, unrelated)

grep -rn 'ComboVerifications' src/ | grep -v '\.Test/'
   → one doc comment. Nothing reads it when deciding to update.

And on a live instance (memex.systemorph.com), Admin/UpdatePolicy reads:

{ "requireCiGreen": true, "latestAvailableTag": "3.0.0-rc8.ci.5363",
  "checkedAt": "2026-08-25T20:11:08Z", "comboVerifications": [] }

Empty, because nothing ever writes it. The poller is alive and sees a newer build; nothing verifies whether that build can serve the modules this instance has landed.

What it cost tonight

memex.systemorph.com is trapped between two failing states, and a user hit it as a rendering bug (/ClientB/EmailThomas → "Importing a module script failed"). Its module-shipped assets 404 while the same assets are 200 on the cloud portal:

                                                     systemorph   cloud
Radzen.Blazor/Radzen.Blazor.js                          404        200 (140 KB)
MeshWeaver.Blazor.OpenStreetMap/OpenStreetMapView.razor.js  404    200
MeshWeaver.Blazor.EntityViews/…styles.css               404        200
MeshWeaver.Blazor.Analysis/…styles.css                  404        200

Every statically-referenced asset on that page is 200, and every dynamic import from core is 200 on both portals — the misses are exclusively module-shipped assets.

It cannot be fixed one half at a time, because an optional parameter added to a record's primary constructor replaces the signature:

roll the image forward → new platform + its old landed modules → MissingMethodException → host aborts (this is what crashlooped it at ci.5355 earlier tonight, and was reverted)
re-fetch its bundles → new modules + its old platform → the identical abort from the other side

Its two halves are consistently old, which is exactly why it is up. A combo verification would have said so before either move — that is precisely the question InstanceComboVerifier answers and nobody asks.

What NOT to do

The self-update poller (memex/Memex.Portal.Shared/SelfUpdate/SelfUpdateHostedService.cs) carries a deliberate, load-bearing invariant:

🔁 wedges-to-zero: NEITHER mesh touch may gate the roll-forward … a fresh image is precisely what recovers a degraded pod.

So do not simply make the roll conditional on a mesh-read verdict — that would reintroduce the wedge that invariant exists to prevent. Tonight's evidence does qualify it (a fresh image does not recover a pod whose modules cannot bind), but reconciling the two is a design decision, not a patch:

run the verification and record it (populating ComboVerifications) without gating the emergency roll-forward path, and/or
gate the catalog / fleet-driven update — where a human or the pipeline chooses a target — while leaving self-recovery ungated.

Why this issue rather than a PR

The verifier is public core; the decision point is the private overlay; and the invariant above means the wiring is a judgement about which path may be blocked. I did not want to guess that in someone else's subsystem — but the gap should not stay invisible, because the evidence that a gate did not run is the absence of evidence.

Related: #2234 (atomic module/platform movement), #2235 (the release broadcast that also had no caller).

Picked this up as part of the "gate exists, never fires" cluster and measured what wiring it would actually take. Confirming the issue's own conclusion — this needs a decision, not a patch — but with the blockers named precisely, so the decision can be made without re-deriving them.

The read side is already complete; only the write is missing

This is a smaller gap than "wire up the verifier" suggests. Everything downstream of the verdict exists and is tested:

UpdatePolicyNodeType.RecordVerification(IMessageHub, ComboVerification) — cold, writes as System through GetMeshNodeStream(NodePath).Update(...), upserts by CandidateTag, orders by VerifiedAt desc, truncates to MaxRecordedVerifications (8), touches only ComboVerifications. Zero production callers; its only caller is ComboVerdictRecordingTest.
UpdatePolicySettingsTab.StatusMarkdown already reads it — content.VerificationFor(tag) at line 187, then a three-way switch on Green/Red/_. Nine tests pin the rendering, including "a Green with caveats never renders as an unqualified pass".

So the admin tab is already written to display a verdict that nothing ever produces.

Why the poller cannot simply call the verifier

InstanceComboVerifier.Verify(combo, imageRef, workRoot, candidateTag) needs three things a portal pod does not have:

Docker. The production gate seam is DockerImageGate in tools/MeshWeaver.ComboVerifier/ — an OutputType=Exe project referenced by nothing. Memex.Portal.Shared does reference MeshWeaver.PluginCatalog, so InstanceComboVerifier / InstanceComboAssembler / ComboVerification are already on the portal's compile path — but the docker orchestration is not, deliberately.
A filesystem workRoot to materialise every module into.
Repo read credentials for the module sources.

That is why the design put it off-cluster, and it is the real reason it has no caller — not an oversight in the poller.

The invariant is NOT the obstacle people assume

Worth stating, because it is the thing the issue was most careful about. The class doc says the poller makes "only two mesh touches"; it is one behind — it makes four (ReadPolicyStream, RecordAvailable, RecordHold, BuildCompletionTicks), and two of them are writes added after the invariant was written. RecordAvailable carries the pattern verbatim (#1020):

🚨 BOOKKEEPING, NOT A GATE. RecordAvailable writes two cosmetic fields … the ROLL-FORWARD does not depend on them. Chaining Apply after it with .SelectMany made a failed status write abort the update: the portal sat 37 h on a stale image. … surface the fault (never swallow it) and carry on to Apply … Concat, not Merge — the record still lands FIRST when it succeeds.

A record-only RecordVerification, .Catched into Observable.Return(Unit.Default) with a naming warning and .Concated (not .SelectManyed) into the roll path, is structurally identical to the two writes already sanctioned. SelfUpdatePollerResilienceTest already subclasses the poller to fault-inject ReadPolicyStream/RecordAvailable, so "this new write cannot gate the roll" is provable in the existing harness.

So the wedges-to-zero invariant does not block the recording half. What it blocks is making the roll decide on the verdict — and that is the separate, larger call.

The decision, as narrowly as I can put it

Given docker/workRoot/credentials, the verdict has to be produced off the pod. Which leaves the question the issue already framed, restated as a choice of producer:

(a) CD-side producer. mw-combo-verify runs where docker exists, and lands the verdict via an authenticated channel — which is what the runbook already does BY HAND (CandidateReleaseProtocol.md has an operator landing combo-verdict.json through the MCP patch tool, i.e. the doc already concedes there is no automatic writer). Making that step automatic is the smallest change that turns the admin tab from decorative into live, and gates nothing.
(b) Gate the catalog/fleet-driven update — where a human or the pipeline picks a target — while leaving self-recovery ungated. This is the half that would have prevented the memex.systemorph.com trap, and it is a policy call about which path may be blocked.

(a) is available today and is not a judgement about someone else's subsystem; (b) is. I did not implement either, for the reason the issue gives — but the constraint list above is now measured rather than assumed, and (a) does not need the invariant relitigated.

One thing that would be cheap and is not blocked by any of this: InstanceComboReader (the "what does this instance run" half) is DI-registered and pod-side, and the instance's combo alone — with no image to verify against — is already more than comboVerifications: [] tells an admin today.

Re-measured against origin/main @ e5ce9628e (2026-08-26) as part of triaging this alongside #1979 — a related "built, tested, never called" report. Confirming the prior investigation on this issue holds, and naming one new blocker.

Still true, unchanged:

InstanceComboVerifier.Verify( — zero callers outside its own file and InstanceComboVerifierTest.
UpdatePolicyNodeType.RecordVerification — zero production callers; only ComboVerdictRecordingTest.
UpdatePolicySettingsTab.StatusMarkdown already reads content.VerificationFor(tag) and renders all three verdicts correctly — the admin tab is still wired to display a verdict nothing produces.
InstanceComboReader.Read() — DI-registered (PluginCatalogConfigurationExtensions.cs:120), fully tested (InstanceComboReaderTest), zero non-test callers. The "cheap, unblocked" win the last investigation named (surfacing the instance's own combo, no verification, no gating) is still unimplemented.

New blocker for that one cheap win: the only place to render it is memex/Memex.Portal.Shared/Settings/UpdatePolicySettingsTab.cs, and memex/Memex.Portal.Shared/ is now mid-extraction — 044a85618 ("The GUI leaves the platform: the view packs and the web clients", merged today) already moved every Blazor view and web client out to Systemorph/MeshWeaver.Plugins, and other sessions are actively restructuring what's left in that directory. Editing it right now risks losing the change to a concurrent file move rather than landing it. This settings tab itself wasn't touched by 044a85618 (it's a platform admin tab, not a "view"), but the directory-wide caution still applies.

No code change made this session. The core finding stands: producing a verdict needs Docker + a materialisation root + repo credentials the portal pod does not have, so the write side has to run off-pod (CD-side), and gating the roll on it vs. only recording it is the policy decision the original issue asked someone to make — not re-derived here, just re-confirmed. The InstanceComboReader surfacing win is now blocked twice over: still nobody's decision to gate anything on it, and its one home is a directory under active restructuring. Recommend picking this back up once the GUI-extraction program settles, or landing it wherever UpdatePolicySettingsTab (or its equivalent) ends up.

Two prior investigations on this issue concluded "decision, not patch" and named the blockers. Both were right about the blockers. But before choosing between wiring it up and deleting it, I ran the gate — for the first time in the 17 days it has existed. That produced a third answer that supersedes both options, because you cannot wire up a gate that reports false Greens, and you cannot delete it as redundant when the invariant is only partly covered.

It never had a caller — and its production seam had no test

Re-measured, and one detail sharper than previously recorded: InstanceComboVerifier, mw-combo-verify and UpdatePolicyNodeType.RecordVerification all arrived in one commit (6e517afe1, 2026-08-10). So this was never a caller lost in a refactor — "restore" was never on the table. GitHub code search across the org returns 5 files for InstanceComboVerifier, all its own definition, test and CLI; .github/ references mw-combo-verify zero times.

The part nobody had noticed: DockerImageGate — the only production seam, the 210 lines that build the docker argv and parse the report — had no test of any kind. No test project referenced tools/MeshWeaver.ComboVerifier. The folding is thoroughly fixture-tested; the thing that actually runs never was.

Running it: it works, and it was wrong in the way a gate is worst wrong

A real combo (AppleWeather @ ad6c2d39 out of MeshWeaver.Plugins) against a real candidate (mw-plugin-test:main): assembled, compiled, rendered, 38/38 tests, Green in six seconds. The machinery is sound.

It also passed by testing the wrong architecture. A candidate tag is a multi-arch manifest list; docker run without --platform silently picks the host's architecture, and docker reports the same manifest LIST digest whichever variant ran:

list   sha256:4a63eda…   ← what the verdict recorded
amd64  sha256:ab6efc31…  ← what AKS runs — never executed
arm64  sha256:e353c397…  ← what actually ran

main-cd.yml already bakes one lane per architecture precisely because those bytes differ (four reference assemblies differ; each resolves its own framework build identity), and every other docker run … --entrypoint /app/mw-plugin-test in the repo passes --platform. This one did not — so the runbook in CandidateReleaseProtocol.md, followed exactly on an arm64 laptop, yields a false Green about the architecture the fleet serves. That is the "false pass on its first day" failure, and it would have been the first thing anyone wiring this up inherited.

Fixed in #2475: --platform pinned on every call (default linux/amd64), the presence probe made per-platform, and the platform recorded on the verdict as verifiedPlatform. Same combo, same tag, before → after: Green → NotVerifiable (the emulated amd64 tester exits 139 on an arm64 host — the honest answer is that the question could not be answered there). Both negative controls proven red, including the subtle one: --platform moved after the image ref still passes a presence check while being invisible to docker.

What changed under this issue on 2026-08-26 — the day after it was filed

Worth recording for whoever makes the remaining call, because the coverage picture is not what it was when this was written:

5507585e4 — publish-bake now bakes and gates every plugin against the same image in the same run, asserts identity equality, and publishes bundles + image atomically. That is the combo check for the CD-shipped combo (image × plugins tip).
f60dc19df — check-record-signatures.py refuses, at PR time, the exact break this issue opens with (a record's primary constructor gaining a parameter); its docstring cites the same LanguageModelCatalogSource incident.

So the residual uncovered case is narrower than in August: per-instance content at refs CD does not ship — which is exactly what mw-combo-verify is for.

Two things that read as covering this and do not: verify-images asserts images exist, not that they work; landed-modules-gate compiles plugin source at tip and is blind to binary breaks by construction. And the pod-side backstop is still unarmed — PreWarm__GateReadiness: "false" in both deploy/aks/values.aks.yaml and deploy/helm/values.yaml, so item 1 of this doc's own "What actually has to change" is still open.

Still open — deliberately not decided in #2475

The producer. It needs docker, a materialisation root and repo credentials a portal pod does not have (off-cluster by design), and whether a verdict may block a roll runs into the poller's wedges-to-zero invariant. That remains the maintainer's call. What is now true is that when someone wires it, the thing being wired reports the truth — and the seam has a regression guard so it cannot silently drift back.

Merge pull request #2475 from Systemorph/fix/2274-combo-gate

Re-measured on `origin/main` 2026-08-29: this issue's gap is unchanged — `034be6f8a` does not close it

Flagging because 034be6f8a's subject carries (#2274) and reads like a fix. Its body says Refs #2274, not fixes: it corrected a false Green found while running the gate by hand, and explicitly leaves the subject of this issue alone.

What DID land ✅

tools/MeshWeaver.ComboVerifier/DockerImageGate.cs:49 — public const string FleetPlatform = "linux/amd64";, --platform pinned on every docker call, per-platform presence probe
src/MeshWeaver.PluginCatalog/ComboVerification.cs:118 — public string? VerifiedPlatform { get; init; }
DockerImageGateArgsTest

What this issue asked for — still nothing calls the gate ❌ (all git grep on origin/main)

InstanceComboVerifier — hits are its own file, InstanceComboVerifierTest, tools/MeshWeaver.ComboVerifier/Program.cs:191 and doc comments. No production caller.
UpdatePolicyNodeType.RecordVerification (memex/Memex.Portal.Shared/SelfUpdate/UpdatePolicyNodeType.cs:215) — only caller is test/Memex.Portal.Shared.Test/ComboVerdictRecordingTest.cs:265.
InstanceComboReader — DI-registered at src/MeshWeaver.PluginCatalog/PluginCatalogConfigurationExtensions.cs:120, zero non-test callers.
git grep 'combo-verify' origin/main -- .github/ → 0 hits. Systemorph/Memex code search for InstanceComboVerifier / combo-verify / RecordVerification → 0 (a control query on that repo returns 17, so search works).
deploy/aks/values.aks.yaml:253 — PreWarm__GateReadiness: "false"; the pod-side backstop is still unarmed.

src/MeshWeaver.Documentation/Data/Architecture/CandidateReleaseProtocol.md:288-300 still documents it as a hand-run operator step, which concedes there is no automatic writer.

The open part is a decision — record-only, or gate the catalog-driven update — not an implementation detail. Keep open.

Heads-up for anyone arming `PreWarm__GateReadiness` off this issue

This issue records PreWarm__GateReadiness: "false" as the unarmed pod-side backstop, so it is a natural place to decide to arm it. Do not arm it on the chart's word about the startup budget — the chart is wrong about its own value.

deployment.yaml asserts, twice, that the startup probe budget is 10 × 180 = 30 min ("which managed envs set to…"). values.yaml ships 5 × 60 = 5 minutes, and memex-cloud was measured live running exactly that on 2026-08-30. The gate's own prerequisite 2 is a budget covering a full cold bake — which the chart derives from production Loki as ~570 s (~2.4 s per NodeType × ~240 types, sequential). Arming the gate on a 5-minute probe does what the chart itself warns about: "Kubernetes kills the pod mid-bake, every time, forever."

Being fixed alongside #2787: the prose now describes what ships, and PreWarmGateReadinessGuard makes the coupling arithmetic — arming the gate fails the build unless the budget, PreWarm__DynamicTypes, maxUnavailable: 0 and the /health startup probe all move in the same change.

Re-measured on current `main`: the premise has moved, the conclusion stands — and it is now harder to see

Re-ran this issue's own greps rather than trusting a 5-day-old measurement (a habit earned tonight: #2640's top candidate turned out to have been fixed 50 minutes after its measurement window closed). Some of this has genuinely advanced.

What now exists that did not

UpdatePolicyNodeType has both storage seams:

Writer — RecordVerification(hub, verdict), upserting by tag, bounded to MaxRecordedVerifications, written through GetMeshNodeStream(...).Update under AsSystem.
Reader — VerificationFor(tag).

Plus ComboVerdictRecordingTest, which round-trips a verdict through the hub serializer, pins upsert-by-tag, and pins the bound. That is real work and the issue's "nothing reads it" line no longer describes the tree.

What has not changed

RecordVerification  callers → test/Memex.Portal.Shared.Test/ComboVerdictRecordingTest.cs  (only)
VerificationFor     callers → memex/Memex.Portal.Shared/Settings/UpdatePolicySettingsTab.cs:206  (only)
InstanceComboVerifier callers → none outside its own file and doc comments

So in production:

Nothing writes a verdict. RecordVerification has no non-test caller, so ComboVerifications is still empty on every instance — for the same reason as when this issue was filed, one layer further in.
The one production reader is a settings-tab display, not a gate.
The decision path does not consult it at all. VersionSelect.PickTarget(tags, policy, requireCiGreen) decides on the policy kind and the CI-green tag filter; PlatformUpdateStatus consults policy.IsHeld(latest) — a manual hold. Neither calls VerificationFor.

🚨 Why this is worse than the original state, not better

When the mechanism was one uncalled class, "it is not wired" was visible from a single grep. Now there is a writer, a reader, a test suite and a settings tab — the shape of a working feature from every angle except the one that matters. A reader who sees the verdict rendered in the settings UI will reasonably conclude the gate is live.

And the UI reads a field nothing writes, so it can only ever render the empty state. That is the failure mode where a control covers only the empty case and never sees the populated one: it looks correct forever, precisely because the thing it displays never arrives.

What closing this actually requires

Not "call the verifier" — three things, and the middle one is the load-bearing one:

Something in the update path runs InstanceComboVerifier for the candidate tag.
Its verdict is recorded via RecordVerification (this seam is ready and tested).
VersionSelect.PickTarget / PlatformUpdateStatus consult VerificationFor(candidate) and refuse to roll on Red, with NotVerifiable handled as its own outcome — ComboVerification's doc is explicit that the three verdicts are never to be conflated, and a NotVerifiable treated as Green reproduces exactly the trap this issue was filed about.

Steps 1 and 2 without step 3 would make the settings tab show real verdicts while the instance still rolls regardless — which would look like the issue was fixed. Worth stating in whatever change closes this that step 3 is the acceptance criterion, and that a test asserting a Red verdict actually blocks a roll is the one that proves it.

Merge pull request #2999 from Systemorph/fix/combo-gate-wiring

Navigation Menu

Repository navigation

The combo gate exists and is never called — nothing verifies an image against the modules an instance actually runs #2274

The gap

What it cost tonight

What NOT to do

Why this issue rather than a PR

The read side is already complete; only the write is missing

Why the poller cannot simply call the verifier

The invariant is NOT the obstacle people assume

The decision, as narrowly as I can put it

It never had a caller — and its production seam had no test

Running it: it works, and it was wrong in the way a gate is worst wrong

What changed under this issue on 2026-08-26 — the day after it was filed

Still open — deliberately not decided in #2475

Re-measured on `origin/main` 2026-08-29: this issue's gap is unchanged — `034be6f8a` does not close it

Heads-up for anyone arming `PreWarm__GateReadiness` off this issue

Re-measured on current `main`: the premise has moved, the conclusion stands — and it is now harder to see

What now exists that did not

What has not changed

🚨 Why this is worse than the original state, not better

What closing this actually requires

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

The combo gate exists and is never called — nothing verifies an image against the modules an instance actually runs #2274

Description

The gap

What it cost tonight

What NOT to do

Why this issue rather than a PR

Activity

rbuergi commented on Aug 25, 2026

The read side is already complete; only the write is missing

Why the poller cannot simply call the verifier

The invariant is NOT the obstacle people assume

The decision, as narrowly as I can put it

rbuergi commented on Aug 26, 2026

rbuergi commented on Aug 27, 2026

It never had a caller — and its production seam had no test

Running it: it works, and it was wrong in the way a gate is worst wrong

What changed under this issue on 2026-08-26 — the day after it was filed

Still open — deliberately not decided in #2475

rbuergi commented on Aug 29, 2026

Re-measured on origin/main 2026-08-29: this issue's gap is unchanged — 034be6f8a does not close it

rbuergi commented on Aug 30, 2026

Heads-up for anyone arming PreWarm__GateReadiness off this issue

rbuergi commented on Aug 30, 2026

Re-measured on current main: the premise has moved, the conclusion stands — and it is now harder to see

What now exists that did not

What has not changed

🚨 Why this is worse than the original state, not better

What closing this actually requires

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Re-measured on `origin/main` 2026-08-29: this issue's gap is unchanged — `034be6f8a` does not close it

Heads-up for anyone arming `PreWarm__GateReadiness` off this issue

Re-measured on current `main`: the premise has moved, the conclusion stands — and it is now harder to see