From d09659502235d9431a2912eee0564167657b4cc4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20B=C3=BCrgi?= <6334612+rbuergi@users.noreply.github.com> Date: Thu, 3 Sep 2026 14:49:21 +0200 Subject: [PATCH 1/2] docs: the self-update schema wall, and why a promoted tag is not a deployable tag Two findings from the 2026-09-03 fleet work lived only in a GitHub issue thread and session notes. Committing them, per the conserve-work-products rule. 1. Every schema-bumping release is un-takeable by self-update, BY CONSTRUCTION. Self-update patches the portal image only; the migration is a run-once Job that only `helm upgrade` mints; DbVersionGate then correctly refuses to serve ahead of the schema. An install rolls forward until it meets its FIRST schema-bumping release and stops there, silently -- Available=True keeps the old ReplicaSet serving while the new one never reaches Ready. Structural, not a bad build: clearing one instance at one tag clears today and re-arms for the next bump, and memex-cloud is behind the same wall at ci.7621 only because it has not arrived yet. Consequential because memex is the control instance for the release wave (#3185): it can freeze arbitrarily far behind the platform it announces, and no control-plane change in a schema-bumping release reaches it without an operator. The durable remedy is an OPEN decision -- the three candidate options are stated neutrally and the page deliberately does not pick one (#3207). 2. A promoted tag is not a deployable tag; the SEAL is the discriminator. `Promote: tag the full set` green means the three platform images carry the tag, `Verify every image shipped` green means they are pullable -- neither says the plugin modules exist for that framework identity. Measured on CD runs 33746020109 (ci.7669) and 33749847612 (ci.7674): both promote-green, verify-green, `Plugins: bake + seal` FAILURE on the one-producer guard (#3175), with `Register the publication with memex` skipped -- so the availability predicate has no record to read either. Image COUNT was never the question. New page: Doc/Architecture/SelfUpdateSchemaWall. Cross-linked from Deployment (router), DeploymentAKS ("Migration under self-update" and the safe-target trap), ContinuousDeliveryContract (a new subsection carrying finding 2's rule, plus its See also), and the Architecture topic map. Also corrects one stale sentence in ContinuousDeliveryContract: the roll has patched the portal Deployment ONLY since #2797, not "the portal and migration Deployments" -- it contradicted DeploymentAKS and sent at least one investigation looking for a migration leg that does not exist. No What's New entry: internal architecture/ops documentation, nothing a user sees. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01G4xPnGYEbdu8AVvR5jytyj --- .../Data/Architecture.md | 2 +- .../ContinuousDeliveryContract.md | 43 +++- .../Data/Architecture/Deployment.md | 1 + .../Data/Architecture/DeploymentAKS.md | 17 ++ .../Data/Architecture/SelfUpdateSchemaWall.md | 194 ++++++++++++++++++ 5 files changed, 254 insertions(+), 3 deletions(-) create mode 100644 src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md diff --git a/src/MeshWeaver.Documentation/Data/Architecture.md b/src/MeshWeaver.Documentation/Data/Architecture.md index 6617a02c45..15bb7862d3 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture.md +++ b/src/MeshWeaver.Documentation/Data/Architecture.md @@ -77,7 +77,7 @@ The catalog below lists every page; these are the load-bearing reads per theme. | **Plugins & content delivery** | [Plugins](Plugins) — node repos from git, no NuGet | [Plugin Manual](PluginAuthoring) — author · publish · install · own registry · [Plugin Registry](PluginRegistry) — memex re-serves plugins over REST · [Webhook Inbox](WebhookInbox) — external services deliver into {target}/_Inbox · [Plugin Packaging](PluginPackaging) — bundles, the framework identity, and the `Release` node that links a release to its assemblies per architecture · [Install Readability](InstallReadability) — the two doors an install can open, and the cover-grant deadlock detector · [Static Repo Import](StaticRepoImport) · [Node Type Compilation](NodeTypeCompilation) | | **Reliability & wedges** | [Error Propagation & Wedges](ErrorPropagationAndWedges) — drive wedges to 0 | [Action-Block Wedge Prevention](ActionBlockWedgePrevention) · [Riding Out a ShuttingDown Address](RidingOutAShuttingDownAddress) — the one transient NACK, and the two axes a ride-out must bound separately · [Hub Initialization Failure](HubInitializationFailure) · [Orleans Stream Pub-Sub Durability](OrleansStreamPubSubDurability) — a publish with no subscriber succeeds, so a cross-silo reply can vanish with nothing logged · [Durable Streams Are Mesh Nodes](DurableStreamsViaMeshNodes) — the design that retires the memory stream without a provider · [The Pod-Hub Claim Must Be Re-Asserted](PodHubClaimReassertion) — a claim asserted once into a directory that is re-partitioned on every membership change is lost silently, and forever · [Oversized Delivery Refusal](OversizedDeliveryRefusal) — a message too large for its transport destroys the connection carrying it; refuse at the producer, never raise the limit · [An Unreachable Store Is Not a Refusal](StoreUnreachableIsNotARefusal) — one classification, three consumers; reporting an availability failure as a verdict is how a retried create becomes a duplicate · [Reading a Silo Eviction](ReadingASiloEviction) — a heartbeat newer than the suspect votes is not proof the silo was healthy; the control arm that tells a correct eviction from a false positive | | **Testing & debugging** | [Writing Tests](WritingTests) | [Negative Controls](NegativeControls) — a pin is only a pin if it fails against the defect · [Reactive Test Assertions](ReactiveTestAssertions) · [Test State Isolation](TestStateIsolation) · [Disposable-mesh e2e](DisposableMeshE2E) · [Debugging Message Flow](DebuggingMessageFlow) · [Debugging Disposal & Leaks](DebuggingDisposalAndLeaks) · [Ambient Test-Host Hangs](AmbientTestHostHangs) — what decides whether a killed test host can be diagnosed at all, and the readings of it already falsified · [In-Mesh Tests and the Seal](InMeshTestsAndTheSeal) — a Tests area no required context executes is a latent trunk red the seal detonates fleet-wide; how to measure a gate before requiring it | -| **Deployment & ops** | [Deployment](Deployment) (the router) | [AKS](DeploymentAKS) · [Container Apps](DeploymentContainerApps) · [Local Dev Workflow](LocalDevWorkflow) · [Onboarding a New Environment](OnboardingNewEnvironment) · [Release & Self-Update Strategy](ReleaseStrategy) · [The Continuous Delivery Contract](ContinuousDeliveryContract) — all-or-nothing publication; verify the image, never the tick · [Bake Identity Mismatch](BakeIdentityMismatch) — why a green CD can publish a bake no portal adopts, and the one rule that keeps two images of one commit on one address · [Release Availability Gates](ReleaseGates) — one predicate; never roll or build into a release a package cannot survive · [Combo Gate Wiring](ComboGateWiring) — the roll consults the combo verdict; Red refuses, and "could not find out" is neither · [Release Process](ReleaseProcess) · [Repository Dependency Direction](RepositoryDependencyDirection) — the platform never depends on a plugin repo; the inventory of every edge that still does · [The Cross-Repo Pair Gate](CrossRepoPairGate) — a removal here reds a plugin repo's trunk hours later; the deleting half lands LAST · [The Merge Queue](MergeQueue) — one entry built at a time so nothing churns, and a steward that re-queues an ejected PR on evidence and never re-runs · [Carving Projects Out Of Core](CarvingProjectsOutOfCore) — what a SOURCE move costs and what it does not · [Red-Log Watching & Ticketing](LogWatchTriage) — every `fail:`/`crit:` becomes exactly one triaged issue · [Measuring a Live Portal Read-Only](MeasuringALivePortalReadOnly) — the four read-only instruments, and why an absence needs a coverage fact before it counts as evidence · [Chart Ownership and the Runner Pool](ChartOwnershipAndRunnerPool) — why the chart's gate is here, what a relocation must carry, and why path-filtering it is unsafe | +| **Deployment & ops** | [Deployment](Deployment) (the router) | [AKS](DeploymentAKS) · [Container Apps](DeploymentContainerApps) · [Local Dev Workflow](LocalDevWorkflow) · [Onboarding a New Environment](OnboardingNewEnvironment) · [Release & Self-Update Strategy](ReleaseStrategy) · [The Continuous Delivery Contract](ContinuousDeliveryContract) — all-or-nothing publication; verify the image, never the tick · [The Self-Update Schema Wall](SelfUpdateSchemaWall) — every schema-bumping release is un-takeable by self-update, the stall is invisible, and a promoted tag is not a deployable tag · [Bake Identity Mismatch](BakeIdentityMismatch) — why a green CD can publish a bake no portal adopts, and the one rule that keeps two images of one commit on one address · [Release Availability Gates](ReleaseGates) — one predicate; never roll or build into a release a package cannot survive · [Combo Gate Wiring](ComboGateWiring) — the roll consults the combo verdict; Red refuses, and "could not find out" is neither · [Release Process](ReleaseProcess) · [Repository Dependency Direction](RepositoryDependencyDirection) — the platform never depends on a plugin repo; the inventory of every edge that still does · [The Cross-Repo Pair Gate](CrossRepoPairGate) — a removal here reds a plugin repo's trunk hours later; the deleting half lands LAST · [The Merge Queue](MergeQueue) — one entry built at a time so nothing churns, and a steward that re-queues an ejected PR on evidence and never re-runs · [Carving Projects Out Of Core](CarvingProjectsOutOfCore) — what a SOURCE move costs and what it does not · [Red-Log Watching & Ticketing](LogWatchTriage) — every `fail:`/`crit:` becomes exactly one triaged issue · [Measuring a Live Portal Read-Only](MeasuringALivePortalReadOnly) — the four read-only instruments, and why an absence needs a coverage fact before it counts as evidence · [Chart Ownership and the Runner Pool](ChartOwnershipAndRunnerPool) — why the chart's gate is here, what a relocation must carry, and why path-filtering it is unsafe | | **Contributing docs** | [Authoring Documentation](AuthoringDocumentation) | [Docs Follow The Functionality](DocsFollowTheFunctionality) — which repo a page belongs in, and what pins the rest here · [Specifying Software](SpecifyingSoftware) · [Glossary](/Doc/Glossary) | | **Licensing** | [Dependency Licensing](DependencyLicensing) — Apache-2.0/MIT compatible only; the CI gate that enforces it | | diff --git a/src/MeshWeaver.Documentation/Data/Architecture/ContinuousDeliveryContract.md b/src/MeshWeaver.Documentation/Data/Architecture/ContinuousDeliveryContract.md index cbc327103d..694fb4e45a 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture/ContinuousDeliveryContract.md +++ b/src/MeshWeaver.Documentation/Data/Architecture/ContinuousDeliveryContract.md @@ -112,11 +112,17 @@ moving pointer like `latest`, untagging *destroys* it rather than reverting it. | **C** | **`memex-portal-ai:`** — one manifest PUT, the last thing the pipeline does | a single PUT either happened or did not; there is no half-armed state | Phase C is the **arming write** and nothing else is, because `SelfUpdateHostedService` lists tags for -`memex-portal-ai` **only**, picks the newest `^\d+\.\d+\.\d+` one, and patches the portal *and* -migration Deployments to it. Everything the roll will need — the matching +`memex-portal-ai` **only**, picks the newest `^\d+\.\d+\.\d+` one, and patches the portal Deployment +to it. Everything the roll will need — the matching `memex-migration:`, the bake image that certified these node types — is already tagged by phase A. **Do not move that step, and do not add anything after it.** +🚨 **The roll patches the portal and NOTHING else** — since #2797 there is no second PATCH, and a +guard in MeshWeaver.Plugins fails the build if one comes back. The matching `memex-migration` image +that phase A tagged is therefore *available* to the roll but never *run* by it: only `helm upgrade` +mints the migration Job. That is why a release which bumps the schema is un-takeable by self-update +at all — [The Self-Update Schema Wall](/Doc/Architecture/SelfUpdateSchemaWall). + One deliberate coupling: the GHCR mirror sits in **phase B**, so a GHCR outage *blocks* the release rather than silently leaving the plugins repo's `latest` stale. `check-image-set.sh` only observes ACR, so a GHCR miss placed after the arming write would never be reconciled and nothing would notice. @@ -310,6 +316,38 @@ what sets that tail. the rolls, so each published set restarts every install that selects it. That is the reason the tick is hourly and not faster, and it is tracked as #1778 rather than papered over with a slower CD. +### A promoted tag is not a deployable tag — the seal is the discriminator + +The rule above stops at the image. **A tag whose images all exist, and are all pullable, can still be +unusable**, and the two checks that sound like they say otherwise are green in exactly that state: + +- **`Promote: tag the full set (all-or-nothing)`** green means the three platform images + (`memex-portal-ai`, `memex-migration`, `mw-plugin-test`) carry the tag. +- **`Verify every image shipped`** green means they are pullable. +- **Neither says anything about the plugin modules.** The binding job is + **`Plugins: bake + seal the publication for this identity`**, which seals the module set *for that + platform identity*. It runs after the promote — see "After the promote" below — and, as recorded + there, the reconciler does not heal it. + +Measured on two consecutive runs, 2026-09-03 — `33746020109` (`ci.7669`) and `33749847612` +(`ci.7674`), both seals FATALing on the one-producer guard (#3175): + +```text +Promote: tag the full set (all-or-nothing) success +Verify every image shipped success +Plugins: bake + seal the publication for this identity FAILURE + └─ Register the publication with memex skipped +``` + +Both tags are **platform-present, plugin-modules-absent** for their framework identity. Rolling to +one produces the half-broken state the standing availability rule forbids — *all plugins available +for the correct platform version, else nothing goes* +([Release Availability Gates](/Doc/Architecture/ReleaseGates)). And note the skipped step: a failed +seal never registers the publication with memex, so the availability predicate has no record to +read. **The number of images promoted was never the question.** The operator-facing checklist for +choosing a target is [The Self-Update Schema Wall](/Doc/Architecture/SelfUpdateSchemaWall) → +"What makes a tag a safe target". + ## Property 1a — how the set already spans repos, and what the GUI move actually breaks 🚨 **Correction, 2026-08-26.** An earlier draft of this section claimed the shipped artefacts have @@ -604,3 +642,4 @@ green lane red. - [Release Process & Versioning](/Doc/Architecture/ReleaseProcess) — where the version number comes from. - [Deployment](/Doc/Architecture/Deployment) — the route router (AKS vs Container Apps). - [Deploying Plugin Changes](/Doc/Architecture/DeployingPluginChanges) — what the `mw-plugin-test` leg is for. +- [The Self-Update Schema Wall](/Doc/Architecture/SelfUpdateSchemaWall) — which releases an install can take by itself, and the three conditions a tag must clear before it is a safe helm target. diff --git a/src/MeshWeaver.Documentation/Data/Architecture/Deployment.md b/src/MeshWeaver.Documentation/Data/Architecture/Deployment.md index a8671340c8..c75271c826 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture/Deployment.md +++ b/src/MeshWeaver.Documentation/Data/Architecture/Deployment.md @@ -22,6 +22,7 @@ MeshWeaver has **two distinct deploy routes**. They target different infrastruct | **Database backups** & disaster recovery — managed PITR, geo-redundancy, restore | [DatabaseBackups.md](/Doc/Architecture/DatabaseBackups) | | Understand the release model, merge gates, version channels, and **policy-driven self-update** | [ReleaseStrategy.md](/Doc/Architecture/ReleaseStrategy) | | Know what CD **guarantees** about a published image set — all-or-nothing publication, the promote ordering, the self-healing reconciler — and why you verify the IMAGE and never the green tick | [ContinuousDeliveryContract.md](/Doc/Architecture/ContinuousDeliveryContract) | +| Work out whether an install can actually **take** the newest release — the schema boundary self-update cannot cross, why the resulting stall is invisible, and the three conditions a tag must clear before it is a safe `helm upgrade` target | [SelfUpdateSchemaWall.md](/Doc/Architecture/SelfUpdateSchemaWall) | | Ship a code update to the `memex` portal on the shared AKS cluster | [DeploymentAKS.md](/Doc/Architecture/DeploymentAKS) | | Deploy an Aspire-orchestrated `test`/`prod` Container Apps environment | [DeploymentContainerApps.md](/Doc/Architecture/DeploymentContainerApps) | | Understand the private-AKS-cluster architecture & operations behind the shared portal | [MemexCloudDeployment.md](/Doc/Architecture/MemexCloudDeployment) | diff --git a/src/MeshWeaver.Documentation/Data/Architecture/DeploymentAKS.md b/src/MeshWeaver.Documentation/Data/Architecture/DeploymentAKS.md index b05b38c425..3adeb793b3 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture/DeploymentAKS.md +++ b/src/MeshWeaver.Documentation/Data/Architecture/DeploymentAKS.md @@ -312,6 +312,14 @@ construction, un-takeable by self-update. CD does build and push a correctly-tag path ever runs it. `main-cd.yml` says as much: *"a migration that can never run surfaces only at the deploy that needs it."* Only `helm upgrade` mints the Job. +> **This is a property of the fleet, not of one namespace, and the remedy is an OPEN decision.** An +> install rolls itself forward until it meets its first schema-bumping release and stops there; an +> install that has not stopped has not arrived yet, not been configured differently (`memex-cloud` +> served `ci.7621` healthily the same day for exactly that reason). Clearing one instance at one tag +> clears today and re-arms for the next bump. Why that matters for the control instance, what the +> three candidate remedies trade, and how to pick a target when an operator does carry an install +> across, are on [The Self-Update Schema Wall](/Doc/Architecture/SelfUpdateSchemaWall). + **What that looks like when it fires** (memex, 2026-09-03 — MeshWeaver#3207): self-update rolled the portal to three successive builds needing `db_version` 55 against a database at 54. Each new pod hit `DbVersionGate`, logged `Critical`, and exited; the ReplicaSet never went Ready and the rollout @@ -334,6 +342,15 @@ must hold a roll, the decision belongs BEFORE the portal is patched — that is that guard prescribes, and re-attempting a migration PATCH after the portal has already rolled is the defect #2797 removed. +🚨 **And where a `helm upgrade` genuinely is the sanctioned path, choosing its target is a separate +problem with its own trap.** A tag whose three platform images are all promoted and all pullable can +still have no plugin modules for its framework identity — `Promote: tag the full set` and `Verify +every image shipped` both go green in that state, and the job that discriminates is `Plugins: bake + +seal the publication for this identity`. Check the seal, the ancestry of the fix you need, and a +`memex-migration` image at the SAME tag, before naming any build a target: +[The Self-Update Schema Wall](/Doc/Architecture/SelfUpdateSchemaWall) → "What makes a tag a safe +target". + **Startup ordering — what actually happens if the portal outruns the schema.** Two mechanisms exist, and only the first is a wait: diff --git a/src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md b/src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md new file mode 100644 index 0000000000..9c76eeb55a --- /dev/null +++ b/src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md @@ -0,0 +1,194 @@ +--- +Name: The Self-Update Schema Wall +Category: Architecture +Description: Why every schema-bumping release is un-takeable by self-update, why the resulting wedge is invisible from outside, and the three conditions a tag must clear before it is a safe helm target. +Icon: +--- + +# The Self-Update Schema Wall + +**A pull-based self-update carries the IMAGE and nothing else. So an instance rolls itself forward +release after release until it meets the first one that bumps the database schema — and then it +stops there, still serving, with nothing on the outside to say it stopped.** + +This page is about the *property*, not the incident: which releases an instance can take by itself, +which it cannot, and how to pick a target when an operator has to carry one across. The +runbook-level mechanics — the exit code that lies, the rollout strategy that saves the service, the +guard that keeps the updater honest — are on +[Deployment — AKS](/Doc/Architecture/DeploymentAKS) → "Migration under self-update", and are not +repeated here. + +## The mechanism, in four lines + +| | | +|---|---| +| The updater patches | `memex-portal-deployment` (container `memex-portal`) — **one** workload, and it says so in its own success line | +| The migration is | a run-once `Job` the chart renders per Helm revision. Only `helm upgrade` mints one; nothing in the continuous path ever runs it | +| The portal checks | `DbVersionGate` reads `admin.mesh_nodes.db_version` **once** at startup and, if it is below the `ExpectedDbVersion` compiled into the build, logs `Critical` and stops the application | +| So a schema-bumping release is | image-forward, schema-behind — the new pod fails closed and exits, and the rollout never completes | + +CD *does* build and push a correctly tagged `memex-migration` image on every run, tag for tag with +`memex-portal-ai`. That is not the gap. The gap is that no continuous path ever runs it: there is no +migration leg for self-update to get wrong, because there is no migration leg at all. + +## Why it is invisible + +The same rollout policy that protects the service is what hides the failure. + +With `maxUnavailable: 0` and `maxSurge: 1`, the previous ReplicaSet keeps serving while the new one +never reaches Ready. The Deployment then reports: + +```text +Available=True MinimumReplicasAvailable +Progressing=False ProgressDeadlineExceeded +``` + +Every outside-in probe — the portal answers, users work, `/alive` is green — reports health, because +the instance *is* healthy. It is healthy at the **old build**. + +On `memex`, 2026-09-03 ([#3207](https://github.com/Systemorph/MeshWeaver/issues/3207)), three +self-updates wedged this way between 07:20Z and 10:27Z (`ci.7647`, `ci.7651`, `ci.7658`), the last +one on a 5-minute back-off at 14 restarts, each attempt writing a ~685 MB core dump. Nothing +alerted. It surfaced only because an unrelated config-drift audit went looking at the ReplicaSets. + +> 🚨 **"The portal is serving" and "the portal took the release" are different questions, and health +> only answers the first.** The signal that discriminates is the **running image tag** — read it back +> off the Deployment and compare it with the newest promoted tag — plus a rollout sitting at +> `ProgressDeadlineExceeded`. That is the same rule as +> [verify the IMAGE, never the tick](/Doc/Architecture/ContinuousDeliveryContract), pointed at the +> cluster instead of at the pipeline. + +## Why it is structural, not a bad build + +None of `ci.7647`, `ci.7651`, `ci.7658` is defective. Any release that raises `ExpectedDbVersion` +produces exactly this, and every release that does not raise it is taken normally. The wall is a +property of the update mechanism, not of a build. + +Two consequences follow, and both are easy to miss: + +- **Clearing one instance at one tag clears TODAY, and nothing else.** An out-of-band `helm upgrade` + runs the Job, the schema advances, the already-attempted portal starts — and the instance resumes + self-updating until the *next* schema bump, where it stops again. The remedy is per-occurrence by + construction; it does not change the property. +- **An instance that has not hit the wall is not configured differently — it has not arrived yet.** + `memex-cloud` served `ci.7621` and was healthy on the same day, for the single reason that no build + it had selected needed schema 55. Its next selection past `ci.7647` meets the wall identically. + Every install stands on the same wall; only the arrival time differs. + +So the question worth asking of the fleet is not "is anything wedged?" but **"for each instance, is +its `db_version` below the expected version of the newest release it would select?"** — the instances +that answer yes are already behind the wall whether or not they have noticed. + +## The control instance stands on it too + +Since [#3185](https://github.com/Systemorph/MeshWeaver/issues/3185) the release wave belongs to +memex: every publishing pipeline ends with one call to the control instance, which **registers** the +release as a durable node and **publishes** `meshweaver-framework-released` / +`meshweaver-upstream-published` to the subscribed repositories. The whole shape is in +[Deployment](/Doc/Architecture/Deployment) → "How a release reaches the fleet" and +[The Release Event Bus](/Doc/Architecture/ReleaseEventBus). + +Be precise about what the wall does to that, because the alarming reading is the wrong one: + +- **The control plane does not stop.** The wedge freezes the *image*; the old pods keep serving, so + the inbox keeps accepting, registering and broadcasting throughout. +- **What it does do** is let the control instance fall arbitrarily far behind the platform it + announces. Every subsequent release it registers and broadcasts to the fleet is one it cannot take + itself, and any change to the control plane's own machinery — an inbox watcher, the broadcaster, + a schema-backed registration table — that ships in a schema-bumping release cannot reach the + instance that runs it without an operator. + +That is the structural gap in the release-wave story: the one instance the fleet's delivery +coordination depends on has no self-service path across a schema boundary. Tracked alongside +[#3207](https://github.com/Systemorph/MeshWeaver/issues/3207); +the wave itself is `Systemorph/MeshWeaver.Plugins#1241` (merged) and `Systemorph/Memex#173` (open). + +## What makes a tag a safe target + +Whenever an operator does carry an instance across the wall, **picking the target is a separate +problem from running the upgrade** — and the checks that sound sufficient are not. + +A safe target clears all three of these, each **measured**, none inferable from the tag: + +| # | Condition | How you establish it | +|---|---|---| +| 1 | It carries the code fix you need | **By ancestry** — is the fix's merge commit an ancestor of the tag's build commit? Tag ordering is not ancestry, and a higher `ci.` is not evidence | +| 2 | `memex-migration` exists at the **same** tag | `helm-release.yml`'s own `--set` block insists on it, in its own words: *a migration from a different build than the code that will run against it is how a schema lands half-applied* | +| 3 | `Plugins: bake + seal the publication for this identity` is **GREEN** on that tag's CD run | The seal is what publishes the plugin modules for the platform identity. Its absence is invisible in the registry | + +Condition 3 is the one that gets skipped, because two other checks sound like it and are green +without it. On 2026-09-03, CD runs `33746020109` (`ci.7669`) and `33749847612` (`ci.7674`) both +recorded: + +```text +Promote: tag the full set (all-or-nothing) success +Verify every image shipped success +Plugins: bake + seal the publication for this identity FAILURE + └─ Register the publication with memex skipped +``` + +Both seals FATALed on the one-producer guard (#3175). Both tags are therefore **platform-present, +plugin-modules-absent** for their framework identity — precisely the half-broken state the standing +availability rule forbids: *all plugins deployed to an instance must be available for the correct +platform version; if not, nothing goes* +([Release Availability Gates](/Doc/Architecture/ReleaseGates)). A session preparing the remedy above +named `ci.7674` as verified having confirmed that three platform images existed, and retracted it +before anyone acted. **Image count was never the right question.** + +Note the skipped step in that transcript: when the seal fails, the publication is never registered +with memex, so the availability predicate has no record to read. The seal and the availability +check are one fact seen from two sides, which is why one call answers condition 3 for a live +instance — ask the portal, per +[Deployment — AKS](/Doc/Architecture/DeploymentAKS) → "Self-update ops": + +```bash +curl -s -H "Authorization: Bearer $MWI_KEY" \ + "https:///api/plugins/is-updatable?version=" | jq +``` + +`isUpdatable: false` names the blocking packages; `indeterminate: true` means the check could not +run and is **not** clearance. Why a green Promote can sit above an unsealed publication at all is +[Bake Identity Mismatch](/Doc/Architecture/BakeIdentityMismatch) and +[The Continuous Delivery Contract](/Doc/Architecture/ContinuousDeliveryContract) → "A promoted tag is +not a deployable tag". + +## The durable remedy is an OPEN decision + +**Nothing here has been decided, and this page deliberately does not pick.** The options on the +table, with what each one costs: + +| Option | Shape | What it trades | +|---|---|---| +| **(a) Self-update triggers the helm run at a schema boundary** | The instance detects that its target needs a schema it does not have and drives the migration itself | Closes the gap end-to-end. Gives the in-pod updater a much larger blast radius — it would have to hold a credential or a workflow trigger that can mint a Job, which is exactly the surface the current one-workload design keeps small | +| **(b) Schema-bumping releases are FLAGGED, and the instance holds** | The release advertises the schema it needs; an instance that cannot satisfy it declines to roll and asks for the helm run instead of rolling into a wedge | Turns a silent stall into an explicit, reported hold — the fleet-visible state the current failure lacks. Does not remove the operator step; needs the expected version to become a published property of a release rather than a constant compiled into an image | +| **(c) Status quo — schema bumps are always operator-run** | Keep the property, make it legible and alarmed | Cheapest and safest. Leaves the control instance without a self-service path, and leaves "did anyone notice?" as a monitoring problem rather than a mechanism one | + +Two constraints any answer has to respect, both already standing directives: + +- **Rolls go through CD.** The first remedy proposed on #3207 — running `helm-release.yml` against a + hand-picked `image` — was **withdrawn** by its own author for exactly this reason: a hand-pinned + tag is what the roll-via-CD directive exists to prevent, and a portal serving a consistently old + build is a safe state, not an incident. +- **Fail-closed stays.** `DbVersionGate` refusing to serve ahead of its schema is the behaviour that + keeps a half-migrated database from being written to. No option may weaken it; the argument is + only about who runs the migration and when. + +The decision is the maintainer's. Until it is made, the operative facts are: schema-bumping releases +are un-takeable by self-update, an instance behind the wall is serving and safe, and a target chosen +to carry one across must clear all three conditions above. + +## See also + +- [Deployment — AKS](/Doc/Architecture/DeploymentAKS) — "Migration under self-update": the runbook + detail, the `exitCode 139` that is really SIGABRT, and the guard that keeps the updater to one + workload +- [The Continuous Delivery Contract](/Doc/Architecture/ContinuousDeliveryContract) — what a promote + guarantees, what it does not, and why the reconciler does not heal a failed bake publication +- [Release Availability Gates](/Doc/Architecture/ReleaseGates) — the one predicate, and the release + marker that makes a release's framework identity knowable from outside its own image +- [Bake Identity Mismatch](/Doc/Architecture/BakeIdentityMismatch) — how a fully green CD run can + publish a bake no portal will ever adopt +- [Release & Self-Update Strategy](/Doc/Architecture/ReleaseStrategy) — update policies, channels, + and what an install selects +- [The Release Event Bus](/Doc/Architecture/ReleaseEventBus) — the event is a wake-up, the mesh is + the truth From 5d909d121613d91e0a107637197202acf81e715a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20B=C3=BCrgi?= <6334612+rbuergi@users.noreply.github.com> Date: Thu, 3 Sep 2026 14:55:59 +0200 Subject: [PATCH 2/2] docs: #3185 is a pull request, link it as one Copilot review. The reference pointed at /issues/3185; #3185 is the PR that made the release wave memex's. GitHub would redirect, but the URL should say what the thing is. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01G4xPnGYEbdu8AVvR5jytyj --- .../Data/Architecture/SelfUpdateSchemaWall.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md b/src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md index 9c76eeb55a..1a9b4336b8 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md +++ b/src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md @@ -81,7 +81,7 @@ that answer yes are already behind the wall whether or not they have noticed. ## The control instance stands on it too -Since [#3185](https://github.com/Systemorph/MeshWeaver/issues/3185) the release wave belongs to +Since [#3185](https://github.com/Systemorph/MeshWeaver/pull/3185) the release wave belongs to memex: every publishing pipeline ends with one call to the control instance, which **registers** the release as a durable node and **publishes** `meshweaver-framework-released` / `meshweaver-upstream-published` to the subscribed repositories. The whole shape is in