docs(deploy): "Migration under self-update" describes behaviour #2797 removed by rbuergi · Pull Request #3208 · Systemorph/MeshWeaver

rbuergi · 2026-09-03T12:01:29Z

Corrects Doc/Architecture/DeploymentAKS → "Migration under self-update", which describes a mechanism #2797 removed. Context: #3207.

The stale claim

The section says the in-pod updater patches two Deployments — the portal and memex-migration-deployment — and that the second 404s and THROWS, poisoning the verdict of the portal roll that already succeeded.

KubernetesDeploymentUpdater.PatchToVersionAsync today:

await PatchDeploymentImageAsync(
    _options.PortalDeployment, _options.PortalContainer, _options.PortalImage(versionTag), ct);
_logger?.LogInformation(
    "[SelfUpdate] patched {Portal} to {Tag}. The database migration is a helm-run Job "
    + "(memex-migration-<revision>) and is NOT rolled from here; DbVersionGate refuses to "
    + "serve if this image is ahead of the schema.", …);

One PATCH, portal only, and it states its own limitation. SelfUpdatePatchesOnlyPatchableWorkloadsGuard fails the build if anyone re-adds a second.

This is not a tidy-up. The stale text cost an investigation today — it sends the reader hunting a 404 on a migration leg, when the actual condition is that there is no migration leg at all. Recorded as an explicit correction rather than silently rewritten, because "prose asserts a mechanism that no longer exists" is the failure mode, not the wording.

The real gap, restated

A self-update moves the image and nothing else, so a schema-bumping release is un-takeable by self-update by construction. CD does build and push a correctly-tagged memex-migration image on every run — tag-for-tag with memex-portal-ai (3.0.0-rc9.ci.7658, .7669, .7671 all verified present in ACR). Nothing in the continuous path ever runs it. main-cd.yml already says so:

"a migration that can never run surfaces only at the deploy that needs it"

What it looks like when it fires

From memex today (#3207): three successive self-updates to builds needing db_version 55 against a database at 54; each pod failed DbVersionGate, logged Critical and exited; ProgressDeadlineExceeded; a pod crash-looping on a 5-minute back-off writing a ~685 MB core dump per attempt.

Two things the old text got wrong about the symptom, both of which cost time and are now stated where the failure is described:

It predicts "the namespace has no serving portal". It had one throughout. maxUnavailable: 0 with maxSurge: 1 kept the previous pods serving while the new ReplicaSet failed to go Ready — a fail-closed boot became a stalled rollout. That is the setting doing its job, and it is why this class is recoverable rather than an incident.
The container reports exitCode 139, which reads as SIGSEGV. The dump records NT_SIGINFO … signo 6 — SIGABRT. It is a managed fail-closed shutdown; the OrleansProvisioningGate cancellation and Hosting failed to start after it are the shutdown's own noise, not the fault.

Disposition

States plainly that a portal held behind its schema is a SAFE state, that rolls go through CD (a hand-pinned tag is what the roll-via-CD directive exists to prevent), and that where the gate must hold a roll the decision belongs before the portal is patched — the shape the guard prescribes and the one #2797's defect violated.

Docs only; no behaviour change. DocumentationLinkIntegrityTest passes.

🤖 Generated with Claude Code

…removed The section says the in-pod updater patches TWO Deployments — the portal and `memex-migration-deployment` — and that the second 404s and THROWS, poisoning the verdict of the portal roll that already succeeded. #2797 removed that second PATCH. The updater now patches the portal and nothing else, says so in its own success line, and SelfUpdatePatchesOnlyPatchableWorkloadsGuard fails the build if anyone re-adds one. This is not a tidy-up. The stale text cost an investigation today: it sends the reader looking for a 404 on a migration leg, when the actual condition is that there is no migration leg at all. Recorded as a correction rather than silently rewritten, because "prose asserts a mechanism that no longer exists" is the failure mode, not the wording. Restates the real gap: a self-update moves the image and nothing else, so a schema-bumping release is un-takeable by self-update by construction. CD builds and pushes a correctly-tagged memex-migration image on every run — tag-for-tag with memex-portal-ai, verified in ACR — and nothing in the continuous path ever runs it. Only `helm upgrade` mints the Job. Adds what it looks like when it fires, from memex today (#3207): three successive rolls to builds needing db_version 55 against a database at 54, each pod failing DbVersionGate and exiting, ProgressDeadlineExceeded, a pod crash-looping on a 5-minute back-off writing a ~685 MB core dump per attempt. Two things the old text got wrong about the symptom, both of which cost time: * it predicts "the namespace has no serving portal". It had one throughout. maxUnavailable:0 with maxSurge:1 kept the previous pods serving while the new ReplicaSet failed to go Ready, turning a fail-closed boot into a stalled rollout. That is the setting doing its job and is why this class is recoverable rather than an incident — so it is now stated where the failure is described. * the container reports exitCode 139, which reads as SIGSEGV. The dump records signo 6 — SIGABRT. It is a managed fail-closed shutdown; the OrleansProvisioningGate cancellation and "Hosting failed to start" after it are the shutdown's own noise, not the fault. And states the disposition: a portal held behind its schema is a SAFE state, rolls go through CD, and where the gate must hold a roll the decision belongs BEFORE the portal is patched — the shape the guard prescribes and the one #2797's defect violated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

🟡 Changes recommended

The updated section introduces an internal inconsistency with later text in the same document and references a guard identifier that is not resolvable within this repository.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Updates the AKS deployment runbook (DeploymentAKS.md) to correct the “Migration under self-update” section, removing/recording a previously described self-update mechanism that no longer exists and documenting the real failure mode when the portal image advances past the DB schema.

Changes:

Rewrites “Migration under self-update” to state that self-update patches only the portal Deployment and does not run migrations.
Adds a concrete symptom description (DbVersionGate fail-closed startup, rollout stall characteristics, exit-code vs signal nuance).
Clarifies operational implications: schema-bumping releases must go through helm upgrade (CD path), and why a stalled rollout can be recoverable with surge-first settings.

File summaries

File	Description
src/MeshWeaver.Documentation/Data/Architecture/DeploymentAKS.md	Corrects self-update vs migration behavior and documents the real “portal ahead of schema” failure mode and recovery characteristics.

Review details

Files reviewed: 1/1 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

… rollout strategy prevents Copilot review on #3208, two real defects in my own edit. 1. I cited `SelfUpdatePatchesOnlyPatchableWorkloadsGuard` with no location. It is not in this repo — it lives in MeshWeaver.Plugins (`src/Memex.Hosts.Test/`), as does the updater it guards (`src/MeshWeaver.SelfUpdate.Aks/KubernetesDeploymentUpdater.cs`), so `git grep` here finds only the page naming it. Verified both paths exist there and neither appears in any core `.cs`. A bare symbol a reader cannot find reads as a phantom guard — the failure mode this very section is about. 2. I added the rollout-strategy explanation and left the older paragraph asserting flatly that "the namespace has no serving portal", so the page contradicted itself two screens apart. Whether the namespace still serves is decided by the rollout strategy, not by the gate: with maxUnavailable:0 and maxSurge:1 the previous ReplicaSet keeps serving and the failure is a stalled rollout, which is what memex measured on 2026-09-03. Without it — or on a fresh install or full restart, where there is no healthy ReplicaSet to fall back on — it genuinely is an outage. Both cases are now stated, so the reader can tell which one they are in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-03T12:15:07Z