memex cannot self-update: the portal image is ahead of its schema (db_version 54 < 55) and only helm upgrade can run the migration Job

Found while auditing config drift for #2355. Not drift — a live wedge, and it blocks that cleanup.

memex cannot self-update: the image walked past its schema

crit: Memex.Portal.Distributed.DbVersionGate[0]
      DB migration incomplete: admin.mesh_nodes.db_version=54 < expected 55.
      Refusing to start the portal until the DB is fully migrated.

The gate is doing exactly its job — a portal ahead of its schema refuses to serve rather than serving a half-migrated one. What follows it in the log (an OrleansProvisioningGate cancellation, Hosting failed to start, then a 685 MB core dump) is downstream noise of the shutdown, not the fault.

🚨 exitCode 139 on the container, but the dump records NT_SIGINFO … signo 6 — SIGABRT, not SEGV. Anyone triaging this from the exit code alone will go looking for a native crash that is not there.

Three self-updates today, all wedged the same way

time (UTC)	ReplicaSet	image	ready
09-02 23:14	`5c8c476d54`	`ci.7632`	2/2 — serving
09-03 07:20	`6c74846cf8`	`ci.7647`	0
09-03 08:20	`5d55558945`	`ci.7651`	0
09-03 10:27	`54f4c96b6b`	`ci.7658`	0 — CrashLoopBackOff, 14 restarts

Available=True    MinimumReplicasAvailable
Progressing=False ProgressDeadlineExceeded

Users are unaffected — maxUnavailable: 0 keeps the ci.7632 pods serving, which is the rollout policy working. But memex has been unable to take a new build since ~07:20Z, and each attempt leaves a pod crash-looping on a 5-minute back-off and writing a ~685 MB core dump to /data/dumps.

Why the schema never advances — the actual deadlock

There is no migration Job in either namespace. kubectl get jobs -n memex returns only assembly-cache-prune.

That is structural, not an accident:

Self-update rolls the IMAGE only. It patches the Deployment from inside the pod. It cannot create the migration Job, so it can never carry the schema forward with the code.
Only helm upgrade creates the migration Job — the run-once Job the chart defines (per AGENTS.md, and memex-migration CrashLoopBackOff in all three prod namespaces (50/53/38 restarts) — run-once workload modelled as a Deployment #1788 is what happens when it is modelled as a Deployment instead).

So every schema-bumping release is, by construction, un-takeable by self-update. The instance rolls forward until it meets a build that needs a new schema, and then wedges — silently, because Available=True and the portal keeps serving the old build.

The fix is the sanctioned path, and its preconditions check out

helm-release.yml (mode deploy) is built for exactly this, and its --set block is explicit that the migration Job must ride the same tag as the portal — "a migration from a different build than the code that will run against it is how a schema lands half-applied."

Two preconditions verified before proposing it:

The stale overlay pins are NOT a hazard. Both overlays still pin portal.image/migration.image at 3.0.0-rc8.ci.5068, but helm-release.yml applies --set after both -f files, so the live (or requested) image wins. I checked this specifically because a rollback from ci.7632 to rc8.ci.5068 would have been catastrophic — it does not happen. (The pins are still worth refreshing so the committed file is not misleading, but they are inert.)
The matching migration image exists. memex-migration Job cannot pull: helm-release derives its tag from the PORTAL image, and the two ACR repos do not share a tagging convention #2555 was a pull failure because the two ACR repos do not share a tagging convention. Not the case here — meshweaver.azurecr.io/memex-migration carries 3.0.0-rc9.ci.7658, .7669 and .7671, matching memex-portal-ai tag for tag.

Suggested remedy

Run helm-release.yml against memex with an explicit image at the target build. It sets portal.image to that build and derives migration.image from the same tag, so helm upgrade runs the migration Job at the matching version, the schema moves 54 → 55, and the already-attempted portal starts.

Worth confirming afterwards, per the deployment runbook: the Job logged Database migration completed. Version: 55, and the portal serves HTTP 200.

memex-cloud is behind the same wall

It serves ci.7621 today and is healthy, so it has not met a build needing schema 55 yet. Its next self-update past ci.7647 hits this identically. Whatever is done for memex should be done for it in the same pass, rather than waiting for it to wedge.

And it blocks #2355 / Memex#148

Any inline-env: deletion changes the pod template and spawns a new ReplicaSet, which lands on the same gate. The drift cleanup cannot proceed until the schema is current.

_{Measured 2026-09-03 on the live cluster: pod/RS state, deployment conditions, the crashed container's previous logs, kubectl get jobs, and ACR tag existence.}

Correcting my own remedy: this is a KNOWN, DOCUMENTED gap, and the fix is not a manual deploy

I suggested running helm-release.yml with an explicit image. Withdrawing that. It is a manual tag pin, and it contradicts the standing directive — "can we please see that we roll always via cd? this stabilizes" (maintainer, 2026-08-25). A hand-pinned roll is the thing that directive exists to stop, and "a consistently-OLD portal is a SAFE state": memex is serving ci.7632 with Available=True, so there is nothing here that justifies bypassing CD.

The root cause is already written down — and it has now fired

Doc/Architecture/DeploymentAKS → "Migration under self-update" describes this exactly:

🚨 That second patch does not currently run a migration, and this is a known gap, not a working mechanism. … the updater and its RBAC target memex-migration-deployment, which the chart does not render at all … it 404s, and KubernetesDeploymentUpdater.PatchDeploymentImageAsync THROWS on any non-success status. The portal is patched FIRST, so the roll itself still happens and the fault surfaces afterwards, on the migration leg only. Either way a self-update rolls the portal image while the schema stays where it was.

So the news in this issue is not the mechanism — it is that the gap has stopped being theoretical. It fired three times on memex today (ci.7647, ci.7651, ci.7658), and main-cd.yml already anticipated the shape in its own comments:

"a migration that can never run surfaces only at the deploy that needs it" — main-cd.yml:1164

CD builds and publishes a correctly-tagged memex-migration image on every run (3.0.0-rc9.ci.7658, .7669, .7671 all exist, matching memex-portal-ai tag for tag). Nothing in the continuous path ever applies it. Continuous delivery ships a migration it never runs.

One thing the doc predicts that did NOT happen, and it matters

The doc says a portal outrunning its schema means "the namespace has no serving portal". That is not what happened, and the reason is worth recording: maxUnavailable: 0 + maxSurge: 1 kept the two ci.7632 pods serving while the new ReplicaSet failed to go Ready. The rollout policy converted a fail-closed boot into a stalled rollout instead of an outage — exactly what that setting is for, and the same property the roll-via-CD note flags as "what keeps a fatal boot from becoming an outage — verify it before ANY roll."

So the live severity is lower than the doc's worst case: no outage, a stalled rollout plus a pod crash-looping on a 5-minute back-off writing a ~685 MB core dump per attempt.

The fix, stated as a change to the pipeline rather than an operator action

The updater's migration leg targets the wrong object and runs in the wrong order:

Wrong object — it patches memex-migration-deployment (the legacy shape). The chart renders a run-once batch/v1 Job, memex-migration-<revision>. The updater should CREATE that Job at the target tag and await completion, not patch a Deployment that does not exist.
Wrong order — the portal is patched first. Schema-then-code is the only safe sequence; today the code is guaranteed to land before the schema, which is precisely how the image outruns the gate.
RBAC — the portal ServiceAccount's grant is resourceNames-scoped to memex-migration-deployment. Creating and awaiting a Job needs batch/jobs rights in its own namespace.

Done that way the schema and the image move together in one CD-driven pass, which is the same atomicity argument that produced the roll-via-CD directive in the first place.

Until that lands, the honest status is: memex stays on ci.7632 and that is the safe state. It is not an outage and should not be "improved" by a hand roll.

_{Root cause corrected 2026-09-03 after re-reading the deployment doc and main-cd.yml. The measurements in the issue body stand; the suggested remedy in it does not.}

The target build is now available and verified: `3.0.0-rc9.ci.7674`

The remedy in this issue needs a build that satisfies three things at once. As of now one does, and I have checked each rather than inferred it.

1. It carries the #3202 fix, so it does not ship the sign-in 503.

This is the constraint the issue could not have known about — it was filed at 11:28, and the freeze that followed matters here. Every memex-portal-ai image built after ~08:03Z returns 503 for every signed-in user (OnboardingMiddleware.LoadUserRoles issues an unanchored nodeType:AccessAssignment … scope:subtree, which Plugins#1231's UnanchoredQueryException correctly refuses). Deploying most of today's builds would trade a wedge for an outage.

ci.7674 is built from core 8ffbe4762, which is the merge commit of #3206 (fix/3202-sign-in-reads-are-anchored). Verified by ancestry, not by tag order.

2. It is a COMPLETE promoted set, so the migration Job and the portal ride the same build — the condition helm-release.yml's own --set block insists on ("a migration from a different build than the code that will run against it is how a schema lands half-applied"):

repository	`3.0.0-rc9.ci.7674`
`memex-portal-ai`	✅ `sha256:4f2bc53e35b6…`
`memex-migration`	✅ `sha256:0722416f1c63…`
`mw-plugin-test`	✅ `sha256:ad18945d16bc…`

3. The matching migration image exists — memex-migration carries 3.0.0-rc9.ci.7674, so this is not the #2555 shape (a pull failure from the two ACR repos not sharing a tagging convention).

🚨 Sequencing — this cannot be run unilaterally today

A fleet freeze is in effect and is not mine to lift. Both production portals are deliberately pinned with Admin/UpdatePolicy = None (memex.meshweaver.cloud at ci.7621, memex.systemorph.com at ci.7651) pending the #3202 fix being merged AND in a promoted image. That condition is now satisfied by ci.7674 — but the decision to lift belongs to the session holding the freeze, and the helm upgrade itself is a production action that needs an explicit human go-ahead.

So the order is: freeze lifts → then this runs at ci.7674, not the other way round.

One correction worth carrying

The issue's triage note is right and worth repeating because it will mislead anyone reading the exit code alone: exitCode 139 here is SIGABRT (signo 6), not SEGV. There is no native crash to hunt. The 685 MB dump and the OrleansProvisioningGate cancellation are shutdown noise downstream of DbVersionGate refusing to start — which is the gate working correctly.

And the structural half is the real finding

Self-update rolls the IMAGE only … it can never carry the schema forward with the code.

That means every schema-bumping release is un-takeable by self-update, by construction — the instance rolls forward until it meets one, then wedges silently behind Available=True. Fixing memex at ci.7674 clears today's wedge; it does not stop the next schema bump doing this again, on either instance. memex-cloud is behind the same wall and has simply not met a schema-bumping build yet, so it should be done in the same pass rather than left to wedge on its own.

🚨 Correction to my previous comment — `ci.7674` is NOT a safe target. Do not roll to it.

My previous comment named 3.0.0-rc9.ci.7674 as the verified target. That was wrong, and I am retracting it before anyone acts on it. Credit to the session running the build-process work for catching it.

What I checked, and what I failed to check

I verified that ci.7674 (a) carries the #3202 sign-in fix — true, it is built from core 8ffbe4762, the merge commit of #3206 — and (b) exists as three platform images: memex-portal-ai, memex-migration, mw-plugin-test. I called that a "complete promoted set".

That definition is necessary but not sufficient. The binding condition is a sealed Plugins publication for the platform identity, and ci.7674 has none. Measured on its CD run 33749847612:

Promote: tag the full set (all-or-nothing)                    success
Verify every image shipped                                    success
Plugins: bake + seal the publication for this identity        FAILURE
  └─ Register …                                               skipped

The seal FATALed on the #3175 one-producer guard:

compile: FATAL — module(s) MeshWeaver.Markdown.Collaboration are composed with --module
AND shipped by the platform host at '/app' … 0/0 NodeType(s) compiled, 0 bundle(s)

So ci.7674 is platform image present, plugin modules absent for that identity (framework sddbf8fa0…). Rolling to it produces exactly the half-broken state the maintainer's rule forbids: all plugins available for the correct platform version, else nothing goes.

The corrected target

The first tag safe for both the freeze lift and this issue's helm upgrade is the first CD run after fix/3175-markdown-collaboration-one-producer merges — the one-producer fix. That run is the first that can produce a complete sealed set. (It is also the #2543 control arm, so it is being watched closely for other reasons.)

That PR is not open yet. Until then there is no safe target, and this issue's remedy waits — not because of the schema, but because rolling to any current tag would swap a silent wedge for missing plugin modules.

The lesson worth keeping

"Complete promoted set" is a phrase that reads as sufficient and is not. Three images can be promoted all-or-nothing while the publication that makes them usable never sealed — Promote and Verify every image shipped both go green in that state. A CD run's conclusion, and even a green Promote, says nothing about whether the modules for that identity exist. The check that discriminates is the Plugins: bake + seal job, or the sealed publication for the framework identity — not the presence of tags.

Nothing has been rolled, no hold lifted, no cluster touched. The structural half of this issue is unaffected and still stands: schema-bumping releases remain un-takeable by self-update, and memex-cloud is behind the same wall at ci.7621.

The structural half of this issue is now committed documentation rather than living only in this
thread: Doc/Architecture/SelfUpdateSchemaWall
(src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md), merged in #3215 at 13:30Z.

It carries the finding as stated here — self-update patches the image, the migration is a run-once Job
only helm upgrade creates, so an instance self-rolls forward until it meets a schema-bumping release
and then wedges silently behind Available=True — plus why it is structural rather than a bad build,
the interaction with making memex the release-control instance, and the three candidate remedies
left explicitly open (self-update drives the migration at a schema boundary / releases are flagged
and the instance holds / status quo made legible and alarmed). It deliberately picks none; that is
still this issue's decision.

It also carries the target-selection half, which is what nearly went wrong here: a safe helm upgrade
target must clear three conditions, each measured — carries the fix by ancestry (not tag
ordering), has memex-migration at the same tag, and has a GREEN Plugins: bake + seal on
that tag's CD run. ci.7674 was proposed as this issue's target on the strength of the first two;
its seal was RED, so it was platform-present / modules-absent and would have traded a wedge for a
half-broken portal.

One correction landed with it: ContinuousDeliveryContract.md claimed the roll patches "the portal
and migration Deployments" — false since #2797, and contradicting DeploymentAKS.md. That sentence
is part of why this wall stayed invisible: it tells a reader self-update carries migrations.

Cross-links were added from Deployment.md, DeploymentAKS.md ("Migration under self-update"),
ContinuousDeliveryContract.md and the Architecture.md topic map, so the next person arrives at it
without knowing it exists.

Resolved — and the resolution REFUTES the structural claim in this issue. Re-measured 2026-09-04 05:5x UTC.

Before acting on the proposed helm-release.yml run, I re-ran this issue's own measurement. memex is no longer wedged:

deployment.apps/memex-portal-deployment   2/2   meshweaver.azurecr.io/memex-portal-ai:3.0.0-rc9.ci.7693
memex-portal-deployment-77898c4947-6mffn   3/3   Running   0 restarts   12h
memex-portal-deployment-77898c4947-bfnft   3/3   Running   0 restarts   12h

The three crash-looping ReplicaSets this issue tabulated are all scaled to 0 and superseded:

ReplicaSet	image	desired	was
`6c74846cf8`	ci.7647	0	0 ready, crash-looping
`5d55558945`	ci.7651	0	0 ready, crash-looping
`54f4c96b6b`	ci.7658	0	0 ready, CrashLoopBackOff, 14 restarts
`77898c4947`	ci.7693	2	← serving, 0 restarts, 12 h

memex-cloud is on the same image (3.0.0-rc9.ci.7693, 3/3 Running). Both portals answer:

https://memex.systemorph.com/    HTTP 200 in 0.187s
https://memex.meshweaver.cloud/  HTTP 200 in 0.196s

DbVersionGate stops the app with a LogCritical when the schema is behind, so two pods Running with zero restarts for 12 h is itself the proof that db_version reached the expected value — a portal ahead of its schema cannot reach Ready.

🚨 The interesting part: no `helm upgrade` ran

$ helm history memex -n memex
27   Sat Aug 29 17:27:23 2026   superseded   Upgrade complete
28   Sun Aug 30 06:54:38 2026   deployed     Upgrade complete

Revision 28 is from Aug 30 — five days before the schema advanced. So the schema was carried forward between 09-03 07:20Z and 09-03 ~17:45Z without the sanctioned helm upgrade path this issue proposes.

That contradicts this issue's central structural claim:

Self-update rolls the IMAGE only. It patches the Deployment from inside the pod. It cannot create the migration Job, so it can never carry the schema forward with the code. … So every schema-bumping release is, by construction, un-takeable by self-update.

That was true when this was filed and is no longer true. Doc/Architecture/DeploymentAKS now records the change:

Since 2026-09-03 the SELF-UPDATER also mints one (memex-migration-su-<tag>) before every automatic roll — see Database Migration Procedure for the routine, the one-time RBAC grant it needs, and the recovery when a pod refuses on db_version.

So the deadlock this issue names was closed by the self-updater learning to mint its own migration Job, landing the same day this was filed. The instance then healed itself: it re-attempted, the Job ran, the gate passed, and ci.7693 came up.

One honest limit on the evidence: I did not directly observe the memex-migration-su-* Job — ttlSecondsAfterFinished cleans it up and the Kubernetes event window has long since rolled past 12 h, so kubectl get jobs now shows only assembly-cache-prune. What is directly measured is the outcome (schema advanced, gate passed, portal serving) and the absence of the alternative explanation (no helm revision since Aug 30). If someone wants the Job itself on the record, catch it on the next schema-bumping release.

Not closed by this, and still worth having

The overlay pins are still stale at 3.0.0-rc8.ci.5068 for portal.image/migration.image. This issue verified they are inert (helm-release.yml applies --set after both -f files), and that is still true — but a committed file that misstates the running image is a trap for the next reader. Worth refreshing separately.
The ~685 MB core dumps the crash-looping pods wrote to /data/dumps during the wedge are presumably still there. Worth a sweep.

Closing as resolved.

Navigation Menu

Repository navigation

memex cannot self-update: the portal image is ahead of its schema (db_version 54 < 55) and only helm upgrade can run the migration Job #3207

memex cannot self-update: the image walked past its schema

Three self-updates today, all wedged the same way

Why the schema never advances — the actual deadlock

The fix is the sanctioned path, and its preconditions check out

Suggested remedy

memex-cloud is behind the same wall

And it blocks #2355 / Memex#148

Correcting my own remedy: this is a KNOWN, DOCUMENTED gap, and the fix is not a manual deploy

The root cause is already written down — and it has now fired

One thing the doc predicts that did NOT happen, and it matters

The fix, stated as a change to the pipeline rather than an operator action

The target build is now available and verified: `3.0.0-rc9.ci.7674`

🚨 Sequencing — this cannot be run unilaterally today

One correction worth carrying

And the structural half is the real finding

🚨 Correction to my previous comment — `ci.7674` is NOT a safe target. Do not roll to it.

What I checked, and what I failed to check

The corrected target

The lesson worth keeping

Resolved — and the resolution REFUTES the structural claim in this issue. Re-measured 2026-09-04 05:5x UTC.

🚨 The interesting part: no `helm upgrade` ran

Not closed by this, and still worth having

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

memex cannot self-update: the portal image is ahead of its schema (db_version 54 < 55) and only helm upgrade can run the migration Job #3207

Description

memex cannot self-update: the image walked past its schema

Three self-updates today, all wedged the same way

Why the schema never advances — the actual deadlock

The fix is the sanctioned path, and its preconditions check out

Suggested remedy

memex-cloud is behind the same wall

And it blocks #2355 / Memex#148

Activity

rbuergi commented on Sep 3, 2026

Correcting my own remedy: this is a KNOWN, DOCUMENTED gap, and the fix is not a manual deploy

The root cause is already written down — and it has now fired

One thing the doc predicts that did NOT happen, and it matters

The fix, stated as a change to the pipeline rather than an operator action

rbuergi commented on Sep 3, 2026

The target build is now available and verified: 3.0.0-rc9.ci.7674

🚨 Sequencing — this cannot be run unilaterally today

One correction worth carrying

And the structural half is the real finding

rbuergi commented on Sep 3, 2026

🚨 Correction to my previous comment — ci.7674 is NOT a safe target. Do not roll to it.

What I checked, and what I failed to check

The corrected target

The lesson worth keeping

rbuergi commented on Sep 3, 2026

rbuergi commented on Sep 4, 2026

Resolved — and the resolution REFUTES the structural claim in this issue. Re-measured 2026-09-04 05:5x UTC.

🚨 The interesting part: no helm upgrade ran

Not closed by this, and still worth having

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The target build is now available and verified: `3.0.0-rc9.ci.7674`

🚨 Correction to my previous comment — `ci.7674` is NOT a safe target. Do not roll to it.

🚨 The interesting part: no `helm upgrade` ran