memex-migration Job cannot pull: helm-release derives its tag from the PORTAL image, and the two ACR repos do not share a tagging convention

Measured

memex-migration-26 has been unable to start for 6 h 27 m:

memex-migration-26-txcx6   Pending   meshweaver.azurecr.io/memex-migration:3.0.0-rc8.ci.6105   created 14:51:56Z
memex-migration-26         Running 0/1   6h27m

Events:
  Normal   BackOff  2m24s (x1699 over 6h27m)  Back-off pulling image
                    "meshweaver.azurecr.io/memex-migration:3.0.0-rc8.ci.6105"
  Warning  Failed   2m24s (x1699 over 6h27m)  Error: ImagePullBackOff

The tag has never existed. memex-migration publishes no 3.0.0-rc8.ci.* tag at all —
az acr repository show-tags -n meshweaver --repository memex-migration --orderby time_desc:

staging-4c1917e-33115739143 (+ -linux-x64/-arm64)
main
5574979
1.0.0
staging-5574979-33113798004 (+ arch variants)
be24fa0
staging-be24fa0-33110573023 (+ arch variants)

Commit-shas, staging-*, main, 1.0.0. No ci.N. memex-portal-ai, by contrast, does carry
3.0.0-rc8.ci.6178 etc. The two repositories do not share a tagging convention, and one caller
assumes they do.

Where the wrong tag comes from

MeshWeaver.Plugins/src/MeshWeaver.SelfUpdate.Aks/KubernetesDeploymentUpdater.cs:114-126:

public async Task PatchToVersionAsync(string versionTag, CancellationToken ct)
{
    if (!CanPatch) return;
    await PatchDeploymentImageAsync(
        _options.PortalDeployment, _options.PortalContainer, _options.PortalImage(versionTag), ct);
    await PatchDeploymentImageAsync(
        _options.MigrationDeployment, _options.MigrationContainer, _options.MigrationImage(versionTag), ct);
    _logger?.LogInformation("[SelfUpdate] patched {Portal} + {Migration} to {Tag}.", …);
}

It applies the portal's version tag to the migration image, for a repository that never
receives one.

And it patches a workload the chart does not define

AGENTS.md is explicit, and this is the shape it forbids:

🚨 The migration is a run-once Job, NOT a Deployment … so kubectl set image /
rollout restart against a memex-migration-deployment is a command with no object: the chart
does not define one, and any that exists in a live namespace is a cluster-only orphan of a chart
revision that predates the Job (#1788). Do not resurrect it.

deploy/helm/templates/memex-migration/job.yaml confirms it: kind: Job, name
memex-migration-{{ .Release.Revision }}, image from .Values.migration.image. MigrationWorkloadModelGuard
fails the build if that command returns "to this file, a doc or a deploy script" — it does not
cover C#, so the forbidden shape survived in the one place that executes it against production.

Leading hypothesis for #2553 — labelled as inference, not measurement

#2553 records that the updater has emitted zero log lines in 6.7 h while three builds published.
This method is a candidate explanation and the ordering is suggestive:

the migration patch is awaited second, and the LogInformation sits after it;
so a throw there — a 404 against a Deployment that does not exist, or any patch failure — leaves the
portal patched, the confirmation line unwritten, and an exception propagating into the watch;
a faulted watch then stops checking, which is exactly the silence memex self-update is silent: 3 builds behind, roll floor satisfied 5h ago, and ZERO SelfUpdate log lines in 6.7h #2553 measures.

Timing is consistent: the portal rolled to ci.6105 at 14:27 (the meshweaver.io/self-update-rolled-at
stamp) and migration Job 26 appeared at 14:51. I have not confirmed the throw — there are no
logs to confirm it with, which is the circularity #2553 is about. Verify before fixing on this basis.

Why it is not currently visible as an outage

The portal serves fine (READY 2/2), because DbVersionGate only stops the app when
admin.mesh_nodes.db_version is below DbVersion.Latest, and the schema at ci.6105 is current.
So a migration that can never run is invisible until a deploy actually needs it — at which point the
portal will refuse to serve and this Job will be six hours into a backoff nobody was watching.

The fix, in the order it should be argued

Stop patching a migration Deployment from the self-updater. The migration is a Job that runs on
helm upgrade; a self-update image roll is not a schema event. Deleting that second patch removes
both the bad tag and the forbidden workload model.
Extend MigrationWorkloadModelGuard to C#. Its stated purpose is to stop this shape returning,
and it missed the only executable instance. A guard whose subject moved into a language it does not
scan is the [[a-guard-follows-its-subject-across-a-split]] shape.
Decide the migration tagging convention deliberately — if anything is ever to pin
memex-migration by version, CD must publish that tag; today promote ends at
memex-portal-ai:<version> and the migration repo gets commit-shas only.
Clean up Job 26 once the cause is fixed. It is not urgent on its own, but it is 1699 pull
attempts of noise sitting on top of the real signal.

Correcting this issue — I got the attribution wrong, and the real cause is more specific

Three corrections, in order of how much they change the fix.

1. The bad tag comes from `helm-release.yml`, not from the self-updater

memex-migration-26 is a Job, and only helm upgrade creates one (memex-migration-{{ .Release.Revision }}). The cluster confirms release v26 at 14:51. The self-updater patches Deployments; it cannot have produced this Job.

The actual line is Systemorph/Memex/.github/workflows/helm-release.yml:357:

tag="${LIVE_IMAGE##*:}"                 # ← ci.6105, taken from the PORTAL image
registry="${LIVE_IMAGE%/*}"
sets="--set portal.image=${LIVE_IMAGE} --set migration.image=${registry}/memex-migration:${tag}"

with this rationale directly above it, which is correct and worth preserving:

The migration Job rides the SAME tag as the portal — a migration from a different build than the code that will run against it is how a schema lands half-applied.

The intent is right; the derivation is not. It assumes the two ACR repositories share a tagging convention, and they do not: memex-portal-ai carries 3.0.0-rc8.ci.N, while memex-migration publishes only commit-shas, staging-*, main and 1.0.0. So the derived tag has never existed, and every helm upgrade since has minted a Job that can never pull.

That flips the fix. Do not decouple the two images — the comment explains why that would be worse. Make CD publish the version tag for memex-migration too, so the derivation becomes true. promote currently ends at memex-portal-ai:<version>; the all-or-nothing set should carry the same version tag on every leg that a deploy pins by version.

2. The self-updater's migration patch is a SEPARATE defect, and it is provably a no-op

kubectl -n memex get deploy returns exactly three: k8s-dashboard-deployment, memex-portal-deployment, portal-next-deployment. There is no memex-migration-deployment — so PatchToVersionAsync's second call targets an object that does not exist. Still worth removing (AGENTS.md forbids the workload model, and the LogInformation after it is why a throw there would silence #2553), but it is not what broke this Job.

3. `MigrationWorkloadModelGuard` did NOT miss this — I misread it

Its own doc-comment scopes it out explicitly, and names the sequencing:

What this guard deliberately does NOT assert: that no file MENTIONS memex-migration-deployment. Several must — deploy/aks/SELF-UPDATE.md and Doc/Architecture/DeploymentAKS.md carry the warning that the self-updater still targets that name, and the chart's own RBAC still grants it (removing that grant before the updater stops patching it would turn a harmless call into a 403 mid-roll).

So this was known, deliberately deferred, and the ordering is written down: updater first, RBAC second. My "the guard missed it" framing was wrong and unfair to the guard.

But one word in that note is now falsified by production: "harmless". The call was assumed to be a benign no-op. It targets a Deployment that does not exist, and #2553 measures an updater that has emitted nothing for 6.7 h across three publications. Whether those are connected is still unproven — but "harmless" is an assumption that has never been tested, and it should not stay in the docs unqualified.

Merge pull request #2557 from Systemorph/fix/2555-migration-version-tag

-p:Version= no longer forks the framework build identity

Navigation Menu

Repository navigation

memex-migration Job cannot pull: helm-release derives its tag from the PORTAL image, and the two ACR repos do not share a tagging convention #2555

Measured

Where the wrong tag comes from

And it patches a workload the chart does not define

Leading hypothesis for #2553 — labelled as inference, not measurement

Why it is not currently visible as an outage

The fix, in the order it should be argued

Correcting this issue — I got the attribution wrong, and the real cause is more specific

1. The bad tag comes from `helm-release.yml`, not from the self-updater

2. The self-updater's migration patch is a SEPARATE defect, and it is provably a no-op

3. `MigrationWorkloadModelGuard` did NOT miss this — I misread it

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

memex-migration Job cannot pull: helm-release derives its tag from the PORTAL image, and the two ACR repos do not share a tagging convention #2555

Description

Measured

Where the wrong tag comes from

And it patches a workload the chart does not define

Leading hypothesis for #2553 — labelled as inference, not measurement

Why it is not currently visible as an outage

The fix, in the order it should be argued

Activity

rbuergi commented on Aug 27, 2026

Correcting this issue — I got the attribution wrong, and the real cause is more specific

1. The bad tag comes from helm-release.yml, not from the self-updater

2. The self-updater's migration patch is a SEPARATE defect, and it is provably a no-op

3. MigrationWorkloadModelGuard did NOT miss this — I misread it

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. The bad tag comes from `helm-release.yml`, not from the self-updater

3. `MigrationWorkloadModelGuard` did NOT miss this — I misread it