memex-migration CrashLoopBackOff in all three prod namespaces (50/53/38 restarts) — run-once workload modelled as a Deployment #1788

Activity

1. rbuergi

   added

   bugSomething isn't workingSomething isn't working

   on Aug 17, 2026
2. rbuergi commented on Aug 23, 2026

   

   rbuergi

   on Aug 23, 2026

   ContributorAuthor

   More actions

   Still live: memex-cloud's migration pod hit 87 restarts over 9h today (2026-08-23). Worth noting the log shows the migration SUCCEEDING every cycle — "Database migration completed. Version: 54" → clean shutdown → kubelet restarts it. So the CrashLoopBackOff is purely the run-once-as-Deployment shape this issue describes; the earlier 'auth-mirror deadlock' theory is retired, the DB is current, and the loop is cosmetic noise + wasted restarts.
3. rbuergi

   closed this as completedin #2253on Aug 25, 2026
4. rbuergi commented on Oct 11, 2026

   

   rbuergi

   on Oct 11, 2026

   ContributorAuthor

   More actions

   (re-posted 2026-10-11; originally by @rbuergi on 2026-08-17)

   Investigated on the live cluster — and both obvious remedies are unsafe, so this needs a decision rather than a quick fix.

   What is actually deployed

   namespace	migration Job	migration Deployment (created)	image
   memex	none	2026-05-31	memex-migration:3.0.0-rc4.ci.4147
   memex-cloud	none	2026-06-04	same
   client-i	none	2026-06-04	same

   The chart was fixed in dd49cd1 (2026-06-27) — deployment.yaml deleted, job.yaml added. Every live Deployment predates that commit. So the clusters have simply never been helm-upgraded onto the new chart; they have been kept current by kubectl set image on the legacy object, which is exactly what the deploy docs still instruct. dd49cd1's own FOLLOW-UP note called that out and it was never done.

   Why not just helm upgrade

   🚨 A great deal of this cluster's configuration was patched live and is not in any values overlay — OpenRouter provider flags and model tiers, ModelTier__*, the LinkedIn ClientId on memex, and the SecretProviderClass object/secretObject mappings. A helm upgrade from the current overlay would reset all of it. deploy/aks/envs/<env>/deploy.sh is documented as first-time env setup for exactly this reason.

   Why not just delete the Deployment

   It is load-bearing for self-update. SelfUpdateOptions.MigrationDeployment defaults to memex-migration-deployment and KubernetesDeploymentUpdater.PatchToVersionAsync patches portal then migration by that name. Delete the object and the next self-update roll fails on the migration patch — trading a noisy crash-loop for a broken update path.

   So the fix is a coordinated change, not an action

   Three pieces, and they must land together or the roll breaks:

   1. the chart's Job form actually applied to the three namespaces, without clobbering live-patched config (either fold that config into the overlay first, or apply the Job manifest alone and remove the Deployment);
   2. the self-updater taught to drive the Job (a Job is not patched-and-restarted; a new one is created per version), or explicitly told the migration is out of its scope;
   3. the deploy docs updated — AGENTS.md and DeploymentAKS.md still say set image deployment/memex-migration-deployment + rollout restart, which is the instruction that keeps the legacy object alive.

   Until then the crash-loop is cosmetic-but-costly, as described above: ~360 restarts/day across three namespaces, and "is anything crash-looping in prod?" can never be answered no.
5. rbuergi commented on Oct 11, 2026

   

   rbuergi

   on Oct 11, 2026

   ContributorAuthor

   More actions

   (re-posted 2026-10-11; originally by @rbuergi on 2026-08-21)

   The repo half is already correct — this is cluster-vs-chart drift, and it needs a cluster operation

   What the chart ships today

   On origin/main (46a4378f9) the migration templates are:

   deploy/helm/templates/memex-migration/config.yaml
   deploy/helm/templates/memex-migration/job.yaml      ← kind: "Job"
   deploy/helm/templates/memex-migration/secrets.yaml
   

   There is no deployment.yaml, and git log --diff-filter=D says one has never been deleted from main — job.yaml arrived with a6a9e4cdf (#145) and main has had the run-once form ever since. The Job is shaped exactly as this issue asks:

   kind: "Job"
   metadata:
     name: "memex-migration-{{ .Release.Revision }}"
   spec:
     backoffLimit: 6
     ttlSecondsAfterFinished: 600
     template:
       spec:
         restartPolicy: "Never"

   One correction to the issue text: dd49cd1bf is not an ancestor of main. That commit (2026-06-27) lives on another lineage; the Job form reached main separately. So "the fix already exists and was never applied here" is right about the clusters and misleading about the repo — there is nothing to port, and no PR to open against this repository.

   So what is actually wrong

   The pods in the table are named memex-migration-deployment-…. The current chart does not template a Deployment at all, so those are orphans of a chart revision that predates the Job — a resource living in the cluster that the chart no longer defines. Reconciling that means a helm upgrade against each release plus removing the leftover Deployments in three production namespaces.

   That is system-changing — it drains and restarts workloads in memex, memex-cloud and client-i — so I have not done it, and it should not be done by an agent. Whoever does it should heed the ordering warning already in this issue: KubernetesDeploymentUpdater.PatchToVersionAsync patches portal then migration, and the AdoNet grain-storage work makes the portal's start depend on migration-created tables, so verify per environment that the Job form cannot invert against the portal.

   I could not verify the live state, and I want to be explicit about that

   kubectl is configured here for memexaks-cluster, but the API server is private-link only and does not resolve from this machine:

   Unable to connect to the server: dial tcp: lookup
   memexaks-cluster-ubdb34eu.privatelink.swedencentral.azmk8s.io: no such host
   

   Worth recording how that nearly went wrong: my first probe was kubectl -n <ns> get deploy,job | grep -i migrat, which printed nothing for all three namespaces — indistinguishable from "the orphans are gone and this is fixed". The grep had swallowed the connection error. I only caught it by re-running without the filter. A "no migration pods found" report from a filtered kubectl is not evidence of anything unless the same command proves it can reach the cluster at all.

   So: I cannot confirm whether the 50/53/38 restarts are still accruing. Someone with cluster reach should check before acting.

   This is the case #1709 exists to catch

   A Deployment in the cluster that the chart does not define is precisely a CLUSTER-ONLY divergence, which is one of the three verdicts check-chart-drift.sh reports. Chart Drift has never completed a run (3 for 3 red on the missing MEMEX_DEPLOY_REPO_TOKEN), so this drift has had no detector since the day the Job landed. Worth treating #1709 as the systemic half of this ticket rather than an unrelated CI chore — provisioning it would have surfaced these orphans automatically instead of someone finding them by eye during an incident.