Repository navigation
memex-cloud has not self-updated ~80 min after two publications — no observable self-update activity #2465
Description
Observation
After delivery was restored on 2026-08-26, two images published:
3.0.0-rc8.ci.6007 ← 139fde8 (~00:10Z)
3.0.0-rc8.ci.6008 ← 842e6c5 (~00:40Z)
At 02:00Z — ~80 min after the second, well past SelfUpdate__MinRollInterval: 01:00:00 — memex-cloud had not rolled:
deployment image : meshweaver.azurecr.io/memex-portal-ai:3.0.0-rc8.ci.5670
/api/version : 3.0.0-rc8+2699e4a… (the ci.5670 commit)
pods : 7/7 Running, 0 restarts ← healthy, just old
And in the running pod, over the last 3 hours:
selfupdate-ish log lines: 0
Nothing matching selfupdate / self-update / platform-update. That is the part worth explaining: not a decision not to roll, but no visible activity either way.
Context that may or may not be causal
🚨 Stated as a hypothesis, not an established mechanism — I could not confirm it from the logs available, and the pods' retention does not reach their startup.
MeshWeaver.SelfUpdate.Aksis a landed MODULE, not part of the image. From an earlier pod's startup log:[ModuleLoad] MeshWeaver.SelfUpdate.Aks ← /data/modules/MeshWeaver.SelfUpdate.Aks@5a72a03e/… (source=store, written=2026-08-26 21:04:05Z).- The running pods are 5h51m old (started ~20:09Z) — i.e. before the 20:37–21:10Z module wave. Landed modules require a restart to activate, so these pods are running whatever was active at 20:09.
- The deployment was rolled back from
ci.5876toci.5670after the crashloop (see theOpenAICompatibleModelSyncskew and Per-module isolation (#2234) does not cover HOSTED SERVICES — one incompatible module still kills the portal #2449).
So the install is now: an older image, running modules from an earlier landing, with newer modules staged on disk awaiting a restart that self-update would normally cause.
If the updater is inactive in that state, the consequence is structural and worth naming: an image rollback can strand an install, because the component that would roll it forward ships as a module rather than in the image. An install would then need an operator to move it — which is exactly what "always roll via CD, never by hand" is trying to avoid having to do.
What I did not do, deliberately
No kubectl set image. The rule is that image and landed modules move atomically, and a consistently-old portal is a safe state — forcing a hand roll risks re-creating the very skew that caused tonight's crashloop. Prod is healthy and serving; this is a "stuck old", not an outage.
What would settle it
- Whether
MeshWeaver.SelfUpdate.Aksis active in the running pods (the startup module report would say; retention lost it here). - Whether the updater logs anything at all on a no-op check — if it is silent by design, "0 lines" is not evidence of absence and this issue is only about the roll not happening.
- Whether
UpdatePolicyKindfor this install selects therc8.ci.*channel at all.
Related: #2449 (why the skew wedged the rollout), #2452 (the bake lane rotting while skipped), #2457 (moving the bake off the shared pool).
Activity
rbuergi commentedon Aug 27, 2026
Current state (08:19Z): memex-cloud IS self-updating — and one publication behind is the designed behaviour
memex-cloud's ReplicaSet is 71 minutes old, so it rolled ~07:08Z — it has moved
ci.6011 → ci.6028since this issue was filed.SelfUpdate__MinRollIntervalon that namespace is01:00:00.So the steady state is: at most one roll per hour, and therefore normally one-or-more publications behind, because CD publishes faster than the floor allows delivery. That floor is deliberate — #1780 added it precisely because #1773 made installs check per publication event, which for a few hours left nothing at all between "an image published" and "every portal restarted".
What that does and does not settle
Does not settle the original report. An 80-minute gap exceeds a 60-minute floor, so the window this issue describes is not explained by the throttle alone. My measurement is from now, not from then, and I am not claiming it refutes the observation.
Does establish that the mechanism is functioning today, which narrows the question from "self-update is broken" to "what delayed it during that particular window". Two candidates worth checking against the original timestamps before anything is changed:
3.0.0-rc8.ci.Ntag inside the window, not merely CD runs.Not to be confused with the memex namespace
memexshowsUPDATED=1ofWANT=2: its ci.6011 rollout has been stuck inCrashLoopBackOfffor hours on a DI failure — that is #2443, not self-update. Worth separating, becausereadyReplicas=2makes it look healthy and its serving pods are from the previous ReplicaSet.rbuergi commentedon Aug 28, 2026
Closing as a stale point-in-time observation, superseded by the issues that track the live mechanisms.
Per the measurement in the comment above (2026-08-27 08:19Z): memex-cloud is self-updating — it rolled
ci.6011 → ci.6028at ~07:08Z, and withSelfUpdate__MinRollInterval: 01:00:00the designed steady state is at most one roll per hour, i.e. normally one-or-more publications behind. The original 80-minute window exceeds the floor and was never explained, but it is now only answerable from historical data (whether a COMPLETE image set was actually promoted inside that window, and where the floor's restart-anchored clock stood), and nothing actionable hangs on it.The live self-update problems have their own threads with sharper mechanisms:
PLATFORM_WEBHOOK_URLvsWebhookInbox:Targets), so promoted builds are not announced;If a portal is observed behind again, measure against those three first; reopen only if a distinct fourth mechanism appears.