memex-cloud: routed self-update Restarts re-opened per replica announcement during an in-flight rollout — six ReplicaSets for one image tag in <2h (not a render checksum)

Six ReplicaSets minted for the SAME image tag in under 2 hours — the pod template is being re-rendered non-deterministically

Measured (read-only, `Ops/Status/memex-cloud` version history, versions 3529-3570, 2026-09-24 07:00-09:58Z)

A Kubernetes ReplicaSet hash changes only when the Deployment's pod template changes — never on a replica-count-only change (that stays on the same RS; KEDA scaling is exactly this and does NOT explain the pattern below).

ReplicaSet hash	image tag	first seen	last seen
`76cf776847`	`3.0.0-ci.9263`	07:00	07:32
`8498855cbf`	`3.0.0-ci.9287`	07:00	07:23
`75cd8dc74c`	`3.0.0-ci.9287`	07:13	07:23
`6f9d6848f8`	`3.0.0-ci.9287`	07:21	07:57
`695fb58dc9`	`3.0.0-ci.9287`	07:53	07:57
`86d8c6dfcc`	`3.0.0-ci.9287`	07:57	08:01
`f8dd949d7`	`3.0.0-ci.9287`	08:01	08:55
`959f6754d`	`3.0.0-ci.9291`	08:29	09:13
`589ff8f895`	`3.0.0-ci.9291`	08:45	09:58 (still current)

ci.9263 → ci.9287 is one ordinary roll (one hash each side). But ci.9287 itself spans SIX different ReplicaSet hashes in under an hour, each living only minutes, with no image tag change across any of them. Only two things force a new ReplicaSet with the image held constant: an env/resources/label change on the pod template, or a checksum/config / checksum/secrets annotation on it (#2801 added exactly these two annotations, hashing the rendered config.yaml/secrets.yaml). Since nothing about the deployed image or its declared config changed between these six hashes, the render of config.yaml/secrets.yaml (or another templated field) is producing a different checksum on each helm upgrade with no underlying value actually changing — e.g. an unstable map/dict iteration order, an embedded timestamp, or a value the operator regenerates per Reconcile pass rather than reading back idempotently.

Why it matters

Each of these six churns forces a full pod boot: ~55-70s to PortalReady (measured on other 589ff8f895 pods the same day), a content-types: Degraded window while the NodeType bake catches up, and one more entry in the 38-boots-in-15h count from the observed report. It also multiplies the population of Store/Install / Hosting/InstanceAction / satellite records this replica has NEVER seen (frameworkstale bake outcomes measured on the f8dd949d7/959f6754d replicas the same day), each an operational-noise cost, not a user-visible outage — no crash, no OOM, no probe failure was found in this window.

What I established / did NOT establish

Established: the ReplicaSet-hash sequence above, from the DeploymentStatus version history (not from kubectl). Did NOT establish: which specific rendered field differs between successive helm upgrade invocations of the same tag (I did not read the hosting-deploy/render pipeline's actual Secret/ConfigMap content run-to-run — that needs a live diff of two consecutive helm get values/helm get manifest outputs, which is break-glass and outside a read-only investigation). Did NOT establish whether this is still happening as of 2026-09-25 (the same investigation on the CURRENT ReplicaSet 589ff8f895, running since ~08:41Z 2026-09-24, has not needed a new hash in the following ~20h, so the six-hash burst may have been specific to the 9287→9291 transition window rather than an ongoing steady-state).

Found while classifying the 38-pod-boots-in-15h / 9-boots-in-one-ReplicaSet report on memex-cloud.

2026-09-26 triage: the render-non-determinism hypothesis is REFUTED. Each extra ReplicaSet is a routed self-update Restart.

A kubectl rollout restart stamps kubectl.kubernetes.io/restartedAt on the pod template, which mints a new ReplicaSet hash with the image unchanged. The control instance's Ops/Actions for memex-cloud line up with the table above one for one:

ReplicaSet first seen	action on the control instance (`Ops/Actions/…`)	announcement `detectedAt`
07:00 `8498855cbf` (ci.9287)	`selfupdate-roll-memex-cloud-3-0-0-ci-9287-…` (Done 06:57)	—
07:13 `75cd8dc74c`	`selfupdate-restart-memex-cloud-20260924062546-…` (Done 07:11)	06:25:46
07:21 `6f9d6848f8`	`selfupdate-restart-memex-cloud-20260924055351-…` (Done 07:22)	05:53:51
07:53 `695fb58dc9`	`selfupdate-restart-memex-cloud-20260924075217-…` (dispatched 07:52:21)	07:52:17
07:57 `86d8c6dfcc`	`selfupdate-restart-memex-cloud-20260924075347-…` (created 07:54:40)	07:53:47
08:01 `f8dd949d7`	`selfupdate-restart-memex-cloud-20260924075540-…` (Done 07:58)	07:55:40
08:29 `959f6754d` (ci.9291)	`selfupdate-roll-memex-cloud-3-0-0-ci-9291-…`	—
08:45 `589ff8f895`	`selfupdate-restart-memex-cloud-20260924083938-…` (Done 08:41)	08:39:38

Every restart's reason is the same: "a landed module generation is pending activation (restart-as-activation, #3650) — routed from the control inbox (MeshWeaver#4098)". So checksum/config and checksum/secrets are not implicated, and nothing in the chart render needs changing.

The real question, narrower than the title: why did the router open a Restart per announcement, several minutes apart, while the previous one was still rolling out?

SelfUpdateRouting.Decide (MeshWeaver.Plugins Hosting/Deployment/Source/SelfUpdateRouting.cs) is meant to prevent exactly this. An OPEN Roll or Restart skips, and a Done Restart whose finishedAt ≥ detectedAt counts as delivered. Two observations I can make but not close:

A Restart reaches Done when the operator's rollout restart returns, not when the rollout converges. Every one of these reads "the observer that dispatched it was lost … the phases AFTER the run did not run" and then done, so Verify one generation never ran. A replica that has not yet been replaced (3 replicas, minute-long bakes) still holds its pending generation and announces again after the Restart is Done, with a later detectedAt. That re-announcement is not "delivered" by the clock rule, so a new Restart restarts the rollout again.
…075347 (detected 07:53:47) was opened at 07:54:40 although …075217 had finishedAt 07:54:22 ≥ 07:53:47. By the stated rule that should have been a Skip. The idempotence listing is an index query, so it may still have shown …075217 in a state the rule does not read. I could not confirm this: I did not read the watcher's log lines for that minute.

Also note: restarts detected at 05:53:51 and 06:25:46 were routed only at about 07:11 to 07:22, roughly one hour or more late, by which point the 06:57 Roll had already re-created the pods and delivered them.

Frequency now: a burst of this shape on 09-22 (2), 09-23 (10) and 09-24 (6). Since then there has been ONE restart (…20260925152337), so the steady state is quiet. The cost is pod churn with no outage, which is consistent with sev:M.

Not established: which of (1) and (2) caused the repeated opens, and the watcher's own decision lines for 07:53 to 07:58Z on 09-24. The likely fix is on the Plugins router: a Restart should stay "in flight" until its rollout converges (or its generation is verified), and a re-announcement from a pod that started before the Restart was dispatched should count as delivered by it. That needs a repro against Decide before any code changes, so I am retitling this to the cause and leaving it for that work.

2026-10-08: the in-flight half is fixed; a sequential restart loop recurred 10-03 → 10-05 and has been silent since; the record is now harder to read

Read-only, on memex.systemorph.com (the routing instance until 2026-10-07) and control.systemorph.com (since then).

Mechanism (1) from the 09-26 triage is fixed. A Restart reached Done when rollout restart returned, before the rollout converged. The Restart plan now runs [6/8] Verify one generation before Done. Example: Ops/Actions/selfupdate-restart-memex-cloud-20261005103617-c8f63a22 dispatched at 10:42:27Z, deployment.apps/memex-portal-deployment restarted at 10:43:44Z, and Verify one generation ran 10:44:51 → 10:51:22Z. Then "3/3 ready, 0 restart(s), running … 3.0.0-ci.9961". finishedAt is 10:51:23Z, i.e. after convergence.

But a restart loop recurred without overlap. namespace:Ops/Actions id:selfupdate-restart-* sort:lastModified-desc (limit 40, truncated, so a floor) lists at least 29 routed Restarts of memex-cloud between 2026-10-03 19:17Z and 2026-10-05 10:36Z, often 15–30 minutes apart (e.g. detected 19:17:43 → done 19:24:05, next detected 19:32:44). Each was detected AFTER the previous one finished, so Delivered (Done ∧ finishedAt ≥ detectedAt) correctly did not suppress it. Every reason is the same: "a landed module generation is pending activation (restart-as-activation, #3650)". So this is not mechanism (1) or (2). It is the instance raising a fresh pending activation after each restart, which by the router's rule warrants a new Restart. Each one is a full 3-replica cycle.

Since 2026-10-05 10:36Z there has been no routed Restart of memex-cloud. On 2026-10-07 memex-cloud instead took a Roll per build (ci.10106 → ci.10157, selfupdate-roll-memex-cloud-*, at least 15). Two changes plausibly ended restart-as-activation, but neither is proven:

feat(modules): root services and the builder hook become live-updatable; only boot-time infra may restart (slices 4+5) #6127 ("root services and the builder hook become live-updatable; only boot-time infra may restart") merged 2026-10-05T13:19Z, about 2.7 h AFTER the last restart, so it cannot explain the first hours of the silence.
RollInsteadOfRestart (Plugins 2988371) merged 2026-09-28, BEFORE the burst, so it did not prevent it.

Why "silent since" is now a weak reading. The control lane moved to control.systemorph.com around 2026-10-07 17:xx. memex.systemorph.com now drops announcements to its old inbox: Webhook for unknown/refused target 'Hosting/PlatformBuilds' dropped, at least 88× in 16:51–18:51Z today (Ops/Actions/verify-5757-control-20261008-inbox, cut at 200). On control.systemorph.com, content.deployment:Deployments/memex-cloud lists only two record-change Reconciles today and no self-update action. If memex-cloud still points at the old inbox, its restart-pendings would be among those drops, and the silence would mean nothing about the loop. I did not read memex-cloud's boot line to settle this. See #5757 for the same 404 on build.

Remaining, and the closing condition: once memex-cloud's self-updater hands to control.systemorph.com, 7 days with no run of more than 3 routed Restarts in 24 h on control.systemorph.com.

Re-read 2026-10-09 ~20:00Z on control.systemorph.com. Quiet so far, but the 7-day clock only started today.

memex-cloud's self-updater now hands to the control lane. Ops/Actions/reconcile-memex-cloud-20261009-reapply-v2-after-3188 applied the control inbox URL at 14:27Z. /health self_update on memex-cloud now reads "handed to the control lane (https://control.systemorph.com/api/hooks/Hosting/PlatformBuilds: accepted (200), signature verified)". So the "dropped to the old inbox" caveat from 10-08 no longer applies.

Reading: namespace:Ops/Actions content.deployment:Deployments/memex-cloud lists 13 actions (not truncated) since 10-08 11:32Z. 0 routed Restarts. It lists 4 routed Rolls today (ci.10322 → .10337 → .10340 → .10344, 18:18–19:47Z), one per platform build, plus record-change Reconciles.

Close condition (unchanged): 7 days with no run of more than 3 routed Restarts in 24 h, counted from the handover. Re-check on 2026-10-16.

Closing on the title: a routed Restart is no longer opened while a rollout is in flight. Measured on the control lane's whole record.

What the router does now (Hosting/Deployment/Source/SelfUpdateRouting.cs, DecideRuns): an open Roll or Restart on the deployment skips every further announcement, and a Restart stays open until Verify one generation has seen the rollout converge — it used to read Done when rollout restart returned, which is what let a not-yet-replaced replica's announcement open the next one. Pinned by SelfUpdate_RestartDedupesOnTheOpenRunAndTheClock and the orphan cases in PlatformBuildInboxTests.

Reading, control.systemorph.com, 2026-10-11 ~11:10Z. namespace:Ops/Actions id:selfupdate-restart-* returns 9 routed Restarts, not truncated. That is every one since the control lane took over routing.

deployment	routed Restarts	most in one day
memex-cloud	0	0
memex	4	3 (2026-10-11: detected 03:23:46Z, 05:26:36Z, 05:52:14Z)
control	4	2
build	1	1

No overlap. The three memex restarts of 10-11 finished at 03:37Z, 05:38Z and 06:04Z; each next one was detected after the previous had finished.
Each waits for convergence. selfupdate-restart-memex-20261011055214-…: deployment … restarted at 05:53Z, Verify one generation until 06:04:09Z, then 3/3 ready, 0 restart(s), then Done.
It names its cause. That run's reason (the one I opened) reads "landed MeshWeaver.SelfUpdate.Aks 1.64.7", a module that declares a restart, so this is an activation and not a loop.
memex-cloud takes a Roll per platform build instead (10 listed for 10-10 23:33Z → 10-11 10:34Z, listing truncated).

The 7-day condition from 2026-10-09 is not met by the calendar (it would be on 2026-10-16). I am closing on the mechanism plus this reading instead: six ReplicaSets for one tag needs overlapping restarts, and the record holds none. The sequential loop of 10-03 → 10-05 has not recurred on any deployment; if it does, it is a different defect from this title (an instance raising a fresh pending activation after each restart) and deserves its own issue.

_{Fleet issue wave, Claude Opus 5.5 (agent session).}

Navigation Menu

Repository navigation

memex-cloud: routed self-update Restarts re-opened per replica announcement during an in-flight rollout — six ReplicaSets for one image tag in <2h (not a render checksum) #5677

Six ReplicaSets minted for the SAME image tag in under 2 hours — the pod template is being re-rendered non-deterministically

Measured (read-only, `Ops/Status/memex-cloud` version history, versions 3529-3570, 2026-09-24 07:00-09:58Z)

Why it matters

What I established / did NOT establish

2026-09-26 triage: the render-non-determinism hypothesis is REFUTED. Each extra ReplicaSet is a routed self-update Restart.

The real question, narrower than the title: why did the router open a Restart per announcement, several minutes apart, while the previous one was still rolling out?

2026-10-08: the in-flight half is fixed; a sequential restart loop recurred 10-03 → 10-05 and has been silent since; the record is now harder to read

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

memex-cloud: routed self-update Restarts re-opened per replica announcement during an in-flight rollout — six ReplicaSets for one image tag in <2h (not a render checksum) #5677

Description

Six ReplicaSets minted for the SAME image tag in under 2 hours — the pod template is being re-rendered non-deterministically

Measured (read-only, Ops/Status/memex-cloud version history, versions 3529-3570, 2026-09-24 07:00-09:58Z)

Why it matters

What I established / did NOT establish

Activity

rbuergi commented on Sep 26, 2026

2026-09-26 triage: the render-non-determinism hypothesis is REFUTED. Each extra ReplicaSet is a routed self-update Restart.

The real question, narrower than the title: why did the router open a Restart per announcement, several minutes apart, while the previous one was still rolling out?

rbuergi commented on Oct 8, 2026

2026-10-08: the in-flight half is fixed; a sequential restart loop recurred 10-03 → 10-05 and has been silent since; the record is now harder to read

rbuergi commented on Oct 9, 2026

rbuergi commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Measured (read-only, `Ops/Status/memex-cloud` version history, versions 3529-3570, 2026-09-24 07:00-09:58Z)