You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Self-update: a pinned (UpdatePolicy=None) install rolls on every pod restart — the Startup pass runs under the configured DEFAULT policy before the persisted policy arrives #2797
Admin/UpdatePolicy has read policy: None all day (every version from v1896 @ 07:08Z through v2088 @ 15:19Z). Yet the install rolled twice — to be39fc607 (ci.6664) at ~11:54Z and to 41dd85a24 (ci.6739) at ~15:03Z (ready 15:57Z) — with /api/version proving both.
The node's own history explains it. Every pass whose lastCheckTrigger is Startup evaluates a candidate as if the policy were Continuous, and only the 1 h floor decides whether it rolls:
version
time (Z)
trigger
verdict
1898
08:14
Startup
check FAILED: HttpRequestException: PATCH memex-migration-deployment → …ci.6585 failed: 404 Not Found — it was rolling
1899
08:15
PolicyChange
updates are disabled on this install (Admin/UpdatePolicy = None)
2000
11:55
Startup
ci.6664 is available but this install rolled 00:01:25 ago, inside the 01:00:00 floor — deferring
2036/2039
13:30
Startup
ci.6691 is available but this install rolled 00:03:08 ago … deferring
2040
13:33
PolicyChange
updates are disabled …
None → "Startup evaluates a candidate" → PolicyChange back to disabled, on every pod start.
The StartWith seeds the pipeline with the configured default (Continuous) before the persisted node's value arrives, and the Switch downstream runs a full RunOnce on that seed — ACR listing, candidate selection, and the Deployment PATCH — as the Startup pass. The persisted None arrives a second later as a "PolicyChange" and disables the updater — after the roll has already been issued.
So on a pinned (None) install, every pod (re)start that happens more than MinRollInterval after the last roll rolls the portal to the newest rollable tag. A restart for any reason (OOM, node drain, a probe failure, a Restart InstanceAction) becomes an unrequested upgrade. Pinning is not a pin. This is the exact evidence-confusion #2553 was about, with the opposite sign: an install the admin pinned advances, and the node says it never does.
Second finding, same log line
v1898's failed roll tried to PATCH memex-migration-deployment, which no longer exists — the migration is a run-once Job the chart runs on helm upgrade (#1788). The portal Deployment patch had already been issued by then, so the roll went ahead with no migration for the new image, which is exactly the state DbVersionGate exists to refuse. The updater's roll step still assumes the pre-#1788 topology.
Fix
Do not seed the policy pipeline with the default. Read the persisted node first (EnsureExists already guarantees it exists — the seed was written with DefaultPolicy at creation, so nothing is lost) and run the first pass only on the value actually read. If the read must fall back, fall back to None (fail closed), never to Continuous.
Make the roll step match the chart: no memex-migration-deployment patch; run the migration the way the chart does, or refuse to roll when it cannot.
A guard test: a None node + a fresh service start must produce ZERO ACR calls and zero patches (SelfUpdateVerdict.UpdatesDisabled() as the Startup verdict, not a candidate evaluation).
Duplicate of #2731 (filed 07:11Z, same day, same mechanism — CreatePolicySource's StartWith(default) seed racing the persisted policy). Closing this one so the fix has a single home; the evidence here is folded into #2731 — it is the stronger half (two further rolls at 11:54Z and 15:03Z, and the proof that everyStartup-triggered pass evaluates as Continuous).
The fix is up next as a PR: the seed is removed — the default belongs in the NODE (EnsureExists puts it there when absent), never in the stream — and a new resilience test pins it (node persisted Stable, default Continuous, first read faulted: the first patch must be the Stable tag and the CI tag must never be applied).
#2731 is the earlier report of the first half and #2800 closes both. Closing this one would have orphaned Plugins #979's target. My apologies for the churn.
I had independently written the same seed fix in a worktree before finding #2800 — discarded, since #2800 is identical and already at the head of the merge queue.
Activity
rbuergi commentedon Aug 30, 2026
Duplicate of #2731 (filed 07:11Z, same day, same mechanism —
CreatePolicySource'sStartWith(default)seed racing the persisted policy). Closing this one so the fix has a single home; the evidence here is folded into #2731 — it is the stronger half (two further rolls at 11:54Z and 15:03Z, and the proof that everyStartup-triggered pass evaluates as Continuous).The fix is up next as a PR: the seed is removed — the default belongs in the NODE (
EnsureExistsputs it there when absent), never in the stream — and a new resilience test pins it (node persisted Stable, default Continuous, first read faulted: the first patch must be the Stable tag and the CI tag must never be applied).rbuergi commentedon Aug 30, 2026
Reopening — I closed this wrongly. It is not a duplicate to discard: two open PRs are pointed at it, one per half.
StartWith(default)seed; addsStartupCheck_OnAPinnedInstall_RunsUnderThePersistedPolicy_NotTheConfiguredDefault)memex-migration-deploymentPATCH turning a successful roll into acheck FAILEDverdict#2731 is the earlier report of the first half and #2800 closes both. Closing this one would have orphaned Plugins #979's target. My apologies for the churn.
I had independently written the same seed fix in a worktree before finding #2800 — discarded, since #2800 is identical and already at the head of the merge queue.
Merge pull request #3208 from Systemorph/docs/3207-migration-under-se…