Skip to content

Self-update: a pinned (UpdatePolicy=None) install rolls on every pod restart — the Startup pass runs under the configured DEFAULT policy before the persisted policy arrives #2797

Closed

Description

@rbuergi

What I saw on memex.meshweaver.cloud (2026-08-30)

Admin/UpdatePolicy has read policy: None all day (every version from v1896 @ 07:08Z through v2088 @ 15:19Z). Yet the install rolled twice — to be39fc607 (ci.6664) at ~11:54Z and to 41dd85a24 (ci.6739) at ~15:03Z (ready 15:57Z) — with /api/version proving both.

The node's own history explains it. Every pass whose lastCheckTrigger is Startup evaluates a candidate as if the policy were Continuous, and only the 1 h floor decides whether it rolls:

version time (Z) trigger verdict
1898 08:14 Startup check FAILED: HttpRequestException: PATCH memex-migration-deployment → …ci.6585 failed: 404 Not Found — it was rolling
1899 08:15 PolicyChange updates are disabled on this install (Admin/UpdatePolicy = None)
2000 11:55 Startup ci.6664 is available but this install rolled 00:01:25 ago, inside the 01:00:00 floor — deferring
2036/2039 13:30 Startup ci.6691 is available but this install rolled 00:03:08 ago … deferring
2040 13:33 PolicyChange updates are disabled …

None → "Startup evaluates a candidate" → PolicyChange back to disabled, on every pod start.

Root cause

SelfUpdateHostedService.CreatePolicySource():

.SelectMany(_ => Observable
    .Defer(ReadPolicyStream)
    .RetryWhen(ResubscribeAfterRetryInterval("policy stream"))
    .StartWith(new UpdatePolicyContent { Policy = _options.DefaultPolicy }))   // Continuous
.DistinctUntilChanged(c => (c.Policy, c.RequireCiGreen));

The StartWith seeds the pipeline with the configured default (Continuous) before the persisted node's value arrives, and the Switch downstream runs a full RunOnce on that seed — ACR listing, candidate selection, and the Deployment PATCH — as the Startup pass. The persisted None arrives a second later as a "PolicyChange" and disables the updater — after the roll has already been issued.

So on a pinned (None) install, every pod (re)start that happens more than MinRollInterval after the last roll rolls the portal to the newest rollable tag. A restart for any reason (OOM, node drain, a probe failure, a Restart InstanceAction) becomes an unrequested upgrade. Pinning is not a pin. This is the exact evidence-confusion #2553 was about, with the opposite sign: an install the admin pinned advances, and the node says it never does.

Second finding, same log line

v1898's failed roll tried to PATCH memex-migration-deployment, which no longer exists — the migration is a run-once Job the chart runs on helm upgrade (#1788). The portal Deployment patch had already been issued by then, so the roll went ahead with no migration for the new image, which is exactly the state DbVersionGate exists to refuse. The updater's roll step still assumes the pre-#1788 topology.

Fix

  • Do not seed the policy pipeline with the default. Read the persisted node first (EnsureExists already guarantees it exists — the seed was written with DefaultPolicy at creation, so nothing is lost) and run the first pass only on the value actually read. If the read must fall back, fall back to None (fail closed), never to Continuous.
  • Make the roll step match the chart: no memex-migration-deployment patch; run the migration the way the chart does, or refuse to roll when it cannot.
  • A guard test: a None node + a fresh service start must produce ZERO ACR calls and zero patches (SelfUpdateVerdict.UpdatesDisabled() as the Startup verdict, not a candidate evaluation).

Activity

  1. rbuergi commented on Aug 30, 2026

    @rbuergi
    ContributorAuthor

    Duplicate of #2731 (filed 07:11Z, same day, same mechanism — CreatePolicySource's StartWith(default) seed racing the persisted policy). Closing this one so the fix has a single home; the evidence here is folded into #2731 — it is the stronger half (two further rolls at 11:54Z and 15:03Z, and the proof that every Startup-triggered pass evaluates as Continuous).

    The fix is up next as a PR: the seed is removed — the default belongs in the NODE (EnsureExists puts it there when absent), never in the stream — and a new resilience test pins it (node persisted Stable, default Continuous, first read faulted: the first patch must be the Stable tag and the CI tag must never be applied).

  2. rbuergi commented on Aug 30, 2026

    @rbuergi
    ContributorAuthor

    Reopening — I closed this wrongly. It is not a duplicate to discard: two open PRs are pointed at it, one per half.

    half fix
    the poller evaluating under the configured default before it has read the persisted policy core #2800 (removes the StartWith(default) seed; adds StartupCheck_OnAPinnedInstall_RunsUnderThePersistedPolicy_NotTheConfiguredDefault)
    the dead memex-migration-deployment PATCH turning a successful roll into a check FAILED verdict Plugins #979

    #2731 is the earlier report of the first half and #2800 closes both. Closing this one would have orphaned Plugins #979's target. My apologies for the churn.

    I had independently written the same seed fix in a worktree before finding #2800 — discarded, since #2800 is identical and already at the head of the merge queue.

  3. added a commit that references this issue on Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions