Skip to content

client-i self-update stalled ~37h — a rollout restart masks it, so it will drift again #1020

Closed

Description

@rbuergi

What was observed

Deployed portal images at 2026-08-09 13:00Z, with the ACR build time of each tag:

Portal Was on Tag built Staleness
memex ci.2371 09-Aug 09:11Z ~4 h — consistent with the 6 h poll
memex-cloud ci.2354 09-Aug 07:42Z ~5 h — consistent with the 6 h poll
client-i (prod) ci.2151 08-Aug 00:31Z ~37 h — roughly 6 missed polls

memex and memex-cloud sit exactly where a 6-hour SelfUpdateOptions.PollInterval should put them.
client-i did not, and it is the production portal.

Why this is still open even though client-i now looks healthy

A kubectl rollout restart was issued during unrelated work. The self-update poll fires on startup
(StartWith(-1L)), so client-i jumped straight to ci.2417 and now reports healthy — one ready pod,
zero restarts, HTTP 200.

That masked the fault rather than fixing it. Nothing was diagnosed, so the poller is expected to
stall again and the portal to drift silently. The restart also destroyed the in-pod log evidence
from the stalled period.

Where to look

  • [SelfUpdate] log lines on the client-i pod. memex is known to log at Warning, which hides the
    Information-level [SelfUpdate] lines; check whether client-i does the same before concluding from
    their absence. (Do not change log levels in code to investigate — see AGENTS.md.)
  • The federated credential for subject system:serviceaccount:client-i:memex-portal-sa on UAMI
    memexaks-portal-mi, and its AcrPull grant on the meshweaver registry. A silently-expired or
    mismatched-subject credential makes every ACR poll fail without crashing the portal.
  • VersionSelect.PickTarget — it has previously mis-selected tags (per-RID -linux-x64 /
    -linux-arm64 suffixes sorting above the manifest list, fixed in fix(self-update): roll to the manifest-list tag, not the per-arch RID image #120). Worth confirming it is not
    rejecting or mis-ranking the newer 3.0.0-rc1.ci.N tags.

Trap to record

kubectl rollout restart on a portal is not a neutral "restart the same version" — the startup
poll pulls it forward to the newest image. Capture [SelfUpdate] logs before restarting a portal
suspected of a stalled poller, or the evidence goes with the pod.

Activity

  1. changed the title [-]atioz self-update stalled ~37h — a rollout restart masks it, so it will drift again[/-] [+]client-i self-update stalled ~37h — a rollout restart masks it, so it will drift again[/+] on Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions