Self-update check runs on every build completion on every replica under UpdatePolicy=None, and the replicas fight over Admin/UpdatePolicy.lastCheckedAt (158 checks, 14 refused writes, 2 queue failures in 5 h) #3790

Activity

1. rbuergi

   added

   bugSomething isn't workingSomething isn't working

   on Sep 9, 2026
2. rbuergi commented on Sep 9, 2026

   

   rbuergi

   on Sep 9, 2026

   ContributorAuthor

   More actions

   An independent measurement of the cumulative cost: the node is at version 62,671

   Read live 2026-09-09T07:40Z via get @Admin/UpdatePolicy (a plain node read, no Loki):

   version:          62671
   lastModified:     2026-09-09T07:40:12.966Z
   lastModifiedBy:   system-security
   lastCheckedAt:    2026-09-09T07:39:42.187Z
   lastCheckTrigger: BuildCompletion
   lastCheckVerdict: "updates are disabled on this install (Admin/UpdatePolicy = None);
                      the registry was not listed."
   

   Three things this adds to the 5-hour Loki window already recorded here:

   1. The write count is enormous and cumulative. Every check writes lastCheckedAt, so version: 62671 is a lower bound on how many times this one node has been rewritten. The 158-checks-in-5-h rate measured above is not a burst — it is the steady state, and it has been running long enough to put five figures of versions on a node whose content decision has not changed since heldAt: 2026-09-07T22:27:17Z.

   2. The trigger and verdict are still exactly as reported, ~1.5 h after the window this issue measured: lastCheckTrigger: BuildCompletion, and the verdict is the "updates are disabled" line. So the loop is live now, not a past episode.

   3. The work is provably pointless in this state. lastCheckedAt (07:39:42Z) has moved thousands of times since checkedAt (2026-09-07T22:26:44Z) and heldAt (22:27:17Z) — the two fields that record an actual decision — have not moved at all. Every one of those writes re-derived the same answer: policy is None, nothing to do.

   🚨 Worth noting for whoever fixes it: version is also the thing a reader would use to spot this from outside, without Loki. A policy node whose version climbs by thousands while its heldAt/checkedAt stand still is the signature, and it is one cheap read.
3. rbuergi commented on Sep 9, 2026

   

   rbuergi

   on Sep 9, 2026

   ContributorAuthor

   More actions

   Fixed by #3795, merged to main.

   Under Admin/UpdatePolicy = None the two fleet-paced triggers (BuildCompletion, ModuleSetProposed) are no longer decision points, so they run no check — while Startup, PolicyChange and the safety net still do. On the measured window that takes a pinned install from ~158 checks in five hours to about five, and with it the [MergeGuard] refusals, the OWNER_NACK_REENQUEUEs and the ten-second [UpdateQueue] FAILED stalls, which were all products of the rate.

   Defect 2 as filed — "lastCheckedAt is a per-replica observation written into a shared node by every replica" — is deliberately not changed, and the PR says so: the field describes "this install checked", not "this pod checked", so the shared field is the right model, and a merge guard refusing the older of two concurrent stamps is the framework being right. What made it a storm was the rate.

   Pinned by SelfUpdateChecksOnlyAtDecisionPointsTest — a 15-case truth table over every trigger × every policy, plus the same build-completion event pushed through the same seam under two policies (not a check under None, a check under Continuous). Falsified: removing the filter fails the acceptance test in 298 ms while the other 18 cases still pass. Documented in Doc/Architecture/SelfUpdateTargetSelection §5 and a What's New entry.
4. rbuergi

   closed this as completedon Sep 9, 2026