Retention: the locks hold nightly — apply the decided 30-day window to the paused purge-old-images, read production back, and inventory bundles/active builds as consumers

Current delivery scope — 2026-09-09

This remains the end-to-end retention issue. The platform owner's #3842 rule supersedes the historical pin-based prescription below: module builds resolve released platform sets at run time, same-major adopted fallbacks may keep serving, red/cancelled successors must remain invisible, and readiness is announced. Retain unreferenced continuous artifacts for at least 30 days by age, without a build-count quota; supported official releases follow the underlying .NET lifetime, with plugin counterparty support also required.

The remaining work is split into independently verifiable pieces:

Retention: require a complete and fresh fleet inventory before releasing artifact protection #3858 — complete and fresh fleet inventory, including actual adopted fallback identities. Core fix: retain builds reported as adopted by remote portals #3849 / Plugins#1564 implement the reporting seam; deployment and full fleet coverage still need verification.
Retention: preserve the full artifact set required by supported releases and advertised fallbacks #3860 — resolve and preserve every artifact needed by supported releases, last-green advertised sets, active builds and consumers.
Retention: interlock cleanup with protection updates and active publication #3859 — coordinate protection and cleanup so failed collection or concurrent publication cannot expose required artifacts to deletion.
Verify the 30-day policy on production consumers and the final deployed cleanup task. Core fix: retain bundles by age and released major instead of build count #3843 and fix: enforce a thirty-day assembly-cache retention floor #3846 are merged; that alone is not production-behavior verification.

Keep this parent open until the production task definition and reference-protection behavior have been read back and verified. The split does not authorize destructive cleanup, additional access or an architectural choice without the maintainer's involvement. #3461 remains the separate publisher-versus-publisher overlap issue.

The original incident and previous investigation follow unchanged as historical evidence.

What happened

MeshWeaver.Education main pins the tester image by digest:

mw-plugin-test@sha256:df19f10a…

ACR no longer has it. Every run of that repo's Disposable-mesh gate dies at manifest unknown
before a single content gate executes — including main's own nightly (run 1013). The repo can
merge nothing: Disposable-mesh gate is one of four required contexts in ruleset 19153714
("main pr protection"). Education#250 is the pin bump and is itself blocked on an unrelated wall, so
the repo has been fully wedged.

Root cause

There are two ACR purge tasks on the meshweaver registry, and only one of them learned the
lesson from the last incident.

purge-old-ci-releases (0 4 * * *) — hardened after Memex#122, and it says so in the task
itself:

# memex-website is DELIBERATELY absent from these filters (Memex#122, 2026-08-26).
# … The test for adding any repository here: if this tag were deleted tonight, would anything
# recreate it? The repositories below are continuously republished. A pinned tag is not.
acr purge --filter 'memex-migration:.*' --filter 'memex-bake:.*' --filter 'memex-portal-next:.*' \
          --filter 'memex-portal:.*' --filter 'memex-portal-ai:.*' --ago 30d --untagged

purge-old-images (0 3 * * *) — no comment, no reasoning, and it is the one that bites:

acr purge --filter 'mw-plugin-test:.*' --filter 'memex-migration:.*' --filter 'memex-portal-ai:.*' \
          --filter 'memex-portal-next:.*' --filter 'memex-bake:.*' --ago 7d --keep 10 --untagged

🚨 Why the existing rule does not cover this — a SECOND failure mode

Memex#122's test is "if this tag were deleted tonight, would anything recreate it?" — and
mw-plugin-test passes it. It is republished many times a day; it currently carries latest,
main, 3.0.0-ci.7903, 8358471, 1f652b7 and several staging-… tags. By that test it is a
perfectly legitimate purge target.

It broke anyway, because the danger is not "nothing rebuilds this repo". It is:

Something PINS a specific digest of it.

The whole fleet pins this image by digest — MW_IMAGE_DIGEST and MW_PORTAL_IMAGE_DIGEST in every
satellite's ci.yml, moved deliberately as one set and left stable for weeks at a time. That is the
build process working as designed (Doc/Architecture/ModuleBuildArchitecture): a pin is supposed to
be stable. --ago 7d --keep 10 guarantees that any pin older than about a week, or displaced by ten
newer builds, refers to bytes that no longer exist.

Republishing frequency does not protect a pinned digest — it is what destroys it, because
--keep 10 is measured in newer builds.

So the retention rule needs a second clause:

(Memex#122) If nothing would recreate this tag, never age-purge it.
(this issue) If anything PINS a digest of this repository, the purge must not delete what is
pinned — regardless of age or how many newer builds exist.

Suggested fix

Reconcile retention against the live pin set rather than guessing an age. The pins are readable:
MW_IMAGE_DIGEST / MW_PORTAL_IMAGE_DIGEST in each satellite's .github/workflows/ci.yml
(MeshWeaver.Plugins, .Education, .Reinsurance, .SocialMedia, .Manufacturing) plus whatever core and
Memex pin.

Options, roughly in order of preference:

Protect pinned digests explicitly — a job that reads every satellite's pins and locks those
manifests (or writes them into a keep-list the purge honours). Self-maintaining; a pin move
releases the old one on the next run.
A guard that reds when a pin no longer resolves in ACR — cheap, and it turns an
entire-repo wedge into one red check naming the repo and the digest. This is worth doing
regardless of which retention fix lands, because it catches the next variant too.
Raising --ago/--keep is not a fix — it moves the cliff, and a pin left stable for a quarter
walks off the new one just the same.

🚨 Whatever lands must not depend on a human remembering to add a repository to the right one of two
tasks. The two tasks having diverged is itself the defect; consider whether there should be one.

The triage trap, worth keeping

From Memex#122 and it applies here too: the purge run and the breakage are not adjacent in any
log. Pods keep serving from their nodes' image cache long after the delete, so the failure surfaces
whenever the next pull happens — a reschedule, or in this case a fresh CI runner. Do not expect the
ImagePullBackOff / manifest unknown timestamp to be near the purge.

Evidence

az acr task show --registry meshweaver --name purge-old-images — the filter list above
az acr task show --registry meshweaver --name purge-old-ci-releases — the hardened one
az acr repository show-tags --name meshweaver --repository mw-plugin-test --orderby time_desc
→ latest, main, 1f652b7, 3.0.0-ci.7903, 8358471, staging-1f652b7-34040888432(+arch variants)
MeshWeaver.Education main: pin mw-plugin-test@sha256:df19f10a…, run 1013 (nightly) red at
manifest unknown
Prior incident: Memex#122 (2026-08-26) — memex-website repository deleted entirely, public brand
site 503 for ~11 h, nothing alerted

Blast radius, measured — three repos went down simultaneously

From meshweaver-bd, who was working the resulting wedge:

the purge took Education, Manufacturing and SocialMedia down simultaneously on 2026-09-05T15:29Z

That is three satellite repos' CI dead at once from one retention run — and, as recorded above, in
Education's case including main's own nightly, because Disposable-mesh gate is a required context
and it dies at manifest unknown before any content gate executes.

This is not a historical problem — it is about to recur, and the fix PR is the victim

Education#250's whole wall-1 fix is a pin bump to staging-bbcb22f-34015096490-linux-x64, chosen
deliberately because its plugins seal is self-consistent (0 declines, against 5 from the previous
candidate). With purge-old-images running --ago 7d --keep 10 over mw-plugin-test:.* nightly,
that new pin walks off the same cliff in about a week — and Education goes red on
manifest unknown again, which is precisely the failure #250 exists to fix.

So the retention policy will undo the fix for the retention policy's last victim. That is the
argument for treating this as more than tidy-up.

Why the priority is higher than "a CI annoyance"

It reds an entire repository's CI including its own nightly, so there is no green baseline left
to compare against — the repo loses its control arm at the same moment it loses its gate.
It is silent until the next pull: pods and runners serve from cache long after the delete, so
the purge run and the breakage are never adjacent in any log (Memex#122's lesson, and it applies
identically to a CI runner pulling a tester image).
The satellites' pins are supposed to be stable for weeks — that is the design
(Doc/Architecture/ModuleBuildArchitecture, pins move as one set, deliberately and rarely). The
retention window and the pin lifetime are in direct contradiction, and nothing reconciles them.

Minimum viable mitigation, if the full fix is not immediate

The cheap half is the guard, not the policy: a check that reds when a repo's MW_IMAGE_DIGEST /
MW_PORTAL_IMAGE_DIGEST no longer resolves in ACR. It does not stop the deletion, but it converts
"an entire repo's CI is inexplicably dead, including its nightly" into one named red saying which
digest is gone — which is the difference between minutes and the several hours this cost today.

The guard half is up: #3462 — and the blast radius is wider than the tester

Two of Education's pins beyond MW_TEST_DIGEST are also gone. Measured against the live
registry, 2026-09-06:

GONE  memex-portal-ai@sha256:dab2a7b3a7e1…  ci.yml:MW_PORTAL_DIGEST
GONE  memex-migration@sha256:d81df6cc79ad…  ci.yml:MW_MIGRATION_DIGEST
GONE  mw-plugin-test@sha256:df19f10afc1f…   ci.yml:MW_TEST_DIGEST

All three repositories are in purge-old-images's filter list, so one task took the whole set. The
pin bump has to move all three, not one.

Fleet sweep, with the denominator printed — 8 scanned · 6 pin · 2 do not (core, Memex) · 12 pins
· 9 resolve, 3 gone, 0 indeterminate, 0 malformed.

🚨 #3454's "SocialMedia, Manufacturing, Crm — 0 matches" is a zero with two causes. All three DO
pin: Manufacturing as MW_IMAGE_DIGEST, SocialMedia and Crm inside a $GITHUB_OUTPUT step output
(echo "image-digest=sha256:…"), which no scan of env: blocks can see. With Education's
MW_PORTAL_DIGEST / MW_MIGRATION_DIGEST / MW_TEST_DIGEST, a check keyed on the two common
variable names reports four of the six pinning repositories clean while verifying none of their
pins. The guard extracts by shape, not by name, and prints both halves of the denominator so a
zero cannot hide its cause.

The retention half: cloud-only, so there is no IaC PR to open

Established three ways rather than assumed:

az acr task show returns an EncodedTask with contextPath: null for both tasks — the YAML
is base64 inside Azure, with no source repository.
An org-wide code search for acr purge finds only two MeshWeaver doc pages.
The deployment repo's tree carries no bicep, terraform or task definition; core's
deploy/aks/infra/main.bicep treats meshweaver.azurecr.io as an existing
sharedAcrLoginServer and only creates a registry when that is empty.

So changing retention is a hand-applied change to shared registry infrastructure. Two findings for
whoever applies it, both measured:

acr purge skips locked manifests by default. --include-locked is the documented opt-in to
delete them, and neither task passes it — so
az acr repository update --image <repo>@<digest> --delete-enabled false is a hard protection
against this exact configuration, not a hint. That makes "protect the live pin set" implementable
without changing what the purge deletes in general.
The 30-day task is a no-op for four of its five repositories. purge-old-ci-releases (04:00,
--ago 30d) names memex-migration, memex-bake, memex-portal-next, memex-portal,
memex-portal-ai; purge-old-images (03:00, --ago 7d --keep 10) already swept all of those an
hour earlier except memex-portal. The only repository unique to the aggressive task is
mw-plugin-test — the one that broke. One task, one filter list, one window removes the
"which of two lists?" question by construction.

The exact az commands, the merged task YAML with the reasoning kept in-task, and the argument for
why raising --ago/--keep is not a fix are in the doc page #3462 adds:
Doc/Architecture/PinnedImageRetention.

Leaving this issue open: the guard is the cheap half, the retention change is outstanding and is
the maintainer's to apply.

docs(#3438): name where the pin extractor stops, so an unseen shape i…

fix(#3438): the sweep's own fleet list could lose a page, and "pins" …

docs(#3438): merging the two purge tasks must not tighten memex-porta…

docs(#3438): two purge steps share one task timeout budget, so say so

fix(#3438): an ABSENT repository read as one with no pins, and the sw…

Guard half MERGED — and its first run on `main` reds naming this issue's three digests

#3462 merged as 9dfce19c0, verified on origin/main (the merge queue does not fire Closes
links). The push trigger fired the guard's first real run immediately, against the live registry:
run 34055176037 — and it is
RED, which is the guard working, not a defect.

Assert the inputs exist          success     <- preflight: every input asserted, OIDC + App token OK
Every pinned digest still resolves  FAILURE   <- the sweep, naming what is gone

The denominator, as printed in CI:

    repositories scanned                  34
    …declaring at least one digest pin    6
    …declaring NO digest pin              28
    …whose workflows could not be read    0
    pin DECLARATIONS found                15
    …over how many DISTINCT digests        7
    …declarations that RESOLVE            12
    …declarations that are GONE            3
    …INDETERMINATE (registry not reached)  0
    pin-shaped declarations MALFORMED      0

  GONE  memex-migration@sha256:d81df6cc79ad…  ci.yml:MW_MIGRATION_DIGEST
  GONE  memex-portal-ai@sha256:dab2a7b3a7e1…  ci.yml:MW_PORTAL_DIGEST
  GONE  mw-plugin-test@sha256:df19f10afc1f…   ci.yml:MW_TEST_DIGEST

Both arms proven on the real fleet in one run: the dead arm names Education's three with repository,
declaration and digest; the live arm resolves 12 declarations including
mw-plugin-test@sha256:642f686f… (3.0.0-ci.7917), the wave's new pin. 34 repositories scanned,
not a committed six — the fleet list comes from the App installation, which is org-wide.

🚨 Education's bump must move all three, not just the tester #3438's title names.

Four defects were found and fixed by falsifying the guard rather than reading it

Each is the same shape as the incident: not checked spelled identically to clean.

An absent repository read as one with no pins, and the sweep went green. Listing
.github/workflows answers 404 both for "no workflows" and for "no such repository / no access".
Measured: --repos …,Systemorph/ThisRepoDoesNotExist12345 printed "0 workflow file(s), no digest
pin declared" and exited 0. The repository is now probed before any absence is believed —
absent ⇒ UNREADABLE and red; Systemorph/iac (real, no workflows) still a measured zero.
The fleet list could silently lose a page — gh api --paginate output was re-parsed by
splitting text on braces, swallowing the decode error. Now --jq '.repositories[].full_name'.
"distinct pins" counted declaration SITES. Declarations and distinct digests are now counted
and labelled separately (15 vs 7 above — the fleet pins few manifests from many places).
🚨 The proposed retention merge would have tightened memex-portal. See below.

The retention half — still OPEN, still the maintainer's to apply

Cloud-only, re-established independently: both tasks are EncodedTask with contextPath: null;
org code search for purge-old-images returns 0 hits and "acr purge" only two doc pages here;
Systemorph/iac holds just a README; deploy/aks/infra/main.bicep treats meshweaver.azurecr.io
as an existing sharedAcrLoginServer and builds a registry only when that is empty. There was no
IaC PR to open.

🚨 The correction that matters before anyone applies it. The two filter lists differ by exactly
two repositories and they are not symmetric: mw-plugin-test is unique to the 7-day task (the
one that broke), but memex-portal is unique to the 30-day task and is the production portal
image. The obvious merge — union the filters, keep the stricter window — moves memex-portal to
7d --keep 10 with nobody deciding to. And this guard cannot cover it: memex-portal is pinned
on the other axis, committed image TAGS in the deployment overlays (Memex#141), which a scan of
.github/workflows never enumerates. That is exactly how Memex#122's victim was pinned.

So the proposed task in Doc/Architecture/PinnedImageRetention is behaviour-preserving: one
task, two steps, each keeping the window that repository already has. It removes the second
task, not the second window. The lock set is correspondingly the union of both pin axes.

Also confirmed against upstream Azure/acr-cli: acr purge skips locked manifests by default
and --include-locked is the opt-in to delete them — neither task passes it, so
az acr repository update --delete-enabled false is a hard protection under this exact
configuration. And raising --ago/--keep is explicitly not a fix: it moves the cliff.

Leaving this issue open. The guard is in; the retention change is a hand-applied change to
shared registry infrastructure and is not mine to make. Exact commands, the merged task YAML with
its reasoning kept in-task, and the timeout caveat (two steps share one 3600 s task budget) are all
in Doc/Architecture/PinnedImageRetention.

fix(#3454): the fleet set-consistency arm must not sit dead behind th…

First real run: 34068789826 — RED, naming this issue's three digests

The merge's push fired the job immediately, in report-only mode as designed. Every arm behaved
as specified, against the live registry:

mode: report-only (event: push)

    AXIS 1 — digest pins in .github/workflows
      repositories scanned                     34
      …declaring at least one digest pin        6  (Crm, Education, Manufacturing, Plugins, Reinsurance, SocialMedia)
      …declaring NO digest pin                 28
      …whose workflows could not be read        0
      digest pin SITES found                   20
      …UNCLASSIFIED (I5 — not checked)          0

    AXIS 2 — image TAG pins in deployment overlays
      repositories scanned                     34
      …whose tree could not be read             0
      tag pin SITES found                      11

    UNION — manifests the fleet pins            REGISTRY
      distinct manifests wanted        12         manifests in the registry     2846
      …already protected                2         …currently locked                5
      …TO LOCK                          7         …locked and pinned by NOTHING    3
      …that could NOT be protected      3

  release arm: DISABLED — MW_ACR_RELEASE_UNPINNED is '', not 'true'.

Assert the inputs exist → success (every secret present, App token minted, fleet enumerated).
The purge still skips locked manifests → success (2 tasks, 3 purge steps, none passing
--include-locked). Lock every manifest the fleet pins → failure, on exactly three lines:

Systemorph/MeshWeaver.Education ci.yml:45 `MW_PORTAL_DIGEST`    memex-portal-ai@sha256:dab2a7b3…  does not exist
Systemorph/MeshWeaver.Education ci.yml:46 `MW_MIGRATION_DIGEST` memex-migration@sha256:d81df6cc…  does not exist
Systemorph/MeshWeaver.Education ci.yml:64 `MW_TEST_DIGEST`      mw-plugin-test@sha256:df19f10a…   does not exist

That is the job working, not a defect — it is this issue, and it clears when Education's bump moves
all three. It will red daily alongside the 05:20 guard on the same fact, by construction.

The OIDC credential's READ half is confirmed working — the control probe enumerated 2,846
manifests. Only the WRITE is untested, and it is untested because this run was report-only. The
first apply run (01:00 UTC, or a workflow_dispatch) is the decisive measurement of whether the
Container Registry Repository Writer grant is needed; on the role evidence above it is.

Note the numbers are live: 2,831 → 2,837 → 2,846 manifests across three readings tonight. Every
reading is a snapshot of a moving registry, which is exactly why a hand-maintained keep-list cannot
be right for long.

🚨 The blast radius is wider than a pinned digest — a MOVING tag is outside #3507's model entirely

This issue documents a pinned digest being purged (mw-plugin-test@sha256:df19f10a…, the Education wedge). Tonight produced a second shape the fix does not cover, and it is the more exposed one.

What happened, 2026-09-07 03:00Z

The purge-old-images task removed memex-portal-ai:latest, and every MeshWeaver.Plugins CI run began failing at preflight:

##[error] 'meshweaver.azurecr.io/memex-portal-ai:latest' returned no digest — the bake has
         no reference set. The tester (mw-plugin-test:latest) resolved, so the promoted set
         is incomplete

Attribution is not inferred — the same head e8f315bf9 ran green at ~01:0xZ (run 34071193971) and red at 03:26Z (run 34079642842). Same commit, same workflow, opposite outcome, with the 03:00Z purge in between.

The state is worse than one tag — four of five

repo	`:latest`
`mw-plugin-test`	2026-07-20T00:30:38Z — the only survivor, seven weeks old
`memex-portal-ai`	ABSENT
`memex-migration`	ABSENT
`memex-portal-next`	ABSENT
`memex-bake`	ABSENT

🚨 The survivor is the anomaly, not the casualties. mw-plugin-test:latest survives because that repository republishes rarely enough that ten newer builds have not accumulated since July. It is one busy week from the same fate — and when it goes, both halves of the preflight's pair vanish and the failure changes from "the promoted set is incomplete" to no reference set at all.

Why #3507's job does not cover this

lock-pinned-digests.py derives its set from literal pins — MW_IMAGE_DIGEST, MW_PORTAL_IMAGE_DIGEST, platform-image-digest:, deployment overlays. A moving tag is named by nobody, so it appears in no extraction axis and is locked by nothing. The job is correct about what it models; the model has a hole shaped exactly like latest.

And a moving tag is structurally more exposed than a pinned one: --keep 10 counts newer builds, so every republish spends its own budget. memex-portal-ai now holds 788 manifests. Tonight's CD burst — sets 7962, 7964, 7967, 7969 plus three staging tags each — consumed the keep window in a single evening. 🚨 The mechanism gets worse precisely when the fleet is most active, which is the opposite of when you want retention to bite.

Ownership of the cause, stated plainly

The two-step purge-old-images task with --ago 7d --keep 10 over memex-portal-ai:.* was applied earlier the same night, by me, as this issue's remedy. Its own header predicted this outcome — "--keep 10 counts NEWER BUILDS, so frequent republishing is what DESTROYS a pin rather than protecting it" — and I protected pinned digests with --delete-enabled false while leaving moving tags outside the lock model. The eviction is the first bill for that gap, not a surprise.

What is deliberately NOT done

🚨 No retag. Restoring memex-portal-ai:latest to a September image would pair it with a July tester in whatever resolves that "promoted set" — the image-pair skew class that 503'd every signed-in request on 2026-09-03. Incomplete fails loudly at a preflight; inconsistent fails at a signed-in request. It is a fleet-wide reference-set decision and it belongs to the maintainer, with two parts:

Is latest meant to exist on these repos at all? In practice it exists on one of five, by accident of quietness. If the preflight requires it, the preflight asserts a convention nothing maintains.
If it is meant to exist, all five move together to one promoted set — never one tag to a September image beside a July one.

🚨 Do not raise --ago / --keep. The task's own header says it moves the cliff rather than removing it. The durable fix for a moving tag is the same as for a pinned one — --delete-enabled false, which acr purge skips and neither step passes today — plus teaching #3507's extractor that a tag a lane resolves is as much a pin as a digest a file names.

The purge took `memex-portal-ai:latest` at 03:00Z and blocked every Plugins CI run — fixed by leaving `latest` behind

Recording this here because the remedy was a repo variable, which lives outside git and is otherwise invisible to anyone reading the workflow.

What broke

03:26Z  Plugins main run 34079642842 FAILURE
  ##[error] 'meshweaver.azurecr.io/memex-portal-ai:latest' returned no digest —
            the bake has no reference set. The tester (mw-plugin-test:latest) resolved,
            so the promoted set is incomplete

Attribution is not inferred: the same head e8f315bf9 ran green at ~01:0xZ and failed at 03:26Z. Same commit, same workflow, opposite outcome, with the purge at 03:00Z in between.

The blast radius is wider than this issue documents

This issue records the tester digest (Manufacturing/SocialMedia, 2026-09-05T15:29Z). Measured across all five filtered repositories:

mw-plugin-test:latest      2026-07-20T00:30:38Z   ← the ONLY survivor, seven weeks old
memex-portal-ai:latest     ABSENT
memex-migration:latest     ABSENT
memex-portal-next:latest   ABSENT
memex-bake:latest          ABSENT

🚨 Four of five. And the survivor is not safe — it survives only because that repository republishes rarely enough that ten newer builds have not accumulated since July. It is one busy week from the same cliff, and when it goes, both halves of the preflight's pair vanish and the failure changes from "incomplete set" to "no reference set at all".

🚨 A MOVING tag is more exposed than a pinned one, which is the opposite of the intuition the lock work is built on: every republish spends its keep budget. Tonight's four CD sets plus three staging tags each consumed --keep 10 in a single evening. #3507's job locks pinned digests only; a moving latest is outside its model entirely.

The fix, and why it is not a retag

The preflight derives the portal image from MW_TEST_IMAGE's tag — "Same registry, same tag, sibling repository". So the two must be one matched set. Retagging memex-portal-ai:latest to a September build beside a July tester would have paired two halves from different builds — the image-pair skew class that 503'd every signed-in request on 2026-09-03. Incomplete fails loudly at a preflight; inconsistent fails at a user's request.

Instead, MW_TEST_IMAGE now names a matched, promoted, verified-sealed set:

- meshweaver.azurecr.io/mw-plugin-test:latest
+ meshweaver.azurecr.io/mw-plugin-test:3.0.0-ci.7969

Both halves resolve (2 platforms each, sha256:c0940ed2… / sha256:5899d18d…), and :latest still does not — the negative control.

🚨 7969 was chosen by checking the seal, not by taking the newest. #7967 is newer than 7964 and failed its seal (failure,skipped,success across promote/verify/seal), so "latest version tag" would have been the wrong pick. 7969: run success, promote/verify/seal all success.

What this does not fix

latest is still absent on four repositories and still unmaintained by CD. Whether it should exist at all is a maintainer call — in practice it exists on one of five.
Pinning means Plugins CI no longer tracks new releases automatically; the variable must be bumped like any other pin. That is consistent with how the rest of the fleet pins, and it is why this is recorded rather than left silent.
Do not raise --ago/--keep. The task's own header says it moves the cliff; the durable fix is --delete-enabled false, which acr purge skips and neither step passes today.

Both questions about the lock job's red are settled by measurement — and neither wanted a change. What DID want one is the read-back (PR #3563)

1. The red was CORRECT and it has already cleared itself

Education's main sat at 53f0a65df (2026-09-04T12:12:07Z) until d622e7dc9 merged PR #274 at
2026-09-07T05:52:00Z — d622e7dc9's first parent is 53f0a65df, and that commit's
ci.yml:45/46/64 carries exactly the three purged digests:

53f0a65df  ci.yml:45  MW_PORTAL_DIGEST     sha256:dab2a7b3…
           ci.yml:46  MW_MIGRATION_DIGEST  sha256:d81df6cc…
           ci.yml:64  MW_TEST_DIGEST       sha256:df19f10a…

So at 00:07Z (push) and 01:18Z (schedule) the default branch really did declare three manifests that
no longer exist. I ran the extractor live against Systemorph/MeshWeaver.Education at ~08:20Z today
and it now returns the three you verified as locked:

ci.yml:99   MW_PORTAL_DIGEST     memex-portal-ai   sha256:47dc1750…
ci.yml:100  MW_MIGRATION_DIGEST  memex-migration   sha256:24ea2fc6…
ci.yml:118  MW_TEST_DIGEST       mw-plugin-test    sha256:4cef7b67…

🚨 The trap that made this look like a scanner bug: the bump commit 4c471c2f3 is dated
2026-09-07T00:31:19Z, before the 01:18Z scheduled run — but that is its authoring date on the
branch fix/246-platform-pin, not the moment it reached main. commits?path=… walks all parents
and orders by date, so it interleaves branch commits with trunk ones and reads as a trunk history that
already had the fix. The merge commit's first parent is what actually answers.

2. "Read the default branch rather than every branch" — it ALREADY does, and changing it would be harmful

check-pin-set-consistency.py:scan_remote calls repos/{repo}/contents/.github/workflows with no
?ref=, and the GitHub contents API defaults to the repository's default branch. So the stale
dependabot branches on Education #277/#278/#279 were never in the picture — no branch but main has
ever been scanned.

And it should stay that way. Reading every branch would be wrong in both directions: red forever
on a commit nobody will merge, and — the worse one — a manifest that nothing live pins entering the
lock set on the strength of an abandoned branch, i.e. the derivation protecting bytes for a reason
that no longer exists. The lock set is only sound because its input is what the fleet actually runs.

3. "It should name the stale pin and which repo declares it, not just die" — it already does

Verbatim from run 34072578951:

##[error]Systemorph/MeshWeaver.Education ci.yml:45 `MW_PORTAL_DIGEST`:
         memex-portal-ai@sha256:dab2a7b3a7e1… does not exist in the registry, so it CANNOT be
         protected. Every job that pulls it dies at `manifest unknown` before it runs anything
         (MeshWeaver#3438). Move the pin to a manifest that exists — as one set.

Repository, file, line, variable, digest, consequence and remedy. It also prints
UNPROTECTABLE … in the report body with every declaration site. Locks are applied even on a
degraded run, so those three never blocked the protection of anything else.

I considered splitting the exit into "a pin is stale" vs "this job could not check", so a chronic
stale pin cannot bury a new blocker. I did not do it: the script's 28-sabotage self-test asserts
against plan.blockers in seven arms, it runs on every PR in the fleet, and the change would buy a
report distinction the log already makes while risking a red across every C#-touching PR. If a pin
goes stale for weeks rather than days, that trade is worth revisiting — recording the reasoning rather
than the conclusion.

What DID want a change, and it is load-bearing right now

The job does exactly one thing: set deleteEnabled: false. It believed az's exit code that it
had:

rc, _, err = az(["acr", "repository", "update", …, "--delete-enabled", "false", "-o", "none"])
return (rc == 0, …)          # the only evidence the job had

An exit code is a statement about the request. A write that returns 0 without the attribute moving
leaves the job printing locked … per manifest and Protected N manifest(s) at the end, over
manifests the 03:00 purge can still delete — and the workflow header already claimed it "fails closed
on … a lock that did not take". Nothing enforced it. That is the same failure this whole job exists to
remove, pointed at itself.

PR #3563 makes apply_locks read deleteEnabled back and require false; true is red, and an
INDETERMINATE read-back is not a confirmation. Two --self-test arms drive both cases and were
falsified by deleting the read-back (5 lock(s) were COUNTED although none took).

🚨 This matters now because the WRITE half has never actually run — both runs so far were
report-only or blocked. The first apply run is the decisive measurement of whether locking works at
all, and it must not be able to report success without having achieved it.

Still yours, unchanged and listed rather than invented

The RBAC grant. az acr repository update needs
Microsoft.ContainerRegistry/registries/repositories/metadata/write — in
Container Registry Repository Writer (and Contributor/Owner), in neither AcrPull nor
AcrPush. The registry's three service principals hold only AcrPull, AcrPush and
Container Registry Data Importer and Data Reader (metadata/read only). Until it is granted the
lock step reds once, naming the exact az role assignment create.
The latest-tag retention policy. Four of five filtered repositories have no :latest; the
survivor (mw-plugin-test, 2026-07-20) survives only by being quiet. Two parts, both yours:
is latest meant to exist on these repositories at all — in practice it exists on one of five,
by accident — and if so, all five move together to one promoted set, never one tag to a
September image beside a July one. A moving tag is also outside Every pinned manifest is locked nightly, so retention cannot delete it (#3438 lock half) #3507's lock model entirely, since
no file names it.

🚨 Not proposed, and it would not have worked: --delete-enabled false is not an acr purge
flag. It is a manifest attribute set by az acr repository update, which is exactly what #3507's job
already does — and the purge tasks' own header says so: "PROTECTION IS THE MANIFEST LOCK, not this
policy … acr purge skips locked manifests by default and neither step passes --include-locked."

_{Measured 2026-09-07: the extractor run live against Education, the commit ancestry via compare and the merge commit's first parent, and the run log of 34072578951. No registry mutation performed.}

STAYING OPEN — and Memex#219 does not cover either remaining half

Checked 2026-09-07T12:25Z during a sweep of merged-but-open issues. The hypothesis worth killing
here is "the code half merged and the retention half is tracked on Memex#219, so this is closable".
It is not, and this note exists so nobody closes it on that reading.

The code half is genuinely merged. #3563, merge commit b37d2079d on origin/main
(fbf9e9be1) — apply_locks reads deleteEnabled back and requires false; true is red, and an
indeterminate read-back is not a confirmation.

What Memex#219 actually says (read in full today, state = open, created 2026-09-07T07:35:07Z):
it asks for "a retention policy that cannot delete a tag any committed overlay or any live workload
references — or, if that cannot be expressed, keeping versioned memex-* tags longer than the
current window." That is the versioned-pin problem, split out of Memex#141.

Neither of this issue's two remaining halves appears in it, nor anywhere else in Memex.
Denominator: all 13 open issues in Systemorph/Memex (#116, #119, #132, #148, #152, #165, #172,
#188, #204, #219, #220, #221, #228), titles and bodies searched for
Repository Writer, metadata/write, deleteEnabled, latest tag, :latest — zero hits.

So both remain untracked outside this issue:

The ACR RBAC grant. az acr repository update needs
Microsoft.ContainerRegistry/registries/repositories/metadata/write, which lives in
Container Registry Repository Writer (and Contributor/Owner) and in neither AcrPull nor
AcrPush. This is load-bearing right now: fix(#3438): the pinned-manifest lock reads the attribute back — an exit code is not a locked manifest #3563 just made the lock job refuse to report a lock
it did not achieve, and the write half has never run, so the first apply run reds on the missing
grant.
The latest-tag retention question. Four of five filtered repositories have no :latest; the
survivor (mw-plugin-test, 2026-07-20) survives by being quiet. Two decisions, both the
maintainer's — is latest meant to exist on these repositories at all, and if so do all five move
together to one promoted set. A moving tag is outside Every pinned manifest is locked nightly, so retention cannot delete it (#3438 lock half) #3507's lock model entirely, since no file
names it.

If those two are wanted on Memex rather than here, they need their own issue there — folding
them into #219 would attach a Container-Registry-RBAC action and a tag-hygiene decision to a
retention-window cost trade-off, which is how one of them gets lost.

_{Measured 2026-09-07T12:25Z: merge commit via git log origin/main --merges --grep on

fbf9e9be1; Memex#219 and the 13-issue open set read over REST

(repos/Systemorph/Memex/issues?state=open&per_page=100, page complete — no second page).}

The WRITE half works — first successful `apply` run, and the two-task question is settled

Two things this issue's thread predicted rather than measured are now measured, and both change what
is written above.

1. ✅ The lock actually locks. The RBAC grant landed and the first apply run succeeded

Every earlier run was report-only or blocked, and the page said so: "the write half has never
actually run". The scheduled run of 2026-09-08T01:14Z
(34176008757) is the first
apply run in the job's history to succeed, and it wrote locks:

mode: apply (event: schedule)

    UNION — manifests the fleet pins            REGISTRY
      distinct manifests wanted        18         manifests in the registry     2813
      …already protected               10         …currently locked               26
      …TO LOCK                          8         …locked and pinned by NOTHING   16
      …that could NOT be protected      0

Protected 18 manifest(s) (8 newly locked this run); 0 released.

0 that could NOT be protected is the first end-to-end measurement of the mechanism rather than of
its plan — and it is only evidence because #3563 made apply_locks read deleteEnabled back. Had
it still believed exit codes, that line would have been indistinguishable from 18 manifests the
purge can still delete.

Education's three unprotectable digests are gone from the report: its bump moved all three, to
memex-portal-ai@47dc1750…, memex-migration@24ea2fc6…, mw-plugin-test@4cef7b67….

🚨 …locked and pinned by NOTHING 16 is the standing debt of a lock-only job, and it grows every
time a pin moves. The release arm that would clear it stays disabled on purpose.

2. The second purge task — settled from the task definitions, not from a summary

az acr task show on both, decoding step.encodedTaskContent, 2026-09-08:

Task	`status`	`schedule`	Steps
`purge-old-images`	Enabled	`0 3 * * *`	5 CI repos at `--ago 7d --keep 10 --untagged`; `memex-portal` at `--ago 30d --untagged`
`purge-old-ci-releases`	Disabled	`0 4 * * *`	its old single 30-day step, kept as the record of the Memex#122 reasoning

So this issue's headline is settled, and the settlement is that there is no longer a second task to
teach. The divergence itself was the defect: mw-plugin-test appeared in the aggressive list and
in no other, and whether a new repository landed under a 7-day or a 30-day window depended on
which of two independently-edited lists someone happened to open. Both carry lastModifiedAt
2026-09-06T21:51Z, ninety seconds apart — the edit that collapsed them.

🚨 The committed record is not the registry. .github/acr-retention/*.yaml is a record; nothing
deploys from it, so "the record says Disabled" is not evidence the task is. The reading above was
taken by hand, read-only, against Azure.

The denominator, hand-counted — pins EXAMINED, not only pins at risk

A plain sha256: grep over every workflow file in all 34 org repositories, deliberately not
the shipped extractor, so the two are an independent cross-check:


repositories in the org	34
…with a `.github/workflows` directory	16
…declaring ≥ 1 digest pin	6 (Crm, Education, Manufacturing, Plugins, Reinsurance, SocialMedia)
`sha256:`-shaped tokens found	31
…inside COMMENTS (prose about past incidents)	11
pin declaration SITES	20 — exactly what the lock job reports
…over how many DISTINCT manifests	9
…that EXIST	9
…that are GONE	0
…that exist and are LOCKED	7
…that exist and are UNLOCKED	2 (both MeshWeaver.Plugins', pinned 10:46Z today)

Two things only a hand count shows. Eleven of thirty-one sha256: tokens are comments — several
of them narrating this incident inside the very files it broke — so any "count the digests"
instrument that does not exclude them over-reports by more than a third. And 20 sites collapse to
15 declarations in the shipped report, which dedupes on (name, digest) because both Plugins and
Crm write the same platform-image-digest: value at two lane calls. 9 distinct manifests is what
retention actually has to keep alive, and it is the number the two counts agree on.

The two unlocked ones are not a defect: tonight's 01:00 lock precedes tomorrow's 03:00 purge.

🚨 The gap that was still open, and #3736 closes it

Existence is a LAGGING indicator. By the time a pin reads GONE, the repository is already
wedged. Nothing was a leading indicator — and nothing asserted that the protection was still
protecting, from a process that is not the lock job itself. Each of these left a green wall until a
satellite died at manifest unknown days later: lock-pinned-digests.yml deleted, its schedule:
removed, the metadata/write grant revoked, a manifest unlocked by hand, the cron moved past the
purge. That is this issue's own failure mode one level up.

#3736 makes check-pinned-digests.py read changeableAttributes.deleteEnabled off every pin it
resolves — the same az call, -o json instead of -o none, no extra traffic — and classify
PROTECTED / UNPROTECTED / PENDING-LOCK / INDETERMINATE. Only the registry saying
deleteEnabled: false earns PROTECTED; "the field was not in the response" is INDETERMINATE and
red, because a protection that cannot be read has not been shown to be working.

It also asserts, credential-free and therefore on every pull request rather than only at 05:20,
that lock-pinned-digests.yml still exists with a schedule: and that its cron fires before
every enabled purge step recorded in .github/acr-retention/tasks.json. acr purge skips locked
manifests and that workflow is what locks them, so the order of those two clocks IS the
protection; move the lock to 04:00 and every pin moved since the previous night faces the purge
unprotected, with both jobs still reporting success about their own work.

Falsified rather than asserted, each watched to go red: the lock cron moved to 0 4 * * *; the lock
workflow deleted; two sabotaged schedules driven through the same comparison inside --self-test;
and the classifier made to answer PROTECTED for everything (five arms red, including
SABOTAGE UNDETECTED: an UNLOCKED pinned manifest classified as PROTECTED). The classifier is a
pure function precisely so every branch can be driven — the UNPROTECTED branch cannot be reached
live without unlocking a production manifest, which a test may not do.

PENDING-LOCK is an exemption and it is counted: a pin that moved after the last scheduled lock
has had no opportunity to be protected, so failing it would red on the fleet's ordinary working day.
The proxy softens, never false-alarms, and the exempted count is its own reported line so the arm
cannot go vacuous unnoticed. A missing lock schedule exempts nothing.

What stays open, and it is not code

The residual exposure is structural and is named rather than papered over. Protection is
derived on a daily cadence and the purge does not wait for it: the lock is a GitHub Actions cron,
the purge an ACR timer task, with no interlock between the two systems. So a pin landing in the
01:00–03:00 window meets that night's purge unprotected, and if the lock job does not run at all
(an Actions outage, a failed OIDC login, a revoked grant) the purge still runs at 03:00 regardless.

Neither is a widened window away from being fixed — raising --ago/--keep moves the cliff. Both
close only by making the pin move and its lock one act, which means a cross-repo dispatch from
each satellite and is not built. Until it is, #3736's protection arm is what makes that exposure
visible instead of silent.

And the latest-tag question is unchanged and still the maintainer's — four of five filtered
repositories have no :latest; a moving tag is named by no file, so it is outside #3507's lock model
entirely.

All of it is committed to Doc/Architecture/PinnedImageRetention, which is the durable form; this
comment is a pointer to it.

_{Measured 2026-09-08, read-only throughout: no purge run, no dry run, and no registry mutation of

any kind. az acr task show ×2, az acr repository show per distinct pinned manifest, the run log

of 34176008757, and the org repository list plus every workflow file over REST.}

Governed path merged; production is unchanged. Maintainer decision (2026-10-09): "#3438 registry retention: apply the decided 30-day window + re-enable purge-old-images through a GOVERNED path (InstanceAction on control; if none exists for registry tasks, build that action kind — never a hand az call), then read production back."

Merged:

core fix(retention): lock lane joins each installation's record; apply refuses behind a red lock lane; governed path recorded (#3438, #4230) #6379:
- Fixes the lock-pinned-digests lane. It failed nightly since 2026-10-03 (last run 37869805384) because memex and memex-cloud take memex-portal-ai from their deployment record, which the lane never searched; it now reads each installation's record.
- Adds a purge interlock. A purge can be switched on only when the newest lock-lane run on main is green and less than 36 h old. The repo, workflow and age are constants with no environment override.
- Policies registry-retention and fleet-registry-bounded.
Plugins fix(#3060): a layout area can name the viewer it is rendering for #3251: new action kind RegistryRetention (Hosting 1.62). A dry run is a read-only verify that needs no approval. A real run applies the record, and needs the deployment id typed back plus one approval.
Memex Package install: make re-install of an unchanged snapshot actually write ZERO #712: the aks-ops lane steps for that verify and apply. It also declares a Container Registry Tasks Contributor grant for hosting-operator in infra/modules/registry-tasks-grant.bicep.

Maintainer acts, in order (no agent approves a governed action):

Let control roll onto Hosting 1.62. RegistryRetention must appear as an action kind there.
As Owner, re-grant infra-rbac's Azure ABAC condition so that it may assign role fb382eab-e894-4461-af04-94435c366c3f (Container Registry Tasks Contributor) to hosting-operator. The exact command is in Memex docs/estate-least-privilege.md. Until then, that one step of InfraDeploy deploy fails by name.
On control, run InfraDeploy what-if, then deploy (second-admin approval).
On control, run RegistryRetention with dryRun: true. It reads back the live ACR tasks and the storage lifecycle policy, and needs no approval.
Run RegistryRetention for real (confirmation plus approval). This installs the 30-day window on purge-old-images, which stays disabled.
Once the lock lane is green on main (it is the interlock): make a reviewed change that lifts pause and sets purge-old-images to Enabled, then apply it again with approval, then read production back.

This issue stays open until step 6 has been read back from production.

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-07)

Lock half MERGED — and one production action stands between it and actually protecting anything

PR #3507 merged as deccceceef607ac9038c08a3a80cfc7d155b9482, verified on origin/main (the merge queue does not fire Closes
links). Requirement 1 — protect pinned digests explicitly, without depending on a human
remembering — is implemented, falsified and scheduled.

.github/workflows/lock-pinned-digests.yml runs at 01:00 UTC, two hours before the 03:00 purge,
derives the fleet's live pin set and locks every manifest it names. acr purge skips locked
manifests and neither step of purge-old-images passes --include-locked, so deleteEnabled: false
is a hard protection against this exact configuration.

🚨 LEAVING THIS OPEN. One grant is missing, and it is only the maintainer's to make

Locking is a WRITE. az acr repository update needs the data action
Microsoft.ContainerRegistry/registries/repositories/metadata/write. Measured 2026-09-07:

	holds `metadata/write`?
`AcrPull`	no
`AcrPush` (`pull/read` + `push/write`, no dataActions)	no
`Container Registry Data Importer and Data Reader`	no — `metadata/read`
`Container Registry Repository Writer`	yes

The meshweaver registry's three service principals hold only the first three, measured with
--include-inherited. So on the evidence the nightly lock will be refused until this lands:

az role assignment create --assignee <the AZURE_CLIENT_ID app's object id> \
  --role 'Container Registry Repository Writer' \
  --scope /subscriptions/<sub>/resourceGroups/meshweaver-shared/providers/Microsoft.ContainerRegistry/registries/meshweaver

The job reds once on refusal, naming that command, rather than repeating one error per manifest.
It is a provisioning task, not a workflow defect — and it is the reason this issue stays open.

What a real run would do, live, report-only, 2026-09-07

34 repositories, 2m19s, exit 1 (correctly — see below).

    AXIS 1 — digest pins in .github/workflows       AXIS 2 — TAG pins in deployment overlays
      repositories scanned              34            repositories scanned              34
      …declaring at least one pin        6            …carrying an overlay that pins     2
      …declaring NO digest pin          28            …whose tree could not be read      0
      …whose workflows could not be read 0            overlay files considered          15
      digest pin SITES found            20            tag pin SITES found               11
      …UNCLASSIFIED (I5)                 0

    UNION — manifests the fleet pins                REGISTRY
      distinct manifests wanted         12            manifests in the registry       2837
      …already protected                 2            …currently locked                  5
      …TO LOCK                           7            …locked and pinned by NOTHING      3
      …that could NOT be protected       3

Seven manifests are pinned and unprotected right now, four of them in repositories the 7-day step
sweeps: memex-portal-ai + memex-migration at 3.0.0-ci.7926 (what both live deployments run),
memex-portal-next:3.0.0-next.43, and the client-b overlay's 3.0.0-rc9.ci.7601 pair.

The three unprotectable ones are this issue — Education's memex-portal-ai@dab2a7b3…,
memex-migration@d81df6cc…, mw-plugin-test@df19f10a…, already deleted. Education's bump must move
all three.

🚨 The unlock half is DESIGNED and DOCUMENTED but SHIPS DISABLED, and the first live run showed why

Of the five manifests locked by hand on 2026-09-06, the job would have protected two and listed
the other three as release candidates — the staging-bbcb22f-… set that Education's pending
pin bump will name, locked pre-emptively. No derivation from the live pin set can see a pin
nobody has committed. An enabled release arm would have unlocked the fix for this issue's own
victim, on its first run, and --ago 7d would have finished the job.

So: locks are applied even on a degraded run (a lock cannot destroy anything); releases are computed,
printed and never acted on unless a maintainer sets MW_ACR_RELEASE_UNPINNED=true and the run
carries no blocker. A pre-emptive lock is a legitimate reason for a lock the job cannot derive.

Two things this issue's premise got wrong

memex-portal is not pinned by any deployment overlay. The merged task comment and the doc
justified the two-step split by saying it is (Memex#141). The overlays pin memex-portal-ai,
memex-migration, memex-portal-next and hosting-operator; an org-wide code search for
meshweaver.azurecr.io/memex-portal: returns zero hits and the repository's newest tag is
3.0.0-rc13. The split is still right on plainer grounds — tightening a low-churn production
image repository is a separate decision with its own evidence — but the stated reason is false,
and a false reason is how a future merge talks itself into the union. Corrected in the doc.
The overlay axis is not only memex-portal-shaped: 11 sites over 7 distinct manifests
including hosting-operator and whisper-swiss-german, and one of its two shapes (helm's split
repository:/tag:) is invisible to a single-line scan — the same "not seen reads as not pinned"
confusion, one layer down.

One thing to check before the next promotion

The five hand locks set both deleteEnabled: false and writeEnabled: false. Only the first
is what acr purge skips on; the second additionally refuses a manifest PUT — and release.yml
promotes by retagging (docker buildx imagetools create --tag $ACR/$repo:$VERSION $ACR/$repo:$SHORT),
which is a PUT of an already-present digest under a new tag. Two of the five
(mw-plugin-test@642f686f…, memex-portal-ai@31aab07d…) are the 3.0.0-ci.7917 wave — the set a
3.0.0 promotion would retag. Reasoned from the mechanism, not measured (confirming needs an
actual retag). --write-enabled true undoes it and leaves the purge protection untouched. The new
job locks delete only.

Also landed

The purge tasks are cloud-only (EncodedTask, contextPath: null, zero org-wide code-search
hits), so .github/acr-retention/ now records both decoded definitions byte for byte plus their
schedule/status/timeout, with .github/scripts/acr-retention-tasks.sh show/record/verify/apply
relating the record to the live registry. A credential-free gate on every PR refuses a purge step
that gained --include-locked — which would turn every lock into decoration while the job kept
reporting success.

28 sabotages were run against the three checks and each was watched to go RED. Two vacuous
assertions fell out of that and were fixed: the release arms asserted "nothing was released" against
a fixture with nothing to release (so deleting the guard left the suite green), and the retention
verifier's --include-locked grep fired on the task's own comment saying it does not pass it.

Doc: Doc/Architecture/PinnedImageRetention.

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-12)

Two of the three split issues are CLOSED and folded back here — #4081 landed the mechanism

🚨 Status, 2026-09-12 — the purge is PAUSED, so no date applies. Roland ordered it stopped
while protection was incomplete, and purge-old-images is disabled (az acr task list →
status: Disabled; Memex#219). The pause is the MITIGATION and #4081 is the FIX, which makes
#4081 the precondition for safely turning the task back on rather than a race against a deadline.
A disabled task still reports an Enabled timer trigger at 0 3 * * * — read the task's status.

#3858 and #3860 are closed; #3859 stays open, narrowed to the one thing that cannot be built in
this repository (the destructive schedule itself — see below). The four described one mechanism,
not four pieces of work: a cleanup whose
protection set is derived from a stale or incomplete source deletes something that is still in use.
The 2026-09-09 split called them "independently verifiable", and building them showed they are not
independently deliverable — a fresh, complete inventory is only meaningful once something decides
what INCOMPLETE means, and nothing can be interlocked against a decision with no completeness
verdict. So they were built as one change set, with the detail on each thread; this issue stays open
and is once again the single home for what is left.

What #4081 landed

AXIS 3 — ask the installations. Each one the deployment overlays declare is asked, through its
own unauthenticated /api/version, what it is RUNNING; the closure of image sets built from that
commit is protected in every repository its overlay pins. 🚨 The bare short-sha tag is a moving
tag (core 4c99ec26 produced two image sets ninety minutes apart and 4c99ec2 followed the newer
one), so the axis protects the closure and says so rather than guessing.
The TAG is locked beside the manifest. A tag carries its own deleteEnabled, and acr-cli
deletes a tag on the tag's. Measured: memex-portal-ai:3.0.0-ci.8372 — what both production
overlays pin — read true over a manifest reading false, and 0 of that repository's 1,396
tags were locked. The bytes were protected and the reference was not.
A locked INDEX is expanded to its platform manifests. acr-cli's locked-manifest continue
comes before the walk that ignore-lists an index's children, so a locked index is never walked
and its untagged children are collected out from under it. The corollary is worse: an unlocked,
tagged index is walked — so protecting the index alone made its children less safe.
Completeness is a verdict with a denominator, printed every run — installations expected /
answered / not / declared non-live, manifests, tag references, and the enabled purge windows it is
racing. An installation that did not answer is named, makes the run INCOMPLETE and refuses the
unlock arm.
The same contract in the portal. DeploymentPinnedReferences.Resolve takes the
Hosting/Deployment records as the expected-consumer denominator; missing, unreadable or stale
(> 24 h) refuses the pass naming the instance.
The zero-pin assertion is replaced by an instrument control in both scripts — which is what
reddened this lane on 2026-09-12, two hours before the purge, on a premise No platform pins in module repos — build against the released platform, and be TOLERANT of drift (same major serves) #3842 had expired.

What is left here, and it is deliberately not code

Retention: interlock cleanup with protection updates and active publication #3859 — the two clocks. The lock is an Actions cron at 01:00, the purge an ACR timer task
at 03:00, and nothing makes the second wait for the first. Tracked there rather than here,
because it has its own acceptance criteria. Nothing destructive was run for #3438: retention protects what is CONSUMED, and refuses when it cannot see it all #4081.
purge-old-images carries memex-portal-ai on its 7-day step — the image both production
portals run — while the 30-day step covers only the retired memex-portal repository.
The :latest question. A moving tag is named by no file and is outside this model.
The Container Registry Repository Writer grant, which is what lets the lane write at all.
Released module bundles and active builds as consumers — the two closure members Retention: preserve the full artifact set required by supported releases and advertised fallbacks #3860
named that are still not inventoried.
Production readback of the report receiver (Retention: require a complete and fresh fleet inventory before releasing artifact protection #3858's last criterion) — a verification on a
running control instance, not a code change.
client-b is an expected consumer that has never reported. Measured on the control instance
2026-09-12: three Hosting/Deployment records, all Active, none declaring retirement — and
client-b has never been installed. So the prebuilt-bundle pass refuses there until one field is
written (retired / retiredAt with a reason, in the deployments repo) or client-b is
installed. That refusal IS the mechanism working — "is client-b a consumer?" is the question
retention must answer before it deletes — and it costs nothing today, because the chart
disarms bundle deletion by default. The registry lane already carries the same declaration in
.github/acr-retention/instances.json; the two stores do not infer each other's answer.
A one-off state cleanup is owed and was NOT performed: 20 tag references are unlocked right
now, and the platform manifests under every locked index are unlocked. Tonight's scheduled run
locks them if it is allowed to; no hand command was issued.

Design: Doc/Architecture/ArtifactRetentionInterlock.

#4081 merged as 4764a6072e9aae9622d50413d9df3d1bdced5bfe.

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-13)

The 7-day step is closed, and it was DECIDED rather than a decision — #4213

This issue's open item "purge-old-images carries memex-portal-ai on its 7-day step" has been
carried on the thread and in Doc/Architecture/ArtifactRetentionInterlock as "whether that is the
intended policy is a decision, not a defect". It is not a decision. It was already decided, and
the registry was the one store nobody held to it.

#3842's rule, quoted verbatim in this issue's own body and in #3859's acceptance list:

Retain unreferenced continuous artifacts for at least 30 days by age, without a build-count quota.

Three stores implement it. Two were clamped to it in code and the third was asserted by nothing:

store	where the 30-day floor lives
prebuilt bundles	`PrebuiltBundleRetention.cs:641-642` — `MinimumAge < 30 days ⇒ 30 days`, and `KeepNewestPerSource` documented as no longer driving deletion (#3843)
assembly cache	`AssemblyCacheRetention.cs:331-332` — the same clamp (#3846)
the registry	nowhere. `check_retention_record` asserted only that no step passes `--include-locked`

#3843's own body records the gap in as many words: "The ACR task record and live cloud cleanup are
unchanged."

The negative control

The new gate, against the record as it stood on origin/main this morning:

$ python3 .github/scripts/lock-pinned-digests.py --check-retention-record .
retention record: 2 task(s), 3 purge step(s) inspected.
::error::purge-old-images.yaml: a purge step retains for `--ago 7d` (7 day(s)), under the decided floor of 30 days
::error::purge-old-images.yaml: a purge step carries a BUILD-COUNT QUOTA `--keep 10`
EXIT=1

After the record states the decided window: EXIT=0, still 3 purge step(s) inspected, and the
pre-existing 30-day step passes unchanged — so the gate is not simply reddening everything. Both new
assertions were then falsified: stubbing check_window_policy reds four ARM 31 lines including
"the exact step this repository carried until 2026-09-13 … passed the window policy"; stubbing
check_pause_coherence reds three ARM 31b lines. The --ago 7d --keep 10 step is driven as a
literal, so the arm keeps firing the day the record is right.

🚨 --keep is removed rather than raised, and that is the point. It counts NEWER BUILDS, so a
repository republished many times a day ages its own manifests out in hours whatever --ago says —
this issue's own root cause ("republishing frequency is what DESTROYS a pin") stated as a flag. A
30-day --ago beside a --keep 10 still collects a manifest ten builds old on the day it is
written. The window is not offered as pin protection; the lock is. It is the decided policy, stated
where re-enabling would otherwise install a window nobody chose.

The record is deliberately AHEAD of the registry, and says so

Nothing was applied. The live task is disabled and still carries --ago 7d --keep 10;
tasks.json gains recordAheadOfRegistry, and three things follow from the declaration rather than
from anyone remembering it:

verify prints the declared drift instead of reporting it as an incident — a drift report that is
always red is one nobody reads, which this file learned on 2026-09-07 over a trailing newline —
and reds when a declared task shows NO drift, so the declaration expires by being checked.
record, the one command that overwrites the record from live, refuses without an explicit
OVERWRITE: re-capturing would silently restore the quota window into a file whose comments
explain at length why it must not say that.
The gate above reds on the restored window on the very next pull request.

Also closed: the record can no longer contradict itself

retention_windows prints "PAUSED" only when it finds no enabled step, so a task flipped to
Enabled while pause.inForce was still true printed the window and never mentioned the pause —
a record asserting both that cleanup is stopped and that it runs at 03:00, with nothing red. That is
now a failure, and so is every task Disabled with no declaration explaining it: a stopped
retention and a stale record otherwise read identically, and only one of them is a decision.

Still open here, unchanged

Retention: interlock cleanup with protection updates and active publication #3859 — the two clocks. The re-enable ACT is now interlocked (apply refuses while the pause
stands); the nightly deletion is still not downstream of that night's verdict. Three coordination
mechanisms are costed on Retention: interlock cleanup with protection updates and active publication #3859 with a recommendation; the choice is the maintainer's and needs
an RBAC grant on the shared registry.
Applying the decided window to the registry — the same act as lifting the pause. It costs
storage: dropping --keep 10 and moving 7d → 30d retains strictly more over five repositories
that republish many times a day.
The :latest question, the Container Registry Repository Writer grant, released bundles and
active builds as consumers, the production read-back, client-b's declaration, and the one-off state
cleanup — all as recorded on 2026-09-12.

_{Measured 2026-09-13 (date -u), read-only: no purge, no dry run, no apply, no registry

mutation of any kind. Design: Doc/Architecture/ArtifactRetentionInterlock.}

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-13)

🚨 The protection lane has been RED since 2026-09-13 01:18Z, and its stated reason is FALSE — #4221

Found while establishing "what would have to be true to re-enable safely". It is the answer:
pause.reEnableWhen is "lock-pinned-digests is green", and it is not.

##[error]AXIS 3 — installation `build` runs core c84c6c0 and its overlay
(Systemorph/Memex deployments/aks/build/values.build.public.yaml) pins no image at all,
so there is no repository in which to protect what it runs.

— run 34730206406

That overlay pins two images. The shipped extractors, run over the actual committed bytes:

overlay	ACR pins	foreign pins
`deployments/aks/build/values.build.public.yaml`	0	2 → `cr.meshweaver.cloud` (`memex-portal-ai:3.0.0-ci.8411` + its migration twin)
`deployments/aks/memex/values.memex.public.yaml`	6	0

REGISTRY_HOST_RE is (?P<host>[a-z0-9]+)\.azurecr\.io — right for a lane that locks one ACR, but
it made "pins its images elsewhere" spell identically to "pins nothing". An instrument answering
confidently wrong, and the fix it suggests is to repair an extractor that has no defect.

The cost of the wrong label

Because the run refuses as a whole, memex — six pins in this ACR, the installation actually
exposed to the ACR purge — got no locks either. One instance's registry question held the
other's protection hostage, and the mislabelled red sat on the pause's own re-enable condition.

build is the fleet's build server and the first live installation on cr.meshweaver.cloud.
client-b pins there too and is declared not-installed, which is why this surfaced only now.

🚨 The larger finding, and it is an open item rather than a fix

A second registry now serves live installations, and acr purge cannot reach it. Per the
new-deployment path, cr.meshweaver.cloud is the default for new instances. Everything on this
issue — the lock, the closure, the tag lock, the 30-day window, the interlock — is about
meshweaver.azurecr.io. What retains the fleet's own registry is established by nothing, and
that gap grows with every instance provisioned there.

#4221 does not answer that. It makes the distinction legible: foreign references are extracted
and named, an undeclared one is RED naming the registry, a declared one is accepted and printed
on its own denominator line (…declared OUT OF THIS REGISTRY'S SCOPE) so it can never be counted
among the protected. registry in instances.json is a scope statement, never an exemption from
answering — checked both ways against what the overlay actually pins, refused when it names a
registry the overlay does not use, refused when the installation also pins in this ACR (half in
is not out), refused on a non-live entry and refused without a reason.

Eleven arms, falsified in two directions. Sabotage A — the foreign extractor stubbed to answer
nothing, i.e. the pre-fix behaviour — reproduces this issue's own false sentence verbatim and reds
six of them.

Added to what is still the maintainer's

What retains cr.meshweaver.cloud. Same question this issue answers for the ACR, asked
again about a second store. Nothing today says what keeps those images, or deletes them.

Design: Doc/Architecture/ArtifactRetentionInterlock → A second registry.

_{Measured 2026-09-13 (date -u), read-only: the run log over REST, the two overlays' committed

bytes through this repository's own extractors, and the task record. No purge, no lock, no dry run,

no apply, no registry mutation.}

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-13)

What landed, what is in flight, and the measured cost of the lane being red

Merged

fix(#3438): hold the recorded purge to the decided 30-day window, and interlock re-enabling it #4213 (7be4af5976) — the recorded purge now states No platform pins in module repos — build against the released platform, and be TOLERANT of drift (same major serves) #3842's decided window (at least 30 days
by age, no --keep build-count quota) and a credential-free gate holds every recorded step to
it on every pull request; acr-retention-tasks.sh apply refuses to re-enable a task while the
pause stands. Verified live: the post-merge run of lock-pinned-digests.yml printed
"every recorded step retains for at least 30 days by age with no --keep build-count quota
(No platform pins in module repos — build against the released platform, and be TOLERANT of drift (same major serves) #3842), and the record's pause declaration agrees with the statuses it is recorded beside."

The cost of the lane being red, measured

Lock pinned image digests, last twelve runs:


last successful apply (scheduled) run	2026-09-11T01:13Z
2026-09-12 schedule	FAILED — the expired zero-pin premise #4081 replaced
2026-09-13 schedule	FAILED — installation `build` (#4221)
2026-09-13T19:21Z push (report-only, after #4213)	still FAILED, same `build` cause

So no lock has been written in over two days, and pause.reEnableWhen — "lock-pinned-digests
is green" — has been unmet since. Two consecutive nightly protection runs lost, for two different
causes. The purge being paused is what makes that survivable rather than an incident.

In flight

fix(#3438): an installation served by ANOTHER REGISTRY is named, not read as pinning nothing #4221 — the second-registry fix, plus its review rounds. build pins in
cr.meshweaver.cloud; the lane now names the registry instead of claiming the overlay "pins no
image at all", and instances.json gains a registries disposition table (the unit is the
registry, not the installation, so the declaration can be shown to be complete).
review(#4213): every reader of a declaration fails closed, and the window is read off the purge #4223 / review(#4223): a purge step is read as TOKENS, and null is not absent #4228 — review rounds on fix(#3438): hold the recorded purge to the decided 30-day window, and interlock re-enabling it #4213. The substantive ones: every reader of a declaration
now fails closed on a non-boolean or explicitly-null inForce (one typed quote mark would
have disarmed the apply interlock, the overwrite guard and the coherence gate at once), and a
purge step is read as tokens rather than text — --include-"locked" executes as that option
and matched no grep for it, which defeated the one invariant this lane rests on.

Still the maintainer's, unchanged

Retention: interlock cleanup with protection updates and active publication #3859's coordination mechanism — four options costed on that thread with a recommendation;
it needs an RBAC grant on the shared registry and the disabling of a destructive schedule.
Applying the decided window to the (disabled) live task — the same act as lifting the pause.
NEW: what retains cr.meshweaver.cloud. The fleet's own registry serves at least one live
installation and is the default for new ones; acr purge cannot reach it, and nothing says what
keeps — or deletes — what is in it. Same question this issue answers for the ACR, asked again
about a second store.
The :latest question, the Container Registry Repository Writer grant, released bundles and
active builds as consumers, the production read-back, client-b's declaration, and the one-off
state cleanup — all as recorded on 2026-09-12.

_{Measured 2026-09-13 (date -u), read-only throughout: no purge, no lock, no dry run, no

apply, no registry mutation of any kind.}

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-13)

The second-registry question is now its own issue: #4230

My earlier comment on this thread listed "what retains cr.meshweaver.cloud" as an open item
here. It is now filed separately — #4230 — because it is not a part of this issue: everything
under #3438 (the lock, the closure, the tag lock, the 30-day window, #3859's interlock) is about
meshweaver.azurecr.io, and #4230 is the same question asked about the registry that is now the
default for newly provisioned instances and that acr purge cannot reach.

What it carries, measured across all twelve of the Memex deployment overlays on 2026-09-13:
build is LIVE and pins 0 images in this ACR and 2 in cr.meshweaver.cloud; client-b the
same but not installed; memex 6/0; memex-cloud 3/1 (the one being ghcr.io, third-party).

This issue keeps the ACR-side items unchanged.

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-14)

Open item 3 — "what retains `cr.meshweaver.cloud`" — is ANSWERED and moves off this list

From the 2026-09-13 19:36Z comment, "Still the maintainer's":

NEW: what retains cr.meshweaver.cloud. The fleet's own registry serves at least one live
installation and is the default for new ones; acr purge cannot reach it, and nothing says what
keeps — or deletes — what is in it.

Something says it now: #4230, landed as #4315, designed in
Doc/Architecture/FleetRegistryRetention.
Nothing deletes there — six mechanisms enumerated, measured read-only over 409 files with the
sweep pattern proven on a synthetic control first — and nothing may, until a cleanup can derive a
protected set over what the registry holds. The remaining unknown is one Azure storage read, and it
is carried in the record as a state of its own rather than as a measured absence.

🚨 The part that bears on THIS issue: the rule over there is stricter, and the reason is structural

distribution has no lock. changeableAttributes.deleteEnabled — the thing everything on this
issue rests on — is an ACR feature, so there is no object a protected set can be written into ahead
of a deleter.

That reframes what the lock buys here, and it is worth stating because it is easy to under-read: the
lock is not merely "the protection"; it is persistent state, written by a different process on a
different clock. That is why an incomplete run on this issue's lane is survivable — it writes fewer
locks and last night's still hold — and the lane could be red for two days in September without
anything already protected being at risk.

On a registry with no lock none of that holds, which is why #4230's rule makes "an incomplete
derivation deletes NOTHING" a clause rather than a nicety.

Also landed here, small

The nightly run no longer discards what it derives about the other registry. It prints a
PROTECTED SET — registries this lane cannot lock section — repository, tag and the file that pins
each (4 references, build ×2 and client-b ×2) — with its own incompleteness on its own line. It
locks nothing and needs no new credential; on a lock-less registry the derivation is the only
artifact available, and a derivation nobody can read is not one.

The registries table's cr.meshweaver.cloud entry no longer says "established by nothing"; the
out-of-scope report line still says UNVERIFIED, and must — this lane still protects nothing
there.

Unchanged on this issue

Items 1, 2 and 4 of that comment stand exactly as recorded: #3859's coordination mechanism (an
RBAC grant on the shared registry and the disabling of a destructive schedule — the maintainer's),
applying the decided window to the disabled task (the same act as lifting the pause), and the
:latest question, the Container Registry Repository Writer grant, released bundles and active
builds as consumers, the production read-back, client-b's declaration and the one-off state cleanup.

_{Measured 2026-09-14 (date -u), read-only throughout: no purge, no lock, no dry run, no

apply, no registry mutation of any kind, on either registry.}

🤖 Generated with Claude Code

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-14)

Landed — `e73047d3233cc4816fd15735528a4766feb8d8d5` (#4315). Open item 3 is off this list

The "NEW: what retains cr.meshweaver.cloud" item from the 2026-09-13 19:36Z comment is answered
and moved to #4230, where the close-out lives. Nothing on this issue's ACR side changed.

🚨 One correction that belongs here too, because this thread carried the same framing: the
danger at the fleet registry is unbounded growth, not silent deletion. Nothing deletes there —
six mechanisms enumerated, read-only, with the sweep pattern proven on a synthetic control first.
That reverses what "retention is established by nothing" implied.

The part that bears on THIS issue, and it is worth reading twice

distribution has no lock. changeableAttributes.deleteEnabled — the thing everything on this
issue rests on — is an ACR feature.

That reframes what the lock actually buys here. It is not merely "the protection": it is
persistent registry state, written by a different process on a different clock. That is why an
incomplete run on this lane is survivable — it writes fewer locks and last night's still hold — and
why this lane could be red for two days in September without anything already protected being at
risk. None of that holds on a store with no lock, which is why #4230's rule makes "an incomplete
derivation deletes NOTHING" a clause rather than a nicety.

Also landed here, small

The nightly run no longer discards what it derives about the other registry: it prints a
PROTECTED SET — registries this lane cannot lock section (repository, tag, and the file that pins
each — 4 references, build ×2 and client-b ×2) with its own incompleteness on its own line. It locks
nothing and needs no new credential; on a lock-less registry the derivation is the only artifact
available, and a derivation nobody can read is not one.

The registries table's cr.meshweaver.cloud entry no longer reads "established by nothing" — but
the out-of-scope report line still says UNVERIFIED, and must: this lane still protects nothing
there.

Unchanged and still open on this issue

Retention: interlock cleanup with protection updates and active publication #3859's coordination mechanism — the RBAC grant on the shared registry and the disabling of a
destructive schedule. Costed on that thread with a recommendation; the choice is the maintainer's.
Applying the decided window to the disabled task — the same act as lifting the pause.
The :latest question, the Container Registry Repository Writer grant, released bundles and
active builds as consumers, the production read-back, client-b's declaration and the one-off state
cleanup — all as recorded on 2026-09-12.

_{Measured 2026-09-14 (date -u), read-only throughout: no purge, no lock, no dry run, no

apply, no registry mutation of any kind, on either registry.}

🤖 Generated with Claude Code

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-15)

Triage 2026-09-15 — (c) tracking, NOT complete. Stays open, and two things in the body are now misleading enough to correct.

The two comments of 2026-09-14 already close out open item 3 and end "Unchanged and still open on this issue"; this adds only what they do not say.

1. 🚨 All four checkboxes in the "Current delivery scope" block are stale — three are done, and the list is no longer the denominator

tracked	state (REST, today)
#3858 fleet inventory	closed / completed 2026-09-12
#3860 preserve artifacts for supported releases	closed / completed 2026-09-12
#3859 coordinate protection and cleanup	closed / completed 2026-09-13
#3849, #3843, #3846 (the seams named in the body)	all closed
#4230 — retention for `cr.meshweaver.cloud` (spun out 09-14)	open, not listed in the body

Anyone reading the body today sees three unchecked boxes that are done and misses one open child that is not there. Worth an edit.

2. 🚨 #3859 reads closed-completed while this parent still lists its coordination mechanism as open

The 09-14T14:02Z comment enumerates as still open: "#3859's coordination mechanism — the RBAC grant on the shared registry and the disabling of a destructive schedule … the :latest question, the Container Registry Repository Writer grant, released bundles and active builds as consumers, the production read-back, client-b's declaration and the one-off state cleanup." A closed child whose mechanism the parent lists as outstanding is the kind of pair that makes a later sweep close the parent on "all children green". Either #3859 reopens or the parent stops citing it.

3. The three reasons the parent is genuinely not closable — all measurable in-repo

The decided 30-day window is not in production. .github/acr-retention/purge-old-images.yaml:47-52 says so against itself: "THE RECORD IS AHEAD OF THE REGISTRY HERE, ON PURPOSE… The live task is DISABLED (paused, Memex#219) and still carries --ago 7d --keep 10; acr-retention-tasks.sh apply is what pushes this file onto it." The record is now --ago 30d --untagged with --keep 10 removed, not raised (:64) — correct, and not yet what the registry runs. The parent's own unchecked verification item is exactly this.
The protection lane has written no lock since 2026-09-11. lock-pinned-digests.yml (cron: 0 1 * * *) → lock-pinned-digests.py, whose premise is "acr purge SKIPS LOCKED MANIFESTS BY DEFAULT" (:21); the 09-12 and 09-13 schedules both failed, and pause.reEnableWhen = "lock-pinned-digests is green" has been unmet since.
Production behaviour has not been read back — which is what the fourth item asks for and what none of the merged PRs can supply.

For the record: the pin-protection half asked for in the filing does exist in core now, on all three axes

Protect pinned digests: .github/workflows/lock-pinned-digests.yml + .github/scripts/lock-pinned-digests.py — committed workflow digest pins, deployment-overlay tag pins, and (since #3438: retention protects what is CONSUMED, and refuses when it cannot see it all #4081) each installation's own /api/version. Locks manifest and tag separately (a tag read deleteEnabled: true over a manifest reading false).
A guard that reds when a pin no longer resolves: .github/workflows/pinned-digests.yml + check-pinned-digests.py, with credential-free arms on every PR via dotnet-test.yml:2205/2221/2250/2259/2277.
The purge tasks are now in-repo as records (.github/acr-retention/), so the "two tasks diverged and nobody noticed" defect the filing calls out is structurally addressed — the two live tasks agreeing with the two records is the remaining verification.

So the filing's preferred option landed and its rejected option (raising --ago/--keep) was not taken. What is left is delivery, not design.

_{Read-only: REST issue states, local reads of .github/ on core main. No writes, nothing applied.}

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-15)

Two corrections to the 2026-09-15T05:09Z comment, and the cause of the current red: a new fleet repo, filed nowhere

Read-only re-measurement, 2026-09-15T08:0xZ, REST only.

Correction 1 — "the protection lane has written no lock since 2026-09-11" is wrong. Run history for lock-pinned-digests.yml:

run	when	event	conclusion
34935956561	2026-09-15T06:12:59Z	push	failure
34916617427	2026-09-15T01:16:37Z	schedule	failure
34856846529	2026-09-14T14:37:46Z	push	success
34852827116	2026-09-14T14:00:48Z	push	success
34795570227	2026-09-14T01:19:53Z	schedule	success

The 09-14 schedule run is a complete green apply: mode: apply (event: schedule) → Protected 16 manifest(s) (0 newly locked this run) and 13 tag reference(s) (0 newly locked); 0 released. That matters beyond bookkeeping: pause.reEnableWhen is "lock-pinned-digests is green…", so the re-enable condition was met on 2026-09-14, and the decision that turns on it is now live rather than pending.

Correction 2 — even the failing runs write locks. lock-pinned-digests.py:1778-1780 is explicit ("Locks are applied even on a degraded run"), and run 34916617427 logs 15 writes at 01:22:52Z — 7 locked <repo>@sha256:… plus 8 locked tag …, including memex-portal-ai:3.0.0-ci.8612, the set memex is currently running. So the protection has not lapsed; the run's verdict has.

The cause of the red, which is on no thread in this repo. Both failure annotations on run 34935956561, verbatim:

installation 'memex' is declared by two overlays (Systemorph/Memex deployments/aks/memex/values.memex.public.yaml and Systemorph/ClientA.Memex deployments/aks/memex/values.memex.yaml), so which host answers for it is ambiguous.
AXIS 3 — installation 'client-a' could not be accounted for: its overlay declares the installation but no ingress host, so it cannot be asked what it is running.

Systemorph/ClientA.Memex was created 2026-09-14T21:30:38Z — about 4 h after the last green run, and ~20 h after the green schedule run above. A new deployments repo entered the fleet and reddened the fleet's only registry-protection lane. This is a real (b) inside this tracking issue:

.github/scripts/lock-pinned-digests.py:1075-1079 — build_instances keys installations by the bare Hosting__Deployment id across the whole org, so two repos each declaring memex collide;
:1096-1101 — a live instance with no ingress host gets instance.error → AXIS 3 INCOMPLETE → exit 1;
.github/acr-retention/instances.json — instances[] holds only build and client-b; there is no client-a entry.

Fix sketch. Declare client-a in instances.json with its honest state (not-installed/retired, or live once it has an ingress host) and a reason, exactly the way client-b is declared — that alone clears the second error and is the stricter-by-default direction the file already documents. The first error is a modelling gap and needs code: qualify the instance identity by its source repo (f"{scan.gh_repo}:{ident}", or a repo field on the roster entry) so two deployments repos can each declare a memex without one holding the other's protection hostage; keep the duplicate-id blocker for a genuine collision within one repo. Pair it with a roster arm that reds when a repo's overlay set is unrepresented, so the qualification cannot silently exempt anything — otherwise the fix for "a fork reddens the lane" becomes "a fork is invisible to it".

What is still open here, unchanged. The decided 30-day window is record-only: .github/acr-retention/purge-old-images.yaml:47-52 says so against itself — the live task is DISABLED and still carries --ago 7d --keep 10, while the record is --ago 30d --untagged with --keep removed (:64-69) and tasks.json declares recordAheadOfRegistry.inForce: true. Applying it is the act that lifts the pause, and the instrument is az acr task show --registry meshweaver --name purge-old-images then acr-retention-tasks.sh apply. Also still open: the production read-back of the report receiver (#3858's last criterion — @Hosting/Modules/* reads [] for me, RLS caveat), and released module bundles / active builds as registry-level consumers (FleetRegistryRetention §5.3 — DeploymentPinnedReferences.Resolve yields platform builds only, no registry repository and no bundle digest). Body hygiene: #3858, #3860, #3859, #3849, #3843 and #3846 are all closed while three boxes are still unchecked, and #4230 is open and not listed.

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-15)

2026-09-15 17:4xZ: 193e54ed4f cleared half of the lock-pinned-digests red. The other half is a fact nobody has stated yet.

The newest run, 34956358066 (10:08Z, after 193e54ed4f), fails on exactly one line:

AXIS 3 — installation Systemorph/ClientA.Memex:memex could not be accounted for: https://client-a.meshweaver.cloud/api/version could not be read: URLError: timed out.

The roster now declares Systemorph/ClientA.Memex:client-a, the TODO template, as not-installed, and that one correctly stays quiet. But ClientA.Memex carries a second overlay, deployments/aks/memex/values.memex.yaml, and that one is not a template:

it pins memexaksacrqoqqdqnhlaksg.azurecr.io/memex-portal-ai:3.0.0-ci.8622, from ClientA's own ACR, so it is a foreign registry and never lockable here;
it names host client-a.meshweaver.cloud, which resolves (57.174.148.54) and does not answer. From here, at 17:4xZ, curl --max-time 15 timed out with http 000.

No roster state is honest without knowing whether that portal is up:

not-installed would be a false claim if it is running behind an allowlist in ClientA's estate;
live keeps the blocker, which is correct only if it is supposed to answer from a GitHub runner.

Nothing is changed here, because this is a statement about a client estate that only its owner can make.

Until it is answered, the lane stays red on every push to its paths and on the 01:00Z schedule. Because pause.reEnableWhen is "lock-pinned-digests is green", it is holding the re-enable decision again. The locks themselves are still written on a degraded run (lock-pinned-digests.py ~1778).

🤖 Generated with Claude Code

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-17)

2026-09-17 — the protection is holding nightly; both lanes are red for a third night on something that is NOT retention

The "one production action" from the 2026-09-07 comment is long done. It was the
Container Registry Repository Writer grant (the data action
Microsoft.ContainerRegistry/registries/repositories/metadata/write, held by none of the three SPs
at the time). It landed 2026-09-08 — first successful apply, run
34176008757, "Protected 18
manifest(s) (8 newly locked this run); 0 released".

And the lane is still writing. Run
35169861024 (2026-09-17T01:15Z,
mode: apply):

UNION — fleet pins and official release manifests    TAG REFERENCES
  distinct manifests wanted                34          tag references depended on               26
  …already protected (deleteEnabled false) 34          …already protected                       26
  …TO LOCK                                  0          …TO LOCK                                  0
  …that could NOT be protected              0          …that do NOT resolve                      0
REGISTRY: manifests 4178 · currently locked 91 · locked and pinned by NOTHING 57
THE WINDOW THIS IS RACING: PAUSED since 2026-09-12 — no purge step runs.

That also discharges the 09-12 item "20 tag references are unlocked right now": 26/26 and 34/34 read
protected today. (locked and pinned by NOTHING has grown 16 → 57 since 09-08 — the standing debt of
a lock-only job, unchanged in kind.)

🚨 Both lanes have failed three nights running, on one cause, and it is not retention

lock-pinned-digests.yml (cron: 0 1 * * *): 09-15 34916617427, 09-16 35043350450, 09-17
35169861024. Sole annotation each night: "AXIS 3 — installation
Systemorph/ClientA.Memex:memex could not be accounted for:
https://client-a.meshweaver.cloud/api/version could not be read" (timed out 09-16,
Name or service not known 09-17).
pinned-digests.yml (cron: 20 5 * * *): 09-15 34933010021, 09-16 35059768252, 09-17
35185997013. Annotations name only that repo's two PLATFORM_REF declarations — while its own
counters in the same run read declarations GONE 0, UNPROTECTED 0, INDETERMINATE 0.

Measured here 2026-09-17, and it contradicts both runners: client-a.meshweaver.cloud resolves
to 74.158.7.121 (it was 57.174.148.54 on 09-15 — the record moved) and GET /api/version answers
200 with {"version":"3.0.0+2592e2284c8b34d91322e2d4e6998d17ca113821"}. The portal is live and
answering; what has not answered for three nights is the GitHub runner's reach to it. Its images
live in memexaksacrqoqqdqnhlaksg.azurecr.io, declared out-of-estate in the registries table, so
nothing this lane would lock is at stake either way — yet pause.reEnableWhen is "lock-pinned-digests
is green", so this is holding the re-enable decision hostage for a third night. Nothing is filed
for it.

Still owed, unchanged

The decided 30-day window is record-only. .github/acr-retention/tasks.json →
recordAheadOfRegistry.inForce: true, since: 2026-09-13; purge-old-images.yaml:64-69 is the
record (--ago 30d --untagged, --keep removed) while the live task is Disabled and still
carries --ago 7d --keep 10. Applying it is the same act as lifting the pause — system-changing.
Production read-back of the deployed task definition and the report receiver (Retention: require a complete and fresh fleet inventory before releasing artifact protection #3858's last
criterion).
Released module bundles and active builds as registry-level consumers —
FleetRegistryRetention §5.3: DeploymentPinnedReferences.Resolve yields platform builds only,
no registry repository and no bundle digest. Not built.
The :latest question — a moving tag is named by no file and sits outside the lock model.

Body hygiene: all four "Current delivery scope" boxes are unchecked while #3858, #3860, #3859,
#3849, #3843 and #3846 are all closed/completed, and #4230 is open and not listed.

Verification of the earlier halves on today's main: 9dfce19c0 = #3462 → pinned-digests.yml +
check-pinned-digests.py; deccceceef607ac9038c08a3a80cfc7d155b9482 = #3507 →
lock-pinned-digests.yml + .py. Both present. --check-retention-record run locally: EXIT=0.

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-18)

Triage 2026-09-18 — still open, and two inputs to this issue have moved since the 09-17 comment

Read-only: core origin/main b76f57a79d, GitHub REST, and three unauthenticated /api/version GETs. No registry call, no purge, no lock, no cluster access.

The remaining item is unchanged — measured in-repo, not inferred

.github/acr-retention/tasks.json on origin/main today:

purge-old-images → "status": "Disabled", note "DISABLED 2026-09-12 — see the top-level pause block"
recordAheadOfRegistry.inForce: true, since: "2026-09-13" → "purge-old-images.yaml step 1 records --ago 30d with no --keep; the live (disabled) task still carries --ago 7d --keep 10"
pause.inForce: true, reEnableWhen: "lock-pinned-digests is green …"

So the decided 30-day window is in the record and not in the registry, which is exactly what this issue's title asks for. Nothing to adjudicate into a close.

🚨 1. The lock lane is red a FOURTH consecutive night, and the cause has CHANGED

gh api repos/Systemorph/MeshWeaver/actions/workflows/lock-pinned-digests.yml/runs?per_page=5
  2026-09-18T01:15:42Z schedule failure 35294596409
  2026-09-17T19:10:20Z push     failure 35263279184
  2026-09-17T01:15:39Z schedule failure 35169861024
  2026-09-16T01:15:19Z schedule failure 35043350450

The 09-15 and 09-17 comments name the cause as "client-a.meshweaver.cloud could not be read: URLError: timed out". That is no longer it. Run 35294596409's annotations, verbatim:

AXIS 3 — installation Systemorph/ClientA.Memex:memex runs core 2592e22 and its overlay pins images ONLY in memexaksacr43rzd6faaix36.azurecr.io, every one of them declared third-party …
AXIS 3 — … pins images in memexaksacr43rzd6faaix36.azurecr.io, which .github/acr-retention/instances.json does not account for.

Two things follow. The host answers now (curl --max-time 15 https://client-a.meshweaver.cloud/api/version → http=200, 2026-09-18 06:14Z), so the timeout half is resolved and the lane read a real core sha from it. And the registry it pins is not the memexaksacrqoqqdqnhlaksg… named in the 09-15 comment — a second, undeclared ClientA ACR. The remedy is a registries declaration, not a roster state change. This is filed on no issue in either repo (search/issues for ClientA and for lock-pinned-digests, state:open → only #3355, the release tracker).

Because pause.reEnableWhen is "lock-pinned-digests is green", this is what currently stands between the fleet and re-enabling the purge — i.e. between this issue and its last item.

🚨 2. The roster's `client-b` entry is now factually wrong, and it exempts a LIVE installation

.github/acr-retention/instances.json:306-310 still reads:

"id": "client-b", "state": "not-installed",
"reason": "… client-b.meshweaver.cloud has no DNS record (measured 2026-09-12) … Delete this entry the day it is provisioned."

Measured 2026-09-18 06:14Z:

curl -s --max-time 15 https://client-b.meshweaver.cloud/api/version
  → {"commit":"67cbbe0ee4467ac30f086587c97f62c7b0d13689"}     http 200

and Systemorph/Memex@main mesh/Deployments/client-b.json reads status: "Live" (commit f86d9cc78, 2026-09-16 — "client-b: status Live — provisioned end to end through the Hosting plugin") with updatePolicy: "Continuous" (bdf675ddf, 2026-09-17). The entry's own instruction — "Delete this entry the day it is provisioned" — came due two days ago. Until it is, AXIS 3 exempts an installation that is serving.

(For the record, 67cbbe0ee is an ancestor of main and 1871 commits behind it, on an instance declared Continuous. That is its own question and not this issue's.)

What this issue still owes, unchanged

The decided 30-day window applied to the paused purge-old-images and read back from production, plus the bundle/active-build consumer inventory. Both are gated on the lane being green, which items 1 and 2 above now describe.

_{Read-only triage 2026-09-18 by Claude Opus 5 (1M context). No registry, cluster or deployment mutation of any kind.}

Step 4 is done: production has been read back through the governed path. The RegistryRetention action kind is live on the control instance, and its dry run answered today (2026-10-11, request Ops/Actions/registry-retention-verify-20261011-3438, operations-lane run 38141799080, read-only, no approval).

What the verify printed:

task	live definition against the record
`purge-old-images`	drift, DECLARED (`recordAheadOfRegistry`). Live step 1 is still `--ago 7d --keep 10 --untagged`; the record's is `--ago 30d --untagged`. Step 2 (`memex-portal`, `--ago 30d`) agrees.
`purge-old-ci-releases`	matches the record byte for byte.

compared 2 task definition(s) against the record
3 purge step(s) inspected; none passes --include-locked, so locked manifests are skipped
preflight: nothing in the record's interlocks refuses an apply

So the 30-day window is still not installed on the registry, and nothing has changed there since the 2026-09-22 reading. The same run answered the storage lifecycle row for the fleet registry (present=false); that is recorded on #6499 and in #6502.

What remains is the maintainer's, in this order. No agent approves a governed action.

Confirm the operator identity holds Container Registry Tasks Contributor on the registry (steps 2 and 3 of the 2026-10-10 comment: the ABAC re-grant, then InfraDeploy what-if and deploy). The verify needed only Reader, so today's green run does not show whether that grant has landed. I did not read role assignments.
On the control instance, file RegistryRetention for deployment control with dryRun: false and confirmation: control, and approve it. This installs --ago 30d with no --keep on purge-old-images, which stays disabled.
In core, delete recordAheadOfRegistry from .github/acr-retention/tasks.json (it goes stale by design once the apply lands).
Enabling the purge waits on the lock lane. lock-pinned-digests.yml on main is not green and recent today: the newest run (38101969700, schedule, 01:28Z) was cancelled, the one before it (38091102891) failed, and the last success is 38087852774 at 2026-10-10 21:30Z. The apply refuses an enable until the newest run is green and under 36 hours old.

Leaving this open until step 2 is read back with another verify.

Retention: the locks hold nightly — apply the decided 30-day window to the paused purge-old-images, read production back, and inventory bundles/active builds as consumers #3438

Description

Current delivery scope — 2026-09-09

What happened

Root cause

🚨 Why the existing rule does not cover this — a SECOND failure mode

Suggested fix

The triage trap, worth keeping

Evidence

Activity

rbuergi commented on Sep 6, 2026

Blast radius, measured — three repos went down simultaneously

This is not a historical problem — it is about to recur, and the fix PR is the victim

Why the priority is higher than "a CI annoyance"

Minimum viable mitigation, if the full fix is not immediate

rbuergi commented on Sep 6, 2026

The guard half is up: #3462 — and the blast radius is wider than the tester

The retention half: cloud-only, so there is no IaC PR to open

rbuergi commented on Sep 6, 2026

Guard half MERGED — and its first run on main reds naming this issue's three digests

Four defects were found and fixed by falsifying the guard rather than reading it

The retention half — still OPEN, still the maintainer's to apply

rbuergi commented on Sep 7, 2026

First real run: 34068789826 — RED, naming this issue's three digests

rbuergi commented on Sep 7, 2026

🚨 The blast radius is wider than a pinned digest — a MOVING tag is outside #3507's model entirely

What happened, 2026-09-07 03:00Z

The state is worse than one tag — four of five

Why #3507's job does not cover this

Ownership of the cause, stated plainly

What is deliberately NOT done

rbuergi commented on Sep 7, 2026

The purge took memex-portal-ai:latest at 03:00Z and blocked every Plugins CI run — fixed by leaving latest behind

What broke

The blast radius is wider than this issue documents

The fix, and why it is not a retag

What this does not fix

rbuergi commented on Sep 7, 2026

Both questions about the lock job's red are settled by measurement — and neither wanted a change. What DID want one is the read-back (PR #3563)

1. The red was CORRECT and it has already cleared itself

2. "Read the default branch rather than every branch" — it ALREADY does, and changing it would be harmful

3. "It should name the stale pin and which repo declares it, not just die" — it already does

What DID want a change, and it is load-bearing right now

Still yours, unchanged and listed rather than invented

rbuergi commented on Sep 7, 2026

STAYING OPEN — and Memex#219 does not cover either remaining half

rbuergi commented on Sep 8, 2026

The WRITE half works — first successful apply run, and the two-task question is settled

1. ✅ The lock actually locks. The RBAC grant landed and the first apply run succeeded

2. The second purge task — settled from the task definitions, not from a summary

The denominator, hand-counted — pins EXAMINED, not only pins at risk

🚨 The gap that was still open, and #3736 closes it

What stays open, and it is not code

31 remaining items

rbuergi commented on Oct 10, 2026

rbuergi commented on Oct 11, 2026

Lock half MERGED — and one production action stands between it and actually protecting anything

🚨 LEAVING THIS OPEN. One grant is missing, and it is only the maintainer's to make

What a real run would do, live, report-only, 2026-09-07

🚨 The unlock half is DESIGNED and DOCUMENTED but SHIPS DISABLED, and the first live run showed why

Two things this issue's premise got wrong

One thing to check before the next promotion

Also landed

rbuergi commented on Oct 11, 2026

Two of the three split issues are CLOSED and folded back here — #4081 landed the mechanism

What #4081 landed

What is left here, and it is deliberately not code

rbuergi commented on Oct 11, 2026

The 7-day step is closed, and it was DECIDED rather than a decision — #4213

The negative control

The record is deliberately AHEAD of the registry, and says so

Also closed: the record can no longer contradict itself

Still open here, unchanged

rbuergi commented on Oct 11, 2026

🚨 The protection lane has been RED since 2026-09-13 01:18Z, and its stated reason is FALSE — #4221

The cost of the wrong label

🚨 The larger finding, and it is an open item rather than a fix

Added to what is still the maintainer's

rbuergi commented on Oct 11, 2026

What landed, what is in flight, and the measured cost of the lane being red

Guard half MERGED — and its first run on `main` reds naming this issue's three digests

The purge took `memex-portal-ai:latest` at 03:00Z and blocked every Plugins CI run — fixed by leaving `latest` behind

The WRITE half works — first successful `apply` run, and the two-task question is settled

Open item 3 — "what retains `cr.meshweaver.cloud`" — is ANSWERED and moves off this list

Landed — `e73047d3233cc4816fd15735528a4766feb8d8d5` (#4315). Open item 3 is off this list

🚨 2. The roster's `client-b` entry is now factually wrong, and it exempts a LIVE installation