hosting-image-mirror: copy the pinned platform images from ACR into the fleet registry when it lacks them (Plugins#1722) by rbuergi · Pull Request #4110 · Systemorph/MeshWeaver

rbuergi · 2026-09-12T14:59:25Z

Operator half of MeshWeaver.Plugins#1722 (the plan half — Mirror platform images as the first Provision/Roll step — is the paired Plugins PR; Pairs-with: none — this PR removes no public surface).

The hand step it replaces

Memex docs/new-deployment.md step 4:

crane copy meshweaver.azurecr.io/memex-portal-ai:<tag> cr.meshweaver.cloud/memex-portal-ai:<tag> · crane copy meshweaver.azurecr.io/memex-migration:<tag> cr.meshweaver.cloud/memex-migration:<tag>

What changes

New command hosting-image-mirror --registry <fleet host> --repository <path> --tag <tag> [--also <path>]… --vault … --publisher-secret … [--source <acr host>] [--publisher <user>]:

present in the fleet registry → KEPT, never re-pushed; its digest is compared with the source's and a difference is the fact image_drift (a re-tagged upstream build is the operator's call);
absent → the source is asked (absent there too → refusal naming the sealed set to pin), crane copy runs, and both digests are read back and must match before it reports;
portal AND migration ride the same tag (Memex#141) via --also;
the publisher password (vault) and the ACR token (az acr login --expose-token) reach crane on stdin (--password-stdin) into a per-process DOCKER_CONFIG — never argv, never printed, never a fact;
a non-ACR --source is refused (the fleet's upstream is ACR and the token comes from az); --registry equal to --source is refused.

Dockerfile: adds crane (go-containerregistry v0.20.3, selected per TARGETARCH) — nothing else in the image can do a manifest+blob copy, and the image's own tool check now asserts it.

Rights the operator identity still lacks

AcrPull on meshweaver (plus Reader on the registry resource so az acr login --name meshweaver --expose-token resolves it) — hosting-operator does not hold it today. Until it is granted the step is RED naming the grant and the exact crane copy fallback; nothing is copied and nothing else is touched. Key Vault get on memexcloud-Registry-PublisherPassword is under the existing Key Vault Secrets Officer grant.

Gates run

$ bash deploy/aks/operator/test/run-tests.sh
326 passed, 0 failed          (291 on main; +35 new hosting-image-mirror cases)
$ shellcheck -S warning deploy/aks/operator/bin/hosting-* deploy/aks/operator/test/stubs/image-mirror/*
(clean)

The crane/az stubs record logins by host and user only — never a credential — and the suite asserts no sentinel appears in any output or argv.

hosting-operator.yml: plan-command list, tracked-file floor 23 → 24, stub list, and crane added to the in-image tool assertion. (The floor line conflicts textually with the sibling PRs #4100/#4103/#4105/#4109, which bump the same number; whichever lands last merges main and re-counts.)

Ships how

Hop (C): hosting-operator:<sha> on merge to main; reaches a Provision when operator.image on Deployments/memex moves to it. Merging is not shipping.

🤖 Generated with Claude Code

…he fleet registry when it lacks them (MeshWeaver.Plugins#1722) cr.meshweaver.cloud serves only what was pushed to it; CD publishes to ACR. A record pinning a tag the fleet registry lacked sent a human to `crane copy` as the registry's publisher (runbook step 4) — or, before #1546, into ImagePullBackOff at step 11. This command is that copy, for the portal image and (--also) the migration image, which must ride the same tag (Memex#141): present → KEPT, never re-pushed; digest compared with ACR's, a difference is the fact image_drift (a re-tagged upstream build is the operator's call). absent → the source is asked (absent there too → refused naming the sealed set to pin), crane copy runs, BOTH digests are read back and must match. Credentials: the publisher password from the vault and the ACR token from `az acr login --expose-token` reach crane on stdin (--password-stdin) into a per-process DOCKER_CONFIG — never argv, never printed, never a fact. The ACR short name derives from the source host; a non-ACR source is refused (the fleet's upstream is ACR). Rights: AcrPull on the source registry for the operator identity, not granted today — without it the step is RED naming the grant and the crane commands. Dockerfile: crane (go-containerregistry v0.20.3, per TARGETARCH) — nothing else in the image can do a manifest+blob copy. Tests: 35 cases against crane/az stubs that record logins by host+user only. Workflow: plan list, floor 24, stubs, tool check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

rbuergi · 2026-09-12T15:02:23Z

Corrected the tracked-file floor in hosting-operator.yml: main already tracks 24 files under deploy/aks/operator/bin/ (21 commands + run.sh + _common.sh + _audit.jq) while the guard declared 23 and its comment said "20 commands" — stale, so one lost file would have passed. This PR adds one command, so the floor is now 25 with the comment re-counted. 🚨 The four sibling PRs (#4103, #4105, #4109, #4110) each do the same on the same line: whichever lands after the first must merge main and set the floor to the real count at that point (26, 27, 28), not keep 25.

🤖 Generated with Claude Code

rbuergi · 2026-09-12T15:03:21Z

Why crane rather than the mesh's own registry package, recorded because a reviewer will ask: the in-mesh Container Registry package (the /v2 surface a portal served from blob storage) was retired on 2026-09-08 — no image was ever published into it and portal hosts now answer 404 there (MeshWeaver.Plugins/WhatsNew/2026-09-08-…, MeshWeaver#3353/#3697). cr.meshweaver.cloud is a separate deployed service (distribution + docker_auth), so there is no in-mesh API to push through, and the operator image carried no registry client at all (az, kubectl, helm, pg_dump, jq).

The push credential is deliberately the registry's publisher account, not an instance key: an mwi_ key is what the auth service grants pull for and cannot push. That is the same credential the hand step in MeshWeaver.Plugins#1722 used.

🤖 Generated with Claude Code

Copilot

🔵 Needs a closer look

Critical and moderate findings remain in argument handling, registry error classification, drift detection, validation, and workflow completeness.

Pull request overview

Adds hosting-image-mirror to copy pinned platform images from ACR into the fleet registry with credential isolation, digest verification, tests, and deployment integration.

Changes:

Adds multi-image mirroring, validation, drift detection, and secure authentication.
Adds Crane tooling, Azure/Crane stubs, and behavior tests.
Updates lifecycle documentation and workflow image checks.

File summaries

File	Review summary
`deploy/aks/operator/test/stubs/image-mirror/crane`	Adds Crane test stub; no final comments.
`deploy/aks/operator/test/stubs/image-mirror/az`	Adds Azure CLI test stub; no final comments.
`deploy/aks/operator/test/run-tests.sh`	Nit (2 votes): read-back digest mismatch is not tested. Nit (1 vote): copy-failure coverage does not exercise the mismatch guard; add successful-copy/wrong-digest coverage.
`deploy/aks/operator/Dockerfile`	Adds architecture-specific Crane tooling; no final comments.
`deploy/aks/operator/bin/hosting-image-mirror`	Critical (1 vote): missing argument handling can loop indefinitely. Critical (1 vote): registry errors are treated as absent images. Moderate (3 votes): all-present images skip source drift comparison. Moderate (2 votes): case-sensitive self-mirror validation can be bypassed. Moderate (1 vote): source validation permits multi-label ACR hosts. Nit (2 votes): read-back digest mismatch lacks test coverage.
`deploy/aks/INSTANCE-LIFECYCLE.md`	Nit (1 vote): documents `AcrPull` but omits the required ACR `Reader` grant.
`.github/workflows/hosting-operator.yml`	Moderate (1 vote): tracked-file floor is too low for the 22-command plan list and supporting files.

Review details

Suppressed comments (3)

.github/workflows/hosting-operator.yml:69

The plan list below contains 22 hosting-* commands, not 21; with run.sh, _common.sh, and _audit.jq, the tracked-file floor should be 25. At 24, losing one of the 25 files still passes this guard, so a clean checkout can ship an incomplete operator image without CI noticing.

          if [ "$tracked" -lt 25 ]; then
            echo "::error::only ${tracked} file(s) tracked under deploy/aks/operator/bin/ — expected 25 (22 commands + run.sh + _common.sh + _audit.jq). The floor is the REAL count — raise it with every command added, or a lost file passes."

deploy/aks/INSTANCE-LIFECYCLE.md:127

This lifecycle summary says the operator only needs AcrPull, but the command's own refusal and the provisioning requirements also require Reader on the ACR resource so az acr login --name ... --expose-token can resolve the registry. Omitting that grant sends operators to provision an incomplete permission set and leaves the step red.

  argv, never printed. Needs `AcrPull` on the source registry for the operator identity.

deploy/aks/operator/test/run-tests.sh:403

HOSTING_IM_COPY_FAIL=1 makes crane copy exit before a destination is written, so this only covers the command-failure branch. It never exercises the read-back mismatch guard at lines 141–143; a regression that accepts a different destination digest would still pass. Add a stub mode that writes a different destination digest and assert that the command refuses without reporting it as mirrored.

# A copy that does not land as the same digest is a failure, not a pass.
im "$IM_SRC" HOSTING_IM_COPY_FAIL=1 -- "${IM_ARGS[@]}"
[ "$_im_rc" -ne 0 ] && ok "a failed copy fails the step" || bad "copy fail" "exited 0"
rm -rf "$_im_state"

Files reviewed: 6/7 changed files
Comments generated: 0
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-09-12T15:13:19Z