feat(helm): the fleet's own container registry (cr.meshweaver.cloud) as a separate service by rbuergi · Pull Request #3697 · Systemorph/MeshWeaver

rbuergi · 2026-09-08T08:15:05Z

The fleet's own container registry, cr.meshweaver.cloud, as a SEPARATE service in the chart — two off-the-shelf images, zero registry code of ours. Part of MeshWeaver#3353; design page updated (Doc/Architecture/ContainerRegistryInMemex → new status block + "The registry as a separate service").

What is in the chart (`deploy/helm/templates/registry/`, all NEW files; `registry.enabled: false` by default)

object	what
`SecretProviderClass memex-registry-keyvault`	azure provider, same identity idiom as the portal's; objects `certObject`/`keyObject`/`httpSecretObject` (+ `notificationSecretObject` when notifications are on) under aliases `cert.pem`/`key.pem`/`http-secret`/`notification-authorization`, synced into Secret `memex-registry-secrets`
`ConfigMap memex-registry-config`	distribution `config.yml`: storage `azure` with `credentials.type: default_credentials` (workload identity), `delete.enabled: true`, `redirect.disable: false`, `auth.token` → `https://<host>/auth`, `rootcertbundle: /etc/registry/token/cert.pem`, storage-driver health check, `notifications:` rendered only when `notifications.url` is set
`ConfigMap memex-registry-auth-config`	docker_auth `auth_config.yml` (`:5001`, token issuer/expiration 900/cert/key, `disable_legacy_key_id: true`, ONE static `users:` entry = the publisher's bcrypt hash, `ext_auth: validate.sh`, ACL: publisher `push,pull,delete` on everything, `account: "/.+/"` `pull` on everything, anonymous nothing) + `validate.sh` (busybox `wget -S` → 200 exit 0 / 401,403 exit 1 / anything else exit 3 = docker_auth ERROR)
`Deployment memex-registry-deployment` + Service `:5000`	pod label `azure.workload.identity/use: "true"`, `serviceAccountName: registry.serviceAccount`, CSI mount at `/etc/registry/token`, `REGISTRY_HTTP_SECRET` from the synced Secret, `OTEL_TRACES_EXPORTER=none`, readiness `httpGet /`, liveness TCP, config checksum annotation
`Deployment memex-registry-auth-deployment` + Service `:5001`	same CSI mount, `validate.sh` mounted `0555`
`Ingress memex-registry`	ONE ingress on `registry.host`: `/auth` → auth:5001, `/` → registry:5000; `cert-manager.io/cluster-issuer` when set, `proxy-body-size: "0"`, request buffering off, read/send timeouts 3600

helm template FAILS naming the key when enabled: true and any of host, image, authImage, storage.accountName, keyVault.name, keyVault.certObject, keyVault.keyObject, keyVault.httpSecretObject, publisher.passwordBcrypt is empty (all reads go through the memex.registry helper in _registry.tpl), and when notifications.url is set without keyVault.notificationSecretObject. keyVault.tenantId/identityClientId fall back to keyVaultSecrets.*.

Additive elsewhere: a registry: block in values.yaml (inserted before secrets:, away from the sections feat/3353-mirror-consumers-core edits), deploy/helm/values.registry.example.yaml (complete, credential-free), and that example added as a 4th combination to check-chart-invariants.sh so the chart gate renders the registry on every PR.

Values shape

registry:
  enabled: false
  host: ""                      # cr.meshweaver.cloud
  image: ""                     # ghcr.io/distribution/distribution:3.0.0@sha256:4ba3adf47f5c866e9a29288c758c5328ef03396cb8f5f6454463655fa8bc83e2
  authImage: ""                 # cesanta/docker_auth:1.14.0@sha256:98e0307e0d2dcbafe10098814a97d248143b78ca478693f78c3c0d67fe776ab4
  issuer: "memex-registry"
  replicas: 1
  storage: { accountName: "", container: "registry" }
  serviceAccount: "memex-portal-sa"
  keyVault: { name: "", tenantId: "", identityClientId: "", certObject: "", keyObject: "", httpSecretObject: "", notificationSecretObject: "" }
  publisher: { username: "publisher", passwordBcrypt: "" }
  validationUrl: "https://memex.meshweaver.cloud/api/plugins?ref=HEAD"
  notifications: { url: "" }
  ingress: { className: "nginx", clusterIssuer: "", tlsSecret: "memex-registry-tls" }
  resources: { registry: {...}, auth: {...} }

🚨 Deviations from the brief, each measured

docker_auth is pinned to 1.14.0, not 1.13. cesanta/docker_auth:1.13 does not exist on Docker Hub (1.13.0 does), and 1.13.0 CANNOT pair with distribution 3.0: distribution 3 keys rootcertbundle by RFC 7638 JWK thumbprint, docker_auth's default kid is the libtrust id (KF26:LQHR:…) and no x5c is sent, so every token — the publisher's included — was refused with token signed by untrusted key with ID while the token exchange answered 200. 1.14.0 adds token.disable_legacy_key_id: true for this exact case; the config sets it. Verified the failure with 1.13.0 AND with 1.14.0-without-the-flag before the fix.
Digests are multi-arch index digests from docker buildx imagetools inspect (no crane on this host; docker manifest inspect gives the same index).
keyVault.httpSecretObject is also required (the brief's list did not name it): without a shared http.secret, a chunked upload cannot continue on another replica.
validate.sh exits 3, not 2, on a non-definite answer — docker_auth's contract is 0 allow / 1 deny / 2 no-match (try next authenticator) / 3 error; the brief asked for "error, never silently pass", and 3 is what docker_auth logs as an error (bad return code from command: 3).
Notification header: distribution's env override for endpoints[0].headers unmarshals the value as YAML into []string (cannot unmarshal !!str into []string, measured), so the plain vault value is wrapped into list form by the container command at start (MEMEX_REGISTRY_NOTIFICATION_AUTHORIZATION → REGISTRY_NOTIFICATIONS_ENDPOINTS_0_HEADERS_AUTHORIZATION); the config file stays the rendered one.
OTEL_TRACES_EXPORTER=none: distribution 3.0 ships an OTLP exporter on by default and logs traces export … connection refused every 10 s otherwise.
Readiness probes are httpGet / on both (both answer 200; distribution's health wrapper turns that into 503 when the storage driver check fails); liveness is TCP.

Verification

Chart (helm v4.2.4 locally; CI uses v3.16.3): helm lint clean; helm template with values.yaml + values.registry.example.yaml renders all 8 registry objects; default values render ZERO memex-registry* objects; blanking host, publisher.passwordBcrypt, or notificationSecretObject-with-url each fails naming the key. Gates as CI runs them: check-chart-invariants.sh exit 0 (4 combinations, the new one included), check-values-are-read.sh exit 0, test-chart-drift-compare.sh exit 0.

Docker (Colima 29.5.2 linux/aarch64, 2026-09-08 08:10Z) — the two pinned images run with the RENDERED ConfigMaps extracted from helm template (only the storage driver swapped to filesystem, the host to the published localhost ports — 5555/5556, because 5000 is macOS AirPlay on this Mac — and the validation/notification URLs pointed at a python stub answering 200 for one known bearer and 401 otherwise, recording every notification), a self-signed RSA-2048 cert from openssl req -x509, docker_auth's password = htpasswd -nbB publisher example-publisher-password (a $2y$ hash):

GET /      -> 200          GET /v2/ -> 401  Www-Authenticate: Bearer realm="http://localhost:5556/auth",service="localhost:5555"
GET :5001/ -> 200          GET :5001/auth (anon) -> 401
PASS  docker login -u publisher -p example-publisher-password        (exit 0)  ← $2y$ hash accepted
PASS  docker push localhost:5555/test/alpine:1 as publisher          (exit 0)
PASS  docker login -u publisher -p wrong-password                    refused (exit 1), never reached the validator
PASS  docker login -u instance -p <known key>                        (exit 0)  ← stub saw "Authorization: Bearer <key>"
PASS  docker pull localhost:5555/test/alpine:1 as instance           (exit 0)
PASS  docker push localhost:5555/test/alpine:2 as instance           DENIED by ACL (exit 1)
PASS  docker login -u instance -p not-a-key                          refused (exit 1)
PASS  anonymous docker pull                                          refused (exit 1)
PASS  validator stopped → login refused, docker_auth logs "Ext command error: 3 … bad return code from command: 3"
PASS  validator back  → login accepted
PASS  6 notification POSTs at the stub, all with Authorization "Bearer notification-secret-from-vault", actions pull+push

Docs: DocumentationLinkIntegrityTest, WhatsNewEntryIntegrityTest, DocumentationEmbedIntegrityTest run after a -c Release -warnaserror build of MeshWeaver.Documentation.Test (31 projects, 0 warnings, 0 errors).

What's New: 2026-09-08-the-fleet-has-its-own-container-registry.md (Category: Feature).

Pairs-with: none — no public type or member leaves src/; this change is chart, scripts and docs only.

🤖 Generated with Claude Code

…as a separate service Two off-the-shelf images, zero registry code of ours: CNCF distribution 3.0.0 (storage driver azure as the namespace's workload identity, token auth) and cesanta/docker_auth 1.14.0 (one static publisher account by bcrypt hash; every other password validated as a MeshWeaver instance key against the portal via ext_auth). One Ingress on the host: /auth → docker_auth, / → distribution. Off by default; with registry.enabled every required key must be set or the render fails naming it. A complete example overlay is a 4th combination of the chart gate, so the registry renders on every PR. Measured before shipping (real docker login/push/pull against the rendered configs): docker_auth 1.13.0 cannot pair with distribution 3 (libtrust kid vs RFC 7638 lookup) — 1.14.0 with disable_legacy_key_id is the pin; the endpoint headers env override needs YAML-list form; OTLP export is on by default. Part of #3353. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot

🟡 Changes recommended

The new SecretProviderClass syncs memex-registry-secrets using alias names as secretObjects.data[].objectName, which is inconsistent with the chart’s established Key Vault sync pattern and risks leaving required env-backed secrets (e.g. REGISTRY_HTTP_SECRET) empty at runtime.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds an opt-in, chart-managed OCI registry service (cr.meshweaver.cloud) deployed alongside the portal, built entirely from off-the-shelf images (CNCF distribution + cesanta/docker_auth) and wired with Key Vault + workload identity. This change lives in the Helm chart + docs only (no src/ runtime code changes), and is accompanied by a rendered example overlay and chart-gate coverage to keep registry templates exercised on every PR.

Changes:

Introduces registry.* values plus a gated templates/registry/ suite (SecretProviderClass, ConfigMaps, Deployments, Services, Ingress) to run the registry + token service when registry.enabled: true.
Adds a complete values.registry.example.yaml and extends check-chart-invariants.sh so the registry templates are rendered/validated in CI.
Updates architecture + What’s New documentation to reflect the “separate service” decision and measured verification.

File summaries

File	Description
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-08-the-fleet-has-its-own-container-registry.md	Adds a What’s New entry describing the new registry option and behavior.
src/MeshWeaver.Documentation/Data/Architecture/ContainerRegistryInMemex.md	Updates the design page with the “separate service” decision + measured validation notes.
deploy/helm/values.yaml	Adds the `registry:` configuration block (disabled by default) and explanatory comments.
deploy/helm/values.registry.example.yaml	Provides a complete example overlay for CI/template validation.
deploy/helm/templates/registry/_registry.tpl	Centralizes required-value gating and defaults for all registry templates.
deploy/helm/templates/registry/configmap.yaml	Defines distribution + docker_auth configuration, including ext_auth validator script.
deploy/helm/templates/registry/deployment-registry.yaml	Deploys the distribution registry with Key Vault CSI + env wiring and probes.
deploy/helm/templates/registry/deployment-auth.yaml	Deploys docker_auth with Key Vault CSI + validator script and probes.
deploy/helm/templates/registry/ingress.yaml	Adds a single host ingress routing `/auth` to docker_auth and `/` to distribution.
deploy/helm/templates/registry/secretproviderclass.yaml	Adds Key Vault SecretProviderClass and synced Secret for cert/key/http-secret (+ optional notification secret).
deploy/aks/scripts/check-chart-invariants.sh	Ensures CI renders the registry templates via an added values-file combination.

Review details

Files reviewed: 11/11 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        - objectName: cert.pem
+          key: cert.pem
+        - objectName: key.pem
+          key: key.pem
+        - objectName: http-secret
+          key: http-secret
+{{- if $r.notificationSecretObject }}
+        - objectName: notification-authorization
+          key: notification-authorization
+{{- end }}


github-actions · 2026-09-08T08:26:08Z