feat(registry): per-repository bundle authorization, the bundle-fetch init container, and the publisher script by rbuergi · Pull Request #3729 · Systemorph/MeshWeaver

rbuergi · 2026-09-08T12:51:32Z

The chart-and-scripts half of Plugin Bundles in the Registry (design, #3720 — this branch is stacked on it and its diff shrinks to the implementation once that merges). The consumer-side C# (OciRegistryClient, the catalog's artifact field) is #3726; nothing here duplicates it. Refs #3697, Systemorph/Memex#244.

🚧 DRAFT on purpose — core merges are held while a CD bake seals. Not to be armed or merged until the hold lifts.

1. Per-repository authorization at the edge — mechanism (b), labels

docker_auth 1.14.0 has two hooks, and only one can carry this: ext_authz receives the marshalled request (account, type, name, actions, labels) and not the credential, so it cannot ask memex anything at pull time; ext_auth (auth_server/authn/ext_auth.go, Authenticate(user, password) (bool, api.Labels, error)) is handed the password and may return {"labels": {...}} that the ACL matches with ${labels:<name>}. So:

deploy/helm/files/registry-validate.sh (rendered verbatim into memex-registry-auth-config) exchanges the key at POST /api/instances/token with an empty body — 200 authenticates, 401/403 refuses, anything else is exit 3 (error, never a quiet deny) — and emits the answered scope[] as labels in the registry's lowercase grammar: package = <source>/<package> per entry (<source>/* for a whole-source entry, plan-scoped or not), source = <source> for plan-less whole-source entries only.
The ACL (templates/registry/configmap.yaml, first match wins): publisher push,pull,delete on *; plugins/releases any authenticated; plugins/${labels:source} pull (the index); plugins/${labels:package} pull (bundle repositories); every other plugins/… → actions: []; any other repository → any authenticated pull. So Plugins/*@pro reaches plugins/plugins/<pkg> and never plugins/plugins.
plugins/_releases from the design is not a legal repository name (a path component must start alphanumeric — refused by distribution 3.1.1 and by ORAS); it is plugins/releases, and the doc/table say so. releases is a reserved source name in the publisher.
docker_auth runs with -alsologtostderr=true: measured, its docker logs are otherwise EMPTY (glog writes Info to files).

2. `bundle-fetch` init container (`deploy/helm/templates/memex-portal/deployment.yaml`)

Values (all under bundles., gated by _bundles.tpl, which fails helm template naming any missing key): registry (default "" ⇒ nothing renders), sources: [], identity or identityFile, image (ORAS v1.3.4 pinned by index digest, verified with docker buildx imagetools inspect), root (defaults to config.memex_portal.PreWarm__PrebuiltBundleRoot), resources; requires portal.imagePullSecret, mounted as ORAS's --registry-config from the same kubernetes.io/dockerconfigjson Secret the pod pulls its image with.

deploy/helm/files/bundle-fetch.sh resolves plugins/<source>:<identity>, pulls the digest's whole graph into a staging dir (every titled layer lands under its name — bundles, modules/*.module.nupkg, modules/_index, _complete, and every sidecar generically, including one no reader knows yet — so #3727's platform-version file rides along untouched), checks _complete against what landed, then renames into <root>/<identity>/<source>/ (one rename ⇒ whole or absent; _complete is inside the renamed tree, so it becomes visible last). Absent tag ⇒ one log line, nothing written, exit 0. Anything else ⇒ exit 1, pod held.

Identity finding: the brief suggested reading the image's /app framework-mvid file. There is none — the identity is computed by FrameworkBuildIdentity from the surface manifests; only the bake lane materialises it as framework-mvid.txt in its own workspace. Hence bundles.identity (the value the lane prints as baked identity:) or bundles.identityFile (a one-line file on the data volume, read at run time). A stale identity is inert (the pre-warm finds no directory for its own identity and compiles).

check-chart-invariants gains a sixth combination (testdata/values.bundle-fetch.yaml) and invariant 13 (init container and portal mount the same volume at the pre-warm root; BUNDLES_ROOT equals it; the projected docker config is the pod's imagePullSecrets Secret; the script ConfigMap is rendered and hashed into the pod template). check-values-are-read.sh stays green (119/119).

3. `.github/scripts/push-bundle-publication.sh`

--registry <host> --source <name> --identity <id> --dir <publication dir> [--tag-run <run>] [--release <version>] [--registry-config …] [--plain-http]. Order = the seal: each bundle as an image manifest (.zip application/vnd.meshweaver.bundle.v1.zip, .module.nupkg application/vnd.meshweaver.module.v1.nupkg) tagged <identity> in plugins/<source>/<package> and copied by digest into plugins/<source>; the sidecar manifest (every non-bundle file as a titled layer; config blob application/vnd.meshweaver.publication.v1+json with identity/source/sourceCommit/repository/architecture/release/bundles(package,digest,module)/files); the index by digest; then tags <identity>-<run> and <identity> last. Refuses no _complete, empty _complete, a listed bundle/module absent, an unspellable name. org.opencontainers.image.created pinned to the epoch so the same bytes yield the same index digest (asserted). Prints bundle-publication: index=… bundles=N and one bundle=<pkg> digest=… module=yes|no per bundle. Standalone — not wired into a workflow.

4. Docs

Design page: consumer table + a bundle-fetch section with the values table, Authorization rewritten for the labels mechanism, Publishing names the script, Verification names the harness. What's New: Category: Feature.

Verification

Docker proof (Colima, real ghcr.io/distribution/distribution:3.1.1 + cesanta/docker_auth:1.14.0, the ConfigMap helm renders, a stub /api/instances/token answering per key): python3 .github/scripts/test-bundle-registry.py → 83 assertions, 0 failures. Publisher facts/tags/index/config; idempotent digest; three refusals write no tag; fetch layout byte-identical to the source dir (10/10 files), unsealed source writes nothing, identityFile, re-run replaces the shelf; matrix: full (Plugins/* + Reinsurance/UWDeepfield) pulls index + pkg + releases + image, not reinsurance/education; pro (Plugins/*@Pro) pulls pkg not the index; edu (Plugins/Edu) reaches plugins/plugins/edu (404, authorized) and is denied pkg.a; 403 key refused; anonymous refused; installation accounts cannot push; keys never appear in docker_auth's log; fetch through the edge: edu/pro exit 1 writing nothing, full materialises byte for byte.
check-chart-invariants.sh: all 6 combinations self-consistent. check-values-are-read.sh: 119/119. check-workflow-timeouts.py: 0 violations (new job 20 min). check-workflow-shell.py, check-pr-secret-preflight.py: clean. ORAS linux-amd64 tarball sha256 re-downloaded and matches the pin CI will verify.
test/MeshWeaver.Documentation.Test builds -c Release -warnaserror (0 warnings, 0 errors); DocumentationLinkIntegrityTest, WhatsNewEntryIntegrityTest, DocumentationEmbedIntegrityTest, WorkflowJobIfContextGuard, WorkflowRunTriggerBranchFilterGuard: 14 passed; the rebuilt DLL contains the new entry and the edited page (grep-verified, not an up-to-date no-op).

Not touched, per the hold on other sessions' files: main-cd.yml, node-repo-publish-bake.yml, publish-bake-bundles.sh, deploy/aks/operator/*, ShippedPrebuiltBundles.cs, PrebuiltAssemblySeeder.cs, SealedSyncGate.cs.

Pairs-with: none — no public type or member leaves src/; the change is chart, shell and docs.

🤖 Generated with Claude Code

…bundle-fetch init container, and the publisher script Doc/Architecture/PluginBundlesInTheRegistry, the chart and scripts half (the consumer-side C# is #3726). * Authorization: docker_auth 1.14.0's ext_auth hook (deploy/helm/files/registry-validate.sh) exchanges the instance key at memex once, at login, and turns the answered licence scope into token labels — package (<source>/<package>, <source>/* for a whole-source entry) and source (<source>, plan-less whole-source entries only). The static ACL grants pull on plugins/${labels:package} and plugins/${labels:source}, denies every other plugins/ object, and keeps plugins/releases and image repositories open to any authenticated account; the publisher keeps push/pull/delete on *. ext_authz cannot carry this: it receives the request, never the credential. docker_auth now runs with -alsologtostderr so its verdicts reach kubectl logs. * bundle-fetch init container (bundles.registry/sources/identity|identityFile/image/root, off by default, gated by _bundles.tpl): pulls plugins/<source>:<identity> with ORAS (pinned by digest) using the pod's imagePullSecret as its docker config, stages, checks _complete, renames into <PreWarm__PrebuiltBundleRoot>/<identity>/<source>/ — the layout ShippedPrebuiltBundles reads. Absent tag ⇒ one log line, nothing written, exit 0; anything else exits 1 and holds the pod. check-chart-invariants gains a sixth combination with it on and invariant 13 (shelf = pre-warm root, credential = pull secret). * .github/scripts/push-bundle-publication.sh: one sealed publication → bundle manifests (zip + module layers), a sidecar manifest (every non-bundle file as a titled layer, facts in its config blob), the index by digest, then <identity>-<run> and <identity> LAST. Refuses a directory without _complete or with a listed file absent. Standalone, not wired into a lane. * .github/scripts/test-bundle-registry.py (chart-gate, "Bundle registry scripts (executed)"): runs the three scripts against the real distribution 3.1.1 and docker_auth 1.14.0 images with the ConfigMap the chart renders and a stub key exchange; 83 assertions, byte-level diff of the materialised layout, full licence matrix. ORAS pinned by version + sha256 per platform. * Docs: the design page's consumer table, init-container section, Authorization and Publishing describe what ships; What's New (Feature). Refs #3720, #3697, #3726, Systemorph/Memex#244.

Copilot

🟡 Changes recommended

bundle-fetch.sh currently allows ./..-style source names that can turn rm -rf "$staging_root/$source" into deletion outside the intended staging directory, and the validator label grammar should be tightened to match the registry’s component rules.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR implements the chart/scripts half of “Plugin Bundles in the Registry”: it adds per-repository bundle authorization at the registry edge, a bundle-fetch init container that materializes sealed publications into the pre-warm filesystem layout, and a standalone publisher script plus an execution harness and documentation to validate the end-to-end behavior.

Changes:

Add per-repository authorization for plugins/<source> and plugins/<source>/<package> using docker_auth ext_auth labels populated from the instance-token exchange.
Add the bundle-fetch init container (templating + script) to pull publications by identity into PreWarm__PrebuiltBundleRoot before the portal starts.
Add a publisher script (push-bundle-publication.sh), an executed CI harness (test-bundle-registry.py), and docs/What’s New entries describing the new flow.

File summaries

File	Description
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-08-plugin-bundles-are-pulled-from-the-fleet-registry.md	Adds a user-facing What’s New entry for bundle materialization + authorization.
src/MeshWeaver.Documentation/Data/Architecture/PluginBundlesInTheRegistry.md	Documents OCI artifact layout, edge authorization, bundle-fetch, and publishing.
src/MeshWeaver.Documentation/Data/Architecture/ContainerRegistryInMemex.md	Adds a “Related” link to the new bundle-registry design page.
deploy/helm/values.yaml	Introduces `bundles.*` values (registry, sources, identity/identityFile, root, resources).
deploy/helm/templates/registry/deployment-auth.yaml	Ensures docker_auth logs reach stderr and passes `MEMEX_VALIDATION_URL` to the validator hook.
deploy/helm/templates/registry/configmap.yaml	Updates ACL ordering and switches validate script to a rendered file (`registry-validate.sh`).
deploy/helm/templates/memex-portal/deployment.yaml	Wires the `bundle-fetch` init container, volumes, mounts, and a checksum roll trigger.
deploy/helm/templates/memex-portal/bundle-fetch.yaml	Adds ConfigMap that renders `bundle-fetch.sh` verbatim.
deploy/helm/templates/memex-portal/_bundles.tpl	Adds a gating helper that validates `bundles.*` keys and returns a normalized config block.
deploy/helm/files/registry-validate.sh	Implements docker_auth `ext_auth` validator that exchanges instance keys for scopes and emits labels.
deploy/helm/files/bundle-fetch.sh	Implements init container logic to resolve/pull publications by identity and atomically lay them out on disk.
deploy/aks/scripts/testdata/values.bundle-fetch.yaml	Adds a values fixture to exercise rendering of bundle-fetch resources in invariants.
deploy/aks/scripts/check-chart-invariants.sh	Adds a new invariant combination to render bundle-fetch-enabled config.
deploy/aks/scripts/check-chart-invariants.py	Adds invariant #13 asserting mounts/paths/credential/script consistency for bundle-fetch.
.github/workflows/chart-gate.yml	Adds a job that executes the end-to-end bundle registry harness in CI.
.github/scripts/test-bundle-registry.py	Adds a harness that executes publisher/fetch/validator against real images + rendered chart config.
.github/scripts/push-bundle-publication.sh	Adds a standalone publisher that pushes the publication index/manifests in seal order (tag last).

Review details

Files reviewed: 17/17 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+  repo_source=$(printf '%s' "$source" | tr '[:upper:]' '[:lower:]')
+  case "$repo_source" in
+    *[!a-z0-9._-]*|'') die "source '$source' is not a valid registry name component" ;;
+  esac


+  # Only what the registry's name grammar can spell becomes a label; anything else is dropped,
+  # which denies rather than matching something unintended.
+  case "$source" in *[!a-z0-9._-]*|'') continue ;; esac
+  case "$package" in '*') ;; *[!a-z0-9._-]*|'') continue ;; esac


github-actions · 2026-09-08T12:59:47Z