Skip to content

Registry: pin distribution 3.1.1 — 3.0.0 signs default_credentials as a shared key - #3710

Merged
meshweaver-cloud[bot] merged 1 commit into
mainfrom
fix/registry-distribution-3-1-1
Sep 8, 2026
Merged

meshweaver-cloud[bot] merged 1 commit into
mainfrom
fix/registry-distribution-3-1-1

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

The measured defect

On the first real deploy of the fleet registry (memex-cloud, 2026-09-08, Systemorph/Memex#248) both registry pods stayed 0/1: the storage health check logged 403 AuthenticationFailed … including the signature — the shared-key error — although the rendered config says credentials.type: default_credentials and the pod carried the workload identity correctly.

Root cause is upstream. distribution v3.0.0's registry/storage/driver/azure/auth.go (newClient) dispatches CredentialsTypeDefault together with CredentialsTypeSharedKey to newSharedKeyCredentialsClient — an empty account key, signed. v3.1.0 routes it to newTokenClient (DefaultAzureCredential). Verified by reading both tags' source.

The change

  • deploy/helm/values.registry.example.yaml, values.yaml (comment), templates/registry/_registry.tpl (the required hint): the pin is ghcr.io/distribution/distribution:3.1.1@sha256:bca24727f4002e51f959c18c42e816e4d1078198081a9837e16b8b7d7e43ebf8 (multi-arch index digest, crane digest) — the image that is live.
  • Doc/Architecture/ContainerRegistryInMemex → "The registry as a separate service": one paragraph stating the floor (default_credentials needs distribution ≥ 3.1.0) with the symptom in one line.
  • What's New: Category: Fix entry (an operator enabling the registry on 3.0.0 gets pods that never become ready).

Verified locally

  • deploy/aks/scripts/check-chart-invariants.sh: all 5 combinations render, including "AKS overlay + the fleet registry" with the example values. CI runs no helm lint.
  • dotnet build test/MeshWeaver.Documentation.Test -c Release -warnaserror: 0 warnings, 0 errors; the Documentation DLL postdates the edits and carries distribution:3.1.1 (1), the floor paragraph (1), the new entry (2) and distribution:3.0.0 (0).
  • DocumentationLinkIntegrityTest, WhatsNewEntryIntegrityTest, DocumentationEmbedIntegrityTest with --no-build: 12/12 passed, fresh .trx.

Pairs-with: none — chart example and docs only

Refs #3697, #3353, Systemorph/Memex#248.

🤖 Generated with Claude Code

… a shared key

On the first real deploy of the fleet registry (memex-cloud, 2026-09-08) both
registry pods stayed 0/1: the storage health check logged
`403 AuthenticationFailed … including the signature` — the shared-key error —
although the config says `credentials.type: default_credentials` and the pod
carried the workload identity correctly.

Root cause is upstream: distribution v3.0.0's azure driver (`auth.go`,
`newClient`) dispatches CredentialsTypeDefault together with
CredentialsTypeSharedKey to the shared-key client, signing every request with
an empty account key. v3.1.0 routes it to the token client
(DefaultAzureCredential). Verified by reading both tags' source.

- deploy/helm/values.registry.example.yaml, values.yaml, _registry.tpl: the pin
  is ghcr.io/distribution/distribution:3.1.1@sha256:bca24727… (multi-arch index
  digest, `crane digest`) — the image that is live.
- ContainerRegistryInMemex.md: one paragraph stating the floor (≥ 3.1.0) and the
  symptom the next reader will recognise.
- What's New: a Fix entry.

Refs #3697, #3353, Systemorph/Memex#248.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 8, 2026 09:45
@meshweaver-cloud
meshweaver-cloud Bot enabled auto-merge September 8, 2026 09:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes are a consistent pin/documentation update across chart examples, required hints, and docs, with no functional code changes and no inconsistencies found in the updated references.

Pull request overview

Pins the Helm chart’s optional in-cluster container registry to a distribution version that correctly uses workload identity (credentials.type: default_credentials) for Azure storage auth, and documents the minimum supported distribution version to prevent non-ready registry pods in real deployments.

Changes:

  • Bump pinned distribution image reference from 3.0.0 to 3.1.1 (example values + required-value hint + documented reference).
  • Document the minimum distribution version (≥ 3.1.0) and the observed failure mode on 3.0.0.
  • Add a “What’s New” Fix entry capturing the operator-visible impact.
File summaries
File Description
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-08-registry-pins-distribution-3-1-1.md Adds a Fix entry describing the readiness failure on 3.0.0 and the new 3.1.1 pin.
src/MeshWeaver.Documentation/Data/Architecture/ContainerRegistryInMemex.md Updates the design doc’s referenced registry image to 3.1.1 and documents the ≥3.1.0 floor for workload identity auth.
deploy/helm/values.yaml Updates the commented, digest-pinned distribution image reference to 3.1.1.
deploy/helm/values.registry.example.yaml Updates the example overlay’s distribution image pin to 3.1.1.
deploy/helm/templates/registry/_registry.tpl Updates the required-error hint to reflect the 3.1.1 digest-pinned image shape.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

494 tests  ±0   494 ✅ ±0   42s ⏱️ -1s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit afa9a3a. ± Comparison against base commit 6044910.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

254 tests  ±0   254 ✅ ±0   2m 15s ⏱️ -22s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit afa9a3a. ± Comparison against base commit 6044910.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

1 073 tests  ±0   1 073 ✅ ±0   1m 35s ⏱️ +2s
    2 suites ±0       0 💤 ±0 
    2 files   ±0       0 ❌ ±0 

Results for commit afa9a3a. ± Comparison against base commit 6044910.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

1 666 tests  ±0   1 472 ✅ ±0   2m 7s ⏱️ ±0s
    4 suites ±0     194 💤 ±0 
    4 files   ±0       0 ❌ ±0 

Results for commit afa9a3a. ± Comparison against base commit 6044910.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    5 files  ±0      5 suites  ±0   3m 10s ⏱️ -2s
1 703 tests ±0  1 703 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 704 runs  ±0  1 704 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit afa9a3a. ± Comparison against base commit 6044910.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 183 tests  ±0   1 183 ✅ ±0   4m 16s ⏱️ -4s
    3 suites ±0       0 💤 ±0 
    3 files   ±0       0 ❌ ±0 

Results for commit afa9a3a. ± Comparison against base commit 6044910.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Test Results

   16 files  ±0     16 suites  ±0   14m 6s ⏱️ -29s
6 373 tests ±0  6 179 ✅ ±0  194 💤 ±0  0 ❌ ±0 
6 374 runs  ±0  6 180 ✅ ±0  194 💤 ±0  0 ❌ ±0 

Results for commit afa9a3a. ± Comparison against base commit 6044910.

@meshweaver-cloud
meshweaver-cloud Bot added this pull request to the merge queue Sep 8, 2026
Merged via the queue into main with commit 6b2fe2a Sep 8, 2026
36 checks passed
@rbuergi
rbuergi deleted the fix/registry-distribution-3-1-1 branch September 13, 2026 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants