Repository navigation
ci: stop pulling Docker Hub anonymously — drop the unused pgvector pre-pull, add a digest-pinned GHCR mirror - #6400
Conversation
…e-pull, add a digest-pinned GHCR mirror
Core PR runs went red on Docker Hub's anonymous pull limit (toomanyrequests),
which is counted per egress IP of the shared hosted-runner pool.
- dotnet-test.yml: delete the per-shard pre-pull of pgvector/pgvector:pg17. No
core suite starts a container any more (the PostgreSql suites left core in the
test move), so the pull fetched an unused image and was the only failure.
- main-cd.yml: control acceptance passes --postgres with the ACR dockerhub/ copy;
the job is already logged in to ACR.
- dockerhub-mirror.{list,yml} + scripts/dockerhub-mirror.sh: copy declared Docker
Hub images by digest to ghcr.io/systemorph/dockerhub/, read back and compare;
`ref` resolves a declared image (undeclared = red, no docker.io fallback). PR
jobs read it with GITHUB_TOKEN, so no secret is needed (forks, Dependabot). The
Chart Gate consumers move onto it in the follow-up, once main has synced it.
- Doc/Architecture/DockerHubInCi.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Scheduled failures are not reported durably, duplicate declarations are ambiguous, and documentation overstates current coverage.
9 open findings
What changed in this PR
Introduces registry mirrors to reduce Docker Hub rate-limit failures in CI.
Changes:
- Removes the unused pgvector pre-pull.
- Adds GHCR mirror synchronization and lookup tooling.
- Routes control-image acceptance through the existing ACR mirror and documents the strategy.
| File | Description |
|---|---|
.github/dockerhub-mirror.list |
Declares digest-pinned mirror images. |
.github/scripts/dockerhub-mirror.sh |
Resolves and synchronizes mirror entries. |
.github/workflows/dockerhub-mirror.yml |
Adds mirror validation and scheduled synchronization. |
.github/workflows/dotnet-test.yml |
Removes the obsolete pgvector pre-pull. |
.github/workflows/main-cd.yml |
Uses ACR-hosted pgvector for acceptance tests. |
src/MeshWeaver.Documentation/Data/Architecture.md |
Adds the architecture-page index entry. |
src/MeshWeaver.Documentation/Data/Architecture/DockerHubInCi.md |
Documents the mirror strategy and exceptions. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| entries() { | ||
| [ -f "$LIST" ] || { echo "::error::dockerhub-mirror: no list at $LIST" >&2; return 1; } | ||
| local line n=0 | ||
| while IFS= read -r line || [ -n "$line" ]; do | ||
| n=$((n + 1)) | ||
| line="${line%%#*}"; line="$(printf '%s' "$line" | tr -d '[:space:]')" | ||
| [ -n "$line" ] || continue | ||
| if [[ ! "$line" =~ ^([a-z0-9]+([._-][a-z0-9]+)*(/[a-z0-9]+([._-][a-z0-9]+)*)*):([A-Za-z0-9_][A-Za-z0-9._-]{0,127})@(sha256:[0-9a-f]{64})$ ]]; then | ||
| echo "::error::dockerhub-mirror: $LIST line $n is not <name>:<tag>@sha256:<64 hex>: '$line'" >&2 | ||
| return 1 | ||
| fi | ||
| printf '%s:%s %s\n' "${BASH_REMATCH[1]}" "${BASH_REMATCH[5]}" "${BASH_REMATCH[6]}" | ||
| done < "$LIST" | ||
| } |
There was a problem hiding this comment.
Fixed in 005a011: entries() now refuses a second line for the same <name>:<tag> (red, naming the line), so sync and ref can never disagree. New self-test arm: "a second digest for the same name:tag is RED, never first-wins".
| schedule: | ||
| # Daily: a mirror whose package was deleted or whose retention dropped a manifest is red the | ||
| # same day, not on the next pull request that needs it. | ||
| - cron: "40 4 * * *" | ||
| workflow_dispatch: |
There was a problem hiding this comment.
Not adding a tracking-issue reporter in this change. sync is self-restoring: on each daily run it copies any declared digest the mirror is missing. A failure it cannot fix also reaches its consumers immediately, because dockerhub-mirror.sh ref and the pull of the pinned digest go red in the Chart Gate step, naming the mirror line. So nothing stays silently unavailable. A sentinel issue is worth adding if a scheduled red ever goes unnoticed, but it is not needed for the fix.
| # dockerhub-mirror.sh — the ONE route by which CI reaches a Docker Hub image | ||
| # (Doc/Architecture/DockerHubInCi). |
There was a problem hiding this comment.
Fixed in 005a011: the header now scopes the script to the runtime images declared in the list, and names FROM base layers as out of scope.
| # red no code change could cause or cure. Retrying is not a fix (the limit is a window, not a | ||
| # blip); the fix is to stop asking Docker Hub from pull requests at all. This workflow is now the | ||
| # ONLY place CI touches docker.io, once per new digest, from main or the schedule. |
There was a problem hiding this comment.
Fixed in 005a011: the workflow header now says "for every image DECLARED in the list", excludes path-filtered FROM base layers, and names the Chart Gate follow-up.
| # Daily: a mirror whose package was deleted or whose retention dropped a manifest is red the | ||
| # same day, not on the next pull request that needs it. |
There was a problem hiding this comment.
Fixed in 005a011: the schedule comment and the doc now say the job RESTORES a deleted package or manifest, and is red only when it cannot restore it.
| - [Rendering a chart you are not allowed to fully configure](ChartDriftRenderWithoutSecrets) — 39 runs, 39 failures, zero verdicts: the check may hold two of the deploy's three value sources and the chart correctly refuses that subset; the placeholder that unblocks the render, the two-render proof that no compared object depends on it, and the bake gate that was off on both production namespaces the moment a verdict finally appeared | ||
| - [Configuring an instance from Aspire](ConfiguringAnInstanceFromAspire) | ||
| - [The Dependabot Secret Store](DependabotSecretStore) | ||
| - [Docker Hub in CI](DockerHubInCi) — hosted runners pull Docker Hub anonymously against a per-IP limit they share with every tenant; CI never asks it from a pull request: a digest-pinned mirror on GHCR read with the job's own token, or the ACR copy where a job is already logged in |
There was a problem hiding this comment.
Fixed in 005a011: the index line now says the mirror is the route for the runtime images CI jobs declare, and points at the exceptions still out of scope.
| **A CI job never pulls from Docker Hub anonymously.** GitHub-hosted runners reach Docker Hub | ||
| from a shared pool of egress IPs. The anonymous pull limit is counted per IP, so other tenants | ||
| spend it before we do. A `toomanyrequests` answer is therefore a red that no commit causes and | ||
| no commit cures. Retrying does not help: the limit is a window of hours, not a blip of seconds. |
There was a problem hiding this comment.
Fixed in 005a011: the opening invariant is narrowed to declared runtime images, and What this does not cover now lists the staged Chart Gate follow-up next to the FROM exceptions.
| `dockerhub-mirror.yml` is the only place CI talks to `docker.io`. It runs on `main` and on the | ||
| schedule, once for each new digest. |
There was a problem hiding this comment.
Fixed in 005a011: the sentence now reads "For a declared image, dockerhub-mirror.yml is the only place CI talks to docker.io".
| The sweep read every failed job log in `Systemorph/MeshWeaver` and `Systemorph/MeshWeaver.Plugins` | ||
| over the window the run search returned, and counted the logs that contain `toomanyrequests`. The | ||
| counts, and the window each count covers, are in the pull request body. |
There was a problem hiding this comment.
Fixed in 005a011: the measured result is now recorded on the page itself, with its exact window: 2026-10-09 20:46:30–20:58:57Z, 7 of 8 failed core runs, 23 test-shard jobs plus 5 Chart Gate jobs. The page says explicitly that the sweep stopped before older logs were read (because of the secondary rate limit), and that it claims no count for the rest of the week or for MeshWeaver.Plugins.
… scope the claims to declared runtime images Review: a second line for one name:tag let sync mirror both while ref served the first — now red. The read-back no longer swallows stderr (workflow-shell gate). Wording narrowed to what the change enforces: declared images only, Chart Gate in the follow-up, FROM base layers out of scope, deletion is restored not red, and the measured window stated with what was NOT read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Why
Core PR runs went red on Docker Hub's anonymous pull limit. On #6381 (Refs #6381), all six
Run testsshards and Chart GateBundle registry scripts (executed)failed withtoomanyrequests: You have reached your unauthenticated pull rate limit. GitHub-hosted runners share their egress IPs with every other tenant, and the anonymous limit is counted per IP, so no commit can cause this red or cure it. Retrying a window of hours does not help either. The root fix is to stop pulling Docker Hub from pull requests.Inventory: what pulled Docker Hub anonymously (core + shared node-repo lanes)
dotnet-test.yml→ Pre-pull Testcontainers images (every shard, every PR)pgvector/pgvector:pg17main-cd.yml→ control acceptance (control-image-acceptance.shdefault)pgvector/pgvector:pg17--postgres meshweaver.azurecr.io/dockerhub/pgvector/pgvector:pg17. The job is already ACR-logged-in (OIDC), and this is the copy the runner sets' shared Postgres already runs.chart-gate.yml→ Bundle registry scripts (executed)cesanta/docker_auth:1.14.0@sha256:98e0…chart-gate.yml→ Observability valuesgrafana/loki:2.9.3sha256:eb92…)clients.yml,s1-extract-image.yml,whisper-image.ymlimage builds (FROM python/node/ubuntu)node-repo-*lanesTESTCONTAINERS_HUB_IMAGE_NAME_PREFIX=meshweaver.azurecr.io/dockerhub/plusdocker loginin module-pack). Nothing else in them names a Docker Hub image.Which mirror, and why
meshweaver.azurecr.io/dockerhub/(az acr importcopies; there is no ACR cache rule, becauseaz acr cache createneeds a Docker Hub credential) is used where a job is already logged in to ACR. In core that is main only: core has no ACR password, and its OIDC federated credentials matchmainalone (Doc/Architecture/DependabotSecretStore).cr.meshweaver.clouddenies anonymous pulls and holds no Docker Hub images. Reading it from a PR would need the publisher (push) password, so it is not used.ghcr.io/systemorph/dockerhub/is used for pull-request jobs. It is the fleet's existing mirror of its own images, the same precedent assatellite-compat-imagein main-cd. A job reads it with its ownGITHUB_TOKEN, so there is no secret to provision, and forks and Dependabot pull the same way. Every entry is pinned by digest.dockerhub-mirror.yml(main push + daily + dispatch) copies each digest withimagetools createand reads it back.dockerhub-mirror.sh refis red for an undeclared image and never falls back to docker.io. There are no retries anywhere.The Chart Gate consumers move to the mirror in a follow-up PR, after this merge has synced the mirror on main. Switching them here would make this PR's own gate depend on packages that do not exist yet.
Verification
dockerhub-mirror.sh --self-test: 7/7 pass. Negative control: with a wrong prefix, 2 arms fail and rc=1.shellcheckis clean.actionlintis clean on the new workflow.check-workflow-timeouts.py: 0 violations.check-pr-secret-preflight.py: 0 violations.Measured frequency
Failed core runs created 2026-10-09 20:46:30–20:58:57Z: 7 of 8 carried
toomanyrequests(3 Build and Test runs = 23 test-shard jobs; 4 Chart Gate runs = 5 bundle-registry jobs). The sweep was stopped before reading older logs, because log downloads were feeding the secondary rate limit the concurrent agents were already hitting. So no count is claimed for the rest of the week or for MeshWeaver.Plugins (1,758 and 1,298 failed runs listed, not read). Recorded on Doc/Architecture/DockerHubInCi → Measured.Pairs-with: none — no public C# surface changes (workflow, script and doc only).
🤖 Generated with Claude Code