Skip to content

[CI] Use Blacksmith Docker layer caching for image builds - #2

Merged
mrubens merged 1 commit into
developfrom
ci/blacksmith-docker-layer-cache
Jul 8, 2026
Merged

[CI] Use Blacksmith Docker layer caching for image builds#2
mrubens merged 1 commit into
developfrom
ci/blacksmith-docker-layer-cache

Conversation

@mrubens

@mrubens mrubens commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Problem

GHCR image builds almost never hit their Docker layer cache. Diagnosed from run 28962359037:

  • The publish workflow used type=gha buildx caching with mode=max across four scopes (app/worker × amd64/arm64). The exported layers total more than GitHub's 10GB per-repo Actions cache cap (the repo was sitting at 10.55GB with LRU eviction active).
  • GHA cache reads are scoped to the run's own ref plus the default branch, so publishing the same commit from develop, main, and a v* tag re-exported the full multi-GB cache set three times under three refs, evicting each other.
  • Net effect: nearly every build (ARM especially) rebuilt the full image from scratch (~8-10 min) and then spent ~3.5 minutes uploading a cache export that rarely survived to the next run.
  • CI's Dockerfile build pulled from a gha scope that no workflow writes, so it never hit either.

Change

Switch both workflows to Blacksmith's persistent builder (useblacksmith/setup-docker-builder@v1 + useblacksmith/build-push-action@v2, a drop-in fork of docker/build-push-action):

  • Layer cache lives on an NVMe sticky disk mounted into the runner, keyed per repo/Dockerfile/arch: no 10GB cap, no ref scoping, no cache export upload step.
  • Dropped the now-unneeded cache-from/cache-to directives.
  • The manifest-merge job keeps stock docker/setup-buildx-action since it only runs imagetools create.
  • Documented the caching setup in the deployment guidance doc.

The existing native-runner matrix (no QEMU) and push-by-digest flow are unchanged; the Blacksmith action exposes the same digest output.

Expected impact

Warm builds should drop from ~8-10 min per arch to roughly image-push time. The first run per Dockerfile/arch is a cold cache and will still take full build time.

Validation

  • pnpm lint (prettier) passes on the edited workflows; pre-push hook (lint:fast, check-types:fast, knip) passed.
  • CI on this PR exercises the new actions via the Dockerfile build job.
  • The publish path only runs on develop/main/tag pushes, so the first real verification will be the next develop push (expect a cold build) and the one after it (expect cache hits).

The GHCR publish builds used type=gha buildx caching with mode=max across
four scopes (app/worker x amd64/arm64). The exported layers overflow
GitHub's 10GB per-repo Actions cache, and GHA cache reads are scoped to
the run's own ref plus the default branch, so develop/main/tag publishes
of the same commit each re-exported the full cache set and evicted each
other. In practice every build (ARM especially) rebuilt from scratch and
then spent ~3.5 minutes uploading a cache export that rarely survived
until the next run. CI's Dockerfile build read from a gha scope that no
workflow writes, so it never hit either.

Switch both workflows to Blacksmith's persistent builder
(useblacksmith/setup-docker-builder + useblacksmith/build-push-action),
which mounts an NVMe-backed layer cache on the runner keyed per
repo/Dockerfile/arch: no size cap, no ref scoping, and no cache export
upload step.
@roomote-roomote-v1

roomote-roomote-v1 Bot commented Jul 8, 2026

Copy link
Copy Markdown

No code issues found. See task

Reviewed the swap to Blacksmith's persistent builder in .github/workflows/CI.yml and .github/workflows/publish-ghcr.yml plus the deployment.md note. Both the CI docker-build job and the publish build matrix run on Blacksmith runners (blacksmith-4vcpu-ubuntu-2404 / -arm), which is the prerequisite for useblacksmith/setup-docker-builder and useblacksmith/build-push-action to mount the NVMe layer cache, so dropping cache-from/cache-to is safe. The push-by-digest outputs, digest export, and 2-arch manifest-merge flow are unchanged, and the merge job correctly keeps stock docker/setup-buildx-action since it only runs imagetools create.

Two non-blocking observations (no change requested):

  • useblacksmith/setup-docker-builder@v1 / useblacksmith/build-push-action@v2 are third-party actions pinned to floating major tags. This matches the repo's existing convention for docker/* and actions/*, but SHA-pinning third-party actions is a stronger supply-chain posture if you ever tighten it.
  • Correct behavior now depends on these jobs always running on Blacksmith runners; if a future runs-on change moves them to GitHub-hosted runners, the cache benefit (and possibly the builder setup) would silently regress.

@mrubens
mrubens merged commit db49120 into develop Jul 8, 2026
@mrubens
mrubens deleted the ci/blacksmith-docker-layer-cache branch July 8, 2026 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants