Every registration is a cold build (`--pull --no-cache`), and two `--build-arg` values are missing their `=`

[NicolasRouquette]

Two problems visible in one job log. They are unrelated in mechanism but share an origin — the docker build invocation — so they are filed together.

1. --pull --no-cache makes every registration a full cold build

The command line, as echoed by the script itself:

[CI] Build for: container-soil-moisture-workflows:v0.18.5-sm-bench-avs-fit-cpu and file soil-moisture-workflows
 docker build --pull --no-cache --rm --force-rm -f docker/Dockerfile -t container-…-cpu --build-arg … .

Measured from that one job:

• --pull re-fetches the base image on every registration: a 295.57 MB layer plus eight smaller ones, 27.2 s wall clock, before step 2 of 19 begins.
• --no-cache re-executes all 19 Dockerfile steps, including every one that reaches the network.

The second cost is not just time. It makes each registration depend on third-party availability. Same repository, same day, a different algorithm:

#8 [ 3/19] RUN pip install backoff
#8 0.746 Collecting backoff
#8 8.335 ERROR: Could not install packages due to an OSError:
         HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded
         … (Caused by ResponseError('too many 502 error responses'))
ERROR: failed to solve: process "/bin/sh -c pip install backoff" did not complete successfully: exit code: 1
[ERROR] Failed to build docker container for: container-soil-moisture-workflows:v0.18.5-sm-bench-avs-fit-cpu

An eight-second PyPI hiccup failed a registration. With a layer cache that step would not have run at all. Registering N algorithms out of one repository means N independent coin flips against PyPI, and against every other host the Dockerfile touches.

What makes --no-cache look unintended: the caller already computes a cache-busting argument and passes it alongside --no-cache:

+ CACHE_BUST=1787101139          # $(date +%s)
… --build-arg CACHE_BUST=1787101139 …

CACHE_BUST exists precisely to invalidate the layers that must be re-run, at the point in the Dockerfile where they must be re-run. --no-cache invalidates everything and renders it inert. Two mechanisms are doing the same job and the blunt one wins.

Suggestions

• Drop --no-cache and let CACHE_BUST do what it was added for.
• If a cold build must stay available, make it opt-in per job — an environment variable, or a commit-message flag in the style of the existing SKIP_IMAGE_BUILD convention.
• --pull can be kept or replaced. BuildKit already resolves the base to a digest (FROM …@sha256:01e74ed2… appears in the log), so pinning gives the same freshness guarantee without the re-pull on every job.
• Independently of the above, network-touching RUN steps deserve retries and a BuildKit cache mount — e.g. RUN --mount=type=cache,target=/root/.cache/pip pip install --retries 10 --timeout 60 backoff. That belongs in the generated Dockerfile rather than here, but it is the same failure being defended against.

Where does this line live?

Not in this repository. --pull and --no-cache appear in no revision of build-container.bash on any ref here — 12 revisions of that file, develop being the only branch. The copy that actually ran is /home/ops/verdi/ops/container-builder/build-container.bash inside hysds/verdi:v5.2.0, whose container-met.py matches 01422ad. So the image ships a modified copy. Happy to move this issue wherever that copy is maintained — a pointer would be welcome either way.

2. Two --build-arg values are missing their =

The CI wrapper's own set -x trace and the command echoed by build-container.bash agree, so this is argv and not a log artefact:

+ export MINICONDA_INSTALLER=https://repo.anaconda.com/miniconda/Miniconda3-py311_24.7.1-0-Linux-x86_64.sh
+ export MINICONDA_INSTALLER_SHA256=a098a5b1581d8fd078c430b82e27106602223e335efef708a124e723814d120c
…
--build-arg MINICONDA_INSTALLERhttps://repo.anaconda.com/miniconda/Miniconda3-py311_24.7.1-0-Linux-x86_64.sh
--build-arg MINICONDA_INSTALLER_SHA256a098a5b1581d8fd078c430b82e27106602223e335efef708a124e723814d120c

The variables are exported correctly. The arguments are then built as --build-arg NAME$VALUE instead of --build-arg NAME=$VALUE.

docker build --build-arg NAME without an = means "inherit NAME from the environment". So these two declare build args literally named MINICONDA_INSTALLERhttps://repo.anaconda.com/… and MINICONDA_INSTALLER_SHA256a098a5b…, look them up in the environment, find nothing, and pass nothing. Docker does not treat this as an error — at most an unconsumed-build-arg warning at the end of a successful build.

Consequence: the pinned Miniconda installer URL and its SHA-256 never reach the build. A Dockerfile that uses them silently falls back to its ARG defaults, and the checksum pin — the part that carries the supply-chain guarantee — is not in effect.

The fix is one character in each:

--build-arg MINICONDA_INSTALLER="${MINICONDA_INSTALLER}"
--build-arg MINICONDA_INSTALLER_SHA256="${MINICONDA_INSTALLER_SHA256}"

Same caveat on location as above: this comes from the .gitlab-ci.yml in the generated register-job-hysds-v4 repository, not from this one. Reported here because that file and build-container.bash are two halves of one pipeline.

---

Footnote: both problems were only visible because of the echo " docker build … $@ ." that #1 proposes removing. If that echo does go, consider echoing argument names rather than dropping the line — it is the only record of what was actually passed.