Skip to content

Pinning sandbox.agent.version drops AWF images from digest-pinned to tag-only, with no supported way to restore them #56135

Description

@prpercival

Summary

Pinning sandbox.agent.version to anything other than the compiler's default silently drops the AWF infrastructure images from digest-pinned to tag-only. There is no supported way to supply the digests back.

Reproduction

Default (gh-aw v0.86.2, which defaults to AWF v0.27.44) — digests present:

"imageTag":"0.27.44,squid=sha256:83e48bbe…,agent=sha256:0d727725…,api-proxy=sha256:b50fbadb…,cli-proxy=sha256:c064d159…"

Pin any other version:

sandbox:
  agent:
    version: v0.28.8
"imageTag":"0.28.8"

Every AWF container is now resolved through a mutable tag.

Why this is hard to avoid

Both obvious escape hatches are closed:

  • sandbox.agent.imageTagerror: Unknown property: imageTag. Valid fields are: agent, config, mcp, type
  • sandbox.agent.args (which could pass AWF's own --image-tag, a flag that does accept the digest form) → error: strict mode: 'sandbox.agent.args' is not allowed because it is an internal implementation detail

Turning off strict to reach the second one trades a much larger set of protections for this one, which is not a good exchange.

Why we hit it

runner.topology: arc-dind on AWF v0.27.44 fails with A20 — runc cannot create the /dev/null credential-mask mountpoints under /host$HOME:

error mounting "/dev/null" to rootfs at "/host/home/runner/_work/_temp/gh-aw/home/.npmrc":
make mountpoint ... openat .npmrc: read-only file system

That is fixed by gh-aw-firewall#7244, first released in AWF v0.28.0 (2026-08-12). v0.27.44 was published 2026-08-05, six days earlier, so the compiler's default cannot run the arc-dind topology it documents as supported. Moving off the default is the only option, and doing so costs digest pinning.

Suggested fixes

Any one of these would resolve it:

  1. Advance the default AWF version past v0.28.0, so arc-dind works out of the box with digests intact. This seems worth doing regardless of the rest.
  2. Resolve digests at compile time for whatever version is pinned — the ghcr index manifest is one authenticated HEAD per image, and the result is exactly the <tag>,role=sha256:… string the config already accepts.
  3. Accept operator-supplied digests, either by allowing sandbox.agent.imageTag or by permitting --image-tag through args without disabling strict mode.

(1) plus (2) would mean nobody has to think about this.

Workaround in use

Pinning v0.28.8 and accepting tag resolution, with a script that prints the digests we would pin if the compiler would take them:

./scripts/resolve-awf-imagetag.sh 0.28.8
# 0.28.8,squid=sha256:9ede5177…,agent=sha256:0a94ad1b…,api-proxy=sha256:531fb75f…,cli-proxy=sha256:c10f37b8…,build-tools=sha256:f17baa27…

Environment

gh-aw v0.86.2 (also reproduced on v0.85.4)
AWF default v0.27.44
AWF pinned v0.28.8
Topology runner.topology: arc-dind, ARC gha-runner-scale-set 0.14.2
Host Talos Linux, kernel 6.18.36-talos, Kubernetes 1.36

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions