Filing against github/gh-aw-firewall (AWF). All source permalinks point at commit 1aeabeff, which is the commit tagged v0.27.44 - the version in the table below. This report touches two projects, so every cross-reference is repo-qualified: github/gh-aw-firewall#N for AWF (TypeScript), github/gh-aw#N for the gh-aw compiler (Go).
Summary
With runner.topology: arc-dind, the agent container cannot be created. AWF emits 30 /dev/null credential-hiding overlays whose targets live under /host$HOME/, plus 30 more under the un-prefixed $HOME/. No writable mount at /host$HOME survives filterAgentVolumesForSysroot, and the gh-aw compiler mounts the parent of $HOME read-only. runc therefore cannot create the mountpoints.
Two independent failure modes follow from the same read-only /host$HOME:
B is only observable after working around A, but neither depends on the secondary gh-aw observation at the end of this report.
Environment
|
|
| AWF |
v0.27.44 |
| gh-aw compiler |
v0.85.4; the same mount emission verified on v0.86.1 |
| Engine |
copilot, runner.topology: arc-dind |
| Runners |
ARC gha-runner-scale-set, containerMode: dind, Kubernetes |
$HOME |
/home/runner/_work/_temp/gh-aw/home (${RUNNER_TEMP}/gh-aw/home), set by gh-aw for arc-dind |
| Docker Compose |
two plugins are present. The runner image ships v5.4.0 in /usr/local/lib/docker/cli-plugins; the gh-aw "Install Docker Compose plugin" step installs a pinned v2.36.2 into ${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins, which the Docker CLI prefers over /usr/local/lib. That step runs before gh-aw redirects HOME, so the target is the runner's own /home/runner/.docker/cli-plugins, not the arc-dind home. Agentic runs therefore orchestrate with v2.36.2; the plain probe workflow quoted below reports the image's v5.4.0. |
| Docker Engine |
client 29.6.1 (runner image), server 29.7.2 (docker:dind sidecar), containerd v2.3.3, runc 1.4.3 |
How our deployment deviates from stock ARC
We run an in-house wrapper chart around ARC, so the deviations are listed explicitly rather than hidden behind "should be equivalent":
- Docker socket at
/run/docker/docker.sock instead of /var/run/docker.sock; DOCKER_HOST=unix:///run/docker/docker.sock. The dind-sock volume is mounted read-only in the runner container.
- Custom runner image (Ubuntu-based, ARC runner binaries plus in-house build tooling).
- For build-type scale sets the pod mounts an
.npmrc and a cache volume into the runner container. Neither is under ${RUNNER_TEMP}/gh-aw.
- Otherwise the pod shape mirrors upstream: privileged
docker:dind sidecar, shared work volume at /home/runner/_work, unprivileged runner container, no hostNetwork.
Why we believe none of this causes the failure: sysroot staging - and with it the volume filter - is gated on the topology alone. isSysrootEnabled returns config.runnerTopology === 'arc-dind', and that result is passed to assembleSysrootService, which returns early when false and otherwise runs the filter. Neither DOCKER_HOST nor the socket path is consulted on that path. The socket path surfaces in exactly one place, a diagnostic: the Non-standard DOCKER_HOST unix socket or AWF_DIND=1 detected warning quoted below is emitted by network-options.ts#L69-L71 and only suggests --docker-host-path-prefix. On stock ARC that warning may stay silent while the volume list is built identically. Two separate points about that warning, since its wording invites conflation. First, we do not set AWF_DIND anywhere - not in the workflow, not in the pod spec; it does not appear in the generated compose either. Second, the condition behind the warning is resolveDockerHostPathPrefix, where dindHint = env['AWF_DIND'] === '1' || isSiblingDaemonSocket(env). In our case the second disjunct fires: isSiblingDaemonSocket treats any unix:// socket outside ['unix:///var/run/docker.sock', 'unix:///run/docker.sock'] as a sibling daemon, and ours is /run/docker/docker.sock. On stock ARC the socket is in that list, so the warning would stay silent - while, as shown above, the volume filter runs identically.
The agent volume list is likewise built from the topology (volume-builder.ts:52); the mount that is missing is dropped by a string comparison against workDir and effectiveHome. The socket path does affect which gh-aw runtime patches apply - see the secondary observation - but not the volume list. If you disagree, this is the first thing worth challenging.
Steps to reproduce
- On an ARC scale set in dind container mode, add this workflow:
---
on: workflow_dispatch
runs-on: <your-arc-dind-scale-set>
runner:
topology: arc-dind
engine:
id: copilot
permissions:
contents: read
copilot-requests: write
tools:
bash: true
timeout-minutes: 10
---
# Hello from the self-hosted runner
Run `uname -a` and report the output verbatim.
gh aw compile and commit both files.
- Dispatch it.
The agent job fails in the "Execute GitHub Copilot CLI" step before the engine starts. No workflow-level configuration was applied beyond what is shown above.
What happens
Six runs are referenced in this report. R1-R5 ran on the same ARC scale set; R6 is the GitHub-hosted contrast.
|
What |
Used for |
| R1 |
the unmodified workflow above |
the failure, the attached compose and resolved config |
| R2 |
R1 with .npmrc pre-created on the runner |
shows the next policy entry failing |
| R3 |
R1 with all 30 paths pre-created |
failure mode B, the entrypoint |
| R4 |
R3 plus a sandbox.agent.mounts attempt |
a user-supplied mount spec being double-prefixed, and the split-filesystem probe warning |
| R5 |
a plain, non-agentic workflow |
Docker and Compose versions, plugin provenance |
| R6 |
the same workflow body on ubuntu-latest, no runner.topology |
the contrast below |
Sysroot staging is active and AWF flags the split filesystem up front:
[WARN] ⚠️ Non-standard DOCKER_HOST unix socket or AWF_DIND=1 detected — this typically indicates an ARC/DinD
[WARN] setup where the runner and Docker daemon have separate root filesystems.
[WARN] If bind mounts fail, set --docker-host-path-prefix to the path prefix where the runner filesystem
[WARN] is visible inside the daemon (e.g. --docker-host-path-prefix /tmp/gh-aw).
[INFO] No /host/etc/hosts mount found (sysroot-stage mode); topology peers rely on extra_hosts + container DNS
R1 - the sysroot init container completes, then the agent container cannot be created (failure mode A):
Container awf-sysroot-stage Exited
Container awf-agent Starting
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/dev/null" to rootfs at "/host/home/runner/_work/_temp/gh-aw/home/.npmrc": create mountpoint for /host/home/runner/_work/_temp/gh-aw/home/.npmrc mount: make mountpoint "/host/home/runner/_work/_temp/gh-aw/home/.npmrc": openat .npmrc: read-only file system
[ERROR] Failed to start containers: Error: Command failed with exit code 1: docker compose up -d --pull never
R2 - it is not one file. Pre-creating .npmrc moves the failure to the next entry of the policy list:
error mounting "/dev/null" to rootfs at "/host/home/runner/_work/_temp/gh-aw/home/.azure/service_principal_entries.json"
R3 - after pre-creating all 30 paths, the containers start and pass the API-proxy health checks, and failure mode B appears:
[health-check] ✓ Connectivity established
[entrypoint] Pre-seeding JVM build tool proxy configuration (squid-proxy:3128)...
mkdir: cannot create directory '/host/home/runner/_work/_temp/gh-aw/home/.m2': Read-only file system
B matters for scoping a fix: repairing only the credential overlays would leave the entrypoint failing on the same read-only path.
Run IDs exist but live in a private repository, so the artifacts are pasted below instead of linked.
Contrast: the same workflow on a GitHub-hosted runner
Same workflow body, same engine, same AWF sandbox, with runs-on: ubuntu-latest and no runner.topology. It passes: the sandbox starts and the agent completes.
### uname -a
Linux 6.17.0-1020-azure #20~24.04.1-Ubuntu SMP x86_64 GNU/Linux
### whoami
runner
### node --version
v24.18.0
### docker --version
Docker version 28.0.4, build b8034c0
Caveat on what this does and does not localise: it changes two variables at once - runner type and the absence of runner.topology: arc-dind. It does not prove ARC is irrelevant; it does show that the workflow, the engine and credential hiding are fine when the sysroot filter is not active, and that filter is gated on runnerTopology === 'arc-dind' alone.
The DinD side of our ARC pod is otherwise healthy. A plain, non-agentic workflow on the same scale set reports a working daemon and the image's Compose plugin:
Client: Version: 29.6.1
Server: Engine: Version: 29.7.2 containerd v2.3.3 runc 1.4.3
Docker Compose version v5.4.0
DOCKER_HOST=unix:///run/docker/docker.sock
srw-rw---- 1 root docker 0 /run/docker/docker.sock
Because a Compose 5.x line is easy to mistake for a misread, its provenance from R5:
$ docker compose version --format json
{"version":"v5.4.0"}
$ docker info --format '{{range .ClientInfo.Plugins}}{{.Name}} {{.Version}} {{.Path}}{{"\n"}}{{end}}'
buildx v0.35.0 /usr/local/lib/docker/cli-plugins/docker-buildx
compose v5.4.0 /usr/local/lib/docker/cli-plugins/docker-compose
$ ls -l /usr/local/lib/docker/cli-plugins/
-rwxr-xr-x 1 root root 49357644 Aug 3 16:18 docker-compose
$ ls -l /home/runner/.docker/cli-plugins/
ls: cannot access '/home/runner/.docker/cli-plugins/': No such file or directory
The binary comes from the docker/compose v5.4.0 release (asset docker-compose-linux-x86_64, published 2026-08-03 - matching the file date), installed into the runner image. $HOME/.docker/cli-plugins/ does not exist on a plain run; the gh-aw step creates it, which is why agentic runs use its pinned v2.36.2 instead.
Root cause
filterAgentVolumesForSysroot drops every writable mount whose target is /host$HOME:
${workDir}-chroot-home:/host$HOME:rw - source starts with the workDir prefix (rule broadened in github/gh-aw-firewall#5857)
- the compiler's home mount - after
/host prefixing in buildCustomVolumeMounts, source equals effectiveHome, so relPath === ''
HOME_TOOL_SUBDIRS and ~/.copilot - relPath starts with /.
github/gh-aw-firewall#5734 states the assumption behind the filter: "Drop home directory mounts targeting /host${effectiveHome}/... (sysroot provides writable home)", and explicitly keeps /dev/null overlays.
The assumption holds only while nothing shadows that subtree of the sysroot volume. Here the gh-aw compiler mounts ${RUNNER_TEMP}/gh-aw - the parent of $HOME - read-only at /host${RUNNER_TEMP}/gh-aw, so the writable sysroot volume is covered exactly where $HOME lives.
The four decisive lines from the generated compose, agent service:
- sysroot:/host:rw
- /home/runner/_work/_temp/gh-aw:/host/home/runner/_work/_temp/gh-aw:ro
- /home/runner/_work/_temp/gh-aw/sandbox/agent:/host/home/runner/_work/_temp/gh-aw/sandbox/agent:rw
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.npmrc:ro
- the sysroot volume is writable at
/host;
- the compiler's read-only mount covers
$HOME's parent;
- a child
rw mount does survive the filter and does override the read-only parent - so nested overrides work; it is specifically the home root that is dropped;
- the overlay needs a writable parent that no longer exists.
Each credential file is masked twice, at $HOME and at /host$HOME. Only the /host variants fail, and that is itself evidence for the diagnosis: the un-prefixed targets land on the agent container's own writable rootfs, where runc can create a mountpoint, while the /host variants land inside the compiler's read-only bind mount, where it cannot.
The compiler does pass a writable home mount. Verbatim --mount arguments from the awf invocation in the run log:
--mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro"
--mount "${RUNNER_TEMP}/gh-aw:/host${RUNNER_TEMP}/gh-aw:ro"
--mount "${RUNNER_TEMP}/gh-aw/home:${RUNNER_TEMP}/gh-aw/home:rw"
--mount "${RUNNER_TEMP}/gh-aw/sandbox/agent:${RUNNER_TEMP}/gh-aw/sandbox/agent:rw"
--mount "${GITHUB_WORKSPACE}:${GITHUB_WORKSPACE}:rw"
Four of these five reach the compose with a /host target - three of them correctly prefixed, plus the double-prefixed one described below. The home mount is the only one that does not arrive at all. grep ':/host/home/runner/_work/_temp/gh-aw/home:rw' over the agent service returns nothing.
Note the second argument, which the compiler already writes with a /host prefix (this is R1, with no workflow-level configuration; the separate double-prefixing of a user-supplied sandbox.agent.mounts spec was R4): buildCustomVolumeMounts prefixes it again, so it lands in the compose as /host/host/... and mounts nothing meaningful. It is harmless today - the first argument produces the effective read-only mount - but it is the same double-prefix mechanism that makes sandbox.agent.mounts unusable as a workaround, and it is visible in the attached volume list.
What we already tried
-
--docker-host-path-prefix, as the warning suggests: the compiler does not expose it and deliberately omits it (comment in pkg/workflow/awf_command_builder.go#L62-L67; the same emission sits in pkg/workflow/awf_helpers.go#L297 in the v0.85.4 we compiled with - referencing github/gh-aw#34896, where passing it translated GITHUB_WORKSPACE to a non-existent path). AWF's own probe tries three candidates - ['/host', '/runner', '/tmp/gh-aw'], dind-probe.ts#L26 - and in R4 it reported that none of them resolve:
[WARN] Split runner/daemon filesystem detected but no known prefix worked. Set --docker-host-path-prefix manually if bind mounts fail.
R1's log does not contain this line - only the Non-standard DOCKER_HOST warning quoted earlier. We did not determine why the probe reported in one run and not the other, and we are not claiming the runs differ in behaviour; the quote is attributed to the run it came from.
Even set, it rewrites mount sources (volume-builder.ts:71); it would not reinstate a mount the filter removed.
-
Disabling credential hiding: we found no configuration surface. buildCredentialHidingOverlays is called unconditionally, with no CLI flag or config field gating it.
-
Pre-creating the mask targets on the runner: clears mode A, hits mode B.
-
sandbox.agent.mounts in workflow frontmatter: cannot express the fix. buildCustomVolumeMounts prefixes /host itself, so an explicit /host… target becomes /host/host/…, and an unprefixed spec is dropped by the same filter.
Not tested
- Whether the failure differs on stock ARC with
DOCKER_HOST=unix:///var/run/docker.sock. The reasoning above says the volume list does not depend on it, but we have not measured that variant.
- A third variant - self-hosted ARC without
runner.topology: arc-dind - which would separate "ARC" from "the arc-dind code path". Happy to run it if you want that separation.
Suggested directions
Maintainers know the constraints better; these are starting points, not requests:
- Apply the daemon-visibility test per source rather than dropping the home root categorically. The two dropped home mounts are not equivalent:
${workDir}-chroot-home really does live on the runner's unshared /tmp, so dropping that one is correct; the compiler-supplied ${RUNNER_TEMP}/gh-aw/home sits under _work, which ARC shares with the daemon, so it is visible and safe to keep. Keeping only the second restores a writable /host$HOME without reintroducing the invisible-source problem the filter exists to prevent.
- Or skip credential-hiding overlays under
/host$HOME when no writable /host$HOME survives, and log it instead of failing inside runc. This trades away a security property: without the overlays the agent can read whatever credential files exist under that home. In this topology the home is a fresh directory created per run, but that is an argument for auditing the case, not for silently dropping the masking.
- Or have the compiler avoid mounting
${RUNNER_TEMP}/gh-aw read-only at /host when it contains the chroot home, or mount the home subtree read-write on top of it - option 3 needs a change in github/gh-aw, not here.
Secondary observation
gh-aw gates its runtime chroot config patch on DOCKER_HOST =~ ^tcp:// (pkg/workflow/awf_helpers.go#L34), while the official ARC chart sets a unix socket. The ARC/DinD guide states DinD provides tcp://localhost:2375 automatically, which does not match that chart.
To be precise, because it is easy to conflate: what is missing in that case is the chroot block in the resolved AWF config (identity, binariesSourcePath). Chroot mode in the agent container is on regardless, as noted in the table at the top. Both failure modes above therefore occur independently of this.
What we are asking for
Confirmation that this is a bug in AWF rather than an unsupported configuration, and guidance on which of the directions above you would accept. We are willing to prepare the patch and test it on our ARC runners.
Full agent-service volume list from the generated compose (76 entries)
- /tmp:/tmp:rw
- /home/runner/_work/<repo>/<repo>:/home/runner/_work/<repo>/<repo>:rw
- /sys:/host/sys:ro
- /dev:/host/dev:ro
- /dev/null:/host/dev/null:rw
- /home/runner/_work/<repo>/<repo>:/host/home/runner/_work/<repo>/<repo>:rw
- /tmp:/host/tmp:rw
- /home/runner/_work/_tool:/host/home/runner/_work/_tool:ro
- /dev/null:/host/var/run/docker.sock:ro
- /dev/null:/host/run/docker.sock:ro
- /home/runner/_work/_temp/gh-aw:/host/home/runner/_work/_temp/gh-aw:ro
- /home/runner/_work/_temp/gh-aw:/host/host/home/runner/_work/_temp/gh-aw:ro
- /home/runner/_work/_temp/gh-aw/sandbox/agent:/host/home/runner/_work/_temp/gh-aw/sandbox/agent:rw
- /home/runner/_work/<repo>/<repo>:/host/home/runner/_work/<repo>/<repo>:rw
- /home/runner/_work/_tool:/host/home/runner/_work/_tool:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.docker/config.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.npmrc:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.composer/auth.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.ssh/id_rsa:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.ssh/id_ed25519:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.ssh/id_ecdsa:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.ssh/id_dsa:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.aws/credentials:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.aws/config:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.kube/config:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.azure/credentials:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.azure/msal_token_cache.bin:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.azure/msal_token_cache.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.azure/accessTokens.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.azure/service_principal_entries.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.cargo/credentials:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.cargo/credentials.toml:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.claude/.credentials.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.gemini/oauth_creds.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.gemini/google_accounts.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.gemini/access_tokens.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.config/gh/hosts.yml:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.config/gcloud/credentials.db:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.config/gcloud/access_tokens.db:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.config/gcloud/application_default_credentials.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.config/doctl/config.yaml:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.config/rclone/rclone.conf:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.config/containers/auth.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.config/pulumi/credentials.json:ro
- /dev/null:/home/runner/_work/_temp/gh-aw/home/.config/helm/repositories.yaml:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.docker/config.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.npmrc:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.composer/auth.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.ssh/id_rsa:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.ssh/id_ed25519:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.ssh/id_ecdsa:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.ssh/id_dsa:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.aws/credentials:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.aws/config:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.kube/config:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.azure/credentials:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.azure/msal_token_cache.bin:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.azure/msal_token_cache.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.azure/accessTokens.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.azure/service_principal_entries.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.cargo/credentials:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.cargo/credentials.toml:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.claude/.credentials.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.gemini/oauth_creds.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.gemini/google_accounts.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.gemini/access_tokens.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.config/gh/hosts.yml:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.config/gcloud/credentials.db:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.config/gcloud/access_tokens.db:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.config/gcloud/application_default_credentials.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.config/doctl/config.yaml:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.config/rclone/rclone.conf:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.config/containers/auth.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.config/pulumi/credentials.json:ro
- /dev/null:/host/home/runner/_work/_temp/gh-aw/home/.config/helm/repositories.yaml:ro
- sysroot:/host:rw
Workspace paths are shown as <repo>; everything else is verbatim, from R1, the unmodified workflow shown under Steps to reproduce - no workflow-level workarounds were applied.
Three things in the list look odd at a glance and are explained rather than edited away. First, a few mounts appear twice verbatim - the workspace and _work/_tool - because the compiler passes them explicitly and AWF derives them as well; identical entries are harmless to compose. Second, /host/host/home/runner/_work/_temp/gh-aw:ro is the compiler's already-prefixed argument being prefixed a second time. Third, there is no rw mount whose target is /host/home/runner/_work/_temp/gh-aw/home - that absence is the bug.
The environment: section is omitted: it carries no evidence for this report and contains our repository and cluster identifiers.
The 30 credential paths, from credentialFilesToHide()
Source: mount-policy.ts#L216-L228, expanded from sandbox-mount-policy.json. Each is masked twice - at $HOME and at /host$HOME - which accounts for 60 of the 63 /dev/null mounts in the list above.
.docker/config.json
.npmrc
.composer/auth.json
.ssh/id_rsa
.ssh/id_ed25519
.ssh/id_ecdsa
.ssh/id_dsa
.aws/credentials
.aws/config
.kube/config
.azure/credentials
.azure/msal_token_cache.bin
.azure/msal_token_cache.json
.azure/accessTokens.json
.azure/service_principal_entries.json
.cargo/credentials
.cargo/credentials.toml
.claude/.credentials.json
.gemini/oauth_creds.json
.gemini/google_accounts.json
.gemini/access_tokens.json
.config/gh/hosts.yml
.config/gcloud/credentials.db
.config/gcloud/access_tokens.db
.config/gcloud/application_default_credentials.json
.config/doctl/config.yaml
.config/rclone/rclone.conf
.config/containers/auth.json
.config/pulumi/credentials.json
.config/helm/repositories.yaml
Relevant fields from the resolved AWF config
{
"runnerTopology": "arc-dind",
"workDir": "/tmp/awf-1786432645176",
"containerWorkDir": "/home/runner/_work/<repo>/<repo>",
"enableDind": false,
"networkIsolation": true,
"chroot": null
}
workDir matters because the filter compares mount sources against it: the writable home lives at ${workDir}-chroot-home, which is why the workDir-prefix rule removes it. effectiveHome is not a config field - it comes from getRealUserHome(), i.e. the HOME of the awf process, which gh-aw sets to ${RUNNER_TEMP}/gh-aw/home; every /host$HOME path quoted in this report is that value.
enableDind: false is expected and not a contradiction with containerMode: dind on the ARC side: the field maps to AWF's --enable-dind, i.e. whether AWF starts a Docker daemon inside its own sandbox. We do not use that. The daemon here is the ARC sidecar, which AWF reaches through DOCKER_HOST.
chroot: null is the secondary observation above, not the cause of either failure mode.
Filing against
github/gh-aw-firewall(AWF). All source permalinks point at commit1aeabeff, which is the commit taggedv0.27.44- the version in the table below. This report touches two projects, so every cross-reference is repo-qualified:github/gh-aw-firewall#Nfor AWF (TypeScript),github/gh-aw#Nfor the gh-aw compiler (Go).Summary
With
runner.topology: arc-dind, the agent container cannot be created. AWF emits 30/dev/nullcredential-hiding overlays whose targets live under/host$HOME/, plus 30 more under the un-prefixed$HOME/. No writable mount at/host$HOMEsurvivesfilterAgentVolumesForSysroot, and the gh-aw compiler mounts the parent of$HOMEread-only. runc therefore cannot create the mountpoints.Two independent failure modes follow from the same read-only
/host$HOME:chrootconfig block?docker compose upaborts creatingawf-agenton the first credential overlayvolume-builder.ts:69)mkdir -p "${JVM_HOME_PREFIX}/.m2"; the script runs underset -e, so this kills the container rather than loggingAWF_CHROOT_ENABLED=trueis set unconditionally (api-proxy-environment.ts:42)B is only observable after working around A, but neither depends on the secondary gh-aw observation at the end of this report.
Environment
v0.27.44v0.85.4; the same mount emission verified onv0.86.1copilot,runner.topology: arc-dindgha-runner-scale-set,containerMode: dind, Kubernetes$HOME/home/runner/_work/_temp/gh-aw/home(${RUNNER_TEMP}/gh-aw/home), set by gh-aw for arc-dindv5.4.0in/usr/local/lib/docker/cli-plugins; the gh-aw "Install Docker Compose plugin" step installs a pinnedv2.36.2into${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins, which the Docker CLI prefers over/usr/local/lib. That step runs before gh-aw redirectsHOME, so the target is the runner's own/home/runner/.docker/cli-plugins, not the arc-dind home. Agentic runs therefore orchestrate withv2.36.2; the plain probe workflow quoted below reports the image'sv5.4.0.29.6.1(runner image), server29.7.2(docker:dindsidecar), containerdv2.3.3, runc1.4.3How our deployment deviates from stock ARC
We run an in-house wrapper chart around ARC, so the deviations are listed explicitly rather than hidden behind "should be equivalent":
/run/docker/docker.sockinstead of/var/run/docker.sock;DOCKER_HOST=unix:///run/docker/docker.sock. Thedind-sockvolume is mounted read-only in the runner container..npmrcand a cache volume into the runner container. Neither is under${RUNNER_TEMP}/gh-aw.docker:dindsidecar, sharedworkvolume at/home/runner/_work, unprivileged runner container, nohostNetwork.Why we believe none of this causes the failure: sysroot staging - and with it the volume filter - is gated on the topology alone.
isSysrootEnabledreturnsconfig.runnerTopology === 'arc-dind', and that result is passed toassembleSysrootService, which returns early when false and otherwise runs the filter. NeitherDOCKER_HOSTnor the socket path is consulted on that path. The socket path surfaces in exactly one place, a diagnostic: theNon-standard DOCKER_HOST unix socket or AWF_DIND=1 detectedwarning quoted below is emitted bynetwork-options.ts#L69-L71and only suggests--docker-host-path-prefix. On stock ARC that warning may stay silent while the volume list is built identically. Two separate points about that warning, since its wording invites conflation. First, we do not setAWF_DINDanywhere - not in the workflow, not in the pod spec; it does not appear in the generated compose either. Second, the condition behind the warning isresolveDockerHostPathPrefix, wheredindHint = env['AWF_DIND'] === '1' || isSiblingDaemonSocket(env). In our case the second disjunct fires:isSiblingDaemonSockettreats anyunix://socket outside['unix:///var/run/docker.sock', 'unix:///run/docker.sock']as a sibling daemon, and ours is/run/docker/docker.sock. On stock ARC the socket is in that list, so the warning would stay silent - while, as shown above, the volume filter runs identically.The agent volume list is likewise built from the topology (
volume-builder.ts:52); the mount that is missing is dropped by a string comparison againstworkDirandeffectiveHome. The socket path does affect which gh-aw runtime patches apply - see the secondary observation - but not the volume list. If you disagree, this is the first thing worth challenging.Steps to reproduce
gh aw compileand commit both files.The agent job fails in the "Execute GitHub Copilot CLI" step before the engine starts. No workflow-level configuration was applied beyond what is shown above.
What happens
Six runs are referenced in this report. R1-R5 ran on the same ARC scale set; R6 is the GitHub-hosted contrast.
.npmrcpre-created on the runnersandbox.agent.mountsattemptubuntu-latest, norunner.topologySysroot staging is active and AWF flags the split filesystem up front:
R1 - the sysroot init container completes, then the agent container cannot be created (failure mode A):
R2 - it is not one file. Pre-creating
.npmrcmoves the failure to the next entry of the policy list:R3 - after pre-creating all 30 paths, the containers start and pass the API-proxy health checks, and failure mode B appears:
B matters for scoping a fix: repairing only the credential overlays would leave the entrypoint failing on the same read-only path.
Run IDs exist but live in a private repository, so the artifacts are pasted below instead of linked.
Contrast: the same workflow on a GitHub-hosted runner
Same workflow body, same engine, same AWF sandbox, with
runs-on: ubuntu-latestand norunner.topology. It passes: the sandbox starts and the agent completes.Caveat on what this does and does not localise: it changes two variables at once - runner type and the absence of
runner.topology: arc-dind. It does not prove ARC is irrelevant; it does show that the workflow, the engine and credential hiding are fine when the sysroot filter is not active, and that filter is gated onrunnerTopology === 'arc-dind'alone.The DinD side of our ARC pod is otherwise healthy. A plain, non-agentic workflow on the same scale set reports a working daemon and the image's Compose plugin:
Because a Compose
5.xline is easy to mistake for a misread, its provenance from R5:The binary comes from the
docker/composev5.4.0release (assetdocker-compose-linux-x86_64, published 2026-08-03 - matching the file date), installed into the runner image.$HOME/.docker/cli-plugins/does not exist on a plain run; the gh-aw step creates it, which is why agentic runs use its pinnedv2.36.2instead.Root cause
filterAgentVolumesForSysrootdrops every writable mount whose target is/host$HOME:${workDir}-chroot-home:/host$HOME:rw- source starts with theworkDirprefix (rule broadened ingithub/gh-aw-firewall#5857)/hostprefixing inbuildCustomVolumeMounts, source equalseffectiveHome, sorelPath === ''HOME_TOOL_SUBDIRSand~/.copilot-relPathstarts with/.github/gh-aw-firewall#5734states the assumption behind the filter: "Drop home directory mounts targeting/host${effectiveHome}/...(sysroot provides writable home)", and explicitly keeps/dev/nulloverlays.The assumption holds only while nothing shadows that subtree of the sysroot volume. Here the gh-aw compiler mounts
${RUNNER_TEMP}/gh-aw- the parent of$HOME- read-only at/host${RUNNER_TEMP}/gh-aw, so the writable sysroot volume is covered exactly where$HOMElives.The four decisive lines from the generated compose, agent service:
/host;$HOME's parent;rwmount does survive the filter and does override the read-only parent - so nested overrides work; it is specifically the home root that is dropped;Each credential file is masked twice, at
$HOMEand at/host$HOME. Only the/hostvariants fail, and that is itself evidence for the diagnosis: the un-prefixed targets land on the agent container's own writable rootfs, where runc can create a mountpoint, while the/hostvariants land inside the compiler's read-only bind mount, where it cannot.The compiler does pass a writable home mount. Verbatim
--mountarguments from theawfinvocation in the run log:Four of these five reach the compose with a
/hosttarget - three of them correctly prefixed, plus the double-prefixed one described below. The home mount is the only one that does not arrive at all.grep ':/host/home/runner/_work/_temp/gh-aw/home:rw'over the agent service returns nothing.Note the second argument, which the compiler already writes with a
/hostprefix (this is R1, with no workflow-level configuration; the separate double-prefixing of a user-suppliedsandbox.agent.mountsspec was R4):buildCustomVolumeMountsprefixes it again, so it lands in the compose as/host/host/...and mounts nothing meaningful. It is harmless today - the first argument produces the effective read-only mount - but it is the same double-prefix mechanism that makessandbox.agent.mountsunusable as a workaround, and it is visible in the attached volume list.What we already tried
--docker-host-path-prefix, as the warning suggests: the compiler does not expose it and deliberately omits it (comment inpkg/workflow/awf_command_builder.go#L62-L67; the same emission sits inpkg/workflow/awf_helpers.go#L297in thev0.85.4we compiled with - referencinggithub/gh-aw#34896, where passing it translatedGITHUB_WORKSPACEto a non-existent path). AWF's own probe tries three candidates -['/host', '/runner', '/tmp/gh-aw'],dind-probe.ts#L26- and in R4 it reported that none of them resolve:R1's log does not contain this line - only the
Non-standard DOCKER_HOSTwarning quoted earlier. We did not determine why the probe reported in one run and not the other, and we are not claiming the runs differ in behaviour; the quote is attributed to the run it came from.Even set, it rewrites mount sources (
volume-builder.ts:71); it would not reinstate a mount the filter removed.Disabling credential hiding: we found no configuration surface.
buildCredentialHidingOverlaysis called unconditionally, with no CLI flag or config field gating it.Pre-creating the mask targets on the runner: clears mode A, hits mode B.
sandbox.agent.mountsin workflow frontmatter: cannot express the fix.buildCustomVolumeMountsprefixes/hostitself, so an explicit/host…target becomes/host/host/…, and an unprefixed spec is dropped by the same filter.Not tested
DOCKER_HOST=unix:///var/run/docker.sock. The reasoning above says the volume list does not depend on it, but we have not measured that variant.runner.topology: arc-dind- which would separate "ARC" from "the arc-dind code path". Happy to run it if you want that separation.Suggested directions
Maintainers know the constraints better; these are starting points, not requests:
${workDir}-chroot-homereally does live on the runner's unshared/tmp, so dropping that one is correct; the compiler-supplied${RUNNER_TEMP}/gh-aw/homesits under_work, which ARC shares with the daemon, so it is visible and safe to keep. Keeping only the second restores a writable/host$HOMEwithout reintroducing the invisible-source problem the filter exists to prevent./host$HOMEwhen no writable/host$HOMEsurvives, and log it instead of failing inside runc. This trades away a security property: without the overlays the agent can read whatever credential files exist under that home. In this topology the home is a fresh directory created per run, but that is an argument for auditing the case, not for silently dropping the masking.${RUNNER_TEMP}/gh-awread-only at/hostwhen it contains the chroot home, or mount the home subtree read-write on top of it - option 3 needs a change ingithub/gh-aw, not here.Secondary observation
gh-aw gates its runtime chroot config patch on
DOCKER_HOST =~ ^tcp://(pkg/workflow/awf_helpers.go#L34), while the official ARC chart sets a unix socket. The ARC/DinD guide states DinD providestcp://localhost:2375automatically, which does not match that chart.To be precise, because it is easy to conflate: what is missing in that case is the
chrootblock in the resolved AWF config (identity,binariesSourcePath). Chroot mode in the agent container is on regardless, as noted in the table at the top. Both failure modes above therefore occur independently of this.What we are asking for
Confirmation that this is a bug in AWF rather than an unsupported configuration, and guidance on which of the directions above you would accept. We are willing to prepare the patch and test it on our ARC runners.
Full agent-service volume list from the generated compose (76 entries)
Workspace paths are shown as
<repo>; everything else is verbatim, from R1, the unmodified workflow shown under Steps to reproduce - no workflow-level workarounds were applied.Three things in the list look odd at a glance and are explained rather than edited away. First, a few mounts appear twice verbatim - the workspace and
_work/_tool- because the compiler passes them explicitly and AWF derives them as well; identical entries are harmless to compose. Second,/host/host/home/runner/_work/_temp/gh-aw:rois the compiler's already-prefixed argument being prefixed a second time. Third, there is norwmount whose target is/host/home/runner/_work/_temp/gh-aw/home- that absence is the bug.The
environment:section is omitted: it carries no evidence for this report and contains our repository and cluster identifiers.The 30 credential paths, from credentialFilesToHide()
Source:
mount-policy.ts#L216-L228, expanded fromsandbox-mount-policy.json. Each is masked twice - at$HOMEand at/host$HOME- which accounts for 60 of the 63/dev/nullmounts in the list above..docker/config.json.npmrc.composer/auth.json.ssh/id_rsa.ssh/id_ed25519.ssh/id_ecdsa.ssh/id_dsa.aws/credentials.aws/config.kube/config.azure/credentials.azure/msal_token_cache.bin.azure/msal_token_cache.json.azure/accessTokens.json.azure/service_principal_entries.json.cargo/credentials.cargo/credentials.toml.claude/.credentials.json.gemini/oauth_creds.json.gemini/google_accounts.json.gemini/access_tokens.json.config/gh/hosts.yml.config/gcloud/credentials.db.config/gcloud/access_tokens.db.config/gcloud/application_default_credentials.json.config/doctl/config.yaml.config/rclone/rclone.conf.config/containers/auth.json.config/pulumi/credentials.json.config/helm/repositories.yamlRelevant fields from the resolved AWF config
{ "runnerTopology": "arc-dind", "workDir": "/tmp/awf-1786432645176", "containerWorkDir": "/home/runner/_work/<repo>/<repo>", "enableDind": false, "networkIsolation": true, "chroot": null }workDirmatters because the filter compares mount sources against it: the writable home lives at${workDir}-chroot-home, which is why theworkDir-prefix rule removes it.effectiveHomeis not a config field - it comes fromgetRealUserHome(), i.e. theHOMEof theawfprocess, which gh-aw sets to${RUNNER_TEMP}/gh-aw/home; every/host$HOMEpath quoted in this report is that value.enableDind: falseis expected and not a contradiction withcontainerMode: dindon the ARC side: the field maps to AWF's--enable-dind, i.e. whether AWF starts a Docker daemon inside its own sandbox. We do not use that. The daemon here is the ARC sidecar, which AWF reaches throughDOCKER_HOST.chroot: nullis the secondary observation above, not the cause of either failure mode.