gh-aw compiler: ARC/DinD topology requires daemon-visible path redirections

[lpcox]

Summary

When compiling workflows with topology: arc-dind frontmatter, the gh-aw compiler must redirect all file paths to daemon-visible locations. In ARC/DinD, the runner container and Docker daemon (DinD sidecar) have separate filesystems — they share only the workspace volume (/home/runner/_work/). Files at /tmp, /usr/local/bin, and most of /home/runner are invisible to the Docker daemon and cannot be bind-mounted into the agent container.

The canonical daemon-visible base path is ${RUNNER_TEMP} (/home/runner/_work/_temp), which is under the shared workspace volume.

Validated via canary

All changes below were validated end-to-end on bbq-beets-four-nines/agentic-workflows-canary PR #1559 (branch fix/arc-dind-lock-yml-v0.27.15) using AWF v0.27.20 on ARC runners. Copilot CLI successfully starts and executes workloads.

Required Compiler Changes

1. Redirect all gh-aw artifacts to ${RUNNER_TEMP}/gh-aw/

Artifact	Current location	Required location
Tool cache	/tmp/gh-aw/tool-cache or RUNNER_TOOL_CACHE	${RUNNER_TEMP}/gh-aw/tool-cache
Copilot CLI binary	/usr/local/bin/copilot	${RUNNER_TEMP}/gh-aw/bin/copilot (copy step)
Prompts	/tmp/gh-aw/aw-prompts/	${RUNNER_TEMP}/gh-aw/aw-prompts/ (copy step)
Node binary	setup-node install path	${RUNNER_TEMP}/gh-aw/tool-cache/node/ (copy step)

Why: /tmp, /usr/local/bin, and /home/runner/_work/_tool are all invisible to the Docker daemon in split-fs mode.

2. Single rw mount for all gh-aw data

--mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:rw"

• Must be rw — Copilot writes logs, safeoutputs, and agent state here
• AWF auto-prepends /host to mount targets (chroot adjustment) — do NOT add /host in the lock.yml

3. Set HOME inside user command

export HOME=${RUNNER_TEMP}/gh-aw/home

• /home/runner is read-only in sysroot/chroot mode (comes from sysroot base image, not writable)
• Copilot CLI needs writable HOME for:
  • SEA bundle extraction (~/.cache)
  • Config/auth data (~/.config)
  • Tool state (~/.copilot, .local, etc.)
• Single HOME redirect replaces the need for individual dot-directory mounts

4. Rewrite all paths in user command

All /tmp/gh-aw references in the user command must point to daemon-visible paths:

• Binary: ${RUNNER_TEMP}/gh-aw/bin/copilot
• --prompt-file ${RUNNER_TEMP}/gh-aw/aw-prompts/prompt.txt
• --log-dir ${RUNNER_TEMP}/gh-aw/sandbox/agent/logs/
• --add-dir ${RUNNER_TEMP}/gh-aw/

5. AWF log/audit dirs under daemon-visible path

--proxy-logs-dir ${RUNNER_TEMP}/gh-aw/sandbox/firewall/logs
--audit-dir ${RUNNER_TEMP}/gh-aw/sandbox/firewall/audit

Default /tmp/awf-*/logs isn't daemon-visible, so post-run log access fails.

AWF Runtime Fixes Already Shipped

These PRs fixed AWF's volume/mount handling for sysroot mode:

• PR fix: skip /etc bind mounts when sysroot-stage is active (arc-dind) #5732 (v0.27.18): Skip /etc selective mounts when sysroot enabled
• PR fix: filter split-fs-invisible mounts when sysroot-stage is active (arc-dind) #5734 (v0.27.19): Skip hosts file mount and filter workDir/home-based mounts in sysroot mode
• PR fix: sysroot filter should not drop workspace custom mounts #5739 (v0.27.20): Narrow sysroot home filter to only drop dot-directories (preserve workspace custom mounts)

What the Compiler Does NOT Need to Handle

• Sysroot stage setup → automatic with --runner-topology arc-dind
• Network isolation mode → automatic with sysroot
• Docker host path prefix → auto-detected from DOCKER_HOST

Key Principle

In ARC/DinD split-fs, anything the agent container needs must exist under ${RUNNER_TEMP} (the workspace volume). The compiler's job is to copy/redirect everything there before invoking AWF.

Debugging Reference

The full iterative debugging log is in the canary PR #1559 commits. Key failure modes discovered:

1. /etc/ld.so.cache mount failure → fixed by skipping etc mounts in sysroot
2. Hosts file mount failure → fixed by skipping hosts generation in sysroot
3. Custom mount filtered by sysroot home filter → fixed by narrowing filter to dot-dirs only
4. Node not found → tool-cache redirect + copy to RUNNER_TEMP
5. Copilot binary not found → copy to RUNNER_TEMP
6. SEA extraction EACCES on ~/.cache → writable HOME
7. Silent crash (exit 1, 0B output) → /home/runner read-only, HOME redirect fixes

[lpcox]

Moved to github/gh-aw#42807 — this is a compiler issue, not a firewall issue.