[aw-failures] [P0] awf-agent container fails to start: read-only file system on /tmp/awf-init

[github-actions]

Fix the awf-agent container-init race — a read-only overlay2 mountpoint took down 6 workflows across 3 different agent engines in one 3-minute burst.

Problem statement

At 2026-08-23T22:25:00Z, six independent scheduled/PR-triggered workflows all failed before their agent CLI ever started. Every one hit the identical Docker/runc error trying to start the awf-agent sandbox container:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed:
runc create failed: unable to start container process: error during container init: error mounting
"/tmp/awf-<run-id>/init-signal" to rootfs at "/tmp/awf-init": create mountpoint for /tmp/awf-init mount:
make mountpoint "/tmp/awf-init": mkdirat /var/lib/docker/overlay2/<hash>/merged/tmp/awf-init: read-only file system
[ERROR] Failed to start containers: Error: Command failed with exit code 1: docker compose up -d --pull never
##[error]Agent execution exited with code 1

This is not an agent-engine bug: the failing workflows used Copilot CLI, Claude Code CLI, and Pi CLI, and none of them got far enough to make a single API call (agenticworkflows audit shows cumulative_input_tokens: 0 / 0 turns on every affected run). The failure is entirely in the sandbox/firewall container bring-up step, specifically in mounting /tmp/awf-init onto the container's overlay2 merged directory.

Affected workflows and run IDs

All created at the same 22:25:00Z schedule tick:

• Ponytail Reviewer — §32670511170
• Test Quality Sentinel — §32670511142
• Design Decision Gate 🏗️ — §32670511169
• PR Code Quality Reviewer — §32670511171
• Matt Pocock Skills Reviewer — §32670511125
• Impeccable Skills Reviewer — §32670511184

Confirmed via raw workflow-log grep (identical stack trace, different overlay2 hash and awf-<id> per run) on the first two; the remaining four share the same created_at tick and failed step name and are treated as the same event pending re-audit.

Probable root cause

The runner host's Docker overlay2 backing filesystem went briefly read-only (or a mount race left /tmp partially remounted read-only) right as this scheduling tick tried to bring up 6+ awf-agent containers concurrently. mkdirat ... read-only file system on a merged overlay2 directory is a host/kernel-level symptom, not an application bug — most likely causes, in order of likelihood: (1) transient disk-pressure eviction or ephemeral-disk quota hit on the runner triggering a protective read-only remount, (2) a race in the firewall/sandbox startup script when multiple docker compose up invocations run concurrently on the same host and contend for the same overlay2 driver state, (3) a runner-image regression in how /tmp/awf-init mountpoints are created.

Proposed remediation

1. Add a pre-flight check in the sandbox startup path (docker compose up -d --pull never) that verifies the target overlay2//tmp mount is writable before attempting container creation, and retries once with backoff instead of hard-failing the whole job on the first runc create failed.
2. Instrument the startup wrapper to capture df -h / mount output for /tmp and /var/lib/docker on failure, so the next occurrence tells us definitively whether this is disk-pressure, concurrent-mount contention, or a runner-image defect.
3. If concurrency contention is confirmed (multiple awf-agent containers starting on the same host at the same scheduling tick), stagger or serialize sandbox bring-up per host rather than firing all scheduled workflows at the exact same second.

Success criteria / verification

• No repeat of mkdirat .../tmp/awf-init: read-only file system across a full scheduling cycle after the fix ships.
• If it does recur, the new diagnostic capture (mount/disk state) is present in the failure log so root cause moves from "probable" to "confirmed" on the next occurrence.
• A burst of 6+ simultaneously-scheduled workflows no longer fails as a single correlated event — each either succeeds or fails independently with a distinguishable cause.

References:

• §32670511170
• §32670511142
• §32670511169
  Related to [aw-failures] [P0] awf-agent container-init read-only-fs race takes down 6 workflows across 3 engines #55211

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 131.9 AIC · ⌖ 14.1 AIC · ⊞ 6.4K · ◷

• expires on Aug 30, 2026, 5:22 PM UTC-08:00

[github-actions]

🍪 Issue Monster selected this for Copilot

I've identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

🍪 Om nom nom by Issue Monster · cop10 · 3.25 AIC · ⌖ 1.34 AIC · ⊞ 15.9K · ◷

[github-actions]

This issue was automatically closed because it expired on 2026-08-31T01:22:57.476Z.

Closed by Workflow