[aw-failures] [P0] awf-agent container-init read-only-fs race takes down 6 workflows across 3 engines

[github-actions]

Fix the docker compose up container-init race on the runner host — it took down 6 workflows across 3 different agent engines in one 3-minute burst, and no open issue covers it.

Executive summary

• 12 failed runs in the last 6h window; 5 had pre-fetched detailed logs, all cross-checked with agenticworkflows audit.
• New P0 gap, filed as sub-issue below: awf-agent container fails to start with OCI runtime create failed: ... mkdirat .../merged/tmp/awf-init: read-only file system. Confirmed via raw log grep on 2 of 6 affected runs (Ponytail Reviewer, Test Quality Sentinel) — identical stack, identical mountpoint path pattern, same 3-minute window. This is a runner-host infra failure, not an engine bug: it hit Copilot CLI, Claude Code CLI, and Pi CLI runs simultaneously.
• Already tracked, recurred again: Daily Observability Report hit Codex 401 Unauthorized: auth error code: invalid_project at 23:51 UTC — matches open [aw-failures] [P0] Codex OpenAI invalid_project 401 auth failure — unfixed 3+ days, Daily Cache Strategy Analyzer failing every #54242 ("Codex OpenAI invalid_project 401 auth failure"). No new issue needed; this is now a third affected workflow beyond the two named in [aw-failures] [P0] Codex OpenAI invalid_project 401 auth failure — unfixed 3+ days, Daily Cache Strategy Analyzer failing every #54242's evidence, worth a comment update but not a new filing.
• Untracked, single occurrence, monitor: Daily Harness Experiment Proposer's safe_outputs job failed outright — GitHub App token minting got 404 Not Found on GET /repos/github/gh-aw/installation, then a downstream conclusion job failed with Input required and not supplied: github-token. Distinct root cause from [aw-failures] safe_outputs job hard-fails entire batch on one non-retryable error #53263 (that issue is about non-retryable per-item output errors; this is an app-installation/token-scope failure upstream of any output processing). Only one occurrence this window — flagging for next cycle rather than filing now.
• No existing open issue closed this cycle: none of the 16 open agentic-workflows issues had fresh evidence of a fix landing in this window's failures.

Failure cluster table

Cluster	Severity	Workflows	Runs	Status
awf-agent container init fails: read-only fs on /tmp/awf-init	P0	Ponytail Reviewer, Test Quality Sentinel, Design Decision Gate, PR Code Quality Reviewer, Matt Pocock Skills Reviewer, Impeccable Skills Reviewer	§32670511170, §32670511142, §32670511169, §32670511171, §32670511125, §32670511184	New sub-issue filed
Codex invalid_project 401 auth failure	P0	Daily Observability Report	§32674599146	Tracked by #54242 — recurrence noted, no new issue
safe_outputs GitHub App token 404 → job hard-fail	P1	Daily Harness Experiment Proposer	§32670714843	Monitor — single occurrence, distinct from #53263, re-file if it repeats

Evidence: awf-agent container-init cluster (click to expand)

Identical signature confirmed via raw workflow log grep on two representative runs, 2 minutes apart, two different CLI engines:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed:
runc create failed: unable to start container process: error during container init: error mounting
"/tmp/awf-<id>/init-signal" to rootfs at "/tmp/awf-init": create mountpoint for /tmp/awf-init mount:
make mountpoint "/tmp/awf-init": mkdirat /var/lib/docker/overlay2/<hash>/merged/tmp/awf-init: read-only file system
[ERROR] Failed to start containers: Error: Command failed with exit code 1: docker compose up -d --pull never

agenticworkflows audit on both runs shows cumulative_input_tokens: 0 and 0 turns — the agent CLI never started, confirming the failure is entirely in the sandbox bring-up step, before Copilot/Claude/Codex/Pi ever get invoked. All 6 affected runs share the same created_at: 2026-08-23T22:25:00Z schedule tick, which is the strongest signal this is a single runner-host event (overlay2 filesystem went read-only, likely disk-pressure or a host remount) rather than 6 independent code regressions.

Evidence: Codex invalid_project 401 (click to expand)

WARN codex_core::responses_retry: stream disconnected - retrying sampling request (1/5 in 218ms)...
sampling_error=unexpected status 401 Unauthorized: You do not have access to the project tied to the API key.,
url: (apiproxy/redacted) auth error: 401, auth error code: invalid_project

Same invalid_project code already named in #54242's title. audit shows cumulative_input_tokens: 0 here too — the API key/project binding fails before the first successful turn.

Evidence: safe_outputs GitHub App token 404 (click to expand)

##[error]Not Found - https://docs.github.com/rest/apps/apps#get-a-repository-installation-for-the-authenticated-app
##[error]Unhandled error: Error: Input required and not supplied: github-token

GET /repos/github/gh-aw/installation returned 404 — the app token action could not resolve an installation for this repo/branch context, and the fallback token input was never supplied, so the downstream conclusion job (which reports the failure) also failed.

Existing issue correlation

• [aw-failures] [P0] Codex OpenAI invalid_project 401 auth failure — unfixed 3+ days, Daily Cache Strategy Analyzer failing every #54242 (Codex invalid_project 401, P0, open) — recurred, now spans 3 workflows total (Daily Observability Report added this cycle). Recommend updating its evidence list rather than filing a duplicate.
• [aw-failures] safe_outputs job hard-fails entire batch on one non-retryable error #53263 (safe_outputs hard-fail on non-retryable item, open) — not a match for this cycle's Daily Harness Experiment Proposer failure; that one is an app-installation/token problem, not a per-item output error. Left open, untouched.
• [aw-failures] [P0] Cross-engine agent-CLI segfault (exit 139) — confirmed on Auto-Triage/Pi CLI, now spread to 8 workflows acros #54186, [aw-failures] [P0] Cloud Hypervisor guest-connectivity probe regression post-v0.28.2 — 11 of 16 runs failed (Ponytail Reviewer x [Content trun #54402, [aw-failures] [P0] Serena Go language server crashes on missing Go toolchain — breaks 17 shared/mcp/serena-go.md workflows #54759, [aw-failures] [P1] Design Decision Gate hangs silently mid-tool-use in Claude Code CLI step #53619, [aw-failures] [P0] Claude Code CLI startup crashes — Anthropic API egress blackout #54462, [aw-failures] [P0] Artifact upload fails: "The artifact name activation is not valid" — smoke-test fleet outage #54116, [aw-failures] Smoke Copilot Sub Agents: 'evals' artifact not found — full run failure #53262, [aw-failures] [P1] Design Decision Gate: create_issue fails 'Resource not accessible by integration' on copilot/* PR branches #54898, [regression] Daily CI Regression Audit — 32613340583 #54963, [aw-failures] [aw] Failure Investigator Report — 2026-08-20 18:41 UTC (6h) #54326, [aw] No-Op Runs #48838 — no fresh evidence for or against in this window's failures; left open, untouched.

Fix roadmap

• P0: Fix the docker compose up/runc container-init race — see sub-issue below for the concrete remediation ask (root-cause the overlay2 read-only-fs trigger, add a startup retry/health-check before failing the whole job).
• P0: Update [aw-failures] [P0] Codex OpenAI invalid_project 401 auth failure — unfixed 3+ days, Daily Cache Strategy Analyzer failing every #54242 with the third affected workflow; do not re-file.
• P1: Watch for a second safe_outputs GitHub App token 404 on Daily Harness Experiment Proposer or any other workflow — file a dedicated issue if it repeats.

Sub-issues created

• awf-agent container-init read-only file system failure (P0) — linked below.

References:

• §32670511170
• §32674599146
• §32670714843

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 131.9 AIC · ⌖ 14.1 AIC · ⊞ 6.4K · ◷

• expires on Aug 30, 2026, 5:22 PM UTC-08:00

---

Fix the root cause now — this incident tripled in blast radius since it was last reported

Action: prioritize the awf-reflect.json write race in the container-init path — it just spread from 6 to 20+ workflows in a single 6h window.

New evidence from the last 6h (2026-08-24 00:40–06:40 UTC), all sharing the identical signature EROFS: read-only file system, open '/tmp/gh-aw/sandbox/firewall/awf-reflect.json':

Workflow	Engine	Run
Daily Go Test Parallelizer	Copilot	§32698289132
Schema Feature Coverage Checker	Codex	§32698261215
AI Moderator	Codex	§32696507811
Agent Container Smoke Test	—	§32681869533
Smoke Gemini	Gemini	§32681869906

Plus ~20 more failed runs in the same window across Smoke Codex, Smoke Copilot Small, Design Decision Gate, Metrics Collector, and others — full run-ID list available on request.

Confirmed mechanism via audit-diff (§32698289132 vs last good run §32684304722): the failed run made 0 calls to api.github.com, github.com, proxy.golang.org, raw.githubusercontent.com, sum.golang.org — all present in the healthy baseline. GitHub Core API consumption dropped from 39 calls to 13. This confirms the agent CLI dies immediately after the reflect-config write fails (401 from api-proxy due to missing awf-reflect.json → CLI gives up before doing any real work), not a downstream symptom.

Success criteria: a 6h window with zero EROFS/awf-reflect.json occurrences across smoke workflows, and audit-diff parity on API-call volume vs a known-good baseline run.

Representative log excerpt

[codex-harness] awf-reflect: fetching (apiproxy/redacted) (timeout=60000ms)
[codex-harness] awf-reflect: models fetch returned 401 for (apiproxy/redacted)
[codex-harness] awf-reflect: request failed: EROFS: read-only file system, open '/tmp/gh-aw/sandbox/firewall/awf-reflect.json'
Error: failed to initialize in-process app-server client: Read-only file system (os error 30)

References:

• §32698289132
• §32697229897
• §32684304722 (last good baseline)> Generated by 🔍 [aw] Failure Investigator (6h) · claude · agent · 145.3 AIC · ⌖ 10 AIC · ⊞ 6.4K · ◷

[github-actions]

This issue was automatically closed because it expired on 2026-08-31T01:22:55.076Z.

Closed by Workflow