[aw-failures] [aw] Failure Investigator Report — 2026-08-20 18:41 UTC (6h)

[github-actions]

Fix the Smoke Copilot build fleet now — Go toolchain pin is broken and blocking all 3 smoke variants

One-sentence rationale: go.mod/go tool pin go1.26.5 no longer matches the toolchain image's go1.26.6, so make build fails before any smoke test can run.

This report covers 21 failed runs across the last 6 hours (2026-08-20 13:26–18:41 UTC). 8 distinct root-cause clusters were identified; 5 already have tracking issues (updated below with fresh evidence), 1 new P0 gap gets a linked sub-issue, and 2 are low-priority/non-infra and need no new issue.

Act on this first (P0)

• Merge a toolchain pin fix immediately. go1.26.6 does not match go tool version go1.26.5 breaks make: *** [Makefile:42: build] Error 1 on all 3 Smoke Copilot variants (apikey, AOAI apikey, AOAI Entra). Tracked in the new sub-issue below — fix is a one-line go.mod/toolchain bump.
• Cloud Hypervisor guest-network failures are still active — do not treat [aw-failures] [P0] Cloud Hypervisor guest network never comes up — full egress blackout kills sandbox runs across engines #53935 as resolved. 9 fresh runs across 6 workflows failed with the same guest-connectivity-probe/ECONNRESET signature between 13:26 and 18:41 today. Comment added with full run list.

Cluster table

#	Cluster	Severity	Runs	Workflows	Status
1	Cloud Hypervisor guest-network down	P0	9	Ponytail Reviewer ×4, Architecture Guardian, Daily Malicious Code Scan Agent, CLI Consistency Checker, Daily Go Test Parallelizer, Scout	Existing #53935 — updated
2	Codex CLI invalid_project 401	P1	3	AI Moderator ×2, Daily Cache Strategy Analyzer	Existing #54242 — updated with root cause
3	Pi CLI segfault (exit 139)	P1	1	Auto-Triage Issues	Existing #54186 — updated
4	Claude Code CLI 30/30 invocation cap (429)	P1	1	Design Decision Gate	Existing #53619 — corrected, kept open
5	Go toolchain mismatch breaks build	P0	3	Smoke Copilot, Smoke Copilot - AOAI apikey, Smoke Copilot - AOAI Entra	NEW — sub-issue created
6	approve_workflow_run unhandled in safe_outputs	P1	2	PR Sous Chef ×2	Existing #53263 — updated
7	KVM kernel module not loaded	P2	1	Daily Fact	Not tracked — isolated, no new issue
8	Super Linter content-lint failures	P2	1	Super Linter Report	Not a bug — linter caught real MD013/MD009/MD060/MD034/MD049 violations in specs/security-architecture-spec.md; fix the doc, not the workflow

Evidence detail per cluster

Cluster 1 — Cloud Hypervisor guest-network down (P0): All 9 runs show Cloud Hypervisor guest connectivity probe failed with exit code 4, preceded in most cases by MCP gateway health check failed (attempt N/120): read ECONNRESET. Confirmed via full job logs on §32395535034, §32394421163, §32375962917.

Cluster 2 — Codex invalid_project 401 (P1): 401 Unauthorized ... auth error code: invalid_project returned by the internal API proxy (`(172.30.0.30/redacted) on every Codex CLI call. This is a NEW concrete signature — #54242 previously had no captured error. Daily Cache Strategy Analyzer is a newly-affected workflow not previously listed on that issue.

Cluster 3 — Pi CLI segfault (P1): Auto-Triage Issues run §32402675238 exits 139 (SIGSEGV) inside the Pi CLI engine — matches #54186 exactly.

Cluster 4 — Claude Code CLI rate-limit cap (P1): Design Decision Gate run §32394421229 fails with Maximum LLM invocations exceeded (30 / 30), a non-retryable 429 guard — NOT a silent hang as #53619 currently states. No commit since #53619 opened has touched this workflow's .md/.lock.yml with a targeted fix, so the issue stays open per the tool-denial-limit rule; corrected characterization posted as a comment.

Cluster 5 — Go toolchain mismatch (P0, NEW): All 3 Smoke Copilot variants fail identically: compile: version "go1.26.6" does not match go tool version "go1.26.5" repeated across every package, terminating in make: *** [Makefile:42: build] Error 1. No existing issue covers a build-time toolchain mismatch (#54116 and #53262 cover artifact-upload/naming symptoms, not this). Sub-issue created below.

Cluster 6 — approve_workflow_run unhandled (P1): PR Sous Chef runs §32389691755 and §32381373575 both hard-fail the safe_outputs job with "No handler loaded" for approve_workflow_run — matches #53263 exactly.

audit-diff regression check (Cluster 1, highest severity): Comparing failed run 32404370749 against baseline success 32387174411 shows the failed run never reached o205451.ingest.us.sentry.io or otlp-gateway-prod-eu-west-2.grafana.net (both new/absent) and made only 2 GitHub API calls vs. 34 in the successful baseline — the sandbox never obtained working outbound network access at all, consistent with the guest-connectivity-probe failure occurring before telemetry/API setup.

Existing issue correlation

All 5 matched issues remain open and accurately track active, recurring problems — none are fixed or stale, so none are closed. Each gets a comment with today's confirming run IDs; #53619 additionally gets a root-cause correction (see above).

Sub-issues created

• Go toolchain version mismatch breaks Smoke Copilot build fleet (P0) — linked below.

References:

• §32404370749
• §32375443008
• §32397326401

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 238.8 AIC · ⌖ 7.28 AIC · ⊞ 5.9K · ◷

• expires on Aug 27, 2026, 11:19 AM UTC-08:00

---

Follow-up — 2026-08-21 19:01 UTC (6h) — no new parent needed, three items updated

One-sentence rationale: all 14 failures in this window map to clusters already covered by open trackers or by a same-day merged fix — no P0 gap, so no new report issue was created.

Act on this first

• Don't trust [aw-failures] [P0] Codex OpenAI invalid_project 401 auth failure — unfixed 3+ days, Daily Cache Strategy Analyzer failing every #54242's closure — it's reopened. The invalid_project 401 auth error on the AWF api-proxy was confirmed via audit and recurred 5 more times today (AI Moderator ×4 new + 1 more, Daily Cache Strategy Analyzer ×1) after being closed with no linked fix. Root cause and fix are now documented there.
• Linter Miner's Go/Serena failure (§32508657133, 17:32 UTC) needs no action — already fixed. audit-diff against the prior successful run showed the Serena/Go error is a pre-existing, non-fatal red herring present in both runs; the real cause was model_not_supported for the linter-writer sub-agent, fixed by #54579 which merged at 18:55 UTC — after this run failed. No recurrence expected on the next scheduled run.

Cluster table (this window)

Cluster	Severity	Runs	Workflows	Status
Codex CLI invalid_project 401	P1	6	AI Moderator ×5, Daily Cache Strategy Analyzer ×1	Existing #54242 — reopened with confirmed root cause
Pi CLI segfault-shape recurrence	P1	1	Auto-Triage Issues	Existing #54186 — updated
Serena Go-LS red herring / real cause model_not_supported	P0→fixed	1	Linter Miner	Existing #54571, fixed by #54579 same day — no action
Isolated single-run failures, no shared signature	P2	6	Avenger, Daily Security Observability Report, Daily Fact, Super Linter Report, CI Optimization Coach, Code Scanning Fixer	Not tracked — monitor only

Evidence detail

Codex CLI invalid_project 401: audit on §32515609498 shows firewall clean (74/74 allowed, 0 blocked) and the actual fault: unexpected status 401 Unauthorized ... auth error code: invalid_project from the AWF api-proxy. codex-harness doesn't classify this as an auth failure, so it retries 3x then surfaces bare exit code 1 — explaining why this was previously filed as "no signature captured." Full detail and affected run IDs in #54242.

Linter Miner: audit-diff between failed §32508657133 and last success §32398150992 shows Setup Go/Go toolchain identical and successful in both; the Serena Go-LS error fires identically in both runs (ephemeral container never bundles Go — cosmetic noise). The actual differentiator is failureClass=model_not_supported on the Copilot backend, absent from the success run, and already fixed same-day by #54579.

Isolated cluster: six workflows each failed exactly once with no retrievable detailed logs and no shared error signature between them — consistent with this report's prior isolated/no-action items (KVM module, lint content violations). No new issue opened; will re-cluster if any of these repeats in the next window.

References:

• §32515609498
• §32508657133
• §32512842539> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 338.5 AIC · ⌖ 13.3 AIC · ⊞ 6.4K · ◷