[aw-failures] [P0] Copilot-brokered credentials rejected by awf-agent api-proxy — Codex 404 + Claude 401

[github-actions]

Fix awf-agent api-proxy rejecting Copilot-brokered credentials — 100% failure for every workflow that authenticates via COPILOT_GITHUB_TOKEN/copilot/* model routing

Executive summary

• 28 failed runs in the last 6h window (2026-08-24, ~07:00–13:00 UTC).
• New P0, zero existing tracking coverage: any workflow that authenticates its agent engine through a Copilot-brokered credential — model: copilot/* on the Codex engine, or ANTHROPIC_API_KEY: secrets.COPILOT_GITHUB_TOKEN on the Claude engine — is rejected by the sandboxed api-proxy sidecar (172.30.0.30). Codex gets a 404 ("OpenAI proxy not configured"); Claude gets a 401 ("Not logged in · Please run /login"). Confirmed via audit-diff: the Codex-side regression began between 11:08 and 12:19 UTC today.
• Direct API-key-based Codex workflows (e.g. AI Moderator, which sets a real OPENAI_API_KEY) are not affected by this bug — this isolates the fault to the Copilot-token routing path specifically, not Codex/Claude engines in general.
• Separately, AI Moderator hit 5 read-only-filesystem crashes this window inside its cloud-hypervisor sandbox — same failure class as tracked #55211/#55212 but a different mountpoint and a later time window, so treated as corroborating evidence there rather than a new issue.

Failure cluster table

Cluster	Severity	Workflows hit	Runs (window)	Tracking
Copilot-BYOK → Codex api-proxy 404	P0	Ponytail Reviewer, Issue Monster	5	NEW, filed below
Copilot-BYOK → Claude api-proxy 401	P0	Smoke Claude on Copilot	4 this window (chronic: 0% pass in last 10 runs, ≥5 weeks)	NEW, filed below
cloud-hypervisor read-only-fs on state db	P0	AI Moderator	5	Related to #55211/#55212, comment added
Anthropic egress blackout (stale, 08-21)	P0	—	0 confirmed recurrences this window	Left open — no evidence of fix or recurrence

Evidence

Codex / Copilot BYOK (model: copilot/mai-code-1-flash-picker) — §32727091257 (Ponytail Reviewer), §32726112053 (Issue Monster):

Codex harness log excerpt

[codex-harness] secrets: CODEX_API_KEY=set (length=31) OPENAI_API_KEY=not set
POST (172.30.0.30/redacted) -> 404
"OpenAI proxy not configured (no OPENAI_API_KEY/COPILOT_PROVIDER_API_KEY or OIDC auth)"
... 4 attempts, all 3 retries exhausted — giving up (exitCode=1)

These workflows never set OPENAI_API_KEY at all — by design, they authenticate via Copilot's BYOK model routing (COPILOT_DUMMY_BYOK), not a raw OpenAI key. The .lock.yml for both explicitly --exclude-env OPENAI_API_KEY from the sandbox. Compare with AI Moderator's compiled workflow, which sets OPENAI_API_KEY: ${{ secrets.CODEX_API_KEY || secrets.OPENAI_API_KEY }} and never fails this way — confirming the api-proxy sidecar has no fallback path for Copilot-brokered credentials.

Confirmed via audit-diff (base §32720355441, success @ 11:08:56Z vs compare §32726474623, failure @ 12:19:17Z): downstream telemetry calls that only fire after a real model turn (Sentry, Grafana, ab.chatgpt.com) vanish entirely in the failing run — this is a genuine regression today, not a data-collection artifact.

Claude / Copilot BYOK (ANTHROPIC_API_KEY: secrets.COPILOT_GITHUB_TOKEN) — §32726702179 (Smoke Claude on Copilot):

Claude harness log excerpt

AWF API proxy: 1 of 5 providers configured
| anthropic | 10001 | ✅ reachable |
[claude-harness] awf-reflect: models fetch returned 401 for (apiproxy/redacted)
{"error":"authentication_failed","result":"Not logged in · Please run /login","api_error_status":401}
[claude-harness] attempt 1: authentication failed — not retrying

Firewall log for this run shows api.anthropic.com | 8 allowed | 0 blocked — the network path is healthy. This is a credential-rejection problem, not an egress blackout, and this workflow has a 0% pass rate across its last 10 runs going back to 2026-07-16 — this is chronic, not a new spike.

Existing issue correlation

• #55211 / #55212 (container-init read-only-fs race, opened 2026-08-23 22:25 UTC): related but distinct. Same failure class (EROFS inside the sandbox) but a different mountpoint (/tmp/awf-init overlay2 mount vs. AI Moderator's /tmp/gh-aw/mcp-config/state_*.sqlite + awf-reflect.json inside a cloud-hypervisor VM) and a different, later time window (07:17–07:57 UTC today vs. 22:25 UTC yesterday). Likely a second occurrence of the same underlying class rather than the original bug recurring — left open, corroborating comment added via update_issue.
• #54462 (Claude Code CLI startup crashes — Anthropic egress blackout, stale since 08-21): does not match. That signature is 0/N allowed egress requests; today's Smoke Claude on Copilot failures show 8/8 allowed requests plus a 401 auth response — a distinct credential problem. Left open; no evidence found either way to justify closing as fixed.
• #54186 (SIGSEGV/exit-139), #54402 (Cloud Hypervisor guest-connectivity probe), #54898/#53619 (Design Decision Gate): no matching signatures in this window. No action taken.

Fix roadmap

P0

1. Make the api-proxy sidecar accept Copilot-brokered credentials (COPILOT_GITHUB_TOKEN / COPILOT_DUMMY_BYOK) as valid auth for both the openai (10000) and anthropic (10001) provider routes — it currently 404s/401s them outright. This is 100% reproducible and breaks every model: copilot/* Codex workflow and every Claude workflow authenticating via COPILOT_GITHUB_TOKEN.
2. Investigate the AI Moderator cloud-hypervisor EROFS crash as a probable second instance of the [aw-failures] [P0] awf-agent container-init read-only-fs race takes down 6 workflows across 3 engines #55211/[aw-failures] [P0] awf-agent container fails to start: read-only file system on /tmp/awf-init #55212 container-init race class, at a different mountpoint.

P1 / P2: none new this cycle.

Sub-issues created

• Codex + Claude Copilot-BYOK api-proxy rejection (see linked sub-issue)

References:

• §32727091257 — Ponytail Reviewer, Codex 404
• §32726702179 — Smoke Claude on Copilot, Claude 401
• §32726474623 — audit-diff compare run

Generated by 🔍 [aw] Failure Investigator (6h) · claude · agent · 254.6 AIC · ⌖ 10.4 AIC · ⊞ 6.4K · ◷

• expires on Aug 31, 2026, 5:28 AM UTC-08:00

[github-actions]

This issue was automatically closed because it expired on 2026-08-31T13:28:58.673Z.

Closed by Workflow