[aw-failures] [aw] Failure Investigator Report — 2026-08-11 13:00 (6h)

[github-actions]

Overview

Fix the MCP Gateway's all-or-nothing startup check first — a single flaky external server (Datadog, HTTP 503) killed the entire Smoke OTEL gateway and starved 3 working integrations (Sentry, Grafana, OTLP) of their calls, and this is the 3rd occurrence in the last two weeks. Everything else this cycle is either already tracked, a confirmed still-open regression, or a single-run blip not worth a ticket yet.

Failure cluster table

Cluster	Severity	Workflow	Signature	Status	Runs
MCP gateway hard-fails on one flaky server	P0	Smoke OTEL	datadog ping HTTP 503 → "1 required server(s) failed connectivity check" → gateway terminated	New sub-issue	§31491021479
Bun runtime segfault	P0	Avenger	panic(main thread): Segmentation fault → Bun crash, all retries exhausted	Tracked: #51984	§31492348182
Copilot billing 403	P1	Daily Max Ai Credits Test	Authentication failed with provider ... HTTP 403 before credit-limit logic runs	Tracked: #51589 (confirmed still failing today)	§31486534868
Claude API ConnectionRefused + 10m timeout	P0 (isolated)	Daily Choice Type Test	API Error: Unable to connect to API (ConnectionRefused) ×3 attempts, then step timeout	Watch — 1st failure in 10 runs	§31486386308
Gemini project spend cap	P1 (isolated)	Smoke Gemini	Gemini API 429 RESOURCE_EXHAUSTED: "project has exceeded its monthly spending cap"	Watch — 1st non-skipped run this window	§31490942301
Smoke-test config bugs	P2	Smoke Copilot	safe_outputs: dispatch_workflow missing required message input; add_comment given invalid item number aw_smoke_discussion	Not ticketed — test-data bug, not infra	§31491012230

Evidence

Fix the OTEL gateway design flaw — it treats every configured MCP server as startup-critical. audit-diff against the last success (§31264782373) shows the failed run made zero calls to sentry.io, grafana.net, or the OTLP intake host — all healthy in the baseline (15/13/13 calls respectively) — because gateway startup aborted the instant datadog returned HTTP 503 after 3 retries, even though grafana, safeoutputs, and sentry had already connected successfully. This is the 3rd such failure in two weeks (§31023914914 2026-08-05, §30606870031 2026-07-31).

Re-open the Claude API ConnectionRefused ticket if it recurs. audit-diff on §31486386308 vs. the prior day's success shows the failed run made 2 allowed firewall calls to api.anthropic.com:443 that were refused at the TCP level (ConnectionRefused), while the successful comparator run shows zero calls to that domain at all — meaning the two runs took different network paths to reach the model. First occurrence in 10+ scheduled runs; not ticketed yet, but the routing divergence is worth a second look if it repeats.

Gemini smoke test failure is a billing cap, not a code bug. Google AI Studio returned 429 RESOURCE_EXHAUSTED — "project has exceeded its monthly spending cap." First occurrence (previous runs were all skipped); same class of problem as the already-tracked Copilot billing issue (#51589) but a different provider/account. Not ticketed — raise the spend cap or confirm it's expected, then watch for recurrence.

Existing issue correlation

• [aw-failures] [P0] Bun runtime segfault crashing Claude Code CLI — now hitting 2 workflows #51984 (Avenger Bun segfault) — matches today's Avenger failure exactly. No new action.
• [aw-failures] [P1] Fix Copilot billing 403 masking Daily Max Ai Credits Test — 5+ consecutive daily failures #51589 (Copilot billing 403 masking Daily Max Ai Credits Test) — confirmed still reproducing today with the identical authentication_failed HTTP 403 signature at the same proxy endpoint. Left open, no update needed.
• [aw-failures] [P0] Verify safeoutputs MCP gateway recovery after #51870 #51897 (safeoutputs MCP gateway recovery verification) — not directly evidenced this window; the Smoke Copilot safe_outputs job registered tools and processed 7/12 messages successfully (including a real issue creation), which is inconsistent with the "0 tools registered" failure mode that issue tracks, but that workflow isn't one of the 8 named in the success criteria. Left open pending confirmation on the named workflows.
• [aw-failures] [P0] Step Name Alignment sandbox blocks its own input file — 100% failure #51547 (Step Name Alignment sandbox lockout) — no runs of that workflow in this window. Left open.
• [aw-failures] [P0] Fix Copilot CLI harness segfault (exit 139) killing Agent Performance Analyzer #51789 (Copilot CLI harness segfault in Agent Performance Analyzer) — no runs of that workflow in this window. Left open.

Fix roadmap

• P0 — MCP gateway hard-fail on one server: make non-responding servers degrade individually instead of aborting the whole gateway; a single external 503 shouldn't take Sentry/Grafana/OTLP down with it. Tracked in the sub-issue below.
• P1 — Copilot billing 403: already tracked in [aw-failures] [P1] Fix Copilot billing 403 masking Daily Max Ai Credits Test — 5+ consecutive daily failures #51589, still reproducing daily — needs org billing/Copilot-requests permission fix, not a new ticket.
• P0/P1 — watch, don't ticket yet: Claude API ConnectionRefused on Daily Choice Type Test (re-open as P0 if it recurs); Gemini project spend cap on Smoke Gemini (re-open as P1 if it recurs).
• P2 — not actionable as infra: Smoke Copilot's dispatch_workflow/add_comment failures are test-fixture bugs (missing input, wrong item-number placeholder) — fix in the smoke workflow's own test data, not tracked here.

Sub-issues created

• MCP gateway startup hard-fails when one configured server is unreachable (Smoke OTEL / Datadog 503) — see linked sub-issue.

References:

• §31491021479 — Smoke OTEL failure (representative)
• §31486386308 — Daily Choice Type Test ConnectionRefused
• §31490942301 — Smoke Gemini spend-cap 429

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 124.7 AIC · ⌖ 42.2 AIC · ⊞ 5.3K · ◷

• expires on Aug 18, 2026, 5:20 AM UTC-08:00

---

[aw] Failure Investigator — 2026-08-12 01:00 (6h) cycle

Fix two P1 infra failures now — both are new, untracked, and each hit 2+ workflows or 4 straight retries.

Failure cluster table

Cluster	Severity	Workflows	Signature	Sub-issue
Claude proxy connection-refused	P1	Daily Documentation Healer, Daily Caveman Optimizer	Connection refused reaching Anthropic API via cli-proxy/squid	new (see linked sub-issue)
Ollama BYOK 503	P1	Daily BYOK Ollama Test	HTTP 503 html error on real inference after warm-up passes	new (see linked sub-issue)
Design Decision Gate cutoff	P2	Design Decision Gate 🏗️	§31543322301 log cuts off mid-tool-use, isolated single occurrence, no confirmed error text	none — monitor only
Team Evolution jq parse error	P2	Daily Team Evolution Insights	§31532493511 — agent tool calls succeeded; post-processing step "Parse agent logs for step summary" hit a transient jq parse error	none — monitor only
Stub / low-evidence	P2	Daily Ambient Context Optimizer, Code Scanning Fixer, Daily Code Debt Cleanup — Aider	§31530215524, §31526915222, §31526696700 — no captured logs in prefetch, insufficient evidence	none — re-investigate if recurring

Existing issue correlation: none of today's 8 failed runs match any currently-open agentic-workflows issue (checked against #52139, #51789, #51897, #51547, #51589, #51984, #51033, #52016, #48838, #51987) — no closures made, no evidence any of those are fixed or stale from this cycle's data.

Fix roadmap

• P0: none open this cycle.
• P1: fix Claude Code CLI proxy connection-refused (new sub-issue), fix Ollama BYOK 503 backend availability (new sub-issue).
• P2: monitor Design Decision Gate, Team Evolution Insights jq parser, and the 3 stub workflows for recurrence before opening issues.

Sub-issues created this cycle: 2 (Claude proxy connection-refused, Ollama BYOK 503).

References: §31548182853, §31542632254, §31535557507> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 221.7 AIC · ⌖ 47.4 AIC · ⊞ 5.3K · ◷

---

Overview

Fix the MCP Gateway's all-or-nothing startup check first — a single flaky external server (Datadog, HTTP 503) killed the entire Smoke OTEL gateway and starved 3 working integrations (Sentry, Grafana, OTLP) of their calls, and this is the 3rd occurrence in the last two weeks. Everything else this cycle is either already tracked, a confirmed still-open regression, or a single-run blip not worth a ticket yet.

Failure cluster table

Cluster	Severity	Workflow	Signature	Status	Runs
MCP gateway hard-fails on one flaky server	P0	Smoke OTEL	datadog ping HTTP 503 → "1 required server(s) failed connectivity check" → gateway terminated	New sub-issue	§31491021479
Bun runtime segfault	P0	Avenger	panic(main thread): Segmentation fault → Bun crash, all retries exhausted	Tracked: #51984	§31492348182
Copilot billing 403	P1	Daily Max Ai Credits Test	Authentication failed with provider ... HTTP 403 before credit-limit logic runs	Tracked: #51589 (confirmed still failing today)	§31486534868
Claude API ConnectionRefused + 10m timeout	P0 (isolated)	Daily Choice Type Test	API Error: Unable to connect to API (ConnectionRefused) ×3 attempts, then step timeout	Watch — 1st failure in 10 runs	§31486386308
Gemini project spend cap	P1 (isolated)	Smoke Gemini	Gemini API 429 RESOURCE_EXHAUSTED: "project has exceeded its monthly spending cap"	Watch — 1st non-skipped run this window	§31490942301
Smoke-test config bugs	P2	Smoke Copilot	safe_outputs: dispatch_workflow missing required message input; add_comment given invalid item number aw_smoke_discussion	Not ticketed — test-data bug, not infra	§31491012230

Evidence

Fix the OTEL gateway design flaw — it treats every configured MCP server as startup-critical. audit-diff against the last success (§31264782373) shows the failed run made zero calls to sentry.io, grafana.net, or the OTLP intake host — all healthy in the baseline (15/13/13 calls respectively) — because gateway startup aborted the instant datadog returned HTTP 503 after 3 retries, even though grafana, safeoutputs, and sentry had already connected successfully. This is the 3rd such failure in two weeks (§31023914914 2026-08-05, §30606870031 2026-07-31).

Re-open the Claude API ConnectionRefused ticket if it recurs. audit-diff on §31486386308 vs. the prior day's success shows the failed run made 2 allowed firewall calls to api.anthropic.com:443 that were refused at the TCP level (ConnectionRefused), while the successful comparator run shows zero calls to that domain at all — meaning the two runs took different network paths to reach the model. First occurrence in 10+ scheduled runs; not ticketed yet, but the routing divergence is worth a second look if it repeats.

Gemini smoke test failure is a billing cap, not a code bug. Google AI Studio returned 429 RESOURCE_EXHAUSTED — "project has exceeded its monthly spending cap." First occurrence (previous runs were all skipped); same class of problem as the already-tracked Copilot billing issue (#51589) but a different provider/account. Not ticketed — raise the spend cap or confirm it's expected, then watch for recurrence.

Existing issue correlation

• [aw-failures] [P0] Bun runtime segfault crashing Claude Code CLI — now hitting 2 workflows #51984 (Avenger Bun segfault) — matches today's Avenger failure exactly. No new action.
• [aw-failures] [P1] Fix Copilot billing 403 masking Daily Max Ai Credits Test — 5+ consecutive daily failures #51589 (Copilot billing 403 masking Daily Max Ai Credits Test) — confirmed still reproducing today with the identical authentication_failed HTTP 403 signature at the same proxy endpoint. Left open, no update needed.
• [aw-failures] [P0] Verify safeoutputs MCP gateway recovery after #51870 #51897 (safeoutputs MCP gateway recovery verification) — not directly evidenced this window; the Smoke Copilot safe_outputs job registered tools and processed 7/12 messages successfully (including a real issue creation), which is inconsistent with the "0 tools registered" failure mode that issue tracks, but that workflow isn't one of the 8 named in the success criteria. Left open pending confirmation on the named workflows.
• [aw-failures] [P0] Step Name Alignment sandbox blocks its own input file — 100% failure #51547 (Step Name Alignment sandbox lockout) — no runs of that workflow in this window. Left open.
• [aw-failures] [P0] Fix Copilot CLI harness segfault (exit 139) killing Agent Performance Analyzer #51789 (Copilot CLI harness segfault in Agent Performance Analyzer) — no runs of that workflow in this window. Left open.

Fix roadmap

• P0 — MCP gateway hard-fail on one server: make non-responding servers degrade individually instead of aborting the whole gateway; a single external 503 shouldn't take Sentry/Grafana/OTLP down with it. Tracked in the sub-issue below.
• P1 — Copilot billing 403: already tracked in [aw-failures] [P1] Fix Copilot billing 403 masking Daily Max Ai Credits Test — 5+ consecutive daily failures #51589, still reproducing daily — needs org billing/Copilot-requests permission fix, not a new ticket.
• P0/P1 — watch, don't ticket yet: Claude API ConnectionRefused on Daily Choice Type Test (re-open as P0 if it recurs); Gemini project spend cap on Smoke Gemini (re-open as P1 if it recurs).
• P2 — not actionable as infra: Smoke Copilot's dispatch_workflow/add_comment failures are test-fixture bugs (missing input, wrong item-number placeholder) — fix in the smoke workflow's own test data, not tracked here.

Sub-issues created

• MCP gateway startup hard-fails when one configured server is unreachable (Smoke OTEL / Datadog 503) — see linked sub-issue.

References:

• §31491021479 — Smoke OTEL failure (representative)
• §31486386308 — Daily Choice Type Test ConnectionRefused
• §31490942301 — Smoke Gemini spend-cap 429

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 124.7 AIC · ⌖ 42.2 AIC · ⊞ 5.3K · ◷

• expires on Aug 18, 2026, 5:20 AM UTC-08:00

---

[aw] Failure Investigator — 2026-08-12 01:00 (6h) cycle

Fix two P1 infra failures now — both are new, untracked, and each hit 2+ workflows or 4 straight retries.

Failure cluster table

Cluster	Severity	Workflows	Signature	Sub-issue
Claude proxy connection-refused	P1	Daily Documentation Healer, Daily Caveman Optimizer	Connection refused reaching Anthropic API via cli-proxy/squid	new (see linked sub-issue)
Ollama BYOK 503	P1	Daily BYOK Ollama Test	HTTP 503 html error on real inference after warm-up passes	new (see linked sub-issue)
Design Decision Gate cutoff	P2	Design Decision Gate 🏗️	§31543322301 log cuts off mid-tool-use, isolated single occurrence, no confirmed error text	none — monitor only
Team Evolution jq parse error	P2	Daily Team Evolution Insights	§31532493511 — agent tool calls succeeded; post-processing step "Parse agent logs for step summary" hit a transient jq parse error	none — monitor only
Stub / low-evidence	P2	Daily Ambient Context Optimizer, Code Scanning Fixer, Daily Code Debt Cleanup — Aider	§31530215524, §31526915222, §31526696700 — no captured logs in prefetch, insufficient evidence	none — re-investigate if recurring

Existing issue correlation: none of today's 8 failed runs match any currently-open agentic-workflows issue (checked against #52139, #51789, #51897, #51547, #51589, #51984, #51033, #52016, #48838, #51987) — no closures made, no evidence any of those are fixed or stale from this cycle's data.

Fix roadmap

• P0: none open this cycle.
• P1: fix Claude Code CLI proxy connection-refused (new sub-issue), fix Ollama BYOK 503 backend availability (new sub-issue).
• P2: monitor Design Decision Gate, Team Evolution Insights jq parser, and the 3 stub workflows for recurrence before opening issues.

Sub-issues created this cycle: 2 (Claude proxy connection-refused, Ollama BYOK 503).

References: §31548182853, §31542632254, §31535557507> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 221.7 AIC · ⌖ 47.4 AIC · ⊞ 5.3K · ◷

---

[aw] Failure Investigator — 2026-08-12 07:00 (6h) cycle

Fix the Copilot-engine api-proxy listener and the sandbox'''s over-eager command-injection guard now — both are new P1s this cycle, and the guard bug turned a benign MCP-test finding into a 5-minute timeout.

Failure cluster table

Cluster	Severity	Workflows	Signature	Sub-issue
Copilot proxy ECONNREFUSED (port 10002)	P1	Daily Assign Issue To User	ECONNREFUSED to internal api-proxy:10002/chat/completions despite "listening" + valid key	new — #aw_cprox
Sandbox guard blocks multi-line printf to safeoutputs	P1	GitHub Remote MCP Authentication Test	command-injection false-positive on \n in printf body → repeated rejects → 5-min step timeout	new — #aw_sbxg
Claude Code CLI → Anthropic ConnectionRefused	P1	Daily AgentRx Trace Optimizer	API Error: Connection refused after 10 api_retry attempts	Tracked: #52194 (confirmed still failing)
Step Name Alignment sandbox lockout	P0	Step Name Alignment	Claude Code CLI failure, same signature as tracked issue	Tracked: #51547 (confirmed still failing)
Avenger Bun segfault (presumed)	P0	Avenger	workflow-name match only; logs not re-pulled this cycle	Tracked: #51984 (not independently re-verified)
Container image critical vulnerabilities	P2 — real finding, not an aw bug	Daily Container Image Security Scan	scan gate correctly failed: "Critical vulnerabilities detected in container images"	none — route to image owners
Unclassified, no log evidence	P2 (unknown)	Go Logger Enhancement, Daily AstroStyleLite Markdown Spellcheck, GPL Dependency Cleaner (gpclean), Daily Compiler Quality Check, Daily action/setup/* Security Audit, Daily Regression Audit	prefetch only captured detailed logs for 5 of 12 failed runs; gh run view --json jobs was blocked by the sandbox'''s network policy for the rest	none — re-investigate next cycle once job-level API access is available

Evidence

The Copilot-engine proxy defect is the same class of bug as #52194, just a different listener. audit-diff on §31569260171 vs. the last success (§31361150414) shows zero firewall blocks and no domain changes — the failure is entirely internal to the sandbox'''s api-proxy sidecar, not an external network block. Full harness logs show the Copilot listener logged "listening on port 10002" and passed key validation ~30s before the first request, then refused all 4 chat-completion attempts over the next minute.

The MCP auth-test failure is a harness bug, not an MCP bug. §31567984790'''s actual finding (unknown tool "get_repository") was captured correctly, but the agent'''s two attempts to report it via safeoutputs create_discussion were both rejected by the sandbox'''s command-injection guard for containing literal \n in a multi-line printf body, and the step then timed out after 5 minutes with no retry-path guidance.

Prefetch coverage gap: only 5 of this window'''s 12 failed runs had truncated_error_logs in the deterministic payload; the other 6 (Go Logger Enhancement, AstroStyleLite Spellcheck, GPL Dependency Cleaner, Compiler Quality Check, action/setup Security Audit, Regression Audit) could not be classified — direct gh run view --json jobs calls for those run IDs failed with "error connecting to api.github.com" from this sandbox. Flagging as a gap rather than guessing at root cause.

Existing issue correlation: #52194 confirmed still reproducing (§31568090663); #51547 confirmed still reproducing (§31563267320); #51984 workflow-name match only (§31555885210), not re-verified. #51789, #51897, #51589, #51033, #52139, #52016, #48838, #51987 — no runs from those workflows this window; no evidence to close or update any of them.

Fix roadmap

• P0: none new this cycle — [aw-failures] [P0] Step Name Alignment sandbox blocks its own input file — 100% failure #51547 and [aw-failures] [P0] Bun runtime segfault crashing Claude Code CLI — now hitting 2 workflows #51984 are pre-existing P0s, both still open and still reproducing.
• P1: fix the Copilot-engine api-proxy listener refusing connections on port 10002 (new sub-issue); fix the sandbox command-injection guard'''s false positive on multi-line printf bodies (new sub-issue).
• P2: container image scan is a real vulnerability finding — route to image owners, not this tracker. Re-run job-level log capture for the 6 unclassified workflows before ticketing them.

Sub-issues created this cycle: 2 (#aw_cprox, #aw_sbxg).

References:

• §31569260171 — Daily Assign Issue To User, Copilot proxy ECONNREFUSED
• §31567984790 — GitHub Remote MCP Auth Test, sandbox guard timeout
• §31568090663 — Daily AgentRx Trace Optimizer, [aw-failures] [P1] Fix Claude Code CLI → Anthropic API connection-refused failures (proxy/firewall) #52194 still reproducing> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 165.2 AIC · ⌖ 50.6 AIC · ⊞ 5.3K · ◷