[aw-failures] Daily BYOK Ollama Test 100% red for 8+ days — offline+BYOK api-proxy returns 503 on /v1/models, Copilot CLI gets H [Content truncated due to length]

[github-actions]

Fix the offline+BYOK Ollama proxy path — Daily BYOK Ollama Test has failed every scheduled run for 8+ consecutive days (100% red, 0 turns, 0 tokens).

The Copilot CLI never makes a single model call: the BYOK proxy returns 503 on /v1/models and HTTP 400 (no body) on every completion, so the agent exits 1 with Turns=0. This is a chronic, fully-reproducible failure of a dedicated test workflow and is not covered by any existing open agentic-workflows issue.

Failure cluster

Field	Value
Workflow	Daily BYOK Ollama Test (.github/workflows/daily-byok-ollama-test.lock.yml)
Engine	GitHub Copilot CLI, offline+BYOK mode (`COPILOT_PROVIDER_BASE_URL=(172.30.0.30/redacted)
Severity	P1 — single workflow, 100% red, chronic
Representative run	§28269378808 (2026-06-26)
Comparator	no successful baseline exists — every prior run failed identically
Recurrence	9/9 scheduled runs failed: 28269378808, 28205363843, 28134431373, 28062125221, 27989476537, 27851504668, 27794407926, 27724841546 (6/17 → 6/26)
Turns / Tokens	0 / 0 (agent never executed)

Evidence (representative run 28269378808)

Probable root cause: the offline+BYOK api-proxy at :10002 cannot reach the local Ollama backend (host.docker.internal:11434), so the model list and completions both fail before the agent starts. Firewall analysis shows only host.docker.internal:443 allowed while host.docker.internal:11434 is among the blocked domains.

agent-stdio.log — proxy 503 then Copilot 400, all retries exhausted

[health-check] COPILOT_PROVIDER_BASE_URL=(172.30.0.30/redacted) (offline+BYOK mode)
[health-check] ✓ Copilot CLI offline+BYOK mode configured
[copilot-harness] awf-reflect: models fetch returned 503 for (apiproxy/redacted) retrying (attempt 2/5)
... (all 5 attempts return 503) ...
[warning] awf-reflect models fetch for (apiproxy/redacted) failed after 4 retry attempts: 503
[copilot-harness] attempt 1: process closed exitCode=1 duration=1s stdout=0B stderr=64B
[copilot-harness] attempt 1: outputTail="400 400 status code (no body)\nChanges +0 -0\nDuration 0s"
[copilot-harness] attempt 2: outputTail="400 400 status code (no body)\nDuration 6s"
... retries 1-3 all exit 1 with HTTP 400 ...

audit firewall_analysis — Ollama port blocked

allowed_domains: [ "host.docker.internal:443" ]
blocked_domains:  [ ..., "host.docker.internal", "host.docker.internal:11434", "api.githubcopilot.com", ... ]  (49 total)

The BYOK provider endpoint (Ollama, port 11434) is not in the allow-list — only :443 is. The proxy at :10002 therefore has no upstream to serve /v1/models or completions.

Existing issue correlation

No open agentic-workflows issue tracks this path. Adjacent-but-distinct issues that do not cover it:

• [aw-failures] Copilot BYOK provider HTTP 403 (authentication_failed) discards full Code Simplifier run #41195 — Copilot BYOK HTTP 403 (authentication_failed) on Code Simplifier (auth failure, different engine path, returns 403 not 503/400).
• [aw-failures] AWF firewall startup fails — getaddrinfo EAI_AGAIN awmg-cli-proxy, managed-gateway DNS not resolvable, agent never [Content truncated due to length] #41455 — AWF firewall EAI_AGAIN awmg-cli-proxy DNS (different proxy and error class).

Probable root cause

The offline+BYOK proxy (:10002) cannot reach the local Ollama backend on host.docker.internal:11434. Either (a) the firewall allow-list for this workflow omits :11434 (only :443 is allowed), or (b) Ollama is started but never serves the requested model, so /v1/models 503s and completions 400. The Terminate orphan process: pid (ollama) line in cleanup confirms Ollama is launched but is not reachable/serving by the time the agent runs.

Proposed remediation

1. Add host.docker.internal:11434 (and bare host.docker.internal) to the workflow's network.allowed allow-list so the BYOK proxy can reach Ollama.
2. Gate agent start on a readiness probe: poll (host.docker.internal/redacted) (or proxy /v1/models`) until 200 before spawning Copilot, instead of failing fast on the first 503.
3. Ensure the target model is pulled/loaded in the Ollama container before the agent step; fail the setup step (not the agent step) with a clear message if the model is missing.

Success criteria / verification

• A scheduled Daily BYOK Ollama Test run reaches Turns > 0 and TokenUsage > 0.
• /v1/models returns 200 from the proxy in agent-stdio.log (no 503 retry loop).
• No 400 status code (no body) in the copilot-harness output tail.
• Run conclusion is success for 2+ consecutive scheduled runs.

Other failures in this 6h window (no action — not new P0/P1 gaps)

• Daily yamllint Fixer §28273541137: single run hit maximum LLM invocations exceeded — not retrying. One-off, P2 — monitor; not yet recurring.
• Agentic Workflow Audit Agent §28266654332: first failure after 7 consecutive successes; logs show clean completion. Likely transient/false-red, P2 — monitor.
• Smoke CI (28267993076) and Auto-Triage Issues (28261421100): cancelled (concurrency supersession), not genuine failures.
• Minor cross-cutting noise: EACCES cleanup of /tmp/awf-*-chroot-home produces an exit-1 [WARN] tail in several runs but is not the primary failure cause.

References:

• §28269378808
• §28205363843
• §28134431373

Generated by 🔍 [aw] Failure Investigator (6h) · 144.8 AIC · ⌖ 39.6 AIC · ⊞ 5.2K · ◷

• expires on Jul 3, 2026, 5:36 PM UTC-08:00

[github-actions]

🍪 Issue Monster selected this for Copilot

I have identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

🍪 Om nom nom by Issue Monster · 3.62 AIC · ⌖ 10.1 AIC · ⊞ 6K · ◷

[github-actions]

This issue was automatically closed because it expired on 2026-07-04T01:36:13.710Z.

Closed by Workflow