codex harness: secret diagnostics, missing-key fast-fail, --json streaming; switch dev.md to codex#34459
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…tion, --json streaming, and pre-flight check; update dev.md to codex engine" Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…sh in error message Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨ |
|
📰 BREAKING: Smoke Copilot is now investigating this pull request. Sources say the story is developing... |
|
🌑 The shadows whisper... Smoke Codex failed. The oracle requires further meditation... |
|
🚀 Smoke Gemini MISSION COMPLETE! Gemini has spoken. ✨ Caution agentic threat detected Testing safeoutputs CLI |
|
🚀 Smoke Pi MISSION COMPLETE! Pi delivered. 🥧 |
|
@copilot merge manic recompile fix conflicts, fix lint js, update wasm https://github.com/github/gh-aw/actions/runs/26363799116/job/77605156732#step:6:1 |
Agent Container Tool Check
Result: 12/12 tools available ✅ Overall Status: PASS
|
|
Caution agentic threat detected Smoke Test: Gemini - 26364283367\n- GitHub MCP Testing: ✅\n- Web Fetch Testing: ✅\n- File Writing Testing: ✅\n- Bash Tool Testing: ✅\n- Build gh-aw: ❌\n\nOverall status: FAILWarning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
…ation Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…ixture Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Smoke Claude — Run 26364283342Core #1–#13: ✅ all pass Overall: Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
💥 Automated smoke test review - all systems nominal!
Warning
Firewall blocked 6 domains
The following domains were blocked by the firewall during workflow execution:
accounts.google.comandroid.clients.google.comclients2.google.comcontentautofill.googleapis.comsafebrowsingohttpgateway.googleapis.comwww.google.com
To allow these domains, add them to the
network.allowedlist in your workflow frontmatter:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
💥 [THE END] — Illustrated by Smoke Claude · opus47 6.9M
| provider: | ||
| id: github | ||
| model: claude-sonnet-4-20250514 | ||
| id: codex |
There was a problem hiding this comment.
💡 Smoke test review: switching engine to codex with gpt-5.4 looks clean — consider noting the model rationale in the PR body for downstream reviewers.
| id: github | ||
| model: claude-sonnet-4-20250514 | ||
| id: codex | ||
| model: gpt-5.4 |
There was a problem hiding this comment.
💡 Smoke test review: a comment block above the engine: stanza explaining the migration from the pi runtime would help future maintainers.
Done in
|
There was a problem hiding this comment.
Pull request overview
This PR updates the Codex CLI harness and the Dev workflow configuration to (a) diagnose and fast-fail when required API keys are missing, (b) treat missing-key failures as non-retryable, and (c) ensure codex exec uses --json to emit JSONL events for CI log parsing.
Changes:
- Add missing-API-key detection + preflight key checks to
codex_harness.cjs, plus--jsoninjection forcodex exec. - Add unit tests for missing-key detection,
--jsonarg injection, and updated retry policy. - Switch
.github/workflows/dev.md(and the compiled lock) to thecodexengine /gpt-5.4.
Show a summary per file
| File | Description |
|---|---|
| actions/setup/js/codex_harness.cjs | Add missing-key detection, secret diagnostics, preflight fast-fail, and --json injection. |
| actions/setup/js/codex_harness.test.cjs | Add tests for missing-key detection, --json injection, and retry policy updates. |
| .github/workflows/dev.md | Switch the Dev workflow engine configuration to Codex / gpt-5.4. |
| .github/workflows/dev.lock.yml | Recompiled lockfile reflecting Codex engine + updated runtime wiring/log parsing. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 5/5 changed files
- Comments generated: 3
| // Diagnose API key presence so CI failures can be triaged without exposing secret values. | ||
| const codexApiKey = process.env.CODEX_API_KEY; | ||
| const openaiApiKey = process.env.OPENAI_API_KEY; | ||
| log(`secrets: CODEX_API_KEY=${codexApiKey ? `set (length=${codexApiKey.length})` : "not set"}` + ` OPENAI_API_KEY=${openaiApiKey ? `set (length=${openaiApiKey.length})` : "not set"}`); | ||
|
|
||
| // Pre-flight: require at least one API key before spawning codex. | ||
| // Without a key, codex exits immediately with "Missing environment variable" and every |
| const hadPromptFile = args.includes("--prompt-file"); | ||
| const safeArgs = hadPromptFile && resolvedArgs.length > 0 ? [...resolvedArgs.slice(0, -1), "<prompt omitted>"] : resolvedArgs; | ||
|
|
||
| // Inject --json after `exec` to stream structured JSONL events to stdout, making | ||
| // Codex output machine-readable in CI without affecting the stderr progress stream. | ||
| resolvedArgs = injectJsonFlag(resolvedArgs); | ||
|
|
||
| // Fetch AWF API proxy reflection data before running the agent to capture initial proxy state. | ||
| // This is best-effort: failures are logged but do not affect the agent run. | ||
| await fetchAWFReflect({ logger: log }); |
| sudo -E awf --config "${RUNNER_TEMP}/gh-aw/awf-config.json" --container-workdir "${GITHUB_WORKSPACE}" --mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro" --mount "${RUNNER_TEMP}/gh-aw:/host${RUNNER_TEMP}/gh-aw:ro" ${GH_AW_DOCKER_HOST_PATH_PREFIX_ARGS} --env-all --exclude-env CODEX_API_KEY --exclude-env GH_TOKEN --exclude-env GITHUB_MCP_SERVER_TOKEN --exclude-env MCP_GATEWAY_API_KEY --exclude-env OPENAI_API_KEY --log-level info --proxy-logs-dir /tmp/gh-aw/sandbox/firewall/logs --audit-dir /tmp/gh-aw/sandbox/firewall/audit --enable-host-access --allow-host-ports 80,443,8080 --skip-pull --difc-proxy-host host.docker.internal:18443 --difc-proxy-ca-cert /tmp/gh-aw/difc-proxy-tls/ca.crt \ | ||
| -- /bin/bash -c 'export PATH="${RUNNER_TEMP}/gh-aw/mcp-cli/bin:$PATH" && export PATH="$(find /opt/hostedtoolcache /home/runner/work/_tool -maxdepth 5 -type d -name bin 2>/dev/null | tr '\''\n'\'' '\'':'\'')$PATH"; [ -n "$GOROOT" ] && export PATH="$GOROOT/bin:$PATH" || true && GH_AW_NODE_EXEC="${GH_AW_NODE_BIN:-}"; if [ -z "$GH_AW_NODE_EXEC" ] || [ ! -x "$GH_AW_NODE_EXEC" ]; then GH_AW_NODE_EXEC="$(command -v node 2>/dev/null || true)"; fi; if [ -z "$GH_AW_NODE_EXEC" ]; then echo "node runtime missing on this runner — check runtimes.node in workflow YAML" >&2; exit 127; fi; "$GH_AW_NODE_EXEC" ${RUNNER_TEMP}/gh-aw/actions/codex_harness.cjs codex exec${GH_AW_MODEL_AGENT_CODEX:+ --model "$GH_AW_MODEL_AGENT_CODEX"} -c web_search="disabled" -c fetch="disabled" --dangerously-bypass-approvals-and-sandbox --skip-git-repo-check --prompt-file /tmp/gh-aw/aw-prompts/prompt.txt' 2>&1 | tee -a /tmp/gh-aw/agent-stdio.log |
|
Ug! Smoke test agent was here! Me caveman. Me test all tools. All work good. Fire hot. Code run. 🔥🪨 Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
|
🎉 Ug! Caveman very excited! New discussion born! Me smoke test agent run 26364283374. Me build big binary (29MB!). Me test playwright - see GitHub cave painting. Me find symbols in code rock. All fire hot and working! Cave art: 🪨🔥💻 Smoke rise, tests pass, caveman happy dance! 🕺 Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
There was a problem hiding this comment.
Ug! Caveman review PR. Me find good changes. Missing key now stop early. No waste retries. LGTM. 🪨
Warning
Firewall blocked 6 domains
The following domains were blocked by the firewall during workflow execution:
accounts.google.comandroid.clients.google.comclients2.google.comcontentautofill.googleapis.comsafebrowsingohttpgateway.googleapis.comwww.google.com
To allow these domains, add them to the
network.allowedlist in your workflow frontmatter:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
📰 BREAKING: Report filed by Smoke Copilot · sonnet46 2.2M
|
|
||
| /** | ||
| * Determines if the collected output indicates a missing API key at startup. | ||
| * Codex exits before producing any agent output in this case, so retrying is futile. |
There was a problem hiding this comment.
Caveman like isMissingApiKeyError. Fast-fail before retry loop — club bad config early! 🦴
| const AUTHENTICATION_FAILED_PATTERN = /Authentication failed(?:\s*\(Request ID:[^)]+\))?/i; | ||
|
|
||
| // Pattern to detect a missing API key at startup — Codex emits this before making any API | ||
| // calls when neither CODEX_API_KEY nor OPENAI_API_KEY is available in the environment. |
There was a problem hiding this comment.
Ug! Caveman see new pattern. MISSING_API_KEY_PATTERN good catch. No retry on bad key. 🪨
|
📰 DEVELOPING STORY: Smoke Copilot reports failed to deliver outputs. Our correspondents are investigating the incident... |
Codex was failing with
ERROR: Missing environment variable: \OPENAI_API_KEY`on every attempt because AWF excludes both API keys from the sandbox. The existingAUTHENTICATION_FAILED_PATTERN` didn't match this error, so the harness burned all 3 retries on an unrecoverable config problem before giving up.codex_harness.cjsCODEX_API_KEY=set (length=N)/not setand same forOPENAI_API_KEYat startup; no values exposedisMissingApiKeyError()+MISSING_API_KEY_PATTERN— detectsMissing environment variable: \OPENAI_API_KEY|CODEX_API_KEY`` and breaks the retry loop on any attempt (not just attempt 0)injectJsonFlag()— injects--jsonaftercodex execautomatically so stdout becomes a JSONL event stream per thecodex execnon-interactive mode spec; no-op if already present or subcommand isn'texeccodex_harness.test.cjsNew tests for
isMissingApiKeyError,injectJsonFlag, and the updatedshouldRetrypolicy (missing key is non-retryable on all attempts).dev.mdSwitched engine from
pi/claude-sonnet-4-20250514tocodex/gpt-5.4; lock file recompiled.✨ PR Review Safe Output Test - Run 26364283342
Warning
Firewall blocked 6 domains
The following domains were blocked by the firewall during workflow execution:
accounts.google.comandroid.clients.google.comclients2.google.comcontentautofill.googleapis.comsafebrowsingohttpgateway.googleapis.comwww.google.comSee Network Configuration for more information.