Meta: enable gh-aw-threat-detection v0.4.0 on the external detection path

[davidslater]

Context

github/gh-aw-threat-detection publishes the standalone threat-detect binary that gh-aw uses when features: gh-aw-detection: true is set. Upstream audited the external detector path against gh-aw's inline path (buildDetectionEngineExecutionStep + parse_threat_detection_results.cjs) to confirm that flipping workflows over would not regress checks, logging, or behavior.

Most of the resulting work landed upstream and shipped in v0.4.0. This meta-issue tracks the subset that can only be fixed in gh-aw, because the relevant values are produced by the compiler and never reach the binary.

Upstream sources, all reviewed and consolidated here:

• gh-aw upstream follow-up items from PRs (meta-issue) gh-aw-threat-detection#745 — meta issue, 14 detailed follow-up comments
• Tracking: parity with gh-aw's inline threat detection before migrating workflows to the external detector gh-aw-threat-detection#705 — full parity checklist
• Upstream draft (for github/gh-aw): compiler-side changes needed for external threat-detect parity gh-aw-threat-detection#726 — upstream-authored draft of the gh-aw-side asks

Headline findings

Several documented, user-facing features are currently inert on the external path, silently. None emits a warning; all of them look like they are working. Three items are more than cosmetic:

• v0.4.0 does not currently work at all. The detector aborts with exit 2 before the engine runs, because GITHUB_STEP_SUMMARY is unwritable inside the AWF sandbox (threat-detection: GITHUB_STEP_SUMMARY is unwritable in the AWF sandbox, aborting the v0.4.0 detector with exit 2 #50659). This is the practical blocker for the pin bump.
• A failed detection execution can currently pass the gate on the external path where the inline path would block — fail-open (threat-detection: delegate all conclude branches to threat-detect conclude on the external path #50646).
• A strict-mode infrastructure failure renders to reviewers as [!CAUTION] agentic threat detected (threat-detection: strict-mode conclude failure writes no outputs, rendering infra failures as detected threats #50645).

Work items

#	Issue	Priority	Depends on
1	#50659 — GITHUB_STEP_SUMMARY unwritable in the AWF sandbox, aborts v0.4.0 with exit 2	P0 — blocker	—
2	#50644 — bump external detector pin to threat-detect v0.4.0	P0	#50659
3	#50645 — strict-mode conclude failure writes no outputs	P0	— (land immediately)
4	#50646 — delegate all conclude branches to threat-detect conclude	P0	#50644
5	#50647 — stage prompt-template.txt, prompt-import-tree.json, aw_info.json, comment-memory/	P1	—
6	#50649 — workflow context + detection env vars never reach the execution step	P1	—
7	#50651 — enable and upload the structured JSONL run log	P2	#50644
8	#50653 — external path writes an unused prompt to the step summary	P2	#50659
9	#50654 — max-ai-credits not propagated to the external AWF config	P2	—
10	#50655 — document macOS support, add installer platform-mapping tests	P3	#50644

Note: the GITHUB_STEP_SUMMARY writability caution originally listed as a bullet in #50647 is superseded by #50659 and should be dropped from that issue.

Suggested sequencing

1. threat-detection: strict-mode conclude failure writes no outputs, rendering infra failures as detected threats #50645 now, independently. Live bug, no coordination, no new binary.
2. threat-detection: GITHUB_STEP_SUMMARY is unwritable in the AWF sandbox, aborting the v0.4.0 detector with exit 2 #50659 next. Nothing on v0.4.0 works until this lands, so the pin bump is blocked behind it.
3. threat-detection: bump external detector pin to threat-detect v0.4.* (Wait for v0.4.* to pass local smokes) #50644 once threat-detection: GITHUB_STEP_SUMMARY is unwritable in the AWF sandbox, aborting the v0.4.0 detector with exit 2 #50659 is in and v0.4.0 is promoted (it is currently marked prerelease upstream).
4. threat-detection: stage prompt-template, prompt-import-tree, aw_info.json and comment-memory into the detection dir #50647 + threat-detection: workflow context and detection env vars never reach the external detector execution step #50649 + threat-detection: external path writes an unused prompt to the step summary, duplicating the detector's real one #50653 + threat-detection: max-ai-credits is not propagated to the external detector's AWF config #50654 as one PR. All touch the detection-job step builders, all need the same gh aw compile lock-file refresh, and all are safe against the currently pinned binary. Batching avoids repeated lock-file conflicts. (threat-detection: external path writes an unused prompt to the step summary, duplicating the detector's real one #50653 is grouped here for lock-file locality but should be verified after threat-detection: GITHUB_STEP_SUMMARY is unwritable in the AWF sandbox, aborting the v0.4.0 detector with exit 2 #50659.)
5. threat-detection: delegate all conclude branches to threat-detect conclude on the external path #50646 once threat-detection: bump external detector pin to threat-detect v0.4.* (Wait for v0.4.* to pass local smokes) #50644 lands and the mustFail decision below is recorded.
6. threat-detection: enable and upload the structured JSONL run log on the external path #50651 + threat-detection: document macOS support and add installer platform-mapping tests #50655 last.

Decisions required before merging

Two behavioral questions must be settled explicitly rather than by omission:

1. mustFail (threat-detection: delegate all conclude branches to threat-detect conclude on the external path #50646). threat-detect conclude fails closed in warn mode when the engine step also failed; parse_threat_detection_results.cjs always warns and proceeds, per ADR 29031-threat-detection-resilience-in-warn-mode. Delegating changes behavior for every workflow currently in warn mode, silently, the moment it sets gh-aw-detection: true. Either upstream drops mustFail to match ADR 29031, or gh-aw adopts it on the inline path too and ADR 29031 is amended. Upstream is happy either way and asks only that the decision be recorded.
2. Env vs flags (threat-detection: workflow context and detection env vars never reach the external detector execution step #50649). Upstream added --workflow-name / --workflow-description / --custom-prompt-file specifically so this plumbing cannot be dropped by an env filter, and prefers the file form for the custom prompt. gh-aw should confirm the approach. Note that threat-detection: GITHUB_STEP_SUMMARY is unwritable in the AWF sandbox, aborting the v0.4.0 detector with exit 2 #50659 is direct evidence for the flags side of this argument: step-level env: is not a reliable channel into the sandboxed process.

Lower-stakes decisions, but still worth writing down: whether experiments/ belongs inside the detection surface (#50647), whether the Setup threat detection step is retired entirely on the external path (#50653), and whether non-proxy detector invocation is supported at all (#50654).

Open question spun out of #50659

The agent job uses the same ineffective GITHUB_STEP_SUMMARY redirect mechanism across all five engines. If the runner re-injects the file-command variable there too, agent-step-summary.md has been empty all along and nobody noticed, because no agent treats the write failure as an error. #50659 includes verifying this; if confirmed, it needs its own issue since it touches every engine.

Validated as already fixed — no issue filed

Two upstream asks were checked against main and are already resolved. Recording them so they are not re-filed:

• apiProxy.models missing from the detection AWF config. Upstream reported Copilot model aliases such as auto failing to resolve in the detection job with 400 The requested model is not supported. pkg/workflow/threat_detection_external.go:319 now propagates data.ModelMappings into the detection workflow data, and lines 280-315 additionally resolve the detection model explicitly rather than leaving it as an alias. Upstream's local workaround (pinning safe-outputs.threat-detection.model) can be removed once they re-test.
• Darwin installer support. install_threat_detect_binary.sh already has the full install_darwin_binary() branch. Only the stale header comment and missing tests remain, tracked in threat-detection: document macOS support and add installer platform-mapping tests #50655.

Definition of done

• Every issue above closed or explicitly waived with rationale recorded.
• A side-by-side run of the same workflow on both paths produces equivalent job-log diagnostics, equivalent conclusion / reason / success outputs across the full failure matrix, and an equivalent or better verdict.
• Both decisions above recorded (ADR amendment where applicable).
• docs/src/content/docs/reference/threat-detection.md updated to describe the resulting external-path contract.

[github-actions]

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Assigning to @mnkiefer — pelikhan and lpcox already have 2 issues each this run; mnkiefer is the next most recent active contributor available for this backfill.

Generated by 🔧 Daily Assign Issue To User · auto · 56.7 AIC · ⊞ 5.6K · ◷

[github-actions]

Assignment rationale

Assigned to @pelikhan, who has recently authored several closely related threat-detection compiler PRs (#50746, #50915, #50918, #50924, #50935) covering this same area of the codebase.

Generated by 🔧 Daily Assign Issue To User · auto · 20.5 AIC · ⊞ 5.6K · ◷