Route harness fallback diagnostics through safeoutputs CLI by Copilot · Pull Request #35934 · github/gh-aw

Copilot · 2026-05-30T14:40:37Z

When agentic runs failed after retry exhaustion, harnesses attempted to append fallback diagnostics directly to outputs.jsonl; in read-only teardown states this hit EROFS, dropping structured failure signals. This change moves fallback emission to the safeoutputs CLI path so missing_tool / report_incomplete diagnostics are still emitted through the supported channel — for all three harnesses (Copilot, Claude, Codex).

Shared CLI helper module (safeoutputs_cli.cjs)
- Extracts runSafeOutputsCLI, buildMissingToolAlternatives, emitMissingToolPermissionIssue, and emitInfrastructureIncomplete into a single shared module consumed by all harnesses
- Error summaries use toolName(key1, key2) format for deterministic, value-free context
Fallback emission path
- emitMissingToolPermissionIssue now emits via safeoutputs missing_tool ... in all harnesses
- emitInfrastructureIncomplete now emits via safeoutputs report_incomplete ...
- Removes direct dependency on appending fallback payloads to GH_AW_SAFE_OUTPUTS file for these paths
- claude_harness and codex_harness switch from the old appendFileSync path to the shared CLI path
CLI emission hardening
- Validates CLI argument keys before constructing flags
- Improves error context to include tool name and key summary when CLI invocation fails
- Preserves existing skip behavior when GH_AW_SAFE_OUTPUTS is unset
Permission-denied diagnostics payload shaping
- Keeps denied-command context in alternatives
- Bounds output to safe length with compact overflow marker (... and N more) instead of naive truncation
Tests
- Adds safeoutputs_cli.test.cjs with 15 tests covering the shared module directly
- Updates fallback emission tests to assert CLI invocation contract
- Adds coverage for CLI failure logging path and alternatives formatting (empty list, capped length, overflow marker)

// safeoutputs_cli.cjs — shared by copilot_harness, claude_harness, codex_harness
runSafeOutputs("missing_tool", {
  tool: "tool/permission",
  reason: "missing tool/permission issue: numerous permission denied errors detected",
  alternatives: buildMissingToolAlternatives("Verify token scopes...", deniedCommands),
});

runSafeOutputs("report_incomplete", {
  reason: "infrastructure_error",
  details,
});

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-05-30T15:20:23Z

@copilot merge main and recompile

Copilot

Pull request overview

Routes Copilot harness fallback diagnostics (missing_tool / report_incomplete) through the safeoutputs CLI instead of appending JSONL directly to GH_AW_SAFE_OUTPUTS, so structured failure signals survive read-only teardown states (EROFS). Denied-command context is folded into the alternatives field with bounded length, and tests are updated to assert the CLI invocation contract.

Changes:

Adds runSafeOutputsCLI helper (with key validation and richer error context) and buildMissingToolAlternatives (512-char cap with ... and N more overflow marker).
Switches emitMissingToolPermissionIssue and emitInfrastructureIncomplete from appendFileSync to CLI invocation; preserves skip behavior when GH_AW_SAFE_OUTPUTS is unset.
Updates harness tests for the new CLI contract and adds coverage for empty/capped/overflow alternatives and CLI failure logging.

Show a summary per file

File	Description
actions/setup/js/copilot_harness.cjs	Adds CLI-based safeoutputs emission and bounded alternatives builder; replaces direct file appends in fallback emitters.
actions/setup/js/copilot_harness.test.cjs	Updates fallback tests to assert CLI calls and adds edge-case + failure-path coverage.
.github/workflows/*.lock.yml (8 files)	Regenerated lock files: checkout-line format flipped to `repo X → $GITHUB_WORKSPACE` and refreshed heredoc tag hashes; unrelated to harness changes.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 10/10 changed files
Comments generated: 2

+  try {
+    childProcess.execFileSync(command, commandArgs, { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] });
+  } catch (error) {
+    const err = /** @type {{message?: string, stderr?: string | Buffer}} */ error || {};


+    const argSummary = commandArgs.slice(1, 7).join(" ");
+    throw new Error(stderr ? `safeoutputs ${argSummary} failed: ${message}: ${stderr}` : `safeoutputs ${argSummary} failed: ${message}`);


…ix-silent-failure # Conflicts: # .github/workflows/schema-feature-coverage.lock.yml # .github/workflows/smoke-codex.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-05-30T15:31:40Z

@copilot merge main and recompile

Done — merged origin/main (resolved heredoc-token conflicts in the auto-generated lock files by accepting main's version) and recompiled. All 236/236 workflow files compiled with 0 errors.

github-actions · 2026-05-30T15:37:59Z

@copilot refresh the branch and rerun checks, then summarize any remaining blockers.

Generated by 👨‍🍳 PR Sous Chef · gpt54 11.5M · ◷

pelikhan · 2026-05-30T16:33:43Z

+ * @param {string} toolName
+ * @param {Record<string, string>} args
+ */
+function runSafeOutputsCLI(toolName, args) {


@copilot refactor safeoutputs cli helpers to own file and refactor support for all harnesses

Done in the latest commit. The CLI helpers are now in safeoutputs_cli.cjs which exports runSafeOutputsCLI, buildMissingToolAlternatives, emitMissingToolPermissionIssue, and emitInfrastructureIncomplete. All three harnesses (copilot, claude, codex) now require from this shared module — claude_harness and codex_harness also switch from the old appendFileSync path to the CLI path. A new safeoutputs_cli.test.cjs covers the shared module directly.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-05-30T16:55:48Z

@copilot lint js, lint go

https://github.com/github/gh-aw/actions/runs/26689561214/job/78663464271?pr=35934

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-05-30T17:06:13Z

@copilot lint js, lint go

https://github.com/github/gh-aw/actions/runs/26689561214/job/78663464271?pr=35934

Fixed in the latest commit:

JS lint: ran Prettier --write on copilot_harness.cjs, safeoutputs_cli.cjs, safeoutputs_cli.test.cjs
Go lint: removed the unused skillDownloadLog var and downloadSkillFileFromGitHub/downloadSkillFileViaGHCLI functions from pkg/cli/agent_download.go (dead code introduced via the main merge); fixed the testifylint assert.Len issue in pkg/workflow/claude_tools_helpers_test.go

Initial plan

c1527bc

Copilot AI assigned Copilot and pelikhan May 30, 2026

Copilot started work on behalf of pelikhan May 30, 2026 14:45 View session

Copilot AI linked an issue May 30, 2026 that may be closed by this pull request

[aw-failures] Copilot harness drops failure diagnostics — fallback safe-output emission fails with EROFS (read-only safeoutputs) after retry e [Content truncated due to length] #35888

Closed

Copilot AI and others added 3 commits May 30, 2026 14:50

Plan fallback safeoutputs CLI emission

6a60504

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Use safeoutputs CLI for copilot fallback emissions

1200173

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Harden safeoutputs CLI fallback emission handling

b7fc0a2

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix silent drop of failure diagnostics in Copilot harness~~ Route copilot harness fallback diagnostics through safeoutputs CLI May 30, 2026

Copilot finished work on behalf of pelikhan May 30, 2026 15:15

Copilot AI requested a review from pelikhan May 30, 2026 15:15

pelikhan approved these changes May 30, 2026

View reviewed changes

pelikhan marked this pull request as ready for review May 30, 2026 15:20

Copilot AI review requested due to automatic review settings May 30, 2026 15:20

Copilot started reviewing on behalf of pelikhan May 30, 2026 15:20 View session

Copilot AI reviewed May 30, 2026

View reviewed changes

Copilot started work on behalf of pelikhan May 30, 2026 15:22 View session

Merge remote-tracking branch 'origin/main' into copilot/aw-failures-f…

8db21af

…ix-silent-failure # Conflicts: # .github/workflows/schema-feature-coverage.lock.yml # .github/workflows/smoke-codex.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan May 30, 2026 15:31

Copilot AI requested a review from pelikhan May 30, 2026 15:31

pelikhan reviewed May 30, 2026

View reviewed changes

Copilot started work on behalf of pelikhan May 30, 2026 16:33 View session

Extract safeoutputs CLI helpers to shared module for all harnesses

226f2ae

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~Route copilot harness fallback diagnostics through safeoutputs CLI~~ Route harness fallback diagnostics through safeoutputs CLI May 30, 2026

Copilot finished work on behalf of pelikhan May 30, 2026 16:49

Copilot AI requested a review from pelikhan May 30, 2026 16:49

Merge branch 'main' into copilot/aw-failures-fix-silent-failure

0f77016

Copilot started work on behalf of pelikhan May 30, 2026 16:55 View session

Fix JS Prettier formatting and Go lint issues

0e39751

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan May 30, 2026 17:06

pelikhan merged commit 6ab90a2 into main May 30, 2026
32 checks passed

pelikhan deleted the copilot/aw-failures-fix-silent-failure branch May 30, 2026 17:12

This was referenced May 30, 2026

Smoke Test: Codex - 26690105532 #35975

Closed

chore: bump firewall to v0.25.58 and gateway to v0.3.22 #35973

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Route harness fallback diagnostics through safeoutputs CLI#35934

Route harness fallback diagnostics through safeoutputs CLI#35934
pelikhan merged 8 commits into
mainfrom
copilot/aw-failures-fix-silent-failure

Copilot AI commented May 30, 2026 •

edited

Loading

Uh oh!

pelikhan commented May 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI commented May 30, 2026

Uh oh!

github-actions Bot commented May 30, 2026

Uh oh!

pelikhan May 30, 2026

Uh oh!

Copilot AI May 30, 2026

Uh oh!

pelikhan commented May 30, 2026

Uh oh!

Copilot AI commented May 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		const argSummary = commandArgs.slice(1, 7).join(" ");
		throw new Error(stderr ? `safeoutputs ${argSummary} failed: ${message}: ${stderr}` : `safeoutputs ${argSummary} failed: ${message}`);

Conversation

Copilot AI commented May 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelikhan commented May 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Copilot AI commented May 30, 2026

Uh oh!

github-actions Bot commented May 30, 2026

Uh oh!

pelikhan May 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 30, 2026

Choose a reason for hiding this comment

Uh oh!

pelikhan commented May 30, 2026

Uh oh!

Copilot AI commented May 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented May 30, 2026 •

edited

Loading