canary-rollout sync-issues aborts every scheduled run — unguarded _gh_issue_create under set -e (+ App missing Issues:write) → no blocker issues, no fleet dashboard

Symptom

Every scheduled canary-rollout run ends with ##[warning]issue sync failed (non-fatal) (confirmed across runs 28707434935, 28702819582, 28698109014, 28692311429 — i.e. 100% failure since deploy). Consequences:

No canary-blocker issues are ever filed (the label doesn't even exist in this repo).
The fleet-status dashboard never renders to the job summary.

So the entire auto-triage/observability layer (#1063) is dark, even though promotion gating itself works.

Root cause (code)

cmd_sync_issues runs under set -euo pipefail. For the one BLOCKED agent with no existing issue it hits:

# scripts/canary-rollout.sh:756
num="$(_gh_issue_create "$title" "$body" "canary-blocker")"

_gh_issue_create is gh issue create … 2>/dev/null | grep -oE '[0-9]+$' | tail -1 — under pipefail it returns non-zero whenever gh fails or grep matches nothing. Because line 756 is a bare assignment (not local, which would mask the exit), set -e aborts the whole step — before the fleet dashboard renders (line ~790) and before the intended fallback warning at line 760. Non-blocked agents with no issue print nothing, which is why the step goes silent for ~3 min then dies. Same bug family as #594.

Root cause (config)

The canary-blocker label was never created and no blocker issue was ever filed → the release-manager App appears to lack Issues:write on petry-projects/.github-private, despite the workflow header claiming that scope. Even with the code guard, blocker-issue creation will keep failing until the App installation is granted Issues:write (label + issue create).

Fix

Code (self-fixable, PR incoming): guard line 756 — num="$(_gh_issue_create … || true)" (and audit siblings: _issue_find, the gh issue edit/reopen/close calls) so set -e can never abort the step. The existing empty-num branch already emits the correct Issues:write on the App? warning and the dashboard still renders → graceful degradation: the fleet table appears every run even when issue-filing is denied.
Config (needs org-admin, cannot self-apply): grant the release-manager GitHub App Issues:write on petry-projects/.github-private.

Acceptance

A scheduled run with ≥1 BLOCKED agent renders the fleet-status table to the job summary and exits 0.
With Issues:write granted: one canary-blocker issue per BLOCKED agent, auto-closed on clear.
Without it: warning logged, dashboard still renders, run still green.

Found while monitoring the canary rollout (pr-review-mention was BLOCKED and no blocker issue was filed).

Dev-Lead Implementation Plan

Issue: #1081 — canary-rollout sync-issues aborts every scheduled run — unguarded _gh_issue_create under set -e

Scope

Guard the one bare command-substitution assignment in cmd_sync_issues (num="$(_gh_issue_create …)", scripts/canary-rollout.sh:756) with || true so a failed/denied gh issue create can no longer trip set -e and abort the step — the existing empty-num warning branch already handles graceful degradation and the fleet-status dashboard still renders.

Root-cause detail

_gh_issue_create is gh issue create … | grep -oE '[0-9]+$' | tail -1. Under set -euo pipefail it returns non-zero whenever gh fails or grep matches nothing. Line 756 is a bare assignment (not local), so its non-zero exit propagates to set -e and aborts cmd_sync_issues before the fallback warning (line 760) and before the dashboard render (~line 790). Same bug family as #594.

Sibling audit (already safe — no change needed)

num_state="$(_issue_find …)" (747): _issue_find ends with || echo "" → always returns 0.
gh issue edit --add-label (758, 767): guarded by trailing || true.
gh issue reopen (764): trailing || true.
gh issue edit --title/--body (765): trailing || echo "::warning::…" → returns 0.
gh issue close (775): trailing || true.

Implementation Checklist

scripts/canary-rollout.sh:756 — change num="$(_gh_issue_create "$title" "$body" "canary-blocker")" to num="$(_gh_issue_create "$title" "$body" "canary-blocker" || true)"

Tests to Write

tests/canary_rollout.bats — new test: with a BLOCKED agent and gh issue create failing (Issues:write denied), sync-issues exits 0, logs the Issues:write on the App? warning, and still renders the fleet-status table to the job summary. (Extend _sync_stub so gh issue create can be made to fail via an env flag.)

Test Command

bats tests/canary_rollout.bats

Note on the config half of the issue

Granting the release-manager App Issues:write on petry-projects/.github-private is an org-admin action that cannot be self-applied from this repo; this PR delivers only the code guard (graceful degradation), which is the self-fixable half.

Dev-Lead: Implementation Complete

Plan Execution

Guarded the bare command-substitution assignment at scripts/canary-rollout.sh:756 with || true
Audited the sibling gh/_issue_find calls in cmd_sync_issues — all already guarded (|| true / || echo); no further changes needed
Added a regression test proving graceful degradation when Issues:write is denied
Confirmed red (test aborts non-zero before fix) → green (fix in place)
Full canary_rollout.bats suite + dev-lead-lint.sh (shellcheck --severity=warning on 88 files, agent-profile + CODEOWNERS validation) pass

Test Results

$ bats tests/canary_rollout.bats
1..75
...
ok 66 orchestrator: sync-issues opens ONE blocker issue (with evidence) + writes the fleet summary — no dashboard issue
ok 67 orchestrator: sync-issues degrades gracefully when Issues:write is denied — warns, renders the dashboard, exits 0 (#1081)
ok 68 orchestrator: sync-issues auto-closes a cleared agent's open blocker issue
ok 69 orchestrator: sync-issues prepends a separator newline so the fleet-status header is never concatenated to prior summary content
ok 70 orchestrator: autocut is a no-op when CANARY_AUTO_CUT is not 'true' (kill-switch off)
...
ok 75 orchestrator: autocut is cross-repo aware — cuts v2.1.1 for auto-rebase from the host main HEAD (#1069)
(exit 0 — all 75 pass)

$ bash .dev-lead/scripts/dev-lead-lint.sh
  [lint] shellcheck on 88 file(s)...
  [lint] validating 4 agent profile(s)...
  [lint] validating .github/CODEOWNERS...
  [lint] all checks passed
(exit 0)

Files Changed

scripts/canary-rollout.sh: guarded num="$(_gh_issue_create "$title" "$body" "canary-blocker" || true)" at line 756 so a failed/denied gh issue create (gh error, or grep no-match under pipefail) can no longer trip set -e on this bare (non-local) assignment and abort the step before the Issues:write on the App? warning and the fleet-status dashboard render.
tests/canary_rollout.bats: new test sync-issues degrades gracefully when Issues:write is denied … — a BLOCKED agent with gh issue create failing (via a GH_CREATE_FAIL flag added to the shared _sync_stub gh stub) must exit 0, log the warning, and still render the fleet-status table to GITHUB_STEP_SUMMARY.

Notes

Scope is the self-fixable code half only. The config half — granting the release-manager GitHub App Issues:write on petry-projects/.github-private (label + issue create) — is an org-admin action that cannot be applied from this repo. Until it is granted, the run now degrades gracefully: warning logged, dashboard renders, run stays green. Once granted, one canary-blocker issue per BLOCKED agent will be filed/auto-closed as designed.
Sibling gh issue edit/reopen/close and _issue_find calls were audited and are already guarded (trailing || true / || echo, and _issue_find ends with || echo ""), so no additional guards were required — matching the issue's "audit siblings" ask.

fix(canary-rollout): sync-issues must not abort under set -e when iss…

Navigation Menu

Repository navigation

canary-rollout sync-issues aborts every scheduled run — unguarded _gh_issue_create under set -e (+ App missing Issues:write) → no blocker issues, no fleet dashboard #1081

Symptom

Root cause (code)

Root cause (config)

Fix

Acceptance

Dev-Lead Implementation Plan

Scope

Root-cause detail

Sibling audit (already safe — no change needed)

Implementation Checklist

Tests to Write

Test Command

Note on the config half of the issue

Dev-Lead: Implementation Complete

Plan Execution

Test Results

Files Changed

Notes

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

canary-rollout sync-issues aborts every scheduled run — unguarded _gh_issue_create under set -e (+ App missing Issues:write) → no blocker issues, no fleet dashboard #1081

Description

Symptom

Root cause (code)

Root cause (config)

Fix

Acceptance

Activity

don-petry commented on Jul 4, 2026

Dev-Lead Implementation Plan

Scope

Root-cause detail

Sibling audit (already safe — no change needed)

Implementation Checklist

Tests to Write

Test Command

Note on the config half of the issue

don-petry commented on Jul 4, 2026

Dev-Lead: Implementation Complete

Plan Execution

Test Results

Files Changed

Notes

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions