[Fleet Monitor] petry-projects/.github — .github/workflows/canary-rollout.yml #747

Activity

1. don-petry

   added

   health-check

   dev-leadFor dev-lead agent pickupFor dev-lead agent pickup

   fleet-tracker

   on Jul 16, 2026
2. don-petry commented on Jul 16, 2026

   

   don-petry

   on Jul 16, 2026

   ContributorAuthor

   More actions

   Dev-Lead Implementation Plan

   Issue: #747 — [Fleet Monitor] petry-projects/.github — .github/workflows/canary-rollout.yml

   Root Cause

   The scheduled canary-rollout.yml run (promote-all / evaluate-all) fans _run_json
   out across every agent × tier repo. A tier repo that has never run a given agent's
   workflow makes gh run list --workflow <name> exit non-zero with
   could not find any workflows named <name>. _run_json (added in #738) suppresses
   stderr and treats every non-zero gh exit as a transient blip, so it exhausts its
   retries and fails CLOSED, killing the whole scheduled fleet sweep. This is the 100%
   failure (6/6 runs) — confirmed in run 29476152481, where .github-private (a tier repo
   that runs only 2 of the agents' workflows) triggers the hard error.

   Scope

   Teach _run_json to distinguish "this tier repo has never run this workflow" (a legitimate
   empty result → [], no retry) from a genuine transient/outage failure (retry, then fail
   CLOSED). The fail-closed protection for real outages stays intact.

   Implementation Checklist

   • scripts/canary-rollout.sh — in _run_json, capture gh stderr; if it matches
     could not find any workflows named, return [] immediately (no retry, no fail-closed)

   Tests to Write

   • _run_json returns [] (status 0) when the workflow is not present in a tier repo
   • _run_json returns immediately without retrying on workflow-not-present
   • (regression) existing fails CLOSED test still passes for a genuine outage

   Test Command

   bats tests/canary_rollout.bats (locally: ./node_modules/.bin/bats tests/canary_rollout.bats)
3. don-petry commented on Jul 16, 2026

   

   don-petry

   on Jul 16, 2026

   ContributorAuthor

   More actions

   Dev-Lead: Implementation Complete

   Root Cause

   The scheduled canary-rollout.yml run (promote-all / evaluate-all) fans _run_json
   across every agent × tier repo. When a tier repo has never run a given agent's
   workflow, gh run list --workflow <name> exits non-zero with
   could not find any workflows named <name>. _run_json (from #738) suppressed stderr and
   treated every non-zero exit as a transient blip, so it exhausted its retries and failed
   CLOSED, killing the whole scheduled sweep. Confirmed in run 29476152481: .github-private
   is a tier repo that runs only 2 of the fleet's agent workflows, so every other agent's gate
   read against it hard-failed → exit 1 → 100% failure (6/6).

   Fix

   _run_json now captures gh stderr and, when it matches could not find any workflows named, returns [] immediately (no retry, no fail-closed) — a tier repo with zero runs of
   a workflow legitimately contributes an empty result. Genuine transient/outage failures still
   retry with backoff and then fail CLOSED, so the #738 protection against green-lighting a bad
   promotion is preserved.

   Plan Execution

   • Root-caused the 100% failure from the live run logs (29476152481)
   • Added two failing TDD tests for the workflow-not-present path ([Fleet Monitor] petry-projects/.github — .github/workflows/canary-rollout.yml #747)
   • Implemented the stderr-discriminating fix in scripts/canary-rollout.sh _run_json
   • Full bats suite green; shellcheck clean; dev-lead-lint clean

   Test Results

   226 tests, 0 failures  (bats tests/canary_rollout.bats — 224 pre-existing + 2 new)
   
   ok 2 _run_json: fails CLOSED (non-zero) when gh fails on every attempt (#738)
   ok 3 _run_json: workflow-not-present in a tier repo returns [] (no fail-closed) (#747)
   ok 4 _run_json: workflow-not-present returns immediately without retrying (#747)
   ...
   ok 226 orchestrator: autocut knob override forces MAJOR over a patch-only diff (#712)
   
   shellcheck --severity=warning -x scripts/canary-rollout.sh scripts/lib/canary-rollout.sh → OK
   .dev-lead/scripts/dev-lead-lint.sh → all checks passed
   

   Files Changed

   • scripts/canary-rollout.sh: _run_json discriminates "workflow not present in this tier
     repo" (→ []) from a genuine fetch failure (→ retry, then fail CLOSED).
   • tests/canary_rollout.bats: two new tests covering the workflow-not-present path and the
     no-retry behavior; the existing fails CLOSED test still guards genuine outages.

   Notes

   The fix is safe against masking real failures: [] is returned only when gh explicitly
   reports the workflow does not exist in that repo (a true zero-run state), never on a
   network/5xx/permission error — those still retry and fail CLOSED. No behavior change for
   repos that do run the workflow.
4. don-petry commented on Jul 23, 2026

   

   don-petry

   on Jul 23, 2026

   ContributorAuthor

   More actions

   ✅ Auto-resolved by Fleet Monitor. This workflow's failure rate has been below the 10% threshold (or the workflow was removed) since 2026-07-20 (3 days) — the monitor stopped flagging it. Closing. It will reopen automatically if the workflow crosses the threshold again.
5. don-petry

   closed this as completedon Jul 23, 2026
6. don-petry

   added 2 commits that reference this issue on Aug 3, 2026

   feat: implement issue #747 — [Fleet Monitor] petry-projects/.github —…

   ...

   238654c

   feat: implement issue #747 — [Fleet Monitor] petry-projects/.github —…

   ...

   4ec5d9b