[#811 re-split 2/2] canary-rollout: fail-closed sync-issues (never drop regression tracking on partial data) #820

Activity

1. don-petry

   added

   dev-leadFor dev-lead agent pickupFor dev-lead agent pickup

   on Jul 20, 2026
2. don-petry commented on Jul 20, 2026

   

   don-petry

   on Jul 20, 2026

   ContributorAuthor

   More actions

   Dev-Lead Implementation Plan

   Issue: #820 — [#811 re-split 2/2] canary-rollout: fail-closed sync-issues (never drop regression tracking on partial data)

   Scope

   Make scripts/canary-rollout.sh sync-issues resilient to a run-history fetch failure: instead of aborting to empty state (which silently drops — or worse, auto-closes — a regression's tracked issue), fail closed to a BLOCKED tracked issue with the data gap annotated, and treat a total inability to determine state as a hard error rather than a green no-op.

   Root cause

   cmd_sync_issues computes each agent's state via _frontier_state, which reads run history through _run_json. On a sustained fetch failure _run_json returns non-zero (fail-closed, #738); under set -euo pipefail that aborts _frontier_state to empty output. The unguarded read then either aborts the whole step or leaves state="", so the agent falls into the "not BLOCKED" branch — closing/skipping its blocker issue. The safety net is defeated by exactly the failures it should record.

   Implementation Checklist

   • Add _frontier_state_resilient <agent> wrapper (sync-issues-only) in scripts/canary-rollout.sh: on a fetch-driven data gap, reconstruct candidate/frontier/transition/reusable-differs→triage from tag+blob resolution (no run history), fail closed to BLOCKED, and emit a trailing datagap flag. Return non-zero ONLY when even candidate/frontier is unresolvable (total inability → hard error).
   • Rewire cmd_sync_issues to read from the wrapper (guarded), track a hard_fail flag, annotate the data gap in the blocker title, and return 1 on total inability (still rendering the fleet dashboard first).
   • Thread a data_gap arg into _blocker_body: banner note + "run-history unavailable" cumulative-failures cell + gap-aware evidence.

   Tests to Write

   • sync-issues under a forced partial fetch (gh run list fails every retry, tag/blob resolution intact) with a changed reusable still OPENS a canary-blocker + needs-human issue, triage REGRESSION, body annotates the gap — no silent skip, no auto-close.
   • sync-issues total inability (all gh unresolvable) fails closed: non-zero exit + ::error::, no green no-op.
   • Existing canary_rollout.bats suite stays green.

   Test Command

   npx bats tests/canary_rollout.bats
3. don-petry commented on Jul 20, 2026

   

   don-petry

   on Jul 20, 2026

   ContributorAuthor

   More actions

   Dev-Lead: Implementation Complete

   Plan Execution

   • Added _frontier_state_resilient wrapper — fails closed to a BLOCKED+datagap line on a run-history outage; returns non-zero only on a total inability to resolve candidate/frontier.
   • Rewired cmd_sync_issues to read from the wrapper (guarded), track hard_fail, annotate the gap in the blocker title, and return 1 (after rendering the dashboard) on total inability.
   • Threaded a data_gap arg into _blocker_body (PARTIAL DATA banner + "run history unavailable" cumulative-failures cell + gap-aware evidence).
   • Wired the fetch-failure signal in _run_json (file flag) — no-op for every non-sync-issues read path.

   Root cause (corrected during TDD)

   A sustained gh run list failure does not cleanly abort _frontier_state — command substitutions don't inherit errexit, so it limped on and emitted a misleadingly clean state line. sync-issues then read that agent as "not BLOCKED" and skipped/auto-closed its regression issue. The fix detects the fetch failure via a file flag _run_json writes, then discards the bogus state and fails closed.

   Behaviour now

   • Partial fetch (run history down, tags/blobs OK) + changed reusable → opens/updates a canary-blocker issue, triage REGRESSION, labels dev-lead + needs-human, body carries a PARTIAL DATA banner. No silent skip, no auto-close.
   • Byte-identical reusable under a gap → still tracked as BLOCKED (PRE_EXISTING, report-only) — promotion held until data is readable.
   • Total inability (candidate/frontier unresolvable) → ::error:: + non-zero exit, never a green no-op. Dashboard still renders first.

   Test Results

   $ npx bats tests/canary_rollout.bats
   244 passing, 0 failures    (239 pre-existing + 5 new #820 cases)
   
   $ bash .dev-lead/scripts/dev-lead-lint.sh
   [lint] shellcheck on 27 file(s)... OK
   [lint] all checks passed
   

   New tests (tests/canary_rollout.bats):

   • partial-fetch → REGRESSION needs-human blocker opened, gap annotated
   • partial-fetch UPDATE keeps an existing blocker open (no auto-close)
   • wrapper passes a normal state line through with datagap=0
   • wrapper fails closed (non-zero) on total inability
   • sync-issues ends non-zero when an agent is totally undeterminable

   Files Changed

   • scripts/canary-rollout.sh: _run_json fetch-failure flag; new _frontier_state_resilient; cmd_sync_issues resilient read + fail-closed exit + gap annotation; _blocker_body data_gap arg.
   • tests/canary_rollout.bats: _sync_gap_stub + 5 resilience cases.

   Notes

   • Scope kept to sync-issues per the issue (independent of the feat: implement issue #817 — Clean up upsert output before fleet apply: conditional negation tail + migrated-comment dedup #818 fetch refactor). The REGRESSION note's "a run failed" wording is superseded by the PARTIAL-DATA banner, which states counts are unreliable and triage rests on the reusable diff alone.
   • No commits/PRs opened — left to the CI workflow.
4. don-petry

   added a commit that references this issue on Jul 20, 2026

   feat: implement issue #820 — {"message": "No server is currently avai…

   ...

   65a4f97
5. don-petry

   closed this as completedin #821on Jul 20, 2026
6. don-petry

   added a commit that references this issue on Jul 21, 2026

   fix(canary): cache run-list per (repo,workflow) + not-found→empty (#819…

   ...

   ff5923b
7. don-petry

   added a commit that references this issue on Aug 7, 2026

   feat: implement issue #820 — {"message": "No server is currently avai…

   ...

   9830497
8. don-petry

   added a commit that references this issue on Sep 7, 2026

   feat: implement issue #820 — {"message": "No server is currently avai…

   ...

   956c0c4