Repository navigation
Canary Rollout times out: 6 of the last 7 scheduled runs hit timeout-minutes: 30 and were cancelled — rings stop advancing silently #1175
Description
Summary
The scheduled Canary Rollout workflow (.github/workflows/canary-rollout.yml, cron 33 */4 * * *) has timeout-minutes: 30 on its job. It has timed out on 6 of its last 7 scheduled runs, which GitHub reports as cancelled. So autocut, promote-all and sync-issues mostly do not complete, and rings stop advancing on their own.
Evidence (run history, 2026-09-22 → 2026-09-25)
| Run | Started | Duration | Result |
|---|---|---|---|
| 36072336644 | 09-24 23:21 | 29 min | success |
| 36052410582 | 09-24 20:04 | 38 min | cancelled |
| 36007658811 | 09-24 13:43 | 35 min | cancelled |
| 35959082706 | 09-24 05:14 | 32 min | cancelled |
| 35931746171 | 09-23 23:04 | 35 min | cancelled |
| 35911508450 | 09-23 19:46 | 35 min | cancelled |
| 35869650430 | 09-23 13:48 | 35 min | cancelled |
| 35820845942 | 09-23 05:03 | 22 min | success |
| earlier (09-22) | 12–24 min | success |
Runtime crept from roughly 12–24 min to 29–38 min around 2026-09-23. The one recent success finished at 29 min, just under the cap.
Acceptance criteria
- Find where the time goes, and fix the growth rather than just raising the cap. The successful run's log shows a full fetch of every branch and tag (
fetch-depth: 0+fetch-tags: true), and hundreds ofdev-lead/issue-*branches are listed. Per-agent evaluate calls may also have grown. - Keep a safety timeout, sized with headroom over measured p95.
- A timed-out run surfaces as a failure, or files a blocker issue, not a silent
cancelled. Today nothing tells an operator the promoter has not completed a pass in ~28h.
Activity