Skip to content

Canary Rollout times out: 6 of the last 7 scheduled runs hit timeout-minutes: 30 and were cancelled — rings stop advancing silently #1175

Description

@rachel-petry

Summary

The scheduled Canary Rollout workflow (.github/workflows/canary-rollout.yml, cron 33 */4 * * *) has timeout-minutes: 30 on its job. It has timed out on 6 of its last 7 scheduled runs, which GitHub reports as cancelled. So autocut, promote-all and sync-issues mostly do not complete, and rings stop advancing on their own.

Evidence (run history, 2026-09-22 → 2026-09-25)

Run Started Duration Result
36072336644 09-24 23:21 29 min success
36052410582 09-24 20:04 38 min cancelled
36007658811 09-24 13:43 35 min cancelled
35959082706 09-24 05:14 32 min cancelled
35931746171 09-23 23:04 35 min cancelled
35911508450 09-23 19:46 35 min cancelled
35869650430 09-23 13:48 35 min cancelled
35820845942 09-23 05:03 22 min success
earlier (09-22) 12–24 min success

Runtime crept from roughly 12–24 min to 29–38 min around 2026-09-23. The one recent success finished at 29 min, just under the cap.

Acceptance criteria

  1. Find where the time goes, and fix the growth rather than just raising the cap. The successful run's log shows a full fetch of every branch and tag (fetch-depth: 0 + fetch-tags: true), and hundreds of dev-lead/issue-* branches are listed. Per-agent evaluate calls may also have grown.
  2. Keep a safety timeout, sized with headroom over measured p95.
  3. A timed-out run surfaces as a failure, or files a blocker issue, not a silent cancelled. Today nothing tells an operator the promoter has not completed a pass in ~28h.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions