[automated] This tracks automatic recovery for failed GitHub main CI runs. Measurement, evaluation duration, and health-report changes will be managed separately outside this issue.
Is there an existing issue for this?
Is your feature request related to a problem? Please describe the problem.
Pull request CI automatically reruns failed jobs up to three times, but failed GitHub push runs on main do not have the same recovery path. A transient test, runner, network, container, or service failure can therefore leave main red until another commit happens to trigger a successful run.
CI runs use one concurrency group per ref and do not cancel an in-progress main run. A retry must be suppressed when its SHA has already been superseded; otherwise a stale retry can consume the pending slot or delay validation of newer code.
Describe the solution you'd like
Give failed GitHub main CI runs the same bounded retry cadence as pull requests:
- Rerun failed jobs unconditionally, up to three reruns/four total attempts.
- Immediately before requesting a rerun, verify that the source run SHA is still the current
main SHA, no newer main CI run supersedes it, the source attempt has not advanced, and the attempt remains within the cap.
- Reuse the existing failed-job rerun infrastructure rather than introduce a second retry implementation.
- Preserve the existing
main concurrency behavior. Reruns retain the original GITHUB_REF, so they remain in the CI-refs/heads/main concurrency group; a newer build waits behind an already-started retry.
- Keep the event-driven
Analyze CI Failure workflow and memory/ci-failure-analysis branch as the durable failure/cause history. Report rerun decisions and skip reasons in the retry workflow's logs and summary.
- Continue creating or updating failure-cause issues. Do not assign or delegate remediation work automatically.
Additional context
This work should be tracked under #18036. Durable retry-decision history for external reporting is tracked separately in #20409.
Out of scope:
- Azure DevOps/internal official-build retries;
- CI Shepherd lifecycle or assignment behavior;
- automatic Copilot delegation or PR shepherding;
- public CI-health report changes;
- a new metrics storage service or dashboard.
Acceptance criteria:
[automated] This tracks automatic recovery for failed GitHub
mainCI runs. Measurement, evaluation duration, and health-report changes will be managed separately outside this issue.Is there an existing issue for this?
Is your feature request related to a problem? Please describe the problem.
Pull request CI automatically reruns failed jobs up to three times, but failed GitHub
pushruns onmaindo not have the same recovery path. A transient test, runner, network, container, or service failure can therefore leavemainred until another commit happens to trigger a successful run.CIruns use one concurrency group per ref and do not cancel an in-progressmainrun. A retry must be suppressed when its SHA has already been superseded; otherwise a stale retry can consume the pending slot or delay validation of newer code.Describe the solution you'd like
Give failed GitHub
mainCI runs the same bounded retry cadence as pull requests:mainSHA, no newermainCI run supersedes it, the source attempt has not advanced, and the attempt remains within the cap.mainconcurrency behavior. Reruns retain the originalGITHUB_REF, so they remain in theCI-refs/heads/mainconcurrency group; a newer build waits behind an already-started retry.Analyze CI Failureworkflow andmemory/ci-failure-analysisbranch as the durable failure/cause history. Report rerun decisions and skip reasons in the retry workflow's logs and summary.Additional context
This work should be tracked under #18036. Durable retry-decision history for external reporting is tracked separately in #20409.
Out of scope:
Acceptance criteria:
mainfailed jobs are rerun automatically up to the configured attempt cap.mainconcurrency behavior is preserved and covered by focused tests.