From 458958025e30b57c4e06ffd6831344ff8bb54936 Mon Sep 17 00:00:00 2001 From: Claude Code Bot Date: Tue, 14 Jul 2026 15:40:41 -0500 Subject: [PATCH 1/4] docs(release): SC2 game-day runbook (#503) + tie regression test to SC2 Now that #1230 landed the FORCE_REVIEW ci-failing break-glass on main, document the SC2 invariant (a broken 'next' can't block its own fix), the two mechanisms that guarantee it (channel decoupling + the break-glass), the operational drill, and the automated guard (test_review_one_pr_force_ci_failing.bats). Closes the automatable half of #503; the human-run game-day drill is documented for a post-merge run. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Juznz5V6su81ffSND8fg7s --- docs/release/sc2-game-day.md | 66 +++++++++++++++++++ .../test_review_one_pr_force_ci_failing.bats | 2 + 2 files changed, 68 insertions(+) create mode 100644 docs/release/sc2-game-day.md diff --git a/docs/release/sc2-game-day.md b/docs/release/sc2-game-day.md new file mode 100644 index 000000000..1bc1ce355 --- /dev/null +++ b/docs/release/sc2-game-day.md @@ -0,0 +1,66 @@ +# SC2 game-day — prove a broken `next` can't block its own fix + +**Invariant (Safe Release SC2, epic #495 · story #503):** a broken in-development +version of an agentic workflow **cannot block the PR that fixes its own breakage.** +This is the circular-dependency failure the whole Safe Release strategy exists to +eliminate (`dev-lead`/`pr-review` build, review, and merge changes *to themselves*). + +## Why the invariant holds + +Two independent mechanisms guarantee it; either alone is sufficient, and together +they are belt-and-suspenders: + +1. **Channel decoupling (structural).** New versions are exercised on `next`/`ring0` + while production self-review/dev **duty stays pinned to `stable`** + (`versioning.md`). A break in `next` therefore does not, by construction, take + out the gate that reviews the fix — the fix PR is still evaluated by the last + known-good `stable`. + +2. **Break-glass (operational, #619).** For the residual case where a break *does* + reach the gate that must review its own fix — e.g. a bug in `compute_ci_status` + that marks the fix PR "failing" — a manual `FORCE_REVIEW` run (`@mention` / + `workflow_dispatch`) bypasses **every** soft gate, including the `ci-failing` + gate: + - advisory-bot gate, `CHANGES_REQUESTED`, idempotency, and the review cap + (pre-existing), **and** + - the `ci-failing` gate (`scripts/review-one-pr.sh`, shipped in #1230). + + `FORCE_REVIEW` is **manual-only** (the scheduled sweep dispatches + `force_review=false`), and GitHub's ruleset still blocks the merge on any + failing **required** check, so the break-glass only unblocks the + codeowner-approval gate — it cannot merge a genuinely-broken PR. + +## Automated regression guard + +`tests/test_review_one_pr_force_ci_failing.bats` codifies mechanism (2): + +- a **normal** run on failing CI still skips (`reason=ci-failing`, exit 100) — the + gate is not weakened for day-to-day operation; +- a **`FORCE_REVIEW=true`** run on failing CI **bypasses** the gate — the fix can + be reviewed and approved. + +If a future change re-couples the gate to its own fix, that test fails. + +## Running the game-day (operational drill) + +Run this after any material change to the CI gate (`compute_ci_status`, +`review-one-pr.sh` skip logic) or the channel-pinning layout. + +1. **Break `next` deliberately.** On a scratch branch, introduce a change to the + `next`-channel review gate that marks all PRs as `ci-failing` (e.g. force + `compute_ci_status` to return `failing`). Promote it to `pr-review/next` only + (`cut-release.sh --channel pr-review/next …`), never `stable`. +2. **Open the fix PR** that reverts/repairs the break, targeting `main`. +3. **Confirm the block reproduces.** The unattended cascade skips the fix PR with + `reason=ci-failing` (the broken gate is reviewing its own fix). +4. **Apply the break-glass.** `@mention` the review bot (or `workflow_dispatch` + the trigger with `force_review=true`) on the fix PR. Expect the + `::warning:: force-review … bypassing the ci-failing gate (#619)` line and an + approval from the reviewer identity. +5. **Merge** (required checks green at the ruleset), which restores `next`. +6. **Restore** `pr-review/next` to the repaired version and record the drill + outcome on #503. + +**Pass criteria:** the fix PR is reviewed and approved via the break-glass within +one manual action, with no admin ruleset bypass and no manual file surgery — SC2 +and SC4 (one-action) both satisfied. diff --git a/tests/test_review_one_pr_force_ci_failing.bats b/tests/test_review_one_pr_force_ci_failing.bats index c0520824e..83429d2d4 100644 --- a/tests/test_review_one_pr_force_ci_failing.bats +++ b/tests/test_review_one_pr_force_ci_failing.bats @@ -1,4 +1,6 @@ #!/usr/bin/env bats +# SC2 regression guard (Safe Release epic #495 · game-day story #503, see +# docs/release/sc2-game-day.md): a broken in-dev gate must not block its own fix. # Guards the #619 break-glass in scripts/review-one-pr.sh: # a manual FORCE_REVIEW=true run overrides the ci-failing gate (so a fix to the # CI gate itself can be reviewed through ring-0 self-host), while a normal run From 604fd909ab771756a839645a837c2bd95af6490f Mon Sep 17 00:00:00 2001 From: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com> Date: Tue, 14 Jul 2026 20:45:07 +0000 Subject: [PATCH 2/4] fix(reviews): address review comments [skip ci-relay] --- docs/release/sc2-game-day.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release/sc2-game-day.md b/docs/release/sc2-game-day.md index 1bc1ce355..6fffc67c9 100644 --- a/docs/release/sc2-game-day.md +++ b/docs/release/sc2-game-day.md @@ -55,7 +55,7 @@ Run this after any material change to the CI gate (`compute_ci_status`, `reason=ci-failing` (the broken gate is reviewing its own fix). 4. **Apply the break-glass.** `@mention` the review bot (or `workflow_dispatch` the trigger with `force_review=true`) on the fix PR. Expect the - `::warning:: force-review … bypassing the ci-failing gate (#619)` line and an + `::warning::force-review: … bypassing the ci-failing gate (break-glass, #619)` line and an approval from the reviewer identity. 5. **Merge** (required checks green at the ruleset), which restores `next`. 6. **Restore** `pr-review/next` to the repaired version and record the drill From e70511fb8b2699ef896ce106931b846eb38cd001 Mon Sep 17 00:00:00 2001 From: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com> Date: Tue, 14 Jul 2026 20:49:43 +0000 Subject: [PATCH 3/4] chore: dev-lead update (review-changes) [skip ci-relay] --- docs/release/sc2-game-day.md | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/docs/release/sc2-game-day.md b/docs/release/sc2-game-day.md index 6fffc67c9..a50b05a5d 100644 --- a/docs/release/sc2-game-day.md +++ b/docs/release/sc2-game-day.md @@ -7,14 +7,18 @@ eliminate (`dev-lead`/`pr-review` build, review, and merge changes *to themselve ## Why the invariant holds -Two independent mechanisms guarantee it; either alone is sufficient, and together -they are belt-and-suspenders: +Two mechanisms together guarantee it. For downstream consumer repos pinned to +`stable`, either alone is sufficient; for this repo (ring-0 canary, pinned to +`next`), mechanism (2) is the operative SC2 safety net: 1. **Channel decoupling (structural).** New versions are exercised on `next`/`ring0` - while production self-review/dev **duty stays pinned to `stable`** - (`versioning.md`). A break in `next` therefore does not, by construction, take - out the gate that reviews the fix — the fix PR is still evaluated by the last - known-good `stable`. + while downstream consumer repos keep their self-review/dev duty pinned to + `stable` ([versioning.md](./versioning.md)). A break in `next` does not, by + construction, take out a `stable`-pinned gate — the fix PR is still evaluated by + the last known-good `stable`. **This repo** is the ring-0 canary + (`pr-review-trigger.yml` and `dev-lead.yml` both pin to `next`), so channel + decoupling alone does not provide the SC2 guarantee here — mechanism (2) is + required. 2. **Break-glass (operational, #619).** For the residual case where a break *does* reach the gate that must review its own fix — e.g. a bug in `compute_ci_status` @@ -49,16 +53,18 @@ Run this after any material change to the CI gate (`compute_ci_status`, 1. **Break `next` deliberately.** On a scratch branch, introduce a change to the `next`-channel review gate that marks all PRs as `ci-failing` (e.g. force `compute_ci_status` to return `failing`). Promote it to `pr-review/next` only - (`cut-release.sh --channel pr-review/next …`), never `stable`. + (`cut-release.sh pr-review --channel next --push`), never `stable`. 2. **Open the fix PR** that reverts/repairs the break, targeting `main`. 3. **Confirm the block reproduces.** The unattended cascade skips the fix PR with `reason=ci-failing` (the broken gate is reviewing its own fix). 4. **Apply the break-glass.** `@mention` the review bot (or `workflow_dispatch` - the trigger with `force_review=true`) on the fix PR. Expect the + the trigger with `force_review=true` **and the fix PR's `pr_url`**) on the fix + PR. Expect the `::warning::force-review: … bypassing the ci-failing gate (break-glass, #619)` line and an approval from the reviewer identity. -5. **Merge** (required checks green at the ruleset), which restores `next`. -6. **Restore** `pr-review/next` to the repaired version and record the drill +5. **Merge** (required checks green at the ruleset); the fix lands on `main`. +6. **Advance `pr-review/next`** to the repaired release: + `cut-release.sh pr-review --channel next --push`. Record the drill outcome on #503. **Pass criteria:** the fix PR is reviewed and approved via the break-glass within From 633dd56f560142767cda032bcb4991330b5f9b1b Mon Sep 17 00:00:00 2001 From: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com> Date: Tue, 14 Jul 2026 21:20:02 +0000 Subject: [PATCH 4/4] chore: dev-lead update (review-changes) [skip ci-relay] --- docs/release/sc2-game-day.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/release/sc2-game-day.md b/docs/release/sc2-game-day.md index a50b05a5d..62d6c469f 100644 --- a/docs/release/sc2-game-day.md +++ b/docs/release/sc2-game-day.md @@ -23,12 +23,19 @@ Two mechanisms together guarantee it. For downstream consumer repos pinned to 2. **Break-glass (operational, #619).** For the residual case where a break *does* reach the gate that must review its own fix — e.g. a bug in `compute_ci_status` that marks the fix PR "failing" — a manual `FORCE_REVIEW` run (`@mention` / - `workflow_dispatch`) bypasses **every** soft gate, including the `ci-failing` - gate: + `workflow_dispatch`) bypasses the following soft gates: - advisory-bot gate, `CHANGES_REQUESTED`, idempotency, and the review cap (pre-existing), **and** - the `ci-failing` gate (`scripts/review-one-pr.sh`, shipped in #1230). + **Exception — per-PR automation budget (`enforce_pr_budget`).** `FORCE_REVIEW` + does **not** bypass the `enforce_pr_budget` gate; a FORCE_REVIEW mention that + finds the budget exhausted exits with + `"reason":"automation-budget-exhausted"` before reaching the review cascade. + The budget resets only on a human interaction (comment, push, or approval). + If the fix PR has already exhausted its budget, a human must interact with the + PR first to reset the budget before the break-glass can proceed. + `FORCE_REVIEW` is **manual-only** (the scheduled sweep dispatches `force_review=false`), and GitHub's ruleset still blocks the merge on any failing **required** check, so the break-glass only unblocks the @@ -40,8 +47,13 @@ Two mechanisms together guarantee it. For downstream consumer repos pinned to - a **normal** run on failing CI still skips (`reason=ci-failing`, exit 100) — the gate is not weakened for day-to-day operation; -- a **`FORCE_REVIEW=true`** run on failing CI **bypasses** the gate — the fix can - be reviewed and approved. +- a **`FORCE_REVIEW=true`** run on failing CI **bypasses** the `ci-failing` gate + (exit is not 100; the run is not skipped with `reason=ci-failing`). + + Note: the test asserts the gate is not tripped (exit ≠ 100 and the bypass + warning appears). It does not verify that a review is subsequently submitted + and approved — a downstream engine or posting failure could still prevent + approval even with a passing test. If a future change re-couples the gate to its own fix, that test fails.