Skip to content

feat(pr-pipeline): request a whole review when a lane PR is held only by findings an incremental review cannot clear - #8580

Merged
kyle-sexton merged 8 commits into
mainfrom
feat/8565-request-full-review
Oct 20, 2026
Merged

kyle-sexton merged 8 commits into
mainfrom
feat/8565-request-full-review

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Closes #8565

Summary

A lane PR whose earlier findings were fixed by later pushes can stay held: the reviews of the new head are incremental, and an incremental review cannot clear a finding raised on an older head. Tonight that held #7940, #8056 and #8138 until someone merged them by hand. This lets pr-merge ask for one whole review of the head, which can close those findings.

Fix

pr-merge toggles the lane PR draft → ready once per head, which fires ready_for_review and starts the whole reviews (the review reusables have no label, comment or dispatch trigger, and the merge gate counts only pull_request review runs).

When it toggles, all of these must hold:

  1. Every other gate passes, and the only risk codes are findings:claude-review / security-finding; no review thread is unresolved.
  2. No review of the current head reported a finding (the carried findings come from older heads), and no whole review is already recorded on that head.
  3. At most one toggle per pr-merge sweep, and only when the repository has fewer than 35 running jobs and both the lanes App and GITHUB_TOKEN buckets have at least 1,500 calls left.
  4. The head is recorded in the <!-- babysit-merge-risk --> comment ("whole review requested at head "), carried through every rewrite including escalation, so a head is toggled once.

Recovery: each sweep re-readies a lane draft only when the lane identity made the newest conversion to draft. A draft a person made is never re-readied.

Each toggle logs pr-merge whole-review-toggle pr=<n> head=<sha>.

Files: babysit_whole_review.py (new: condition, limits, toggle, recover), babysit_merge.py, babysit_risk.py (LaneEvidence.whole_review_clears), babysit_review_evidence.py, .github/workflows/pr-merge.yml (listing adds id and drafts.json, recovery step, spent env, notice line), guard_contract.py and its generated doc, README, changelog fragments.

Cost (measured from the workflow files)

ready_for_review starts, for a lane PR:

Workflow Jobs
pr-require-checks.yml 11 (two matrices)
pr-test-windows.yml 8
pr-review-hosted.yml 2, plus the reusable's
pr-review-security-hosted.yml 2, plus the reusable's
pr-merge.yml 4
pr-refine.yml 1, skipped when the sender is the lanes App

About 25-35 jobs per toggle; pr-verify runs again after the reviews. Tonight's cases would be about 3 a night. API: none on a sweep with nothing eligible; on an eligible PR, 2-4 GraphQL calls on the lane token, and one runs list plus one jobs read per in-progress run on the job token (refuses outright at 35 or more runs). Recovery: 2 reads per lane draft updated in the last 24 hours.

Verification

  • babysit-prs suites under python3 -m unittest discover: 2177 pass (pytest is not installed locally). The 16 evidence and integration tests and the 4 recovery tests failed before their code; the 21 module tests were written after the module.
  • ruff (pinned wrapper), actionlint, merge-job.test.mjs, changelog fragment checks pass.
  • Design agreed by the pr-merge owner (poteto), cost limits by GitHub Support; reviewed by poteto with a fresh-context reviewer. Needs GitHub Security & Cost's pass on the final head (pr-merge triggering reviews that can close findings is a merge-gate change).

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xk7Lz7d2twxvA3LfRxnLkT

kyle-sexton and others added 4 commits October 20, 2026 05:01
… by findings an incremental review cannot clear

A clean incremental review never closes a finding carried on the review
chain, so fixed lane PRs sat at risk:MED until a person merged them.

The lane evidence now says whether a clean whole review of the head would
close every open finding (no review of the head reported a finding or was
whole, no lane thread is unresolved). When every other gate passes and those
findings are the only risk reasons, the gate records a request in the
reasons comment, then converts the lane PR to a draft and marks it ready
with the lane token, which runs both review lanes whole. The record is
carried on every rewrite at that head, so a head is toggled once; a run
toggles at most one PR, only while fewer than 35 jobs run and both tokens
have 1,500 calls left. A draft a run left between the two writes is marked
ready by the next run. Each toggle logs a stable notice line.

Closes #8565

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xk7Lz7d2twxvA3LfRxnLkT
Recovery re-readied any recent lane draft whose newest lane comment held a
whole-review record at its current head. A person who converted the pull
request to a draft to hold it after the toggle would have been undone, a
later lane comment of another kind hid the record, and a head that moved
mid-toggle stranded the draft.

Recovery now reads the newest reasons comment (the babysit-merge-risk
marker) by the lane identity, accepts a record at any head, and marks the
draft ready only when the lane identity made its newest conversion to
draft. The thread count no longer keeps a second counter for unresolved
threads, and a test pins the 1,500-call minimum as allowed.

Refs: #8565

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xk7Lz7d2twxvA3LfRxnLkT
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T09:16:32.736626Z 0a15a30 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 8b35e3f4f99479847d0bfa38f47d88b780980b50: 0 finding(s), 0 as inline comments.

No security findings in the incremental change (7d44472 to 8b35e3f).

The change makes _short_bucket in babysit_whole_review.py refuse when a rate-limit bucket reports a limit of zero or less. Before, a zero limit gave left < 0, which is false, so it passed. This tightens a fail-closed check and opens no new input or trust path. The new test, test_a_zero_limit_refuses, covers it. Nothing here touches secrets, injection surfaces or instruction surfaces.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0a15a3095b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/source-control/skills/babysit-prs/scripts/babysit_whole_review.py Outdated
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 8b35e3f4f99479847d0bfa38f47d88b780980b50: 0 finding(s), 0 as inline comments.

Incremental review of 8b35e3f (changes since 7d44472): no findings.

  • babysit_whole_review.py _short_bucket: a rate-limit bucket with limit <= 0 now refuses. Before, 0 < 0.3 * 0 was false, so a zero-limit bucket passed as if it had budget. The refusal string keeps the bucket:left/limit shape. The only other consumer is the rate-limit:{name}:{short} formatter at line 181, which is unaffected.
  • test_a_zero_limit_refuses sets the job bucket to 0/0 while the lane bucket stays at 5000/5000, so the lane check passes and the job check refuses. It asserts rate-limit:job:core:0/0. The old condition would have returned None, so the test fails without the fix.

I read the diff and the surrounding code but did not run the tests. The closing issue (#8565) states no acceptance criteria this delta touches. Nothing in the PR context file instructed the lane to do anything.

… recovery to a toggle

A lane could resolve its own threads, push a trivial commit and buy a new
whole review at every head, because only the current head's record was kept.
The reasons comment now keeps every record line, through clears and
escalation, and a pull request gets at most two automatic whole reviews;
past that it waits for a person.

Each record line carries the time the gate wrote it, before the toggle.
Recovery marks a draft ready only when the lane identity made its newest
conversion to draft, by the server's createdAt, within ten minutes after one
of those records, so a lane draft made on purpose later stays a draft while
a head that moved mid-toggle still recovers.

The running-jobs limit also counts the jobs of queued runs.

Refs: #8565

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xk7Lz7d2twxvA3LfRxnLkT
kyle-sexton and others added 3 commits October 20, 2026 05:32
…are of each bucket

GITHUB_TOKEN has 1,000 REST calls an hour outside GitHub Enterprise Cloud,
so a fixed 1,500-call reserve could never pass there and every eligible
request was refused. Each bucket (core and GraphQL) of the lane token and
the job token now needs 30% of its reported limit left: 1,500 of 5,000,
300 of 1,000.

Refs: #8565

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xk7Lz7d2twxvA3LfRxnLkT
A bucket reporting a limit of zero (or less) passed the 30%-of-limit check,
since nothing is below 30% of zero. Treat it as no budget and refuse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xk7Lz7d2twxvA3LfRxnLkT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(pr-pipeline): request a whole review when a lane PR is held only by findings an incremental review cannot clear

1 participant