Skip to content

fix(babysit-prs): merge gate is blind to an in-flight current-head reviewΒ #1629

Description

@kyle-sexton

πŸ€– Agent-authored β€” filed by the adversarial adjudication of #1614, from a code finding that neither the issue nor its synthesis had reached.

The merge gate can pass while the configured review bot is mid-review of the current head. Nothing in the gate asks whether the reviewer has reviewed this head, and the module that would answer cannot be configured for a reviewer that posts a review instead of a commit status.

The gap, in three parts

1. The merge gate never receives the reviewer configuration. babysit_review_bot_logins reaches exactly two consumers:

  • plugins/source-control/skills/babysit-prs/scripts/pr_queue_snapshot.py:82
  • plugins/source-control/skills/babysit-prs/scripts/request_review.py:55

babysit_merge.py is not one of them. It fetches reviews, review threads, and the branch rules β€” including required_review_thread_resolution β€” but it has no notion of a configured reviewer, so it cannot ask the one question that matters here: has that reviewer reviewed the head I am about to merge?

2. The reviewer-awareness module is all-or-nothing and requires a commit-status context. babysit_review_trigger.py:51-53:

@property
def configured(self) -> bool:
    return bool(self.trigger_phrase and self.reviewer_logins and self.gate_context)

and review_gate_state() (same file, line 243) derives every signal β€” gate_state, workflow_state, request_signal_pending β€” by matching gate_context against StatusContext / CheckRun entries in the check rollup. Its own docstring: "With no configured contexts nothing matches, so every summarized state degrades to absent … the dormant default."

3. This reviewer publishes no such context. chatgpt-codex-connector posts a pull-request review. It creates no commit status and no check run, so there is no gate_context value that could be configured. The module is therefore not merely unconfigured here β€” it is unconfigurable for this reviewer, and reviewer_logins alone cannot activate it because configured demands all three slots.

Net effect: for a reviewer of this shape, no pending-engagement blocker can ever fire, and the merge gate is independently blind regardless.

Observed consequence β€” PR #1594

Codex reviewed five successive heads. Latency from commit to posted review, every round:

Head Committed Reviewed Latency
dddce901 18:43:30Z 18:48:44Z 5m14s
81e5369d 19:56:53Z 20:01:47Z 4m54s
372b9872 21:24:27Z 21:28:29Z 4m02s
3ec45192 21:34:46Z 21:39:04Z 4m18s
edd9cbfb 21:44:00Z 21:49:06Z 5m06s

Mean 4m43s, never below 4m02s. The PR merged at 21:48:40Z β€” 4m40s after its final commit, and 26 seconds before round 5 landed. Round 5 carried two valid findings, one of them a regression the PR itself introduced (both filed as #1613). The gate had no way to know a review was in flight, so it reported ready and the merge proceeded.

This was not a near-miss at the edge of a wide distribution. 4m40s sits below the mean of every prior observation on the same PR.

Note also that round 5 posted after head_ref_deleted. No push-side discipline can prevent a trailing round; only declining to merge inside the reviewer's latency window can.

Why configuration cannot close this

Setting babysit_review_bot_logins alone leaves ReviewTriggerConfig.configured false, so the module stays dormant. There is no gate_context to set. And even a fully configured module would only raise a snapshot blocker β€” babysit_merge.py would still merge on its own verdict, because it never sees the keys.

Proposed shape

A hold in the merge gate that does not depend on a status context: refuse ready when both hold β€”

  • no submitted review or inline review comment whose own commit id equals the current head exists from any login in the configured reviewer set (the same current-head test review-trigger.md already specifies for completion: "a submitted review or inline review comment whose own commit ID equals the current head SHA and whose author has the authoritative GitHub Bot type plus an exact login from <review-bot-logins>"); and
  • the current head is younger than a configured settle window.

Two conventions the design should follow rather than re-derive:

  • Fail open when unconfigured. No reviewer set configured means no hold and no behavior change β€” mirroring how gate_context already degrades to absent. This keeps every repo that has not wired a reviewer exactly as it is today.
  • Bounded, not indefinite. The hold expires with the settle window; a reviewer that never engages must not wedge the PR. review-trigger.md already takes this position for the trigger path: "If the reviewer does not engage after that one request, report it rather than retrying."

The change is strictly in the safe direction β€” it can only withhold a merge, never permit one the gate would otherwise refuse.

Whoever picks this up should decide the settle-window default against fresh measurements rather than inheriting the five samples above, and should confirm the fail-open-when-unconfigured reading before implementing. If either of those turns out to be a real policy call rather than a convention lookup, bounce this to status: needs-decision instead of guessing.

Acceptance criteria

  • babysit_merge.py receives the configured reviewer set and holds ready per the rule above, with the settle window configurable through the existing effective-configuration seam.
  • With no reviewer configured, gate behavior is byte-for-byte what it is today β€” covered by a test.
  • A head with a current-head review from a configured reviewer merges immediately, with no settle wait.
  • The hold expires with the window rather than persisting until a review arrives.
  • reference/safety.md's "Two Gates, One Merge-Ready Authority" reflects the new policy blocker, since that section enumerates what the merge gate adds beyond GitHub's own mergeability.

Context

Surfaced while adjudicating #1614, which attributed the #1594 merge to an unbounded review-round loop. The round count was not the defect β€” every one of the 15 findings across five rounds was valid and distinct. The defect is that the merge fired inside a well-characterized reviewer latency window with no mechanism able to see it. Related: #1594 (the run), #1613 (the findings that arrived 26 seconds late).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.priority: highSignificant impact, or blocks an imminent release; staff this cycle.status: readyTriaged, unblocked, and fully specified; eligible to pick up.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions