Skip to content

Non-required failing check silently deadlocks every merge: template-drift fails on all PRs, review agent skips as ci-failing, no approval can land #1795

Closed

Description

@don-petry

Story

As a maintainer whose PRs are all green on every gate that actually blocks merge,
I want a non-required failing check to stop silently withholding the review agent's approval,
so that an advisory signal cannot convert itself into an unbreakable merge block.

The deadlock, observed live 2026-09-13

Every open PR in this repo is currently unmergeable, and no gate that the ruleset enforces is red.

  1. template-drift fails on every PR. scripts/template_stub_drift.sh compares blobs committed in
    petry-projects/repo-template against this repo's standards-derived baseline:

    ::error::Template stub .github/workflows/copilot-setup-steps.yml has DRIFTED (9c8f93e7146c != 77c99a427679)
    ::error::Template stub .github/workflows/sonarcloud.yml         has DRIFTED (687bf511b01f != d7aa265ffdfe)
    ::error::Template stub .github/dependabot.yml                   has DRIFTED (23d611e8f038 != 3e2daee1f7f8)
    ::warning::Template stub .github/workflows/agent-ingress.yml is MISSING from petry-projects/repo-template
    

    repo-template has not been touched since 2026-08-31 (363190b2). The template did not drift —
    the baseline moved underneath it, and nothing re-seeded it. Same cross-repo shape as the ADR-0006
    schema change on 2026-09-07, where an org-repo commit instantly red-lit every open PR here.

  2. review-one-pr.sh then refuses to review. Its CI gate is all-or-nothing:

    skip: CI checks are failing — will re-evaluate after fixes are pushed
    {"decision":"skip","reason":"ci-failing"}
    
  3. No approval is ever produced, and required_approving_review_count: 1 means the PR cannot merge.

  4. Nobody sees it on main. lint.yml is pull_request-triggered, so template-drift never runs
    on main. Main is permanently green while every PR is blocked — the failure is invisible exactly where
    people look for repo health.

Why this is a defect and not the gate working as intended

template-drift is not a required status check. The main ruleset requires only SonarCloud,
CodeQL, agent-shield / AgentShield, dependency-audit / Detect ecosystems, and
duplicate-decl-gate. GitHub would merge these PRs today if an approval existed.

So the block is created entirely by the review agent treating a non-blocking check as blocking. An
advisory check has escalated itself into a hard merge gate, repo-wide, with no signal saying so.
review-one-pr.sh already reasons in exactly these terms for SKIPPED and NEUTRAL conclusions
("NEUTRAL covers informational checks that don't gate merging") — it just has no notion of
non-required as a third such category.

Acceptance Criteria

  1. The immediate outage is cleared: petry-projects/repo-template is re-seeded via
    scripts/seed-repo-template.sh so template-drift passes, including the missing
    .github/workflows/agent-ingress.yml. This is cross-repo; sequence it as its own step and confirm a
    PR here goes green before closing.
  2. The CI gate distinguishes required from non-required failures. A failing check that the branch
    ruleset does not require must not by itself produce reason: ci-failing. Derive the required set
    from the ruleset/branch-protection API — do not hardcode a list of check names, which would drift
    the same way the template just did.
  3. The gate is not weakened for anything that does block. A failing required check still skips the
    review, unchanged. This changes which failures count, never what the gate does when one counts. The
    FORCE_REVIEW break-glass (Self-host deadlock: force_review bypasses the advisory gate but not the CI gate, so CI-gate fixes can't merge through ring-0 #619) stays exactly as-is.
  4. A non-required failure is reported, not ignored. When the agent proceeds past one, it names the
    check in its review body, so "we reviewed despite a red advisory check" is visible rather than
    silent. Downgrading a signal must not delete it.
  5. The invisibility is fixed too. template-drift (or an equivalent scheduled check) runs somewhere
    that is not pull_request-only, so a standards/template divergence is caught against main instead
    of surfacing as an unexplained block on every contributor's PR.
  6. Covered by tests: a PR whose only failing check is non-required is reviewed; a PR with a failing
    required check is still skipped; the required set is read from the API, not a literal. Offline,
    stubbed — no network.

Dev Notes

References


Evidence added 2026-09-13 ~10:10Z: the model judges "CI green" too, so AC2 alone won't clear it

After stale cancelled checks were cleared on several PRs (09:14–10:04Z), pr-review ran; the code gate was not the blocker there, since FORCE_REVIEW=true on the sweep bypassed it. The deep-review model then escalated on template-drift itself:

The source is the prompts, not only review-one-pr.sh:

  • prompts/deep-review.md:108, deep-review-logic.md:63, -performance.md:63, -security.md:77, -style.md:64: "Check statusCheckRollup for CI status."
  • prompts/deep-review.md:197, -logic.md:136, -performance.md:125, -style.md:124: approve only if "all gates pass (CI green, issue addressed, …)"

statusCheckRollup includes non-required checks. Suggested AC addition: the harness computes the ruleset-required set once (AC2's API read) and passes it into the prompt context. Every deep-review prompt defines "CI green" as required checks passing, and lists non-required failures as informational to name in the review body (AC4), never as a reason to escalate. Test: a non-required-only failure yields approve-eligible, not escalate.

Knock-on cost, observed: dev-lead can't fix a check the PR doesn't cause, so it answers no-changes. Each round burns a review cycle, and PRs hit the 3/3 escalation cap in about 20 minutes (#1788, #1783, #1782).

⚠ AC conflict with #1791 — needs a decision before release

These can't both hold. Decide first whether template-drift should expect agent-ingress.yml in repo-template at all during the ADR-0007 migration (#1723). Then align the other issue's AC before either is released to dev-lead.


Delivery brief — 2026-09-19 (read this first; implement it, do not re-ask)

Scope for this PR is AC #2, #3, #4 and #5 only. AC #1 (re-seed repo-template) is explicitly OUT OF SCOPE — do not run scripts/seed-repo-template.sh. #1812 is still open: the seeder destroys files it does not own (it wiped repo-template's 420-line managed .gitignore and overwrote the drift-allowlisted ci.yml). Running it would trade this outage for a worse one. Leave AC #1 to #1812 and say so in the PR description.

The deadlock is still live, re-measured 2026-09-19

Four PRs (#1783, #1790, #1814, #1815) have every required check green — SonarCloud, CodeQL, agent-shield / AgentShield, dependency-audit / Detect ecosystems, duplicate-decl-gate — and template-drift red. A forced review run (35441769067) produced exactly this:

skip: CI checks are failing — will re-evaluate after fixes are pushed
{"pr":".../pull/1783","decision":"skip","reason":"ci-failing", ...}
Summary: 0 reviews posted, 1 no-ops skipped, 0 failures (1 deferred pending CI/checks)

template-drift is not in the ruleset's required set. A purely advisory check is holding every merge in the repo.

What to implement

  1. compute_ci_status (scripts/review-one-pr.sh, called at line ~176) must ignore failures of checks the branch ruleset does not require. A red non-required check alone must not yield ci-failing.
  2. Read the required set from the API — never hardcode check names. This repo uses a ruleset, so GET /repos/{owner}/{repo}/rules/branches/{branch} is the source; /branches/{branch}/protection returns 404 here. Today that set is exactly the five named above. Hardcoding would drift the same way the template baseline just did — that is the bug, not the fix.
  3. Fail closed on an unreadable required set. If the ruleset API call fails or returns nothing, treat every failing check as blocking (today's behaviour). An API hiccup must never silently widen what gets approved — same reasoning as the router's unreadable-label guard.
  4. Do not weaken anything that genuinely blocks. A failing required check still skips with ci-failing. A failing non-required check must still be visible: log it and name it in the review body, so "ignored" never means "unmentioned".
  5. Keep ci-pending behaviour as it is. The bounded deferral from [Phase 4] pr-review defers indefinitely on another agent's own check run (dev-lead/dispatch) and on zombie checks — ci-pending has no timeout #1427 is separate; do not fold it into this change.

Tests (bats, in the existing suite)

  • a red non-required check with all required green → reviews (no ci-failing skip)
  • a red required check → still skips with ci-failing
  • ruleset API unreadable → fails closed, skips
  • the non-required failure is named in the output

Budget guard

  1. The gate change plus its tests. This is the commit that matters — a green partial PR beats a timed-out run.
  2. Logging/surfacing polish and the PR-description note that AC test issue from agent #1 stays with seed-repo-template.sh destroys files it does not own: wipes the 420-line managed secrets .gitignore baseline and overwrites the drift-allowlisted ci.yml #1812.

Stop after step 1 if budget runs low and list the rest as a follow-up.

Activity

  1. github-actions commented on Sep 13, 2026

    @github-actions
    Contributor

    Thanks for the thorough write-up — the deadlock mechanism is crystal clear and the affected PRs make the urgency plain. One quick question before we dig into the review-one-pr.sh changes: for the immediate unblock (AC #1), do you have a push path to petry-projects/repo-template already open, or does re-seeding via seed-repo-template.sh itself require a PR and review there — and if so, is that repo also currently blocked by this same deadlock?

  2. don-petry commented on Sep 13, 2026

    @don-petry
    CollaboratorAuthor

    Correction and escalation: there is no automated route past this, by design

    Two findings since filing. The first corrects something I implied in the issue body.

    1. force_review is NOT the break-glass for this gate — and must not become one

    I dispatched pr-review-trigger.yml with force_review=true on #1788 expecting it to clear the
    ci-failing gate. It did not, and it never could. pr-review.yml splits the two flags deliberately
    (#1590, whose comment says "do NOT re-merge them"):

    FORCE_REVIEW:    ${{ github.event_name == 'repository_dispatch' && 'true' || 'false' }}
    FORCE_RE_REVIEW: ${{ inputs.force_review || 'false' }}
    • FORCE_REVIEW — the human @mention break-glass. Bypasses every gate. Reachable only through
      repository_dispatch (the mention-listener path).
    • FORCE_RE_REVIEW — what the force_review input actually sets. Clears only the same-SHA
      idempotency no-op, explicitly so that "an automated re-review can never silently convert into a gate
      bypass."

    That design is correct and this issue should not weaken it. But it means the practical situation is
    worse than the body describes: with template-drift red on every PR, no automated path can produce
    an approval in this repo at all.
    The only routes are a human @mention (which bypasses every gate,
    including ones that should hold) or a human approval. AC #2 is therefore the whole fix, not a
    nice-to-have — FORCE_REVIEW is not a workaround anyone can lean on.

    2. The gate also trips on CANCELLED, which this repo generates constantly

    Current non-passing checks on #1788:

    template-drift      FAILURE     (non-required; the cross-repo drift above)
    dev-lead / resume   CANCELLED
    dev-lead / ci-relay CANCELLED
    dev-lead / dispatch CANCELLED
    

    review-one-pr.sh classifies CANCELLED as failing. Superseded dev-lead orchestration jobs are
    cancelled constantly here — that is #1741's subject ("42% of dev-lead runs are cancelled before doing
    any work"). So even with template-drift fixed, a PR can still be held by the cancelled remnants of
    its own superseded orchestration runs.

    The skip message hints the script already has a notion of this — it says "the failing
    (non-superseded) required checks turn green"
    — so the superseded concept exists somewhere in the
    decision path but is not preventing this skip. Worth checking whether that logic is reachable, or
    whether the word is only in the message.

    Consequence for AC #2

    The required-vs-non-required distinction needs to cover conclusion kind as well as check identity:
    a CANCELLED superseded orchestration job is not evidence that the PR is broken. Both belong in the
    same fix, or this reopens the first time a dev-lead run is cancelled on a PR whose checks are otherwise
    green.

    Blocked PRs, unchanged: #1783, #1788, #1790 — all MERGEABLE/BLOCKED, all green on every
    required check, all verified against their acceptance criteria.

  3. added
    dev-leadFor dev-lead agent pickup
    and removed
    needs-human-reviewFlagged by automated PR review agent
    on Sep 19, 2026
  4. don-petry commented on Sep 19, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: implementation deferred — automation PR queue is full

    No pull request was opened for issue #1795. The org-wide
    open-automation-PR queue is currently at its signed-off cap (see
    standards/pr-limits.json in petry-projects/.github).

    This issue stays labeled dev-lead and will be re-attempted automatically once
    the queue drains — no action is required.

  5. added
    dev-leadFor dev-lead agent pickup
    and removed
    dev-leadFor dev-lead agent pickup
    on Sep 19, 2026
  6. don-petry commented on Sep 19, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1795 — Non-required failing check silently deadlocks every merge

    Scope (per 2026-09-19 delivery brief: AC #2, #3, #4, #5 only)

    Make the pr-review CI gate derive the required check set from the branch ruleset API (not a hardcoded list, not the unreliable .isRequired rollup field alone) so a red non-required check (e.g. template-drift) no longer produces ci-failing; fail closed when the ruleset is unreadable; name non-required failures instead of silently ignoring them. AC #1 (re-seed repo-template) is OUT OF SCOPE — left to #1812; seed-repo-template.sh will not be run.

    Implementation Checklist

    Tests to Write

    • tests/test_ci_status.bats: red non-required + all required green (ruleset set supplied) → passing.
    • tests/test_ci_status.bats: red required (name in ruleset set) → failing.
    • tests/test_ci_status.bats: ci_nonrequired_failures names the non-required red check.
    • tests/test_review_one_pr_*: ruleset API stubbed returns 5 names, template-drift red → reviews; ruleset API unreadable → fails closed (skips ci-failing); non-required failure named in output.

    Test Command

    scripts/run-bats.sh tests/test_ci_status.bats tests/test_review_one_pr_nonrequired_gate.bats and bash .dev-lead/scripts/dev-lead-lint.sh

  7. don-petry commented on Sep 19, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    Plan Execution (intended)

    Scope decisions (please confirm)

    Local check output (provisional — not the pushed run)

    shellcheck (review-one-pr.sh, ci-status.sh): clean
    .dev-lead/scripts/dev-lead-lint.sh: all checks passed
    tests/test_ci_status.bats + tests/test_review_one_pr_nonrequired_gate.bats + ci-pending/force-ci-failing: green
    

    Files touched (this workspace, not yet pushed)

    • scripts/lib/ci-status.sh: compute_ci_status gains an optional ruleset-required-name arg (union
      gating); new ci_nonrequired_failures surfacing helper.
    • scripts/review-one-pr.sh: ruleset_required_checks API read (fail-closed), passed into both
      compute_ci_status calls; names non-required failures in the log + review prompt.
    • tests/test_ci_status.bats, tests/test_review_one_pr_nonrequired_gate.bats: coverage.
    • .github/workflows/lint.yml: register the new bats file.

    Notes

    Name-matching against the ruleset context strings is best-effort; the .isRequired union means a
    mismatch can only ever make the gate stricter, never weaker. GitHub's ruleset still enforces required
    checks at merge regardless of this review decision.

  8. don-petry commented on Sep 19, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #1847

    The implementation for issue #1795 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  9. added a commit that references this issue on Sep 19, 2026
  10. added 2 commits that reference this issue on Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions