Skip to content

chore(pr-pipeline): watch pr-verify verdicts lost to permission denials #8432

Description

@kyle-sexton

Summary

Since #8424, a pr-verify run whose model hits any permission denial writes no verdict and fails (owner's choice, option A: fail closed, retryable). The merge lane then refuses the head as verify:missing until a new push or a pr-require-checks rerun judges it again. This issue tracks whether that stalls lane merges in practice.

Signal

pr-verify runs on main that fail with ::error::no verdict: the run hit N permission denial(s) or the run's legs hit N permission denial(s), and lane PRs refused verify:missing behind them.

gh run list -R melodic-software/claude-code-plugins --workflow pr-verify.yml --branch main --status failure -L 50

Decide when

  • Denial failures recur on benign reads (the model reaching for a path outside pr-verify-data): tighten the prompt, or have the lane rerun pr-require-checks on the stalled head automatically.
  • A denial traces to injected PR text: the refusal worked; keep it.

Options considered and rejected for now (README "Verifier verdict"): a fail verdict with the reason, which verify:earlier-fail turns into a block on every later head of the PR; and ignoring denials, the old behavior, which lets an injected read still be judged.

Related

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 19, 2026
  2. github-actions commented on Oct 19, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Small
    Complexity: Low
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Decision comment: #8432 (comment)

    Decision recorded. The owner answered in the Refinement session (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    I should only ever be in the loop ONLY when necessary (intent/vision changed, something came up during implementation that impacts trajectory/alignment, OR a high security risk) - I've covered this all before

    [quote truncated; the full decision is at the link]

    This settles the open questions: the watch moves to a row in .github/recurring-schedule.json, with the 2-in-30-days threshold and the automatic pr-require-checks rerun tried first. The remaining work is that one row, as the acceptance criteria describe.

    Automated triage, applied by the lanes App.

  3. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 19, 2026
  4. removed
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 25, 2026
  5. kyle-sexton commented on Oct 25, 2026

    @kyle-sexton
    MemberAuthor

    Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.

    To hold this issue again, add needs-human (or status: needs-decision) and say why in a comment.

  6. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    priority: mediumReal value, no hard deadline; normal backlog flow.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 25, 2026
  7. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    status: needs-decisionAwaiting a human or maintainer judgment call.
    and removed
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    on Oct 25, 2026
  8. melodic-automation-lanes commented on Oct 25, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed - [ ] criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  9. melodic-automation-lanes commented on Oct 25, 2026

    @melodic-automation-lanes
    Contributor

    The latest triage verdict holds nothing: the issue is at status: ready.

  10. melodic-automation-lanes commented on Oct 25, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: Record the failed pr-verify denial runs on main, then either tighten the prompt or add an automatic pr-require-checks rerun if denials are benign, or close it if they trace to injected text.

    Dependencies

    • #​8424 (merged): the fail-closed denial behavior this watches

    Related work

    Open questions

    1. Should this watch stay an open issue, or move to a maintenance schedule row as #​9701 did for other outside triggers?
      RECOMMENDED: Move it to a schedule row that runs the gh run list query and opens a concrete fix issue only when denials on benign reads recur. Basis: 9701
    2. What threshold of recurrence counts as stalling lane merges, and which fix is chosen first: prompt tightening or automatic pr-require-checks rerun on a verify:missing head?
      RECOMMENDED: Act when 2 or more lane PRs are refused verify:missing from benign-read denials in 30 days; try the automatic rerun first because it does not weaken the injection refusal. Basis: judgment
    3. Can an unattended run inspect the run history, given that deciding needs gh run list output on main that the refinement run cannot fetch?
      RECOMMENDED: No; have a person or a scheduled job with gh access list the failed runs, then reopen this as a concrete task. Basis: .base/.github/scripts/pr-verify/write-verdict.mjs:141

    Sources read

    • .base/.github/scripts/pr-verify/write-verdict.mjs:141: throws 'the run hit N permission denial(s)' so no verdict is written
    • .base/.github/scripts/pr-verify/write-verdict.mjs:153: throws 'the run's legs hit N permission denial(s)' for the multi-leg case
    • .base/docs/conventions/pr-pipeline/README.md:1436: a run with any permission denial yields no verdict
    • .base/docs/conventions/pr-pipeline/CHANGELOG.md:1337: a permission denial in either pr-verify job yields no verdict

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

  11. kyle-sexton commented on Oct 26, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. The owner answered in the Refinement session (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    I should only ever be in the loop ONLY when necessary (intent/vision changed, something came up during implementation that impacts trajectory/alignment, OR a high security risk) - I've covered this all before

    Question asked: Move this watch to a recurring-schedule row and close the issue, acting first with the automatic rerun when benign denials recur?

    1. Does this watch stay an open issue, or move to a schedule row as chore(maintenance): watch outside triggers with schedule rows instead of open issues #9701 did for other outside triggers?
      Decided: Move it to a row in .github/recurring-schedule.json that runs the body's gh run list query and opens a fix issue only when benign-read denials recur. Basis: commit e58dfa3 (chore(maintenance): watch outside triggers with schedule rows instead of open issues #9701) added .github/recurring-schedule.json to "watch outside triggers with schedule rows instead of open issues"; kept, basis made concrete.
    2. What recurrence counts as stalling, and which fix comes first?
      Decided: The threshold is an owner choice; the refinement proposed 2 lane PRs refused verify:missing from benign-read denials in 30 days. Try the automatic pr-require-checks rerun first, since it leaves the injection refusal intact. Basis: body "Decide when" names both fixes and keeps the refusal for injected text; judgment for the number; kept.
    3. Can an unattended run inspect the run history?
      Decided: Not this lane run. The schedule row (item 1) or a person lists the failed runs. Basis: .github/scripts/pr-verify/write-verdict.mjs:141 and :153 throw the two messages the body's signal greps for; kept.

    Acceptance criteria:

    • .github/recurring-schedule.json has a row that names the body's signal (::error::no verdict with the permission-denial messages on pr-verify runs on main) and the two outcomes in its "Decide when" section.

    Checks a person runs:

    • Deciding whether a recorded denial was a benign read or injected PR text.
  12. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    admission-deferredReady for the issue lane; work-unblock admits it when queued work is under the queued cap
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    and removed
    admission-deferredReady for the issue lane; work-unblock admits it when queued work is under the queued cap
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions