Combine #28758 + #28117 + #28147 — Agent Shin auto-close + review-gate label lifecycle

[mateo-berri]

Summary

Combines three open PRs into a single branch targeting litellm_internal_staging:

• #28117 — Agent Shin: LLM-judge + Greptile auto-close (any age, any draft state) + @agent-shin reconsider flow.
• #28147 — Safety fixes on top of feat(triage): Agent Shin — LLM-judge + Greptile auto-close (any age, any draft state) + @agent-shin reconsider flow #28117: reconsider provenance check, 1-day grace period before close, @greptileai post-close hint, SwiftWinds dogfood bypass.
• #28758 — ready for review label lifecycle (review-gate state machine) + the proxy/UI bugfixes that rode along on the demo branch (team-cache refresh, vertex-anthropic model filter, spend-logs route permissions).

How it was assembled

1. New branch off litellm_internal_staging (the base of feat(triage): Agent Shin — LLM-judge + Greptile auto-close (any age, any draft state) + @agent-shin reconsider flow #28117).
2. Merged litellm_fix-agent-shin-reconsider-safety-1af4 (fix(triage): bugbot on #28117 — reconsider safety + 1-day grace + @greptileai post-close + SwiftWinds dogfood #28147) — this brings feat(triage): Agent Shin — LLM-judge + Greptile auto-close (any age, any draft state) + @agent-shin reconsider flow #28117 along since fix(triage): bugbot on #28117 — reconsider safety + 1-day grace + @greptileai post-close + SwiftWinds dogfood #28147 is stacked on it.
3. Merged claude/agent-shin-pr-demo-UhrOx (Agent Shin: "ready for review" label lifecycle + end-to-end demo #28758). Eight files conflicted: Agent Shin: "ready for review" label lifecycle + end-to-end demo #28758 was built on an older snapshot of feat(triage): Agent Shin — LLM-judge + Greptile auto-close (any age, any draft state) + @agent-shin reconsider flow #28117 (its 0ad6836e31 commit literally vendored an old version), so the constants/imports in triage_with_llm.py and close_low_quality_prs.py diverged. Resolved by taking fix(triage): bugbot on #28117 — reconsider safety + 1-day grace + @greptileai post-close + SwiftWinds dogfood #28147's (newer, more complete) versions of the duplicated files and keeping Agent Shin: "ready for review" label lifecycle + end-to-end demo #28758's unique additions (review_gate.yml, test_github_review_gate.py, the proxy/UI changes, issue templates).
4. Manually ported the actual review-gate feature commit (b4d970371e) onto fix(triage): bugbot on #28117 — reconsider safety + 1-day grace + @greptileai post-close + SwiftWinds dogfood #28147's structure since a cherry-pick conflicted: added the review_gate() state machine, the --review-gate / --grace-days / --min-greptile-score CLI flags, new constants (READY_FOR_REVIEW_LABEL, READY/REGRESSED/WITHIN_GRACE markers, AGENT_SHIN_AUTO_CLOSE_MARKER, GREPTILE_BOT_LOGINS, SCORE_PATTERN), and new helpers (add_label, remove_label, extract_greptile_score, parse_iso8601, format_*_comment).

Test plan

• tests/test_litellm/test_github_review_gate.py — 18 tests, all pass
• tests/test_litellm/test_github_triage_with_llm.py — passes
• tests/test_litellm/test_github_close_low_quality_prs.py — passes
• tests/test_litellm/test_github_triage_workflows.py — passes
• python3 .github/scripts/triage_with_llm.py --help shows --review-gate, --grace-days, --min-greptile-score
• uv run black --check clean on edited files

162 GitHub-triage tests pass total.

Reviewer notes

• The three source PRs should stay open until this combined PR merges; closing them prematurely would lose review history.
• .github/workflows/review_gate.yml runs in dry-run mode unless the repo variable AGENT_SHIN_ENABLED is set to true.

🤖 Generated with Claude Code

---

Note

Medium Risk
Automated closing, labeling, and commenting on external PRs/issues can misfire or upset contributors; mitigated by dry-run defaults, grace periods, internal exemptions, and explicit enablement.

Overview
Introduces Agent Shin, an OSS triage stack under .github/scripts/ and matching workflows, gated by repo variable AGENT_SHIN_ENABLED (dry-run until set to true).

Contributor-facing templates now spell out auto-triage rules: bug/feature issue forms require repro, expected vs actual, and proof; the PR template adds problem/expected-actual/QA sections and documents Greptile ≥4/5 plus @agent-shin reconsider.

Core automation: triage_with_llm.py LLM-judges external PRs/issues (linked issue or rubric + QA proof), 24h grace before close, @agent-shin reconsider with bot-only reopen guards and rate limits. close_low_quality_prs.py sweeps open external PRs below Greptile 4/5 (drafts included). review_gate() adds a ready for review label lifecycle (LLM + Greptile, regression handling, auto-close after grace). triage_rollout_heads_up.py posts a one-shot 7-day rollout warning. Shared helpers live in agent_shin_shared.py; _agent_shin_actions.py centralizes dry-run vs real GitHub writes.

Workflows: PR/issue triage, reconsider on comment, review gate (PR events + daily sweep), daily low-quality closer, rollout heads-up on staging merge. Tests cover review gate, Greptile closer, and related triage logic under tests/test_litellm/.

^{Reviewed by Cursor Bugbot for commit dd02b76. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 4 committers have signed the CLA.

✅ mateo-berri
❌ claude
❌ cursoragent
❌ Mateo

---

Mateo seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR combines three branches into a single Agent Shin automation: an LLM-as-judge triage bot that auto-closes low-quality external PRs, a @agent-shin reconsider comment-trigger reopen flow, and a ready for review label state machine (review_gate) that tags passing PRs and removes the tag on regression.

• triage_with_llm.py is the core orchestration script (~1 500 lines) implementing triage(), review_gate(), reconsider logic, grace periods, and all GitHub write helpers; most new workflows invoke it with different flag combinations.
• close_low_quality_prs.py is a separate daily cron script that sweeps all open PRs and closes those where Greptile scored below the threshold, with its own grace-warning flow.
• Five new workflows wire up the triggers (pull_request_target, issue_comment, daily cron) with explicit AGENT_SHIN_ENABLED dry-run gating; 162 mock-only tests are added in tests/test_litellm/.

Confidence Score: 3/5

The workflows contain two issues that can produce real GitHub write operations (PR closures, comments) before the team intends to enable the bot.

In triage_pr_with_llm.yml, OPENAI_API_KEY is always injected into the step regardless of AGENT_SHIN_ENABLED, causing live LLM calls for every inbound external PR the moment the secret is configured. More critically, the IMMEDIATE_CLOSE_LOGINS bypass in triage() ignores the --close flag entirely — the workflow strips --close for pull_request_target events and prints 'No PRs will be closed', but PRs from swiftwinds are still commented on and closed even when the bot is nominally disabled. The review_gate.yml handles the key guard correctly and is otherwise sound. The test suite is thorough and all mock-based.

.github/workflows/triage_pr_with_llm.yml (unconditional API key) and .github/scripts/triage_with_llm.py lines 1383–1406 (IMMEDIATE_CLOSE_LOGINS bypass ignores --close).

Important Files Changed

Filename	Overview
.github/scripts/triage_with_llm.py	New 1545-line Agent Shin orchestration script: LLM-judge triage, reconsider flow, and review_gate() state machine. Contains the IMMEDIATE_CLOSE_LOGINS bypass that ignores --close for swiftwinds, a dead AGENT_SHIN_AUTO_CLOSE_MARKER constant, unfiltered _has_marker() author checks, and a grace-period approach in review_gate() that differs from triage() in ways that close PRs without prior warning on first deployment.
.github/workflows/triage_pr_with_llm.yml	New PR triage workflow using pull_request_target. Exposes OPENAI_API_KEY unconditionally (unlike review_gate.yml which guards the key behind AGENT_SHIN_ENABLED), causing LLM calls even when the bot is supposed to be in dry-run mode.
.github/workflows/review_gate.yml	New review gate workflow (pull_request_target + daily cron). Correctly guards OPENAI_API_KEY behind AGENT_SHIN_ENABLED. DO_CLOSE logic is sound. No significant issues.
.github/scripts/close_low_quality_prs.py	New 645-line daily cron closer. Uses time-since-warning approach for grace period (always warns before closing). IMMEDIATE_CLOSE_LOGINS bypass is in evaluate_pr() but does not bypass dry_run flag at the caller level, so behavior is consistent.
.github/workflows/triage_reconsider.yml	New reconsider workflow triggered by issue_comment. Authorization check (commenter is author or internal collaborator) is correctly gated before any destructive steps.
tests/test_litellm/test_github_review_gate.py	18 new unit tests for review_gate() state machine. All use mocks; no real network calls. Coverage is thorough for labeled/unlabeled/within-grace/past-grace paths.
tests/test_litellm/test_github_triage_with_llm.py	Large test file for triage(), reconsider, grace period, and IMMEDIATE_CLOSE_LOGINS behavior. All mock-based. Explicitly validates swiftwinds bypass as intentional design.
tests/test_litellm/test_github_close_low_quality_prs.py	New mock-only tests for the daily closer. No real network calls. Coverage looks solid for grace period, immediate close, and internal author exemption paths.

_{Reviews (1): Last reviewed commit: "Port review-gate feature from #28758 ont..." | Re-trigger Greptile}

[greptile-apps]

IMMEDIATE_CLOSE_LOGINS bypass fires even when AGENT_SHIN_ENABLED is not set

triage() closes and comments on swiftwinds PRs regardless of whether --close was passed (if not close and not is_immediate: falls through when is_immediate=True). The triage_pr_with_llm.yml workflow always strips --close for pull_request_target events and prints ::notice:: No comments will be posted; no PRs will be closed — but that guarantee is violated for any login in IMMEDIATE_CLOSE_LOGINS. An operator who has not yet set AGENT_SHIN_ENABLED=true and believes the bot is fully inert will still see live PR closures and comment posts for those accounts.

[greptile-apps]

OPENAI_API_KEY exposed unconditionally, burning API budget in dry-run mode

OPENAI_API_KEY is always injected into the step, so the LLM judge is called for every external PR that lacks a linked-issue short-circuit, even when AGENT_SHIN_ENABLED is not set. The sister workflow review_gate.yml guards this correctly with (vars.AGENT_SHIN_ENABLED == 'true' || github.event_name == 'workflow_dispatch') && secrets.OPENAI_API_KEY || '', passing an empty string when the bot is disabled. Without a similar guard here, a high-volume repo in dry-run mode will silently consume LLM quota on every opened/reopened PR.

Suggested change

	OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
	OPENAI_BASE_URL: ${{ vars.OPENAI_BASE_URL }}
	GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	OPENAI_API_KEY: ${{ (vars.AGENT_SHIN_ENABLED == 'true' || github.event_name == 'workflow_dispatch') && secrets.OPENAI_API_KEY || '' }}

[greptile-apps]

AGENT_SHIN_AUTO_CLOSE_MARKER is defined but never used in code logic

The constant AGENT_SHIN_AUTO_CLOSE_MARKER = "I'm **Agent Shin**" is defined with a comment saying "reconsider can recognize the closure as a bot close". However, was_closed_by_agent_shin() uses fetch_last_close_actor() (GitHub close-event actor) to detect bot closures — it never inspects comment text for this marker. The format functions also hardcode the string literal instead of referencing the constant. This creates a false sense that keeping the string in sync matters for correctness, and the constant will diverge silently from the formatter functions over time.

[greptile-apps]

_has_marker does not filter by comment author — user-injected HTML can suppress bot notifications

_has_marker(comments, marker) iterates all comments regardless of who wrote them. A contributor can paste <!-- agent-shin:within-grace --> or <!-- agent-shin:regressed --> in their own PR description or a comment, causing review_gate() to either skip the within-grace notification entirely (so the user misses the courtesy warning before closure) or always post "all clear again" instead of "ready for review" on the first pass. Both effects are benign in practice, but deduplicating on bot-authored comments would make the state machine robust to accidental or deliberate injection.

[greptile-apps]

review_gate() closes PRs older than grace_days without any prior warning on first encounter

Unlike triage(), which always posts a GRACE_COMMENT_MARKER warning on the first low-quality detection and only closes on a subsequent run after GRACE_PERIOD_SECONDS has elapsed, review_gate() closes any PR whose creation age exceeds grace_days (1 day) immediately — regardless of whether a warning was ever posted. On initial workflow deployment, the daily cron sweep will encounter all existing open external PRs that are older than 1 day and close them without any advance notice. close_low_quality_prs.py uses the time-since-warning approach and avoids this problem.

[veria-ai]

PR overview

The PR has worked down several findings, with 3 issues already addressed, but a few low-impact workflow abuse paths remain open. The remaining risks are operational rather than data-compromise risks: external users may be able to trigger unnecessary LLM/API spend, and a reconsider command may reopen items closed by other GitHub Actions because closures share the same bot actor. Overall security posture is improving, but these workflow guards should be tightened before merging.

Open issues (3)

• .github/scripts/triage_with_llm.py:326 — Low: Reconsider can reopen items closed by other GitHub Actions An external issue/PR author can comment @agent-shin reconsider and reo
• .github/workflows/review_gate.yml:20 — Low: Repeated LLM spend from fork PR updates When AGENT_SHIN_ENABLED is true, every synchronize event on a fork PR gets `OPENAI_API_
• .github/workflows/triage_issue_with_llm.yml:58 — Low: LLM spend on automatic dry-run events When AGENT_SHIN_ENABLED is true, automatic issues events receive OPENAI_API_KEY, then l

Fixed/addressed: 3 · PR risk: 5/10

[cursor]

PR triage workflow duplicates review gate LLM calls

Low Severity

Both triage_pr_with_llm.yml and review_gate.yml fire on the same pull_request_target events (opened, reopened). When AGENT_SHIN_ENABLED=true, both workflows call the LLM judge for the same PR on the same event. The triage workflow always strips --close on pull_request_target, making its automatic trigger purely observational — its entire function is already performed by the review gate. This results in paying for two independent LLM evaluations of every opened/reopened PR while only one produces any action.

Additional Locations (1)

• .github/workflows/review_gate.yml#L23-L25

 

^{Reviewed by Cursor Bugbot for commit 19dfdd4. Configure here.}

[cursor]

Review gate omits IMMEDIATE_CLOSE_LOGINS grace bypass

Low Severity

The review_gate() function's close path uses only PR age (age_days >= grace_days) to decide whether to close, without checking IMMEDIATE_CLOSE_LOGINS. In contrast, triage() explicitly checks login.lower() in IMMEDIATE_CLOSE_LOGINS to bypass grace entirely. The PR documentation states these logins "bypass BOTH the grace period AND the dry-run gating," but the review gate still subjects them to the full age-based grace window. A dogfood account opening a fresh PR would receive a "within-grace" notice from the review gate instead of being closed immediately.

Additional Locations (1)

• .github/scripts/triage_with_llm.py#L1331-L1334

 

^{Reviewed by Cursor Bugbot for commit 19dfdd4. Configure here.}

[cursor]

Unused constant misleadingly documented for reconsider detection

Low Severity

AGENT_SHIN_AUTO_CLOSE_MARKER is documented as enabling reconsider to "recognize the closure as a bot close," but was_closed_by_agent_shin() actually uses GitHub event-actor identity checking (not comment-body matching) for that purpose. The constant is never used for operational detection anywhere — only in a test assertion to pin comment text. The misleading documentation could cause a future developer to rely on comment-body matching that doesn't exist.

 

^{Reviewed by Cursor Bugbot for commit 19dfdd4. Configure here.}

[veria-ai]

Low: Reconsider can reopen items closed by other GitHub Actions

An external issue/PR author can comment @agent-shin reconsider and reopen an item closed by another workflow, such as stale or duplicate handling, because closures performed with GITHUB_TOKEN all have the same github-actions[bot] actor. Stamp Agent Shin close comments with a dedicated hidden marker and require the latest close to be accompanied by that marker before allowing the reopen path.

[cursor]

Heads-up script misidentifies regressed PRs as closable

Low Severity

_would_be_closed returns True for any PR result where passing is False, including the regressed-already-notified and would-remove-label actions from review_gate. Those PRs are explicitly promised to stay open (the regression comment says "The PR stays open"), yet the heads-up script would post a warning claiming they'll be auto-closed. The function conflates "not currently passing" with "will be auto-closed."

Additional Locations (1)

• .github/scripts/triage_with_llm.py#L1106-L1108

 

^{Reviewed by Cursor Bugbot for commit 09720a7. Configure here.}

[veria-ai]

Low: Repeated LLM spend from fork PR updates

When AGENT_SHIN_ENABLED is true, every synchronize event on a fork PR gets OPENAI_API_KEY and runs the review gate, so an external PR author can burn LLM/API budget by repeatedly pushing commits. Consider removing synchronize here and relying on the scheduled/manual runs for re-checks, or add a per-PR cooldown before exposing the LLM key.

Suggested change

	# against fork PRs. Fork code is never checked out — only PR metadata is read
	types: [opened, reopened, ready_for_review]

[veria-ai]

Low: LLM spend on automatic dry-run events

When AGENT_SHIN_ENABLED is true, automatic issues events receive OPENAI_API_KEY, then lines 84-96 force dry-run but still execute the Python script; that script calls the LLM whenever the key is set. An external GitHub user can open or reopen issues to trigger paid LLM calls without any bot action. The same pattern exists in .github/workflows/triage_pr_with_llm.yml; keep the key empty for automatic dry-run triggers, or only call the LLM on workflows that can actually act.

Suggested change

	OPENAI_API_KEY: ${{ (vars.AGENT_SHIN_ENABLED == 'true' || github.event_name == 'workflow_dispatch') && secrets.OPENAI_API_KEY || '' }}
	OPENAI_API_KEY: ${{ github.event_name == 'workflow_dispatch' && secrets.OPENAI_API_KEY || '' }}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

There are 5 total unresolved issues (including 4 from previous reviews).

^{Reviewed by Cursor Bugbot for commit dd02b76. Configure here.}

[cursor]

Reconsider reopens PRs the review gate immediately re-closes

Medium Severity

The triage function's linked-issue short-circuit in reconsider mode (has_linked_issue → pass → reopen) never checks the Greptile confidence score, while review_gate requires both a rubric pass and Greptile ≥ 4/5. When the review gate closes a PR because Greptile scored it below the threshold (even though it has a linked issue), the reconsider flow reopens it based solely on the linked issue. The reopened event then re-triggers the review gate, which immediately closes the PR again — producing contradictory comments and a frustrating reopen-close loop bounded only by the 10-minute rate limiter.

Additional Locations (1)

• .github/scripts/triage_with_llm.py#L1104-L1110

 

^{Reviewed by Cursor Bugbot for commit dd02b76. Configure here.}

[mateo-berri]

Superseded by #30433 — recreated on the litellm_-prefixed branch litellm_agent_shin_review_gate with a clearer title and description. Same content, same base (litellm_internal_staging).