fix(triage): bugbot on #28117 — reconsider safety + 1-day grace + @greptileai post-close + SwiftWinds dogfood

[mateo-berri]

Fix-on-top for #28117. Combines two layers of changes:

1. Reconsider safety (original scope) — addresses three concerns Greptile + veria-ai flagged on the @agent-shin reconsider flow.
2. 1-day grace period before close + SwiftWinds dogfood (new scope) — the user explicitly asked for both in chat:

   "I want to give contributors a 1 day grace (specify in the comment) to fix their pr before it closes. They can still @ the bot the request another look (and thus a possible re-opening). It should state in the message all this and that also even after the pr is closed @'ing greptileai works fine. Also, turn on full (non dry run) mode with SwiftWinds. Thats my personal account I want to test this PR with for this week. It has no push permissions to litellm so its a perfect example."
   "For SwiftWinds, just close immediately. Faster iteration that way."

---

Part 1 — Reconsider safety (Greptile + veria-ai feedback on #28117)

Greptile review on #28117 (Confidence Score 3/5): "The reconsider mode in triage_with_llm.py ignores --close and makes real write calls unconditionally; an operator running --reconsider locally without the workflow AGENT_SHIN_ENABLED guard would post comments and reopen PRs. The reconsider workflow also has no check that the closure came from the bot, so it can reopen items closed by maintainers for non-quality reasons."

veria-ai review on #28117 (Risk 5/10): "This PR adds GitHub automation that can close and reopen issues/PRs from a write-token workflow. The reconsider path authorizes the original external author but does not verify that the item was previously closed by Agent Shin, so a contributor can make the bot reopen a PR a maintainer closed for another reason."

1a. Reconsider had no dry-run path (Greptile)

Before: --reconsider ignored --close and always posted comments + reopened on pass. After: reconsider honors close=False the same way regular triage does. Returns would-reopen / would-reconsider-still-failing (with the previewed comment body) for step-summary rendering. The workflow only passes --close when AGENT_SHIN_ENABLED=true.

1b. Reconsider could reopen maintainer-closed PRs/issues (Greptile + veria-ai security)

Before: triage_reconsider.yml only checked the commenter — it did NOT check that the most recent close was performed by Agent Shin. A contributor could @agent-shin reconsider on a PR a maintainer closed for non-rubric reasons (duplicate, security report, design rejection) and have the bot reopen it. After: was_closed_by_agent_shin() inspects the issue events API for the most recent closed event's actor and only permits reopen when the actor matches the configured bot login (default github-actions[bot], overridable via AGENT_SHIN_BOT_LOGIN). Fail-closed on missing events. Returns skip-not-bot-closed; the LLM never runs.

1c. No rate-limiting on the reconsider trigger (Greptile)

Before: every @agent-shin reconsider comment burned CI minutes + an OpenAI API call. After: a 10-minute cooldown via seconds_since_last_reconsider_verdict(), which detects the bot's own verdict marker <!-- agent-shin:reconsider-verdict -->. Inside the cooldown window, triage returns skip-rate-limited and the LLM never runs.

---

Part 2 — 1-day grace period before close

Behavior

When Agent Shin (LLM judge) or the Greptile-score closer flags an external PR/issue as low-quality, the bot now does NOT close immediately. Instead it posts a grace-period warning comment that explicitly states:

• "You have 1 day to address this before this PR is auto-closed."
• During the grace window: update the description and comment @agent-shin reconsider to skip the auto-close.
• After auto-close: @agent-shin reconsider still re-runs triage and reopens; @greptileai also works even after the PR is closed for a fresh re-review.

The warning comment carries an HTML marker <!-- agent-shin:grace-warning --> that both scripts use to coordinate:

• Both .github/scripts/triage_with_llm.py (LLM judge) and .github/scripts/close_low_quality_prs.py (Greptile-score closer) detect the marker.
• A warning posted by one path is recognized by the other on the next run.
• GRACE_PERIOD_SECONDS = 86400 (24h) — defined in both files; tests pin both.

Flow on a real low-quality PR (24h cadence is the daily Greptile cron)

Day	Closer evaluates	Grace marker	Action
1	Greptile 3/5, no warning yet	absent	warn-grace → post warning, do NOT close
1.5	Greptile 3/5, warning 12h old	< 24h	skip-in-grace-period → no-op
2	Greptile 3/5, warning 25h old	≥ 24h	close → post close comment + close PR
2 (alt)	Greptile bumped to 5/5 after fix	≥ 24h	skip-score-ok → PR stays open

evaluate_pr action surface (close_low_quality_prs.py)

New: warn-grace, skip-in-grace-period. Existing actions (close, skip-too-young, skip-optout-label, skip-internal, skip-no-greptile-score, skip-score-ok) are unchanged.

triage() action surface (triage_with_llm.py)

New: warned-grace, would-warn-grace (dry-run preview), skip-in-grace-period. Existing closed / would-close now only fire after the grace window.

---

Part 3 — SwiftWinds dogfood bypass (IMMEDIATE_CLOSE_LOGINS)

The user explicitly named SwiftWinds as their personal account for testing this PR for the week ahead, and explicitly said "for SwiftWinds, just close immediately. Faster iteration that way."

IMMEDIATE_CLOSE_LOGINS = frozenset({"swiftwinds"}) (case-insensitive match) lives in both scripts and bypasses both:

1. The 1-day grace period — close fires on the first detection.
2. The dry-run / AGENT_SHIN_ENABLED workflow gating — the script writes (post comment + close PR) for these logins regardless of whether the workflow passed --close.

This is intentional: SwiftWinds has no push permissions to litellm, which makes it a clean dogfood account, and waiting 24h per iteration would kill the testing feedback loop.

The bypass is centralized in the Python scripts — workflows don't need conditional logic. Static workflow tests (test_github_triage_workflows.py) still pin the AGENT_SHIN_ENABLED = "true" gate for the global population.

---

Files changed

• .github/scripts/triage_with_llm.py

  • Reconsider safety: was_closed_by_agent_shin(), seconds_since_last_reconsider_verdict(), fetch_last_close_actor(), _iter_paginated_json(); bot-closed + rate-limit guards before the LLM call; honors close=False in reconsider mode; RECONSIDER_COMMENT_MARKER on reopen / still-failing comments.
  • Grace period: GRACE_COMMENT_MARKER, GRACE_PERIOD_SECONDS, IMMEDIATE_CLOSE_LOGINS; seconds_since_last_grace_warning(); format_grace_warning_pr_comment() and format_grace_warning_issue_comment(); triage() now posts a warning on the first failing detection and only closes after the window elapses (or for IMMEDIATE_CLOSE_LOGINS); format_pr_close_comment() updated to mention @greptileai works post-close.
  • Refactor: shared _seconds_since_latest_marker_comment() helper underneath both reconsider and grace helpers so the marker-iteration logic lives in one place.

• .github/scripts/triage_reconsider.yml

  • Passes --close only when AGENT_SHIN_ENABLED=true (matches the other Agent Shin workflows); always runs the script so the dry-run verdict appears in the step summary. (Part of original PR scope.)

• .github/scripts/close_low_quality_prs.py

  • Grace period: GRACE_COMMENT_MARKER, GRACE_PERIOD_SECONDS, IMMEDIATE_CLOSE_LOGINS (mirroring the LLM-judge constants); seconds_since_last_grace_warning() (operates on already-fetched comments, with injectable now for tests); format_grace_warning_comment(); evaluate_pr returns warn-grace / skip-in-grace-period / close based on prior warning age; main() posts the warning via new post_grace_warning() and dispatches per-PR pr_dry_run = dry_run and not is_immediate so SwiftWinds bypasses the global --close gate.
  • The standard close comment now mentions @greptileai works even after the PR is closed.

• tests/test_litellm/test_github_triage_with_llm.py

  • Updates existing close-path tests to use the new _stub_grace_aged_out / _stub_grace_no_warning helpers.
  • New tests (15): TestTriageOrchestration::{test_should_post_grace_warning_on_first_failing_run_in_close_mode, test_should_skip_close_inside_grace_window, test_should_dry_run_grace_warning_when_close_false, test_should_skip_grace_for_swiftwinds_login, test_should_close_swiftwinds_even_when_close_flag_false, test_should_match_immediate_close_login_case_insensitively, test_should_not_treat_random_external_login_as_immediate_close}; TestImmediateCloseLoginsConstant; TestGraceWarningCommentText (5 invariants on the user-facing language: 1-day grace, @agent-shin reconsider, @greptileai, "even after the PR is closed", marker presence); TestSecondsSinceLastGraceWarning (3 cases).

• tests/test_litellm/test_github_close_low_quality_prs.py

  • Updates existing close-path tests: first detection now produces warn-grace, not close.
  • New tests (12): grace flow for drafts / brand-new PRs / no-prior-warning, post-grace close after window expires, skip inside window, SwiftWinds bypass + case-insensitivity, TestSecondsSinceLastGraceWarning, TestImmediateCloseLoginsConstant, TestGraceWarningCommentText (1-day, reconsider, @greptileai, marker, post-close mentions).

Test results

$ uv run pytest tests/test_litellm/test_github_triage_with_llm.py \
                tests/test_litellm/test_github_close_low_quality_prs.py \
                tests/test_litellm/test_github_triage_workflows.py
============================= 144 passed in 2.36s =============================

(Was 112 on the previous revision; +32 new tests for the grace path, the SwiftWinds bypass, the new comment language, and the new helper.)

Out of scope

• The grace window length (GRACE_PERIOD_SECONDS = 86400) is a constant. If the team wants per-repo tuning we can wire it through --grace-seconds in a follow-up.
• IMMEDIATE_CLOSE_LOGINS is hardcoded to {"swiftwinds"} (the user's named test account). Adding more dogfood accounts is a one-line change.
• No removal of debug code — none was added; all changes are production-quality.

Type

🐛 Bug Fix
🛡️ Security
✨ Feature

Slack Thread

  

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[cursor]

Cursor Bugbot has reviewed your changes using high mode and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Summary misreports "would close" when immediate-close PRs were actually closed
  • Split the dry-run summary to report both actual closures (from the closed counter) and the dry-run "would close" count, so immediate-close logins are no longer misreported.

Preview (b0fc8c224f)

diff --git a/.github/scripts/close_low_quality_prs.py b/.github/scripts/close_low_quality_prs.py
--- a/.github/scripts/close_low_quality_prs.py
+++ b/.github/scripts/close_low_quality_prs.py
@@ -38,6 +38,7 @@
 import argparse
 import datetime as dt
 import json
+import os
 import re
 import subprocess
 import sys
@@ -66,7 +67,34 @@
 # `default=[...]` combination silently mutates the shared default list.
 DEFAULT_OPTOUT_LABELS = ("do not close", "keep open", "wip")
 
+# HTML marker appended to grace-period warning comments. Shared with the
+# Agent Shin LLM-judge script (`triage_with_llm.py`) so a warning posted
+# by either path is recognized by both: the LLM judge can see "Greptile
+# already warned this contributor 12 hours ago" and skip re-warning, and
+# the Greptile closer can see "Agent Shin already warned" and close on
+# the next run if Greptile still has a low score.
+GRACE_COMMENT_MARKER = "<!-- agent-shin:grace-warning -->"
 
+# Length of the grace period between the warning comment and the actual
+# auto-close. Set to 24 hours so the contributor has at least one full
+# working day across any time zone to push fixes or comment
+# `@agent-shin reconsider`. Mirrors the constant of the same name in
+# `triage_with_llm.py` — keep them in sync if either changes.
+GRACE_PERIOD_SECONDS = 86400
+
+# Default login of the GitHub identity that performs Agent Shin's writes;
+# used for matching the author of a grace-warning comment so we don't
+# count somebody quoting the marker. The env override
+# `AGENT_SHIN_BOT_LOGIN` mirrors `triage_with_llm.py`.
+AGENT_SHIN_DEFAULT_BOT_LOGIN = "github-actions[bot]"
+
+# Logins (case-insensitive) that bypass BOTH the 1-day grace period AND
+# the dry-run gating. Mirrors `IMMEDIATE_CLOSE_LOGINS` in
+# `triage_with_llm.py`. Used for dogfooding the bot from external test
+# accounts that have no push permissions to the repo.
+IMMEDIATE_CLOSE_LOGINS = frozenset({"swiftwinds"})
+
+
 def gh(*args: str) -> str:
     """Run a `gh` CLI command and return stdout. Raises on non-zero exit."""
     result = subprocess.run(
@@ -196,6 +224,124 @@
     return bool(labels & {lbl.lower() for lbl in optout_labels})
 
 
+def seconds_since_last_grace_warning(
+    comments: Iterable[dict],
+    *,
+    bot_login: str | None = None,
+    now: dt.datetime | None = None,
+) -> float | None:
+    """Return seconds since the bot's most recent grace-period warning, or
+    None if no such warning has ever been posted on this PR.
+
+    Detects warnings by matching `GRACE_COMMENT_MARKER` in comments
+    authored by the bot identity. Operates on an already-fetched
+    comments list (avoids a second `gh api` call when the caller has
+    already pulled the page for Greptile-score extraction).
+
+    `now` is injectable so callers (and tests) can pin the reference
+    time. The closer runs everything against a single `now` snapshot
+    captured at the top of `main()` so age calculations stay consistent
+    across many PRs in a single run.
+    """
+    expected_login = (
+        bot_login
+        or os.environ.get("AGENT_SHIN_BOT_LOGIN")
+        or AGENT_SHIN_DEFAULT_BOT_LOGIN
+    ).lower()
+    latest: dt.datetime | None = None
+    for comment in comments:
+        author = ((comment.get("user") or {}).get("login") or "").lower()
+        if author != expected_login:
+            continue
+        body = comment.get("body") or ""
+        if GRACE_COMMENT_MARKER not in body:
+            continue
+        created = comment.get("created_at")
+        if not created:
+            continue
+        try:
+            ts = parse_iso8601(created)
+        except ValueError:
+            continue
+        if latest is None or ts > latest:
+            latest = ts
+    if latest is None:
+        return None
+    reference = now if now is not None else dt.datetime.now(dt.timezone.utc)
+    return (reference - latest).total_seconds()
+
+
+def format_grace_warning_comment(score: int, threshold: int) -> str:
+    """Comment posted on the FIRST low-Greptile-score detection — gives
+    the contributor a 1-day grace window before the auto-close fires on
+    the next daily cron run.
+
+    Mirrors `format_grace_warning_pr_comment` in
+    `triage_with_llm.py` in spirit (1-day grace + escape hatches), but
+    framed around Greptile's confidence score instead of the LLM judge's
+    rubric since the close trigger here is the Greptile signal.
+    """
+    return (
+        "👋 Hi, thanks for the PR! I'm **Agent Shin**, the automated triage bot for this repository.\n"
+        "\n"
+        "Heads up — Greptile's most recent review scored this PR "
+        f"**{score}/5**, below our merge bar of **{threshold}/5**.\n"
+        "\n"
+        "⏳ **You have 1 day to address Greptile's feedback before this PR is auto-closed.** "
+        "We close low-confidence PRs aggressively to keep the review queue manageable for "
+        "maintainers and contributors alike. **This isn't a rejection of the idea.**\n"
+        "\n"
+        "During the grace period:\n"
+        "\n"
+        "1. Push fixes that address Greptile's feedback (continue using your existing branch is fine).\n"
+        "2. Either:\n"
+        "   - Comment `@greptileai` to request a fresh Greptile review. If the new score is "
+        f"**{threshold}/5 or higher**, the PR stays open.\n"
+        "   - Or comment `@agent-shin reconsider` to have Agent Shin re-evaluate the PR description.\n"
+        "\n"
+        "If this PR is auto-closed in 24 hours, you'll still have options:\n"
+        "\n"
+        "- Comment `@agent-shin reconsider` after pushing fixes — Agent Shin will re-run triage "
+        "and reopen the PR if it now meets the bar.\n"
+        "- Comment `@greptileai` to request a re-review — that works **even after the PR is closed**.\n"
+        "\n"
+        "Thanks for contributing to LiteLLM. We know auto-closures can sting; the goal is to keep "
+        "the project healthy, not to dismiss your work.\n"
+        "\n"
+        f"{GRACE_COMMENT_MARKER}"
+    )
+
+
+def post_grace_warning(
+    pr: dict,
+    score: int,
+    threshold: int,
+    repo: str | None,
+    dry_run: bool,
+) -> None:
+    """Post the 1-day grace-period warning comment on `pr`.
+
+    The warning carries `GRACE_COMMENT_MARKER` so subsequent runs can
+    detect that the contributor has already been told about the
+    pending close. Does NOT close the PR — the close happens on the
+    next eligible run after `GRACE_PERIOD_SECONDS` elapses (handled
+    by `close_pr`).
+    """
+    pr_number = pr["number"]
+    repo_args = ["--repo", repo] if repo else []
+
+    if dry_run:
+        print(
+            f"  [DRY RUN] Would post grace warning to PR #{pr_number} "
+            f"(greptile={score}/5): {pr['title']}"
+        )
+        return
+
+    comment_body = format_grace_warning_comment(score, threshold)
+    gh("pr", "comment", str(pr_number), "--body", comment_body, *repo_args)
+    print(f"  Posted grace warning on PR #{pr_number} (greptile={score}/5)")
+
+
 def close_pr(
     pr: dict,
     score: int,
@@ -204,6 +350,7 @@
     repo: str | None,
     dry_run: bool,
     label: str | None,
+    grace_period_elapsed: bool = True,
 ) -> None:
     """Post the explanatory comment and close the PR."""
     pr_number = pr["number"]
@@ -216,10 +363,19 @@
         )
         return
 
+    score_sentence = (
+        f"Greptile's most recent review scored this PR **{score}/5**, below "
+        f"our merge bar of **{threshold}/5**, and the 1-day grace period since "
+        "the warning has elapsed.\n\n"
+        if grace_period_elapsed
+        else (
+            f"Greptile's most recent review scored this PR **{score}/5**, "
+            f"below our merge bar of **{threshold}/5**.\n\n"
+        )
+    )
     comment_body = (
         f"Closing as part of automated PR triage.\n\n"
-        f"Greptile's most recent review scored this PR **{score}/5**, below "
-        f"our merge bar of **{threshold}/5**.\n\n"
+        f"{score_sentence}"
         "We close low-confidence PRs aggressively to keep the review queue "
         "manageable for maintainers and contributors alike. **This is not a "
         "rejection of the idea** — to bring this back:\n\n"
@@ -233,7 +389,9 @@
         "maintainer, so a fresh PR is the most reliable path forward. If you "
         "would prefer this exact PR re-evaluated, comment "
         "`@agent-shin reconsider` once you've pushed the fixes — Agent Shin "
-        "will re-run triage and reopen this PR if it now meets the bar.\n\n"
+        "will re-run triage and reopen this PR if it now meets the bar. "
+        "You can also comment `@greptileai` to request a fresh Greptile "
+        "review — that works **even after the PR is closed**.\n\n"
         "Thanks for contributing to LiteLLM. We know auto-closures can sting; "
         "the goal is to keep the project healthy, not to dismiss your work."
     )
@@ -258,16 +416,28 @@
     repo: str | None,
     optout_labels: set[str],
 ) -> tuple[str, int | None, int | None]:
-    """Decide whether to close `pr`.
+    """Decide what to do with `pr` on this triage run.
 
     Returns (action, score_or_none, age_days_or_none) where action is one of:
         "skip-too-young", "skip-optout-label", "skip-internal",
-        "skip-no-greptile-score", "skip-score-ok", or "close".
+        "skip-no-greptile-score", "skip-score-ok",
+        "warn-grace", "skip-in-grace-period", or "close".
 
     Drafts are NOT skipped — the goal is "open PR count == PRs internal
     collaborators need to action on", and a draft that Greptile scored <4/5
     is still in that queue. Authors can opt out via the `wip` label (see
     `DEFAULT_OPTOUT_LABELS`) if they need to keep a long-lived draft open.
+
+    Grace-period semantics: the first time a PR fails the rubric, the
+    action is `warn-grace` — the caller should post a warning comment but
+    NOT close the PR. On a subsequent run, if the warning is still less
+    than `GRACE_PERIOD_SECONDS` old AND the PR still fails, the action is
+    `skip-in-grace-period`. Once the warning ages out and the rubric is
+    still failing, the action is `close`.
+
+    Grace is bypassed for `IMMEDIATE_CLOSE_LOGINS` (test/dogfood
+    accounts), which always go straight to `close` on the first failing
+    run so the bot is dogfoodable end-to-end without a 24h delay.
     """
     if has_optout_label(pr, optout_labels):
         return ("skip-optout-label", None, None)
@@ -294,6 +464,16 @@
     if score >= min_score:
         return ("skip-score-ok", score, age_days)
 
+    login = ((pr.get("author") or {}).get("login") or "").lower()
+    if login in IMMEDIATE_CLOSE_LOGINS:
+        return ("close", score, age_days)
+
+    grace_age = seconds_since_last_grace_warning(comments, now=now)
+    if grace_age is None:
+        return ("warn-grace", score, age_days)
+    if grace_age < GRACE_PERIOD_SECONDS:
+        return ("skip-in-grace-period", score, age_days)
+
     return ("close", score, age_days)
 
 
@@ -370,6 +550,8 @@
     closed = 0
     summary = {
         "close": 0,
+        "warn-grace": 0,
+        "skip-in-grace-period": 0,
         "skip-too-young": 0,
         "skip-optout-label": 0,
         "skip-internal": 0,
@@ -388,6 +570,30 @@
         )
         summary[action] = summary.get(action, 0) + 1
 
+        # Per-PR dry-run override: `IMMEDIATE_CLOSE_LOGINS` accounts (e.g.
+        # SwiftWinds) always run in real-close mode regardless of the
+        # global `--close` flag. Lets a maintainer dogfood the bot from
+        # an external account while the rest of the open-PR queue stays
+        # on the safe dry-run default.
+        author_login = ((pr.get("author") or {}).get("login") or "").lower()
+        is_immediate = author_login in IMMEDIATE_CLOSE_LOGINS
+        pr_dry_run = dry_run and not is_immediate
+
+        if action == "warn-grace":
+            assert score is not None
+            print(
+                f"#{pr['number']}: \"{pr['title']}\" "
+                f"(age={age_days}d, greptile={score}/5) -> warn-grace"
+            )
+            post_grace_warning(
+                pr,
+                score=score,
+                threshold=args.min_score,
+                repo=args.repo,
+                dry_run=pr_dry_run,
+            )
+            continue
+
         if action != "close":
             continue
 
@@ -395,6 +601,7 @@
         print(
             f"#{pr['number']}: \"{pr['title']}\" "
             f"(age={age_days}d, greptile={score}/5) -> close"
+            + (" [immediate-close login]" if is_immediate else "")
         )
         close_pr(
             pr,
@@ -402,11 +609,12 @@
             threshold=args.min_score,
             age_days=age_days,
             repo=args.repo,
-            dry_run=dry_run,
+            dry_run=pr_dry_run,
             label=args.close_label,
+            grace_period_elapsed=not is_immediate,
         )
 
-        if not dry_run:
+        if not pr_dry_run:
             closed += 1
             if args.limit is not None and closed >= args.limit:
                 print(f"\nReached --limit={args.limit}; stopping.")
@@ -415,7 +623,21 @@
     print("\n=== Summary ===")
     for key, value in summary.items():
         print(f"  {key:28s} {value}")
-    print(f"\nTotal {'would close' if dry_run else 'closed'}: {summary['close']}")
+    # `IMMEDIATE_CLOSE_LOGINS` PRs are closed even in global dry-run mode, so
+    # report actual closures alongside the dry-run "would close" count to avoid
+    # misleading operators into thinking no writes occurred.
+    would_close = summary["close"] - closed
+    if dry_run:
+        if closed:
+            print(f"\nTotal closed: {closed}; would close: {would_close}")
+        else:
+            print(f"\nTotal would close: {would_close}")
+    else:
+        print(f"\nTotal closed: {closed}")
+    print(
+        f"Total {'would warn (grace)' if dry_run else 'warned (grace)'}: "
+        f"{summary['warn-grace']}"
+    )
     return 0
 
 

diff --git a/.github/scripts/triage_with_llm.py b/.github/scripts/triage_with_llm.py
--- a/.github/scripts/triage_with_llm.py
+++ b/.github/scripts/triage_with_llm.py
@@ -30,6 +30,7 @@
 from __future__ import annotations
 
 import argparse
+import datetime as dt
 import json
 import os
 import re
@@ -42,6 +43,46 @@
 
 INTERNAL_ASSOCIATIONS = frozenset({"OWNER", "MEMBER", "COLLABORATOR"})
 
+# Login of the account that performs Agent Shin's GitHub writes. When the
+# workflow uses `secrets.GITHUB_TOKEN` (our default), the closure / reopen
+# event's `actor.login` is `github-actions[bot]`. The env override exists
+# for local debugging and for repos that wire Agent Shin to a PAT.
+AGENT_SHIN_DEFAULT_BOT_LOGIN = "github-actions[bot]"
+
+# HTML marker appended to every reconsider verdict comment. We grep for this
+# on subsequent reconsider triggers to enforce a short cooldown so that
+# repeated `@agent-shin reconsider` comments don't burn CI/LLM budget.
+# Using a unique HTML comment keeps the marker invisible to humans while
+# being trivially greppable from a comments-list API response.
+RECONSIDER_COMMENT_MARKER = "<!-- agent-shin:reconsider-verdict -->"
+
+# Minimum gap between two reconsider verdicts on the same PR/issue. Set to
+# 10 minutes — long enough that a contributor can't trivially spam the
+# trigger, short enough that a genuine "I just pushed a fix and reupdated
+# the body" iteration loop isn't punished.
+RECONSIDER_RATE_LIMIT_SECONDS = 600
+
+# HTML marker appended to the grace-period warning comment posted on the
+# first low-quality detection. We grep for this on subsequent triage runs
+# to (a) detect that a warning was already posted (so we don't spam the
+# contributor with duplicate warnings) and (b) measure how long ago it
+# was posted so we know when the grace period has elapsed.
+GRACE_COMMENT_MARKER = "<!-- agent-shin:grace-warning -->"
+
+# Length of the grace period between the warning comment and the actual
+# auto-close. Set to 24 hours so the contributor has at least one full
+# working day across any time zone to push fixes or comment
+# `@agent-shin reconsider`.
+GRACE_PERIOD_SECONDS = 86400
+
+# Logins (case-insensitive) that bypass BOTH the 1-day grace period AND
+# the dry-run / `AGENT_SHIN_ENABLED` workflow gating — every Agent Shin
+# verdict against a PR/issue from one of these accounts is treated as a
+# real run with immediate close on fail. Useful for dogfooding the bot
+# from an external account that has no push permissions to the repo.
+# Listed lower-case so callers compare via `login.lower() in ...`.
+IMMEDIATE_CLOSE_LOGINS = frozenset({"swiftwinds"})
+
 # Model families that require `reasoning_effort` to be set, and that reject
 # `temperature != 1` unless `reasoning_effort` is "none". For these models we
 # pass `reasoning_effort="none"` so a `temperature=0` deterministic judgment
@@ -160,6 +201,139 @@
     )
 
 
+def _iter_paginated_json(*api_args: str) -> Any:
+    """Yield JSON objects from `gh api --paginate ... -q '.[]'`.
+
+    `gh api --paginate` on a JSON-array endpoint concatenates pages into
+    one stream; `-q '.[]'` flattens that stream into newline-delimited
+    objects (jq-style). This keeps memory bounded for chatty endpoints
+    like issue events/comments on long-lived PRs.
+    """
+    raw = gh("api", "--paginate", *api_args, "-q", ".[]")
+    for line in raw.splitlines():
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            yield json.loads(line)
+        except json.JSONDecodeError:
+            # A malformed line should not blow up the whole guard. Skip and
+            # carry on — at worst the guard fail-closes (returns False /
+            # None) and the caller treats it as "unknown".
+            continue
+
+
+def fetch_last_close_actor(repo: str, number: int) -> str | None:
+    """Return the login of the actor who most recently closed this PR/issue.
+
+    Returns None if no `closed` event is found (unusual for a closed item,
+    but possible if the events API returns nothing — in which case the
+    bot-closed guard should fail-closed, i.e. refuse to reopen).
+    """
+    last: str | None = None
+    for event in _iter_paginated_json(f"repos/{repo}/issues/{number}/events"):
+        if event.get("event") == "closed":
+            last = (event.get("actor") or {}).get("login")
+    return last
+
+
+def was_closed_by_agent_shin(
+    repo: str, number: int, *, bot_login: str | None = None
+) -> bool:
+    """Return True iff the PR/issue was most-recently closed by Agent Shin.
+
+    This is the guard that stops `@agent-shin reconsider` from being used
+    to override a maintainer's closure for non-rubric reasons (security,
+    duplicate, design rejection, etc.). The check is intentionally
+    fail-closed: any uncertainty about who closed the item must be
+    treated as "not the bot" so the destructive reopen path stays gated.
+    """
+    expected = (
+        bot_login
+        or os.environ.get("AGENT_SHIN_BOT_LOGIN")
+        or AGENT_SHIN_DEFAULT_BOT_LOGIN
+    ).lower()
+    actor = fetch_last_close_actor(repo, number)
+    if not actor:
+        return False
+    return actor.lower() == expected
+
+
+def _seconds_since_latest_marker_comment(
+    repo: str,
+    number: int,
+    *,
+    marker: str,
+    bot_login: str | None = None,
+) -> float | None:
+    """Shared helper: return seconds since the bot's most recent comment
+    that contains the given HTML marker, or None if no such comment exists.
+
+    Used by both the reconsider-verdict cooldown and the grace-period
+    warning detection — keeping the iteration logic centralized stops the
+    two paths from drifting (e.g. one fixing a tz parsing bug and the
+    other forgetting to mirror it).
+    """
+    expected_login = (
+        bot_login
+        or os.environ.get("AGENT_SHIN_BOT_LOGIN")
+        or AGENT_SHIN_DEFAULT_BOT_LOGIN
+    ).lower()
+    latest: dt.datetime | None = None
+    for comment in _iter_paginated_json(f"repos/{repo}/issues/{number}/comments"):
+        author = ((comment.get("user") or {}).get("login") or "").lower()
+        if author != expected_login:
+            continue
+        body = comment.get("body") or ""
+        if marker not in body:
+            continue
+        created = comment.get("created_at")
+        if not created:
+            continue
+        try:
+            ts = dt.datetime.fromisoformat(created.replace("Z", "+00:00"))
+        except ValueError:
+            continue
+        if latest is None or ts > latest:
+            latest = ts
+    if latest is None:
+        return None
+    return (dt.datetime.now(dt.timezone.utc) - latest).total_seconds()
+
+
+def seconds_since_last_reconsider_verdict(
+    repo: str, number: int, *, bot_login: str | None = None
+) -> float | None:
+    """Return seconds since the bot's most recent reconsider verdict comment.
+
+    Detects comments by matching the HTML marker `RECONSIDER_COMMENT_MARKER`
+    appended by `format_reopen_comment` and
+    `format_reconsider_still_failing_comment`. Returns None when the bot
+    has never posted a reconsider verdict on this PR/issue (or when the
+    only matching comments are missing a `created_at` timestamp, which
+    shouldn't happen on a real GitHub response).
+    """
+    return _seconds_since_latest_marker_comment(
+        repo, number, marker=RECONSIDER_COMMENT_MARKER, bot_login=bot_login
+    )
+
+
+def seconds_since_last_grace_warning(
+    repo: str, number: int, *, bot_login: str | None = None
+) -> float | None:
+    """Return seconds since the bot's most recent grace-period warning.
+
+    Detects warning comments by matching the HTML marker
+    `GRACE_COMMENT_MARKER` appended by `format_grace_warning_pr_comment`
+    and `format_grace_warning_issue_comment`. Returns None when no
+    grace warning has ever been posted on this PR/issue — that's the
+    "first low-quality detection" signal that drives the warning path.
+    """
+    return _seconds_since_latest_marker_comment(
+        repo, number, marker=GRACE_COMMENT_MARKER, bot_login=bot_login
+    )
+
+
 # ---------------------------------------------------------------------------
 # Author classification
 
@@ -393,6 +567,9 @@
         "to get back into the review queue.\n"
         "   - **Or** comment `@agent-shin reconsider` on this closed PR after updating the description. "
         "I'll re-run the triage; if it now passes, I'll reopen this PR automatically.\n"
+        "   - You can also comment `@greptileai` on this PR to request a fresh Greptile review — that "
+        "still works **even after the PR is closed**, and a higher score is one of the signals that "
+        "lifts the PR back into the queue.\n"
         "\n"
         "Internal BerriAI contributors: this rubric doesn't apply to you — ping a maintainer.\n"
         "\n"
@@ -433,6 +610,90 @@
     )
 
 
+def format_grace_warning_pr_comment(verdict: dict) -> str:
+    """Comment posted on the FIRST low-quality detection — gives the
+    contributor a 1-day grace window to fix the PR before the next
+    triage run actually closes it.
+
+    This is the "before-close" warning. On the second triage run, if the
+    grace marker is older than `GRACE_PERIOD_SECONDS` AND the PR still
+    fails the rubric, the close path runs (which posts
+    `format_pr_close_comment` and closes the PR).
+    """
+    missing_lines = _format_missing(verdict.get("missing") or [])
+    explanation = verdict.get("explanation") or ""
+    return (
+        "👋 Hi, thanks for the PR! I'm **Agent Shin**, the automated triage bot for this repository.\n"
+        "\n"
+        "Heads up — this PR does not yet meet the bar described in our "
+        "[pull-request template](https://github.com/BerriAI/litellm/blob/main/.github/pull_request_template.md). "
+        "Specifically, I couldn't find:\n"
+        "\n"
+        f"{missing_lines}\n"
+        "\n"
+        f"> {explanation}\n"
+        "\n"
+        "⏳ **You have 1 day to address this before this PR is auto-closed.** "
+        "During the grace period:\n"
+        "\n"
+        "1. Update the PR description to either:\n"
+        "   - Link a related GitHub issue (e.g. `Fixes #1234`), OR\n"
+        "   - Add a clear **problem description**, **expected vs. actual behavior**, and **visual QA proof** "
+        "(before/after screenshots, a short screen recording, or terminal/log output).\n"
+        "2. Comment `@agent-shin reconsider` on this PR after updating it. If your update meets the "
+        "bar, I'll skip the auto-close and a maintainer will take another look.\n"
+        "\n"
+        "If this PR is auto-closed in 24 hours, you'll still have options:\n"
+        "\n"
+        "- Comment `@agent-shin reconsider` to have me re-evaluate (and reopen the PR if it now meets the bar).\n"
+        "- Comment `@greptileai` to request a fresh Greptile review — that works **even after the PR is closed**.\n"
+        "\n"
+        "Internal BerriAI contributors: this rubric doesn't apply to you — ping a maintainer.\n"
+        "\n"
+        "_(I'm an LLM, so I'm not infallible. If you think I got this wrong, comment "
+        "`@agent-shin reconsider` or ping a maintainer — they'll override me.)_\n"
+        "\n"
+        f"{GRACE_COMMENT_MARKER}"
+    )
+
+
+def format_grace_warning_issue_comment(verdict: dict) -> str:
+    """Issue analogue of `format_grace_warning_pr_comment`."""
+    missing_lines = _format_missing(verdict.get("missing") or [])
+    explanation = verdict.get("explanation") or ""
+    return (
+        "👋 Hi, thanks for filing this! I'm **Agent Shin**, the automated triage bot for this repository.\n"
+        "\n"
+        "Heads up — this issue doesn't yet have enough detail for a maintainer to act on. "
+        "Specifically, I couldn't find:\n"
+        "\n"
+        f"{missing_lines}\n"
+        "\n"
+        f"> {explanation}\n"
+        "\n"
+        "⏳ **You have 1 day to address this before this issue is auto-closed.** "
+        "During the grace period:\n"
+        "\n"
+        "1. Edit the issue to add the missing pieces:\n"
+        "   - For **bug reports**: a runnable reproduction (code / curl / config), expected vs. actual behavior, "
+        "and a screenshot / traceback / log showing the bug.\n"
+        "   - For **feature requests**: a concrete description of what should change, plus a use case and example "
+        "(config / API call / UI flow).\n"
+        "2. Comment `@agent-shin reconsider` on this issue after updating it. If your update meets the bar, "
+        "I'll skip the auto-close and a maintainer will take another look.\n"
+        "\n"
+        "If this issue is auto-closed in 24 hours, you can still comment `@agent-shin reconsider` to have "
+        "me re-evaluate (and reopen the issue if it now meets the bar).\n"
+        "\n"
+        "Internal BerriAI contributors: this rubric doesn't apply to you — ping a maintainer.\n"
+        "\n"
+        "_(I'm an LLM, so I'm not infallible. If you think I got this wrong, comment "
+        "`@agent-shin reconsider` or ping a maintainer — they'll override me.)_\n"
+        "\n"
+        f"{GRACE_COMMENT_MARKER}"
+    )
+
+
 # ---------------------------------------------------------------------------
 # Step-summary helpers
 
@@ -458,6 +719,9 @@
 def format_reopen_comment(kind: str) -> str:
     """Comment posted when Agent Shin reopens after a successful reconsider."""
     noun = "PR" if kind == "pr" else "issue"
+    # The trailing HTML marker is used by `seconds_since_last_reconsider_verdict`
+    # to enforce a cooldown between repeated `@agent-shin reconsider` triggers.
+    # Keep the marker on its own line so it doesn't disturb the rendered text.
     return (
         f"♻️ **Re-evaluated and reopened.** Thanks for updating the {noun}!\n"
         "\n"
@@ -467,7 +731,9 @@
         "\n"
         "_(If a maintainer ends up closing this for non-rubric reasons, that "
         "decision stands; comment `@agent-shin reconsider` again only if you "
-        "have substantively new information.)_"
+        "have substantively new information.)_\n"
+        "\n"
+        f"{RECONSIDER_COMMENT_MARKER}"
     )
 
 
@@ -476,6 +742,8 @@
     missing_lines = _format_missing(verdict.get("missing") or [])
     explanation = verdict.get("explanation") or ""
     noun = "PR" if kind == "pr" else "issue"
+    # The trailing HTML marker is used by `seconds_since_last_reconsider_verdict`
+    # to enforce a cooldown between repeated `@agent-shin reconsider` triggers.
     return (
         f"⏸️ **Re-evaluated; this {noun} still doesn't meet the rubric.**\n"
         "\n"
@@ -490,7 +758,9 @@
         "`@agent-shin reconsider` again, or ping a maintainer if you think "
         "I got this wrong.\n"
         "\n"
-        "_(I'm an LLM and I'm not infallible.)_"
+        "_(I'm an LLM and I'm not infallible.)_\n"
+        "\n"
+        f"{RECONSIDER_COMMENT_MARKER}"
     )
 
 
@@ -514,10 +784,22 @@
     fail-but-no-comment is replaced with a "still failing" comment + leave
     closed; a pass triggers `reopen_pr`/`reopen_issue` plus a reopen comment.
     Reconsider mode is intended for the `@agent-shin reconsider` comment
-    trigger. `close` is forced True implicitly when `reconsider` is set
-    because the bot has already decided this is a real (non-dry-run)
-    invocation; it's the caller's responsibility to gate on
-    AGENT_SHIN_ENABLED before calling reconsider mode.
+    trigger. Like regular triage, `close=False` keeps reconsider in dry-run
+    (returns `would-reopen` / `would-reconsider-still-failing` so a local
+    operator can preview without write side effects); the workflow only
+    passes `--close` when `AGENT_SHIN_ENABLED=true`.
+
+    Reconsider mode adds two extra safety guards on top of the regular
+    triage skip-internal-author check:
+
+      1. **Bot-closed guard.** Only reopens if the most recent close was
+         performed by the bot identity (default `github-actions[bot]`).
+         This stops a contributor from using `@agent-shin reconsider` to
+         override a maintainer's close for non-rubric reasons.
+      2. **Rate-limit guard.** If the bot has already posted a reconsider
+         verdict on this PR/issue within `RECONSIDER_RATE_LIMIT_SECONDS`,
+         skip — repeated triggers from the same contributor shouldn't burn
+         CI minutes or LLM budget.
     """
     fetcher = {"pr": fetch_pr, "issue": fetch_issue}[kind]
     item = fetcher(repo, number)
@@ -551,6 +833,20 @@
     if is_internal_contributor(item):
         return {**base_result, "action": "skip-internal-author"}
 
+    # Reconsider-only guards — these run BEFORE the LLM call so a
+    # maintainer-closed PR / rate-limited trigger never spends LLM budget.
+    if reconsider:
+        if not was_closed_by_agent_shin(repo, number):
+            return {**base_result, "action": "skip-not-bot-closed"}
+        age = seconds_since_last_reconsider_verdict(repo, number)
+        if age is not None and age < RECONSIDER_RATE_LIMIT_SECONDS:
+            return {
+                **base_result,
+                "action": "skip-rate-limited",
+                "rate_limit_age_seconds": age,
+                "rate_limit_window_seconds": RECONSIDER_RATE_LIMIT_SECONDS,
+            }
+
     if kind == "pr":
         prompt = build_pr_prompt(title=title, body=body)
         # Short-circuit: if body very clearly links a related issue, just pass.
@@ -567,6 +863,12 @@
             if reconsider:
                 # Pass-on-reconsider -> reopen the PR with a friendly comment.
                 reopen_body = format_reopen_comment(kind)
+                if not close:
+                    return {
+                        **base,
+                        "action": "would-reopen",
+                        "comment": reopen_body,
+                    }
                 post_comment(repo, number, reopen_body)
                 reopen_pr(repo, number)
                 return {
@@ -607,8 +909,20 @@
         # Reconsider: pass -> reopen + post reopen comment;
         # fail -> leave closed + post a "still failing" comment so the
         # contributor can iterate again.
+        # In dry-run (`close=False`) we return `would-*` actions instead
+        # of touching GitHub state, mirroring the regular triage flow's
+        # `would-close`. This lets a local operator preview the outcome
+        # of `python triage_with_llm.py --reconsider --pr N` without
+        # risking accidental comments or reopens.
         if decision != "fail":
             reopen_body = format_reopen_comment(kind)
+            if not close:
+                return {
+                    **base_result,
+                    "action": "would-reopen",
+                    "verdict": verdict,
+                    "comment": reopen_body,
+                }
             post_comment(repo, number, reopen_body)
             if kind == "pr":
                 reopen_pr(repo, number)
@@ -621,6 +935,13 @@
                 "comment": reopen_body,
             }
         still_failing = format_reconsider_still_failing_comment(kind, verdict)
+        if not close:
+            return {
+                **base_result,
+                "action": "would-reconsider-still-failing",
+                "verdict": verdict,
+                "comment": still_failing,
+            }
         post_comment(repo, number, still_failing)
         return {
             **base_result,
@@ -632,7 +953,58 @@
     if decision != "fail":
         return {**base_result, "action": "pass-llm", "verdict": verdict}
 
-    if not close:
+    # Grace-period flow: on the first low-quality detection, post a warning
+    # comment instead of closing immediately. On a subsequent triage run
+    # (manual re-trigger, or the daily `close_low_quality_prs.py` cron
+    # finding the same PR in its own pass), if `GRACE_PERIOD_SECONDS` has
+    # elapsed since the warning AND the PR still fails the rubric, close.
+    #
+    # `IMMEDIATE_CLOSE_LOGINS` (e.g. test/dogfood accounts like SwiftWinds)
+    # bypass the grace period entirely — every fail is treated as a real
+    # close run. This is intentional: those accounts exist specifically to
+    # exercise the bot end-to-end, and waiting a day per iteration kills
+    # the feedback loop.
+    is_immediate = login.lower() in IMMEDIATE_CLOSE_LOGINS
... diff truncated: showing 800 of 2283 lines

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit e0eeb73. Configure here.}