CodeRabbit's Security Architecture Review is throttled separately from its code review — findings in both sections must be addressed, and an in-place edit after disposition must re-open the comment #2008

Problem

CodeRabbit now posts two independently throttled outputs in one summary issue comment, which it edits in place:

1. Code review. This part can be rate-limited ("Review limit reached — You've used all free OSS reviews for now…").
2. Security Architecture Review. This part is not throttled with the code review and can carry real findings while the code review shows the rate-limit block.

Our automation treats that comment as a single unit. It also only reacts to newly created comments. As a result, a security finding that CodeRabbit adds by editing an already-dispositioned summary is never addressed and never re-blocks the gate.

Live evidence: PR #2000, comment 5938681830 (coderabbitai[bot], created 19:12Z, edited in place through 21:15Z)

The section markers, in order:

<!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- This is an auto-generated comment: rate limited by coderabbit.ai --> … <!-- end of auto-generated comment: rate limited by coderabbit.ai -->
<!-- recent_review_start --> … <!-- recent_review_end -->
<!-- walkthrough_start --> … <!-- walkthrough_end -->
<!-- final_review_risk_start --> … <!-- final_review_risk_end -->
<!-- architecture_review_start -->
### Security Architecture Review
**Security architecture risk:** _🟡 Moderate_ · up to `769f3`
…
**Retained concerns**
- **Medium · security · inferred:** New exemptions classify an entire registered-reviewer comment as clean when only its opening notice matches. …
<details><summary>Security review details</summary> … **Hardening Proposals** … </details>
<!-- architecture_review_end -->

At the same moment, the rate-limit block covered commits 769f3dd→2c77e41, while the security section still reported a finding at 769f3.

Timeline:

• 19:23Z: Dev-Lead posted disposition 5938896990, disposition=informational ("rate-limit/summary notice — no actionable review finding"). The comment was then minimized RESOLVED.
• ~20:35Z: CodeRabbit edited the same comment to add the Moderate security finding.
• After that: no automation re-opened, re-dispositioned or addressed it. It was fixed only because a human-driven session acted on it manually (reply 5940203474).

CodeRabbit had no review threads on #2000, and its two PR reviews were DISMISSED with empty bodies. The security finding existed only inside the edited summary comment.

Similar stale dispositions:

• feat: implement issue #1994 — Rate-limited PR reviews wait up to ~7h after the limit resets — the retry relies on GitHub's best-effort cron, which fires far less often than every 15 min #1999 (5938846806) described the comment as a "review in progress placeholder"; CodeRabbit later edited it into a rate-limit notice.
• feat: implement issue #1993 — Stop the Codex noise: auto-clear chatgpt-codex-connector "usage limits" notices so they stop blocking the maintainer gate and spawning dev-lead runs #1998 (5938594515) described it as a "review-in-progress / walkthrough summary".

Root causes (code on main)

1. Edits are invisible.
   • dev-lead.yml:42–43 triggers on issue_comment: types: [created] only. So do persona-mention.yml and pr-review-mention.yml.
   • Neither scripts/lib/maintainer-comment-gate.sh (:164–186) nor scripts/lib/comment-disposition-verify.sh compares the comment's updatedAt/lastEditedAt with the disposition's time.
   • Once a comment is minimized RESOLVED it stays cleared, whatever the bot appends later.
2. No section awareness.
   • prompts/dev-lead/fix-bot-comment.md:113–125 dispositions a "rate-limit / 'review limit reached' notice" as informational, and :36 says "A neutral overview is not an actionable finding".
   • Nothing tells the agent to look inside CodeRabbit's sections: architecture_review/Security, Actionable comments posted, Outside diff range comments, nitpicks.
   • The rate-limit notice is what clears the whole comment, findings included.
3. Security findings never become threads, so fix-reviews Phase 1, required_review_thread_resolution and notify_coderabbit_resolve (dev-lead-fix-reviews.sh:308–324) never see them.
4. The registry pattern targets text CodeRabbit doesn't post.
   • The coderabbitai row at scripts/lib/reviewer-sources.tsv:65 matches the "…used up your prepaid credits" form.
   • The live notice is the OSS form ("You've used all free OSS reviews…"), inside a blockquote in the full summary.
   • The pattern never matches a real comment (checked with jq test() on feat: implement issue #1995 — info_status_pattern covers only SonarCloud — Codex/Qodo/CodeRabbit usage-limit notices are still undispositionable and still strand PRs #2000's body). That is fail-closed, so it is safe, but the row's rationale and its tests (tests/test_reviewer_sources.bats:219,266, tests/dev-lead/unit/test_maintainer_comment_gate.bats:347) describe a notice that doesn't occur.
   • Do not "fix" this by matching the OSS notice against the whole summary. That would auto-clear the security section along with it (info_status_pattern covers only SonarCloud — Codex/Qodo/CodeRabbit usage-limit notices are still undispositionable and still strand PRs #1995's "do not widen" rule).
5. Rate-limit detection matches the whole body and keys on creation time.
   • ADVISORY_RATE_LIMIT_RE (scripts/lib/advisory-review-gate.sh:94) is matched against the entire summary, so a walkthrough that merely mentions rate limits counts.
   • detect_advisory_rate_limit (:301–326) picks the latest entry by createdAt/submittedAt, not by edit time.
   • Nothing tells "code review throttled" apart from "security review completed with findings".

Acceptance criteria

1. Edits re-open dispositioned comments.
   • If a registered bot's comment was edited after its covering disposition (lastEditedAt/updatedAt later than the disposition's createdAt), the maintainer comment gate treats it as un-dispositioned. It is not cleared until a fresh disposition covers the current body.
   • Dev-Lead is triggered for such edits, either via issue_comment: [edited] for trusted bots (deduplicated so CodeRabbit's frequent progress edits don't spawn a run each time) or a sweep check.
   • Fail closed when the timestamps can't be read.
2. Both CodeRabbit sections are addressed.
   • The fix-bot-comment and fix-reviews prompts treat a CodeRabbit summary as a set of sections. A rate-limit block covers only its own section.
   • Security Architecture Review findings ("Retained concerns", severity-labelled items, Hardening Proposals) and any "Actionable comments posted" / "Outside diff range" items each get a real disposition: fixed with a verifying commit, or answered/out-of-scope with a reason.
   • Only a body whose every finding-bearing section is empty or "no issues" may be dispositioned informational.
3. A rate-limit notice never clears findings.
   • No info_status_pattern or prompt rule may clear a CodeRabbit comment that contains an architecture_review section with findings, or any other finding-bearing section.
   • Update the coderabbitai row's rationale so it no longer implies the prepaid-credits form is the live notice.
   • Add a regression test built from feat: implement issue #1995 — info_status_pattern covers only SonarCloud — Codex/Qodo/CodeRabbit usage-limit notices are still undispositionable and still strand PRs #2000's real body (rate-limit block plus security finding): it must block, and must be refused on the maintainer-resolve-comment.sh bot path.
4. Section-aware rate-limit detection.
   • CodeRabbit's rate-limit detection looks only inside the rate limited by coderabbit.ai marker block, not the whole body, and uses the comment's last-edit time.
   • When the code review is throttled but the security section is present, the PR still waits on, or addresses, the security findings and is not treated as "no CodeRabbit evidence".
5. Tests:
   • the edited-after-disposition gate case;
   • the feat: implement issue #1995 — info_status_pattern covers only SonarCloud — Codex/Qodo/CodeRabbit usage-limit notices are still undispositionable and still strand PRs #2000-shaped body (throttled code review + security finding) blocks, and dispositioning only the notice does not clear it;
   • a clean CodeRabbit summary (no findings in any section) can still be dispositioned informational;
   • rate-limit detection ignores a walkthrough that merely mentions "rate limit".

Notes

• Related: info_status_pattern covers only SonarCloud — Codex/Qodo/CodeRabbit usage-limit notices are still undispositionable and still strand PRs #1995 / feat: implement issue #1995 — info_status_pattern covers only SonarCloud — Codex/Qodo/CodeRabbit usage-limit notices are still undispositionable and still strand PRs #2000 (anchored info-status patterns; "don't widen" rule), Recurring bot quota/trial notices (Codex usage limits, Qodo trial, CodeRabbit review-limit) still block approval — add info_status_patterns; fix stale maintainer-resolve refusal message #1932 (earlier quota-notice issue), [#1813 follow-up] Harden the comment-disposition gate: workflow-enforced provenance, ADR-0009, retire blanket @coderabbitai resolve, audit trail, no marker-less agent comments #1846 (disposition-gate hardening), dev-lead re-dispositions already-dispositioned PR comments on every pass — "expected exactly one, found N" then deadlocks the maintainer gate for good (#1952, #1953) #1992 / fix(dev-lead): idempotent comment dispositions and duplicate recovery (#1992) #2006 (latest-disposition selection; reuse cdv_select_disposition when comparing against edit time).
• Scope: CodeRabbit is advisory_gate=no, so this does not change the advisory quorum. It concerns whether CodeRabbit's findings are acted on, and whether its rate-limit notice can clear them.