Skip to content

dev-lead falsely reports "all AI engines are currently rate-limited" when the real cause is stale cancelled check runs #461

Closed

Description

@don-petry

Symptom

On PR #453, dev-lead posted (comment 4642769821, 2026-06-07T13:10:43Z):

@don-petry I received your request but all AI engines are currently rate-limited. I'll retry automatically once the rate limit clears.
Rate limit resets at: 2026-06-07T13:40:42Z

No AI engine was rate-limited. The engine ran successfully and decided no changes were needed. The "reset" time is exactly now + 30 minutes — the giveaway is the :42 seconds, which can only come from date -u -d '+30 minutes' (dev-lead-fix-reviews.sh hard-blocker backoff path), never from parse_reset_time (which parses whole-minute "resets H:MMam/pm" strings).

Root causes (two distinct defects)

1. The hard-blocker retry path reuses the rate-limit messaging verbatim

In scripts/dev-lead-fix-reviews.sh, the fix-reviews / review-changes intents handle "engine ran, no changes committed, but has_hard_blockers is true" by writing now+30min to /tmp/dev-lead-rate-limit-reset and calling post_reviews_rate_limited — which unconditionally tells the user "all AI engines are currently rate-limited", regardless of why it was called. The marker reuse (status=rate-limited) is intentional so the retry cron re-dispatches, but the user-facing text is simply false. Introduced in #435.

2. cancelled check runs from superseded workflow runs count as hard blockers forever

fetch_pr_context builds CI_STATUS_JSON from GET /commits/{sha}/check-runs with no dedup by check name. Each triggering event creates a new check suite, so GitHub's default filter=latest does not collapse them. On PR #453's head SHA the data was:

review | cancelled | suite=…8170 | started 13:08:53   (Dev-Lead Agent, concurrency-cancelled)
review | cancelled | suite=…8444 | started 13:08:54   (PR Review Agent, concurrency-cancelled)
review | success   | suite=…8971 | started 13:08:56   (surviving run)

has_hard_blockers / has_tier1_blockers count conclusion == "cancelled", so the two stale runs permanently register as blockers even though the same-named check later succeeded — putting the PR into an endless 30-minute "rate-limited" retry loop.

Notably, the legacy commit-statuses path in the same function already dedupes by context exactly to avoid this ("so a stale failure overwritten by a later success does not appear as a Tier-1 blocker") — the check-runs path just never got the same treatment.

Proposed fix

  1. Dedupe check runs by name in fetch_pr_context, keeping the newest run (branch protection keys required checks by name, so name-level dedup matches GitHub's own merge semantics). A latest cancelled run still counts as a blocker — only superseded runs stop counting.
  2. Parameterize post_reviews_rate_limited with a reason (rate-limit vs blocked) and post honest user-facing text for the blocked path, while keeping the machine-readable status=rate-limited marker token so dev-lead-retry.sh and the e2e scenarios continue to work unchanged.

Not related to

PR #458 (advisory review gate) — open, touches the pr-review agent only. Both defects predate it (#435, #426/#353).

Activity

  1. 3 remaining items

  2. added 4 commits that reference this issue on Jun 25, 2026
    8c8ac4f
    7917e0f
    c6ea71c
    4bf4435
  3. added 9 commits that reference this issue on Aug 2, 2026
    3eaea09
    d9942c5
    b5b2a0a
    5645e6b
    4cb8748
    21ad26e
    3bd7189
    f5fd8e3
    a537d28
  4. don-petry commented on Aug 21, 2026

    @don-petry
    CollaboratorAuthor

    Follow-up filed for the half this issue deliberately deferred: #1568.

    #461 fixed the user-facing prose but explicitly kept the machine-readable token — "while keeping the machine-readable status=rate-limited marker token so dev-lead-retry.sh and the e2e scenarios continue to work unchanged." No follow-up was ever opened for the token itself, and it is still hardcoded in dev-lead-fix-reviews.sh with reason=blocked appended alongside.

    It bit again on 2026-08-21 (PR .github#995): a hold caused by a cascade of self-cancelling dispatches was recorded as status=rate-limited while actual Claude usage was 8% session / 86% weekly. #1568 carries the retry-cron/e2e compatibility requirement that made the original deferral reasonable.

  5. added a commit that references this issue on Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions