fix(queue): prioritize live PR work under GitHub rate pressure - #2847
Merged
Conversation
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
gittensory-ui | 1f3acaa | Commit Preview URL Branch Preview URL |
Jul 03 2026, 10:40 PM |
Adds a bounded ramp-up cap to the foreground-liveness release sweep (#selfhost-queue-liveness): when a large backlog of over-deferred foreground jobs has accumulated (a production incident showed ~190), releasing all of it in one sweep tick lets every released job re-attempt GitHub at once and immediately re-trip the same rate-limit bucket it was deferred for. FOREGROUND_LIVENESS_MAX_RELEASE_PER_SWEEP (default 25) now bounds each sweep to the oldest-first N eligible jobs, draining a large backlog gradually over several ticks instead of in one burst. Investigation into the rest of this issue's scope found most of it already addressed: - Current-head agent-regate-pr reclassification and sticky-cooldown recovery were shipped in the prior self-host queue-liveness PR. - Required-status (branch-protection) policy is already cached durably (GITHUB_BRANCH_PROTECTION_CACHE_TTL_SECONDS, ~20m default). - Webhook/regate coalescing is already correctly head-SHA-scoped (githubWebhookCoalesceKey embeds head_sha for check_run/check_suite/ pull_request events; agent-regate-pr coalesces by repo+PR only, which is correct for that job's semantics since it always reads live PR state at execution time, never a stale snapshot). Deliberately NOT included in this PR, flagged for dedicated follow-up given their correctness-criticality: - Per-installation concurrent GitHub-fetch scheduling (queue-wide concurrency exists; per-installation tracking would be a new coordination mechanism). - Caching check-runs/statuses/check-suites REST reads: the existing client has an explicit, deliberate design decision that these are "mutable review inputs [that] must always reflect the current GitHub state" (src/github/client.ts), since they feed merge/close decisions. A safe version (short TTL + webhook-driven invalidation, explicit bypass preserved at the documented freshness-critical call sites) needs a dedicated design pass, not a rushed reversal of that decision. - Expanding GraphQL rollup coverage beyond the existing GITHUB_STATUS_ROLLUP_GRAPHQL-gated path. Validated: npm run typecheck, git diff --check, and the full foreground-liveness/pg-queue/sqlite-queue test suites (225 tests) all green. Every added line has 100% line and branch coverage.
JSONbored
force-pushed
the
claude/queue-rate-pressure-priority
branch
from
July 3, 2026 22:41
1f3acaa to
2813cc1
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2847 +/- ##
=======================================
Coverage 96.22% 96.22%
=======================================
Files 257 257
Lines 28114 28124 +10
Branches 10218 10219 +1
=======================================
+ Hits 27053 27063 +10
Misses 435 435
Partials 626 626
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FOREGROUND_LIVENESS_MAX_RELEASE_PER_SWEEP, default 25) to the foreground-liveness release sweep from fix(selfhost): keep review queue live and verify configured CI #2843: a large inherited backlog of over-deferred foreground jobs (a production incident showed ~190) now drains gradually, oldest-first, over several sweep ticks instead of releasing everything into one instant — releasing all of it at once lets every job re-attempt GitHub simultaneously and can immediately re-trip the same rate-limit bucket it was deferred for.Investigation findings (scope narrowed accordingly)
Most of this issue's originally-requested scope turned out to already be handled:
agent-regate-prreclassification (live/foreground vs. scheduled-sweep background) and condition-based sticky-cooldown recovery were already shipped in fix(selfhost): keep review queue live and verify configured CI #2843.GITHUB_BRANCH_PROTECTION_CACHE_TTL_SECONDS, ~20m default) — verified insrc/github/client.ts.githubWebhookCoalesceKeyembedshead_shaforcheck_run/check_suite/pull_requestevents (verified in code + existing tests), andagent-regate-printentionally coalesces by repo+PR only, which is correct for that job's semantics — it always re-reads live PR state at execution time, never acts on a stale payload snapshot.Deliberately not included, flagged as dedicated follow-up given correctness-criticality:
check-runs/status/check-suitesREST reads to cut request volume at the source —src/github/client.tshas an explicit, deliberate design decision that these are "mutable review inputs [that] must always reflect the current GitHub state" since they feed merge/close decisions. A safe version (short TTL + webhook-driven invalidation, with explicit bypass preserved at the documented freshness-critical call sites) needs its own design pass rather than a rushed reversal of that decision.GITHUB_STATUS_ROLLUP_GRAPHQL-gated path.Scope
src/**backend changes only, config-driven, no repo-specific hardcodingValidation
npm run typecheckgit diff --checkselfhost-foreground-liveness/selfhost-pg-queue/selfhost-sqlite-queuetest suites (225 tests) greenSafety
site/,CNAME,**/lovable/**CHANGELOG.mdedit