Skip to content

fix(queue): prioritize live PR work under GitHub rate pressure - #2847

Merged
JSONbored merged 1 commit into
mainfrom
claude/queue-rate-pressure-priority
Jul 3, 2026
Merged

fix(queue): prioritize live PR work under GitHub rate pressure#2847
JSONbored merged 1 commit into
mainfrom
claude/queue-rate-pressure-priority

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Add a ramp-up cap (FOREGROUND_LIVENESS_MAX_RELEASE_PER_SWEEP, default 25) to the foreground-liveness release sweep from fix(selfhost): keep review queue live and verify configured CI #2843: a large inherited backlog of over-deferred foreground jobs (a production incident showed ~190) now drains gradually, oldest-first, over several sweep ticks instead of releasing everything into one instant — releasing all of it at once lets every job re-attempt GitHub simultaneously and can immediately re-trip the same rate-limit bucket it was deferred for.

Investigation findings (scope narrowed accordingly)

Most of this issue's originally-requested scope turned out to already be handled:

  • Current-head agent-regate-pr reclassification (live/foreground vs. scheduled-sweep background) and condition-based sticky-cooldown recovery were already shipped in fix(selfhost): keep review queue live and verify configured CI #2843.
  • Required-status (branch-protection) policy is already cached durably (GITHUB_BRANCH_PROTECTION_CACHE_TTL_SECONDS, ~20m default) — verified in src/github/client.ts.
  • Webhook/regate coalescing is already correctly head-SHA-scoped: githubWebhookCoalesceKey embeds head_sha for check_run/check_suite/pull_request events (verified in code + existing tests), and agent-regate-pr intentionally coalesces by repo+PR only, which is correct for that job's semantics — it always re-reads live PR state at execution time, never acts on a stale payload snapshot.

Deliberately not included, flagged as dedicated follow-up given correctness-criticality:

  • Per-installation concurrent GitHub-fetch scheduling — queue-wide concurrency exists; per-installation tracking is a new coordination mechanism worth its own design.
  • Caching check-runs/status/check-suites REST reads to cut request volume at the source — src/github/client.ts has an explicit, deliberate design decision that these are "mutable review inputs [that] must always reflect the current GitHub state" since they feed merge/close decisions. A safe version (short TTL + webhook-driven invalidation, with explicit bypass preserved at the documented freshness-critical call sites) needs its own design pass rather than a rushed reversal of that decision.
  • Expanding GraphQL rollup coverage beyond the existing GITHUB_STATUS_ROLLUP_GRAPHQL-gated path.

Scope

  • src/** backend changes only, config-driven, no repo-specific hardcoding
  • Small, focused diff (7 files)

Validation

  • npm run typecheck
  • git diff --check
  • Full selfhost-foreground-liveness / selfhost-pg-queue / selfhost-sqlite-queue test suites (225 tests) green
  • 100% line and branch coverage on every added line

Safety

  • No secrets/wallet/hotkey/trust-score/reward terms introduced
  • No changes to site/, CNAME, **/lovable/**
  • No CHANGELOG.md edit

@JSONbored JSONbored self-assigned this Jul 3, 2026
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
gittensory-ui 1f3acaa Commit Preview URL

Branch Preview URL
Jul 03 2026, 10:40 PM

Adds a bounded ramp-up cap to the foreground-liveness release sweep
(#selfhost-queue-liveness): when a large backlog of over-deferred
foreground jobs has accumulated (a production incident showed ~190),
releasing all of it in one sweep tick lets every released job
re-attempt GitHub at once and immediately re-trip the same rate-limit
bucket it was deferred for. FOREGROUND_LIVENESS_MAX_RELEASE_PER_SWEEP
(default 25) now bounds each sweep to the oldest-first N eligible
jobs, draining a large backlog gradually over several ticks instead
of in one burst.

Investigation into the rest of this issue's scope found most of it
already addressed:
- Current-head agent-regate-pr reclassification and sticky-cooldown
  recovery were shipped in the prior self-host queue-liveness PR.
- Required-status (branch-protection) policy is already cached
  durably (GITHUB_BRANCH_PROTECTION_CACHE_TTL_SECONDS, ~20m default).
- Webhook/regate coalescing is already correctly head-SHA-scoped
  (githubWebhookCoalesceKey embeds head_sha for check_run/check_suite/
  pull_request events; agent-regate-pr coalesces by repo+PR only,
  which is correct for that job's semantics since it always reads
  live PR state at execution time, never a stale snapshot).

Deliberately NOT included in this PR, flagged for dedicated follow-up
given their correctness-criticality:
- Per-installation concurrent GitHub-fetch scheduling (queue-wide
  concurrency exists; per-installation tracking would be a new
  coordination mechanism).
- Caching check-runs/statuses/check-suites REST reads: the existing
  client has an explicit, deliberate design decision that these are
  "mutable review inputs [that] must always reflect the current
  GitHub state" (src/github/client.ts), since they feed merge/close
  decisions. A safe version (short TTL + webhook-driven invalidation,
  explicit bypass preserved at the documented freshness-critical call
  sites) needs a dedicated design pass, not a rushed reversal of that
  decision.
- Expanding GraphQL rollup coverage beyond the existing
  GITHUB_STATUS_ROLLUP_GRAPHQL-gated path.

Validated: npm run typecheck, git diff --check, and the full
foreground-liveness/pg-queue/sqlite-queue test suites (225 tests)
all green. Every added line has 100% line and branch coverage.
@JSONbored
JSONbored force-pushed the claude/queue-rate-pressure-priority branch from 1f3acaa to 2813cc1 Compare July 3, 2026 22:41
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.22%. Comparing base (48b8a06) to head (2813cc1).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2847   +/-   ##
=======================================
  Coverage   96.22%   96.22%           
=======================================
  Files         257      257           
  Lines       28114    28124   +10     
  Branches    10218    10219    +1     
=======================================
+ Hits        27053    27063   +10     
  Misses        435      435           
  Partials      626      626           
Files with missing lines Coverage Δ
src/selfhost/foreground-liveness.ts 100.00% <100.00%> (ø)
src/selfhost/sqlite-queue.ts 99.52% <100.00%> (+<0.01%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit fa35528 into main Jul 3, 2026
12 checks passed
@JSONbored
JSONbored deleted the claude/queue-rate-pressure-priority branch July 3, 2026 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant