A PR that cannot progress is re-evaluated forever, at full cost
JSONbored/metagraphed#8886, measured on the Orb:
action | reason_code | count | distinct head_sha | first_seen | last_seen
hold | missing_linked_issue | 56 | 1 | 09:49:17 | 10:36:56
56 evaluations of one unchanged commit in 47 minutes — about 1.2 per minute — every one producing the identical verdict. The PR is CONFLICTING, so it cannot merge; it is held, so it does not close; its linked issue was closed by a merged rival (#10168), so the hold never clears. It is in a state it can never leave, and nothing throttles re-entry.
It is not alone. Four PRs produced 115 of 173 decision records (66%) in the post-deploy window:
metagraphed#8886 56
loopover#10157 23
loopover#10155 21 (the label-flap PR, #10164)
loopover#10134 15
This distorts the metrics we steer by
Like-for-like windows:
| window |
records |
distinct heads |
records/head |
| 07:00–09:25 |
107 |
66 |
1.62 |
| 09:25–11:16 |
173 |
42 |
4.12 |
I initially read that as a regression from the coalescing work (#10127). It is not — it is a handful of stuck PRs looping. But it means records_per_head is not a usable health metric while unbounded re-evaluation exists: a few pathological PRs dominate it and mask whatever the fleet is really doing.
Why the existing coalescing does not catch it
The webhook coalescer (#10127) collapses bursts within a window. This is not a burst — it is a steady ~1.2/min drip driven by ordinary events (CI completions, label writes, sibling activity) on a PR whose answer cannot change. Each arrives after the window and is legitimately a distinct delivery.
The missing control is different in kind: not "collapse simultaneous events" but "stop asking a question whose answer has not changed".
Proposal: verdict-stability backoff
When the last N evaluations for a (repo, pull, head_sha) produced the same action + reason_code, back off further re-evaluation of that triple — exponentially, to a cap — and reset immediately on any of:
- a new
head_sha (new commit ⇒ genuinely new question)
- a maintainer action (
reevaluation_actor, maintainer_request)
- a change in the decisive inputs the verdict keyed on
The safety property to preserve: backoff must never make a PR unreachable. A ceiling (say the existing sweep cadence) means a stuck PR is still revisited, just not 1.2×/min.
Note for whoever implements it
record_digest cannot be used to detect "same verdict" — #8886 has 56 distinct digests for 56 identical verdicts, because the digest carries per-evaluation data. Sameness has to be judged on the decision fields (action, reason_code, and now hold_cause), not on the digest.
Value
This caps churn for every stuck PR rather than fixing causes one at a time. #10164 (flap) and #10168 (superseded) each remove one source; this bounds the blast radius of the next one nobody has found yet — and today's rate-limit exhaustion was exactly that blast radius arriving.
Found while re-measuring #10127's effect, which is the only reason the domination by four PRs was visible at all.
A PR that cannot progress is re-evaluated forever, at full cost
JSONbored/metagraphed#8886, measured on the Orb:
56 evaluations of one unchanged commit in 47 minutes — about 1.2 per minute — every one producing the identical verdict. The PR is
CONFLICTING, so it cannot merge; it is held, so it does not close; its linked issue was closed by a merged rival (#10168), so the hold never clears. It is in a state it can never leave, and nothing throttles re-entry.It is not alone. Four PRs produced 115 of 173 decision records (66%) in the post-deploy window:
This distorts the metrics we steer by
Like-for-like windows:
I initially read that as a regression from the coalescing work (#10127). It is not — it is a handful of stuck PRs looping. But it means
records_per_headis not a usable health metric while unbounded re-evaluation exists: a few pathological PRs dominate it and mask whatever the fleet is really doing.Why the existing coalescing does not catch it
The webhook coalescer (#10127) collapses bursts within a window. This is not a burst — it is a steady ~1.2/min drip driven by ordinary events (CI completions, label writes, sibling activity) on a PR whose answer cannot change. Each arrives after the window and is legitimately a distinct delivery.
The missing control is different in kind: not "collapse simultaneous events" but "stop asking a question whose answer has not changed".
Proposal: verdict-stability backoff
When the last N evaluations for a
(repo, pull, head_sha)produced the sameaction+reason_code, back off further re-evaluation of that triple — exponentially, to a cap — and reset immediately on any of:head_sha(new commit ⇒ genuinely new question)reevaluation_actor,maintainer_request)The safety property to preserve: backoff must never make a PR unreachable. A ceiling (say the existing sweep cadence) means a stuck PR is still revisited, just not 1.2×/min.
Note for whoever implements it
record_digestcannot be used to detect "same verdict" — #8886 has 56 distinct digests for 56 identical verdicts, because the digest carries per-evaluation data. Sameness has to be judged on the decision fields (action,reason_code, and nowhold_cause), not on the digest.Value
This caps churn for every stuck PR rather than fixing causes one at a time. #10164 (flap) and #10168 (superseded) each remove one source; this bounds the blast radius of the next one nobody has found yet — and today's rate-limit exhaustion was exactly that blast radius arriving.
Found while re-measuring #10127's effect, which is the only reason the domination by four PRs was visible at all.