orb(webhook): 5,547 deliveries stuck 'queued' are permanently un-redeliverable — 4,900 are check_suite.completed, the auto-merge trigger

[JSONbored]

Verified on the live box

status      | count
------------+--------
processed   | 303074
queued      |   5555
superseded  |    901
error       |    101

5,547 rows are status='queued' with processed_at IS NULL, broken down:

event	action	count	oldest	newest
check_suite	completed	4900	2026-06-29	2026-07-23
pull_request	labeled	318	2026-06-29	2026-07-13
pull_request	synchronize	139	2026-06-29	2026-07-06
pull_request	opened	114	2026-06-29	2026-07-03
pull_request	unlabeled / edited / reopened / ready_for_review	76

Good news: it is quiescent. Newest stuck row is 2026-07-23, and the last 2 hours show 398 processed, 0 stuck. This was a cutover-era incident, not an active one.

Bad news: none of it can ever be recovered, and if it recurs the same trap applies.

The trap

src/github/webhook.ts ~164:

if (existingEvent && existingEvent.status !== "error" &&
    (existingEvent.status === "processed" || existingEvent.payloadHash === payloadHash))
  return "duplicate";

The queued row is written before env.WEBHOOKS.send(message) (~199-205). If the process dies in that window — or the queued job is lost — the row sits at queued with the real payload hash forever. Every GitHub retry and every manual "Redeliver" click hits this check and is discarded with 202 {status:"duplicate"}. The operator's only recovery tool is silently a no-op. superseded (written by pg-queue.ts ~816) fails the same !== "error" test.

There is no watchdog anywhere sweeping webhook_events WHERE status='queued'.

Why 4,900 check_suite.completed matters

maybeReReviewOnCiCompletion is documented in-code as "THE auto-merge / close-on-red TRIGGER" (src/queue/processors.ts ~4646-4652). Nearly five thousand dropped CI-completion events over ~3.5 weeks means a large number of PRs never got their post-CI disposition from the event path and had to wait for the ~2-minute sweep — or never got it at all. This is very likely a major contributor to the historical "ORB isn't handling its work" experience.

Fix

1. Treat queued/superseded rows older than a bound (say received_at > 10 min with no processed_at) as re-enqueueable rather than duplicates.
2. Add a cron sweep that re-sends queued rows having no matching pending/processing queue row.
3. Never suppress a delivery that carries GitHub's redelivery marker — a human clicking Redeliver must always win.
4. One-off: decide whether to replay the 4,900 historical check_suite.completed rows (most of their PRs are long resolved; probably just purge them so the metric is meaningful again).

Acceptance

• Killing the process between the queued-insert and the send, then redelivering from GitHub, results in the event being processed.

[linear-code]

JSO-6260