Skip to content

fix(queue): mark superseded webhook_events rows instead of leaving them stuck at 'queued' forever #3814

Description

@JSONbored

Context

Found during the 2026-07-06 incident audit (parent: #1667). pull_request "opened" is a coalescable action, keyed as github-webhook:pr-refresh:{repo}#{pr}@{headSha}. The Postgres queue's enqueue() coalesces a new message with the same job_key into an existing pending row by fully overwriting its payload — the old row's original deliveryId (and its associated webhook event) is discarded entirely; only the new message's deliveryId will ever be processed and get its webhook_events row advanced to status='processed'. The superseded delivery's webhook_events row (already written as status='queued' before coalescing happens) is left at queued permanently, with nothing ever marking it processed, errored, or superseded.

This is a narrower audit-completeness gap than the D1-failure case (separate issue), but a real one: an operator grepping webhook_events for entries stuck at queued finds real rows with no explanation, and nothing distinguishes "silently coalesced away, handled by a later delivery" from "actually lost."

Requirements

  • When enqueue() coalesces a new message into an existing pending row by job_key, also update the superseded message's own webhook_events row (looked up by its deliveryId, obtainable from the old payload before it's overwritten) to a distinct terminal status (e.g. superseded) rather than leaving it at queued forever.
  • Add a regression test: enqueue two coalescable pull_request/opened-family webhooks for the same repo#pr@headSha before either is claimed; assert the first delivery's webhook_events row ends at a terminal, explanatory status.

Deliverables

  • src/selfhost/pg-queue.ts: mark the coalesced-away deliveryId's webhook_events row as superseded at each coalesce site.
  • src/db/repositories.ts: extend the webhook_events status type if a new superseded status is added.
  • test/unit/selfhost-pg-queue.test.ts (or equivalent): new coalesce-supersede regression case.

Expected outcome

No webhook_events row is left permanently ambiguous at queued after its underlying job was silently overwritten by a coalesced redelivery — every delivery ends at an explanatory terminal status.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions