Problem
In pull mode, the relay drain is scoped by installation, not by enrollment, and the drain is
destructive. A stale enrollment silently steals and deletes a live container's webhooks.
validateOrbRelayEnrollment (src/orb/relay.ts:220-232) resolves any valid secret to its row's
installation_id; the route then calls pullRelayPending(env, enrollment.installationId, { ack })
(src/api/routes.ts:4392). Both the SELECT and the ack-DELETE key on installation_id only
(relay.ts:422-452):
DELETE FROM orb_relay_pending WHERE installation_id = ? AND delivery_id IN (…)
SELECT … FROM orb_relay_pending WHERE installation_id = ? ORDER BY created_at, delivery_id LIMIT ?
There is no enrollment/consumer column, and forwardOrbEvent enqueues each event once per
installation. So every container holding any enrollment secret for that install drains the same queue
and ack-deletes what it took.
The push path was hardened for exactly this multi-enrollment case (relay.ts:533-544's deterministic
ORDER BY election, #1783). The pull path was not.
Trigger
Any of: enrollment-secret rotation (see the sibling issue — the old secret stays valid forever), a
blue/green container swap, or a rebuilt container issued a fresh secret while the old container is still
inside its ~minute drain timer (src/server.ts:1290-1301). The old container drains pull_request
events, acks them, and processes them under its own possibly-stale config; the new container polls an
empty queue.
Impact
Contributor PRs are silently never gated by the intended instance — no error, no DLQ, no
orb_relay_events_dropped log (that only covers the push-retry table). The events were delivered, just
to the wrong consumer. A PR that should have been closed stays open; one that should have been reviewed
never is. The losing drainer just sees "empty", which src/selfhost/monitored-work.ts:98-102 records as
healthy progress.
Independently reported by two separate audit passes, which is worth noting: the failure is invisible
enough that it reads as normal operation from every side.
Requirements
- Add
claimed_by_enroll_id to orb_relay_pending (or fan out enqueues per enrolled consumer) so each
enrollment drains its own cursor.
- At minimum, scope both the SELECT and the DELETE by
enroll_id.
- Fixing the enrollment lifecycle (sibling issue) reduces the exposure but does not close this: a
blue/green swap still produces two live consumers briefly.
- Emit a metric when more than one enrollment is live for an installation.
Test Coverage Requirements
99%+ patch coverage, branch-counted; a regression test with two live enrollments asserting each drains
only its own events.
Links & Resources
maintainer-only — webhook-ingress correctness.
Problem
In pull mode, the relay drain is scoped by installation, not by enrollment, and the drain is
destructive. A stale enrollment silently steals and deletes a live container's webhooks.
validateOrbRelayEnrollment(src/orb/relay.ts:220-232) resolves any valid secret to its row'sinstallation_id; the route then callspullRelayPending(env, enrollment.installationId, { ack })(
src/api/routes.ts:4392). Both the SELECT and the ack-DELETE key oninstallation_idonly(
relay.ts:422-452):There is no enrollment/consumer column, and
forwardOrbEventenqueues each event once perinstallation. So every container holding any enrollment secret for that install drains the same queue
and ack-deletes what it took.
The push path was hardened for exactly this multi-enrollment case (
relay.ts:533-544's deterministicORDER BYelection, #1783). The pull path was not.Trigger
Any of: enrollment-secret rotation (see the sibling issue — the old secret stays valid forever), a
blue/green container swap, or a rebuilt container issued a fresh secret while the old container is still
inside its ~minute drain timer (
src/server.ts:1290-1301). The old container drainspull_requestevents, acks them, and processes them under its own possibly-stale config; the new container polls an
empty queue.
Impact
Contributor PRs are silently never gated by the intended instance — no error, no DLQ, no
orb_relay_events_droppedlog (that only covers the push-retry table). The events were delivered, justto the wrong consumer. A PR that should have been closed stays open; one that should have been reviewed
never is. The losing drainer just sees "empty", which
src/selfhost/monitored-work.ts:98-102records ashealthy progress.
Independently reported by two separate audit passes, which is worth noting: the failure is invisible
enough that it reads as normal operation from every side.
Requirements
claimed_by_enroll_idtoorb_relay_pending(or fan out enqueues per enrolled consumer) so eachenrollment drains its own cursor.
enroll_id.blue/green swap still produces two live consumers briefly.
Test Coverage Requirements
99%+ patch coverage, branch-counted; a regression test with two live enrollments asserting each drains
only its own events.
Links & Resources
src/orb/relay.ts~220-232, ~422-452, ~533-544;src/api/routes.ts~4392;src/selfhost/monitored-work.ts~98-102;src/server.ts~1290-1301maintainer-only — webhook-ingress correctness.