Context
Found during the 2026-07-06 incident audit (parent: #1667). drainOrbRelayWithMonitor (src/selfhost/monitored-work.ts) iterates over relayed events and calls args.enqueue(...) for each with no try/catch around the call. enqueueWebhookByEnv swallows its own anticipated failures (missing queue binding, queue send failure) and returns a string result rather than throwing — but two of its recordWebhookEvent calls are not wrapped in try/catch, so a D1/Postgres write failure there (exactly the class of failure a D1-near-cap or Postgres-blip scenario produces) throws uncaught out of enqueueWebhookByEnv, up through this for-loop, aborting the entire remaining batch for that drain tick. Every event still in the batch after the failing one is never attempted this tick.
This is a distinct bug from the "malformed event silently filtered" issue (already covered in #3812) — this one is a downstream write failure propagating up and killing sibling events that were otherwise perfectly valid.
Requirements
- Wrap the
args.enqueue(...) call inside drainOrbRelayWithMonitor's loop in its own try/catch, so one event's failure logs + increments a per-event failure metric and continues to the next event, instead of throwing out of the whole batch.
- Ensure a per-event enqueue exception does not get added to
pendingAck (so it is retried on the next drain, consistent with the existing "enqueue_failed" non-throwing handling).
- Add a test asserting that when
enqueue throws for the 2nd of 3 events in a drain batch, events 1 and 3 both still get processed/acked appropriately rather than the whole tick aborting after event 2.
Deliverables
src/selfhost/monitored-work.ts: try/catch around the enqueue call in drainOrbRelayWithMonitor's loop.test/unit/selfhost-monitored-work.test.ts: new case for a mid-batch enqueue throw.
Expected outcome
A single event's downstream write failure during pull-mode relay drain no longer silently discards every subsequent event in that same batch; each event is isolated and independently retried or acked.
Context
Found during the 2026-07-06 incident audit (parent: #1667).
drainOrbRelayWithMonitor(src/selfhost/monitored-work.ts) iterates over relayed events and callsargs.enqueue(...)for each with no try/catch around the call.enqueueWebhookByEnvswallows its own anticipated failures (missing queue binding, queue send failure) and returns a string result rather than throwing — but two of itsrecordWebhookEventcalls are not wrapped in try/catch, so a D1/Postgres write failure there (exactly the class of failure a D1-near-cap or Postgres-blip scenario produces) throws uncaught out ofenqueueWebhookByEnv, up through this for-loop, aborting the entire remaining batch for that drain tick. Every event still in the batch after the failing one is never attempted this tick.This is a distinct bug from the "malformed event silently filtered" issue (already covered in #3812) — this one is a downstream write failure propagating up and killing sibling events that were otherwise perfectly valid.
Requirements
args.enqueue(...)call insidedrainOrbRelayWithMonitor's loop in its own try/catch, so one event's failure logs + increments a per-event failure metric and continues to the next event, instead of throwing out of the whole batch.pendingAck(so it is retried on the next drain, consistent with the existing "enqueue_failed" non-throwing handling).enqueuethrows for the 2nd of 3 events in a drain batch, events 1 and 3 both still get processed/acked appropriately rather than the whole tick aborting after event 2.Deliverables
src/selfhost/monitored-work.ts: try/catch around the enqueue call indrainOrbRelayWithMonitor's loop.test/unit/selfhost-monitored-work.test.ts: new case for a mid-batch enqueue throw.Expected outcome
A single event's downstream write failure during pull-mode relay drain no longer silently discards every subsequent event in that same batch; each event is isolated and independently retried or acked.