orb(observability): the queue-backlog alerts deactivate during the DB incident they exist to catch, and both DLQ-rate surfaces are structurally always zero

[JSONbored]

Problem

Two distinct mechanisms make ORB's queue and DLQ alerting go quiet at exactly the moment it should fire.

1. Every queue-backlog alert deactivates during a DB incident

renderMetrics wraps each gauge sampler in a bare swallow (src/selfhost/metrics.ts:326-334):

try { … } catch { /* a failing sampler must not break the scrape */ }

A throwing sampler emits no series at all — no error counter, no effect on up. Every queue gauge is
a live DB read: gauge("loopover_queue_pending", () => backend.queue.size()), and
pressureSignals() → maintenancePressureSignals(Date.now()) → pool.query(...)
(src/server.ts:846-847, 873-889; src/selfhost/pg-queue.ts:1618-1620).

So when Postgres is down, the pool is exhausted, or statements time out: /metrics still returns 200
(counters and non-DB gauges render), so up{job="loopover"}==0 never fires and LoopoverTargetDown
stays inactive. Meanwhile loopover_queue_pending, loopover_queue_live_pending,
loopover_queue_live_runnable_now, loopover_queue_oldest_*_age_seconds and loopover_queue_dead all
vanish, so LoopoverQueueBacklogHigh, LoopoverLiveQueueStarved, LoopoverLiveQueueNoRunnableWork,
LoopoverMaintenanceStarved, and LoopoverDeadLetterBacklogHigh/Persisting
(prometheus/rules/alerts.yml:89,106,122,139,155,185) all evaluate over an empty set and go inactive.
Grafana reads "No data", indistinguishable from idle.

2. Both DLQ-rate surfaces are structurally always zero

• src/review/ops.ts:242,250 reads review_audit.event_type = 'dead_lettered'. The two writers of
  review_audit emit 'gate_decision' (parity-wire.ts:184) and the outcome/reversal types
  (outcomes-wire.ts:293). Nothing writes 'dead_lettered' — only these readers and a test that
  mocks the row. So h.dlqCount is permanently 0 and the operator alert "⚠️ N review(s)
  DEAD-LETTERED — these were dropped and need a re-queue" (src/review/alerts.ts:190-195) can never
  fire.
• loopover_dlq_dead_lettered_recent (src/server.ts:848 → src/selfhost/dlq-recent.ts:16 →
  src/db/repositories.ts:3389) counts github_app.dlq_dead_lettered audit events, written only by
  processDlqBatch (src/queue/dlq.ts:41), which is reached only from the Cloudflare queue() handler
  (src/index.ts:55). server.ts calls worker.fetch and worker.scheduled but never
  worker.queue — the self-host runtime dead-letters via UPDATE … SET status='dead' plus a stdout
  line (pg-queue.ts:1420-1433). The gauge is a constant 0 on every ORB instance.

Note the same gating means the webhook self-heal re-drive in dlq.ts:49-67 — explicitly gated on
isSelfHostedReviewRuntime(env) — is never executed by the self-host process either.

Impact

The six alerts that exist to catch "the review queue has stopped moving" switch off when it stops moving,
and both dedicated DLQ-rate surfaces are dead. Only the raw loopover_queue_dead depth gauge and
loopover_jobs_dead_total survive — and the former is one of the gauges that vanishes under (1).

Requirements

• Increment loopover_metrics_sampler_errors_total{metric} in the catch block and alert on it; emit a
  -1 sentinel series so a gauge's absence is itself visible.
• Add a generic up == 0 for 10m rule by (job) — today only job="loopover" has one, so all ten other
  scrape jobs (postgres, backup, qdrant, redis, node-exporter, cadvisor, …) go silently inert when their
  exporter dies. The backup case is the sharpest: LoopoverBackupMissing requires the exporter to be up
  and reporting zero, so a dead backup exporter means backups can stop with no alert. Add
  absent(loopover_backup_files) alongside it.
• Write a review_audit row with event_type='dead_lettered' from the self-host dead-letter path, or
  repoint the alert at loopover_jobs_dead_total.
• Back loopover_dlq_dead_lettered_recent with the self-host queue's own dead counter.
• Decide whether the dlq.ts webhook self-heal should run on self-host; today it cannot.
• Fix alertmanager/alertmanager.yml:175, whose inhibit example matches alertname="LoopOverTargetDown"
  (capital O) against a real rule named LoopoverTargetDown — an operator who uncomments it gets a rule
  that silently never inhibits. Extend test/unit/alerts-metric-name-references.test.ts to assert every
  alertname="…" in alertmanager.yml, including commented blocks, resolves to a real rule.

Test Coverage Requirements

99%+ patch coverage, branch-counted; both arms of the sampler error path, and a test that a self-host
dead-letter increments the surface the alert reads.

Links & Resources

• src/selfhost/metrics.ts ~39, ~326-334; src/server.ts ~846-848, ~873-889;
  src/selfhost/pg-queue.ts ~1420-1443, ~1618-1620; src/review/ops.ts ~242, ~250;
  src/review/alerts.ts ~190-195; src/queue/dlq.ts ~41, ~49-67; src/db/repositories.ts ~3389;
  prometheus/rules/alerts.yml ~33, ~89-185, ~316, ~328; alertmanager/alertmanager.yml ~175

maintainer-only — alerting integrity.

[linear-code]

JSO-6314