orb(observability): the relay-drain staleness alarm is suppressed by its own worst failure — 'never drained' reads -1, and a deploy silences a firing alert

[JSONbored]

Problem

If the relay drain has never once succeeded, the staleness alarm reads -1 and cannot fire. The
alarm is suppressed by its own worst failure mode — total webhook starvation.

lastDrainAtMs is stamped only inside the success path, after await args.drain(...) returns
(src/selfhost/monitored-work.ts:103). The gauge maps "never drained" to a sentinel:

src/server.ts:1259-1261:  relayDrainState?.lastDrainAtMs == null ? -1 : …

and the Prometheus rule is (prometheus/rules/alerts.yml:409):

loopover_orb_relay_register_consecutive_failures >= 3 or loopover_orb_relay_drain_seconds_since_last > 1800

-1 is not > 1800. The in-app escalation gate has the identical hole:

src/selfhost/monitored-work.ts:230:  if (args.drainLastAtMs === null) return false;   // "insufficient signal to escalate"

loopover_orb_relay_drains_total also increments only on success, so there is no failure counter to
notice either, and consecutiveFailures tracks registration, not drain.

Trigger

Pull mode; registration succeeds (broker reachable) but every drain tick throws — a 4xx on the drain
endpoint, a broker-side schema change, an ack-payload rejection. consecutiveFailures stays 0,
lastDrainAtMs stays null forever, both alarms stay quiet.

There is a second, more routine path: the relay was already broken and the container restarted (any
deploy). The previously-climbing seconds_since_last gauge resets to -1 and a firing alert goes
quiet. Given ORB's deploy cadence, a real outage can be silenced by the next deploy.

Impact

Zero GitHub webhooks reach ORB: no PRs reviewed, no gate decisions, no auto-merge, no auto-close — and
neither the Prometheus rule nor isOrbRelayRegistrationAlerting says anything. This is the highest-blast-
radius silent failure in the runtime, and the monitor for it is off precisely when it matters.

Requirements

• Count drain failures: increment loopover_orb_relay_drains_total{result="failed"} (or a dedicated
  loopover_orb_relay_drain_consecutive_failures) on the throw path.
• Make "never drained since boot" alertable — emit seconds-since-process-start when lastDrainAtMs
  is null rather than -1, so a never-drained instance ages into the existing threshold naturally.
• Give monitored-work.ts:230 the same treatment: never-drained past a boot grace period must escalate,
  not return false.
• Audit the other -1-sentinel gauges for the same shape — loopover_clock_skew_sample_age_seconds is
  a confirmed sibling (referenced by zero alert rules and zero dashboard panels).

Test Coverage Requirements

99%+ patch coverage, branch-counted. Both arms of never-drained / previously-drained, plus a regression
test that a drain which only ever throws produces a firing condition.

Links & Resources

• src/server.ts ~1259-1261; src/selfhost/monitored-work.ts ~103, ~230; prometheus/rules/alerts.yml ~409
• Same class as orb(review): forceAiReview does not bypass the AI-review lock — the re-run button is silently inert behind an orphaned lock (root cause of #9000) #9008 (a placeholder satisfying the presence check that suppresses the alarm for its own
  absence) — worth fixing as a pattern, not a one-off.

Boundaries

Relay drain observability only. No change to drain/ack semantics (the multi-enrollment drain-stealing
defect is a separate issue).

maintainer-only — observability on the webhook-ingress critical path.

[linear-code]

JSO-6303