fix(observability): LoopoverOrbExportErrorRateHigh runbook references a metric that doesn't exist

[JSONbored]

Context

prometheus/rules/alerts.yml defines a LoopoverOrbExportErrorRateHigh alert (in the loopover-orb
group) whose expr correctly measures the ratio of loopover_orb_export_errors_total to
loopover_orb_events_exported_total + loopover_orb_export_errors_total. Its runbook annotation,
however, tells the responding operator:

"Verify Orb API credentials/connectivity and check level=error logs around orb export.
Recorded-but-unexported events (loopover_orb_events_recorded_total vs _exported_total diverging)
confirm a stuck exporter."

loopover_orb_events_recorded_total does not exist anywhere in this codebase. Only two Orb-export
metrics are actually registered and incremented, both in src/selfhost/metrics.ts /
src/selfhost/orb-collector.ts:

• loopover_orb_events_exported_total (orb-collector.ts:232, incr("loopover_orb_events_exported_total", {}, results.length))
• loopover_orb_export_errors_total (orb-collector.ts:215, :219)

Grepping src/** for orb_events_recorded returns nothing. An operator paged by this alert who follows
the runbook literally and queries loopover_orb_events_recorded_total in Prometheus/Grafana gets an
empty series and no way to "confirm a stuck exporter" as instructed — the comparison the runbook describes
was never implemented, or the metric was renamed after the runbook text was written and the annotation
was never updated to match.

This is a dangling reference in a hand-authored diagnostic string, not something
scripts/validate-observability-configs.mjs catches — that script only checks PromQL expr fields for
balanced brackets/dangling operators (see its own doc comment), not free-text annotations.runbook/
description content, and no other test in test/unit/alerts*.test.ts cross-checks annotation text
against src/selfhost/metrics.ts's DEFAULT_METRIC_META registry.

Requirements

• Fix the LoopoverOrbExportErrorRateHigh runbook text in prometheus/rules/alerts.yml so it references
  only metrics that actually exist. The straightforward fix: since no "recorded" counter exists, rephrase
  the diagnostic to use the two real metrics already in the expr (e.g. point the operator at comparing
  loopover_orb_export_errors_total growth against loopover_orb_events_exported_total staying flat, or
  at the level=error logs already mentioned earlier in the same sentence) — do not invent and register a
  new loopover_orb_events_recorded_total metric just to make the existing prose true, unless a
  maintainer specifically wants that (out of scope for this issue; flag it as a comment instead if
  genuinely ambiguous).
• Add a regression check that would have caught this: extend the existing observability-config validation
  (scripts/validate-observability-configs.mjs and/or a new test/unit/*.test.ts) to scan every
  loopover_*-prefixed metric name mentioned inside annotations.summary / description / runbook
  strings in prometheus/rules/alerts.yml, and assert each one is either a key in
  DEFAULT_METRIC_META (src/selfhost/metrics.ts) or one of the explicitly-external prefixes already
  called out in the file's own comments (loopover_backup_*, loopover_d1_*, loopover_miner_* — see
  the comments at the top of the loopover-d1 and loopover-miner-* rule groups for the sanctioned
  external-source list).

Deliverables

• prometheus/rules/alerts.yml's LoopoverOrbExportErrorRateHigh runbook text references only
  real, currently-registered metric names.
• A new or extended test asserting every loopover_* metric name referenced in any alert's
  annotation text in prometheus/rules/alerts.yml is either registered in
  src/selfhost/metrics.ts's DEFAULT_METRIC_META or matches one of the documented external-source
  prefixes, so a future stale/renamed metric reference in annotation prose fails CI.

Test Coverage Requirements

prometheus/rules/alerts.yml is a YAML config file outside src/**/coverage.include, so the Codecov
patch gate does not apply to the YAML edit itself. The new/extended validation code (wherever it lands —
scripts/validate-observability-configs.mjs is currently untested-by-Codecov-patch only insofar as it's
invoked via npm run selfhost:validate-observability; if the new check is added as TS/JS under a path
Codecov measures, or as a test/unit/** spec) must hit 99%+ patch coverage: both the pass case (current
alerts.yml, no dangling references) and a fail case (a fixture alert file with one annotation
referencing a fabricated metric name) to prove the check actually fails when it should.

Expected Outcome

An operator paged by LoopoverOrbExportErrorRateHigh gets a runbook that names metrics they can actually
query. A future PR that renames or removes a loopover_* metric without updating every alert annotation
that mentions it by name fails CI instead of shipping another silent dead reference.

Links & Resources

• prometheus/rules/alerts.yml (the loopover-orb rule group, LoopoverOrbExportErrorRateHigh alert)
• src/selfhost/metrics.ts (DEFAULT_METRIC_META, the registered-metric source of truth)
• src/selfhost/orb-collector.ts (lines ~215, ~219, ~232 — the two real Orb-export counters)
• scripts/validate-observability-configs.mjs (existing PromQL-syntax-only validator; explicitly scoped
  out of semantic/annotation checks per its own module doc comment)
• test/unit/alerts-job-failure-ratio-formula.test.ts and sibling test/unit/alerts-*.test.ts files for
  the existing pattern of testing individual alert-rule formulas

[linear-code]

JSO-4000