Context
Found during the 2026-07-06 incident audit (parent: #1667). During today's incident, the scheduled re-gate sweep silently stopped producing audit events for JSONbored/gittensory and JSONbored/metagraphed specifically for over an hour — while it kept succeeding for every other registered repo — and nothing in the system noticed. The operator had to manually query raw Postgres for ~2 hours to discover this, then manually INSERT rows directly into _selfhost_jobs to force recovery, bypassing every normal enqueue path.
Confirmed by exhaustive search: agent.sweep.regate/agent.sweep.fanout audit events (src/queue/processors.ts) are never read back by any dashboard, alert, or ops route anywhere in the repo. The existing anomaly-alerting module (src/review/alerts.ts, detectAnomalies) covers manual-rate spikes, DLQ counts, calibration drift, and disputed closes — nothing about the sweep/fan-out trigger itself going quiet. src/selfhost/sentry.ts's SENTRY_MONITOR_NAMES is a fixed 5-name list with no per-repo dimension — the regate sweep runs as ordinary logic inside the single scheduled-loop monitor, which only reports "error" if the whole 2-minute tick function throws. A sweep that silently produces zero candidates/dispatches for one specific repo does not throw — it just does nothing — so the scheduled-loop check-in reports "ok" every tick regardless of per-repo health.
Requirements
- Add a periodic watchdog (piggybacking the existing ~2-min cron tick or its own schedule) that, for every repo eligible for the sweep (the same set
fanOutAgentRegateSweepJobs already computes), checks whether that repo has produced any agent.sweep.regate/agent.sweep.fanout audit event within a configurable threshold (e.g. 3x expected cadence, default ~30-60 min).
- When a repo is found stale beyond the threshold and has open, non-draft PRs eligible for regate, emit a structured Sentry-forwarded error naming the repo and how long it's been silent.
- On the same detection, directly enqueue a fresh, targeted
agent-regate-sweep job scoped to that specific repo — this is the actual self-heal action replacing the manual _selfhost_jobs INSERT the operator had to perform today.
- Wrap each repo's check in its own try/catch so a DB blip checking one repo doesn't stop the watchdog checking others.
- Add a gauge (e.g.
gittensory_regate_sweep_seconds_since_last{repo}) and a Prometheus alert rule + Grafana panel so a stalled repo is visible proactively, not only via a Sentry page.
- Distinguish "sweep ran, found 0 stale candidates" (healthy) from "sweep did not run this tick" (the actual failure) so a quiet repo with no work isn't falsely flagged.
Deliverables
- New watchdog function (e.g.
checkAgentRegateSweepLiveness or a new src/review/sweep-watchdog.ts), wired into the existing scheduled cron in src/index.ts.
- New DB read helper for the most recent
agent.sweep.* audit row per repo.
src/selfhost/metrics.ts: new gauge metric.prometheus/rules/alerts.yml: new alert rule.grafana/dashboards/gittensory.json: new panel.- Unit tests: (a) a repo stale beyond threshold triggers the Sentry log + targeted re-enqueue; (b) a repo with a recent audit event (including a legitimate "denied"/"queued" one) does not false-alarm; (c) a repo with genuinely zero open PRs is not flagged as broken.
Expected outcome
A repo whose scheduled sweep silently stops is detected within one watchdog cycle, gets both a Sentry-visible alert and an automatic targeted re-enqueue of its sweep — closing the exact gap that required manual database surgery today. This also directly covers the discovery of orphaned PRs like #3782/#3793 (zero audit_events, zero DB row) that a stalled sweep would otherwise never find.
Context
Found during the 2026-07-06 incident audit (parent: #1667). During today's incident, the scheduled re-gate sweep silently stopped producing audit events for JSONbored/gittensory and JSONbored/metagraphed specifically for over an hour — while it kept succeeding for every other registered repo — and nothing in the system noticed. The operator had to manually query raw Postgres for ~2 hours to discover this, then manually INSERT rows directly into
_selfhost_jobsto force recovery, bypassing every normal enqueue path.Confirmed by exhaustive search:
agent.sweep.regate/agent.sweep.fanoutaudit events (src/queue/processors.ts) are never read back by any dashboard, alert, or ops route anywhere in the repo. The existing anomaly-alerting module (src/review/alerts.ts,detectAnomalies) covers manual-rate spikes, DLQ counts, calibration drift, and disputed closes — nothing about the sweep/fan-out trigger itself going quiet.src/selfhost/sentry.ts'sSENTRY_MONITOR_NAMESis a fixed 5-name list with no per-repo dimension — the regate sweep runs as ordinary logic inside the singlescheduled-loopmonitor, which only reports "error" if the whole 2-minute tick function throws. A sweep that silently produces zero candidates/dispatches for one specific repo does not throw — it just does nothing — so the scheduled-loop check-in reports "ok" every tick regardless of per-repo health.Requirements
fanOutAgentRegateSweepJobsalready computes), checks whether that repo has produced anyagent.sweep.regate/agent.sweep.fanoutaudit event within a configurable threshold (e.g. 3x expected cadence, default ~30-60 min).agent-regate-sweepjob scoped to that specific repo — this is the actual self-heal action replacing the manual_selfhost_jobsINSERT the operator had to perform today.gittensory_regate_sweep_seconds_since_last{repo}) and a Prometheus alert rule + Grafana panel so a stalled repo is visible proactively, not only via a Sentry page.Deliverables
checkAgentRegateSweepLivenessor a newsrc/review/sweep-watchdog.ts), wired into the existing scheduled cron insrc/index.ts.agent.sweep.*audit row per repo.src/selfhost/metrics.ts: new gauge metric.prometheus/rules/alerts.yml: new alert rule.grafana/dashboards/gittensory.json: new panel.Expected outcome
A repo whose scheduled sweep silently stops is detected within one watchdog cycle, gets both a Sentry-visible alert and an automatic targeted re-enqueue of its sweep — closing the exact gap that required manual database surgery today. This also directly covers the discovery of orphaned PRs like #3782/#3793 (zero audit_events, zero DB row) that a stalled sweep would otherwise never find.