Context
Found during the 2026-07-06 incident audit (parent: #1667). In fanOutAgentRegateSweepJobs (src/queue/processors.ts), the per-repo candidate loop calls resolveRepositorySettings(env, repoFullName) and getLatestRegatedAt(env, repoFullName) with no try/catch. If either throws for any single repo (a transient DB read hiccup, a malformed settings row — exactly the class of failure likely during a D1-near-cap or Postgres-blip window), the exception propagates out of the whole function: the Promise.all fan-out never runs, no repo gets a sweep job that tick, and the agent.sweep.fanout audit event is never written — so there's no record the tick even attempted to run.
The identical operation is already hardened elsewhere in this codebase: opsScanRepos (src/review/ops-wire.ts) wraps the same resolveRepositorySettings call in a per-repo try/catch specifically because "a settings blip on one repo must not abort the whole scan." This fix is missing here.
On the self-hosted Postgres queue, a top-level agent-regate-sweep trigger job hitting this unguarded throw repeatedly can burn through its full retry + dead-letter-revive budget (~60-90 min) before permanently dying — requiring exactly the kind of manual _selfhost_jobs row-insert the operator had to perform during today's incident to force recovery.
Requirements
- Wrap the per-repo body of the candidate loop (settings resolution,
isConvergenceRepoAllowed/isAgentConfigured check, getLatestRegatedAt/isRegateSweepDraining check) in a try/catch mirroring opsScanRepos's pattern.
- Log a structured error (reaching the existing
forwardStructuredLogToSentry pipeline) on a per-repo skip-due-to-error, and skip only that repo for the tick — every other repo in the candidate set must still get evaluated and fanned out normally.
- Add a
skippedErrored counter alongside the existing skippedDraining, included in the agent.sweep.fanout audit event's metadata, so a partial-failure tick is visible in the audit trail instead of looking identical to a fully healthy one.
- Audit sibling fan-out functions with the same shape (
fanOutBacklogConvergenceSweepJobs, fanOutRepoDocRefreshSweepJobs, the signal-snapshot fan-out) for the same unguarded-per-repo-call pattern and fix consistently if found.
Deliverables
- Per-repo try/catch in
fanOutAgentRegateSweepJobs's configuration loop (src/queue/processors.ts).
- Regression test in
test/unit/queue.test.ts: one repo's settings/DB read rejects among several configured repos; assert the other repos still get their sweep job and the fan-out audit event still gets written.
- Same audit + fix applied to sibling fan-out functions if the same gap exists.
Expected outcome
A transient DB/settings read failure for one repo no longer silently zeroes out the entire tick's sweep for every other repo on the instance. Any repo skipped due to error is visible via a Sentry-forwarded log line and an audit-event counter instead of vanishing with no trace.
Context
Found during the 2026-07-06 incident audit (parent: #1667). In
fanOutAgentRegateSweepJobs(src/queue/processors.ts), the per-repo candidate loop callsresolveRepositorySettings(env, repoFullName)andgetLatestRegatedAt(env, repoFullName)with no try/catch. If either throws for any single repo (a transient DB read hiccup, a malformed settings row — exactly the class of failure likely during a D1-near-cap or Postgres-blip window), the exception propagates out of the whole function: thePromise.allfan-out never runs, no repo gets a sweep job that tick, and theagent.sweep.fanoutaudit event is never written — so there's no record the tick even attempted to run.The identical operation is already hardened elsewhere in this codebase:
opsScanRepos(src/review/ops-wire.ts) wraps the sameresolveRepositorySettingscall in a per-repo try/catch specifically because "a settings blip on one repo must not abort the whole scan." This fix is missing here.On the self-hosted Postgres queue, a top-level
agent-regate-sweeptrigger job hitting this unguarded throw repeatedly can burn through its full retry + dead-letter-revive budget (~60-90 min) before permanently dying — requiring exactly the kind of manual_selfhost_jobsrow-insert the operator had to perform during today's incident to force recovery.Requirements
isConvergenceRepoAllowed/isAgentConfiguredcheck,getLatestRegatedAt/isRegateSweepDrainingcheck) in a try/catch mirroringopsScanRepos's pattern.forwardStructuredLogToSentrypipeline) on a per-repo skip-due-to-error, and skip only that repo for the tick — every other repo in the candidate set must still get evaluated and fanned out normally.skippedErroredcounter alongside the existingskippedDraining, included in theagent.sweep.fanoutaudit event's metadata, so a partial-failure tick is visible in the audit trail instead of looking identical to a fully healthy one.fanOutBacklogConvergenceSweepJobs,fanOutRepoDocRefreshSweepJobs, the signal-snapshot fan-out) for the same unguarded-per-repo-call pattern and fix consistently if found.Deliverables
fanOutAgentRegateSweepJobs's configuration loop (src/queue/processors.ts).test/unit/queue.test.ts: one repo's settings/DB read rejects among several configured repos; assert the other repos still get their sweep job and the fan-out audit event still gets written.Expected outcome
A transient DB/settings read failure for one repo no longer silently zeroes out the entire tick's sweep for every other repo on the instance. Any repo skipped due to error is visible via a Sentry-forwarded log line and an audit-event counter instead of vanishing with no trace.