fix(queue): isolate per-repo failures in the scheduled regate sweep fan-out

[JSONbored]

Summary

• fanOutAgentRegateSweepJobs had two unguarded failure paths that could silently abort the ENTIRE scheduled sweep tick for EVERY managed repo, not just the one that failed:
  1. The per-repo settings/draining-check loop (resolveRepositorySettings, getLatestRegatedAt) had no try/catch — a transient D1 read error for a single repo threw out of the for loop before any repo reached the dispatch stage.
  2. The dispatch loop wrapped every repo's env.JOBS.send(...) in a single Promise.all — one repo's send rejecting (a transient queue-send error) rejected the whole Promise.all, which both aborted the sends still in flight for OTHER repos and skipped the trailing recordAuditEvent call that records this fan-out's own outcome, making the failure invisible in the audit log.
• Wraps the settings/draining-check loop body in a try/catch: a failing repo is logged (sweep_fanout_repo_check_failed) and skipped, and the loop continues to the next repo (that repo is picked up again on the next cron tick, since it never got a convergence marker).
• Wraps each dispatch send in its own .catch(...): a failing repo's dispatch is logged (sweep_fanout_dispatch_failed) and swallowed, so the Promise.all never rejects on a per-repo send failure — every other repo's dispatch still completes, and the fan-out's own audit event is always recorded.
• Adds a skippedErrored count alongside the existing skippedDraining count in the fan-out's audit-event metadata, so a repeated settings-check failure for one repo is now directly observable instead of silent.

Closes #3807. This was the leading hypothesis from today's incident audit for why the scheduled sweep stalled for hours across every managed repo simultaneously.

Scope

• The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked an issue, or this is small enough that the summary explains why an issue is not needed.

Validation

• git diff --check
• npm run actionlint
• npm run typecheck
• npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A — no auth/session/CORS surface touched.)
• API/OpenAPI/MCP behavior is updated and tested where needed. (N/A — no API/OpenAPI/MCP surface changed.)
• UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. (N/A — backend-only change, no UI touched.)
• Visible UI changes include a UI Evidence section below. (N/A — no visible UI change.)
• Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.41%. Comparing base (fd8b3da) to head (9d6f069).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3827   +/-   ##
=======================================
  Coverage   93.41%   93.41%           
=======================================
  Files         325      325           
  Lines       32844    32850    +6     
  Branches    12030    12030           
=======================================
+ Hits        30680    30686    +6     
  Misses       1530     1530           
  Partials      634      634           

Files with missing lines	Coverage Δ
src/queue/processors.ts	94.45% <100.00%> (+0.01%)	⬆️

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[loopover-orb]

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

_{Review updated: 2026-07-06 13:28:26 UTC}

2 files · 1 AI reviewer · no blockers · readiness 100/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
This PR correctly fixes the two failure-isolation bugs described: it wraps the per-repo settings/draining-check body in try/catch (logging `sweep_fanout_repo_check_failed` and continuing to the next repo) and wraps each dispatch send in its own `.catch` so a single repo's rejection no longer collapses the `Promise.all` and skips the trailing `recordAuditEvent`. The new `skippedErrored` counter is threaded into the audit metadata, and the two added regression tests genuinely exercise the real failure paths (mocking `resolveRepositorySettings` to throw, and mocking `env.JOBS.send` to reject for one repo) rather than fabricating unreachable states, verifying both that the sibling repo still gets dispatched and that the audit event still records with the correct counts. Control flow inside the try/catch (the two `continue` statements and the final `configured.push(repo)`) is unchanged in logic from the original and remains correct inside a for-loop try/catch.

Nits — 6 non-blocking

• The two `console.error` calls in the new catch/`.catch` handlers (processors.ts around the try/catch and dispatch `.catch`) are consistent with existing structured-logging conventions elsewhere in this file/`index.ts`, but consider extracting a small shared helper since this is now the third near-identical `JSON.stringify({level:'error', event, ...})` block in this function's vicinity.
• A repo that fails its settings/draining check every tick (e.g. a persistently misconfigured repo, not just a transient error) will silently retry forever with only a log line — consider surfacing `skippedErrored` staying nonzero across consecutive audit events as an alerting signal rather than just a per-tick count.
• `errorMessage(error)` is used in the new catch blocks — confirm it's already imported/defined elsewhere in processors.ts (not shown in this diff) since it isn't part of the visible hunk.
• Consider whether `skippedErrored` repos should be distinguished in the audit metadata by repo name (not just a count) to speed up incident triage next time, mirroring how `sweep_fanout_repo_check_failed` already logs `repository` per-occurrence.
• The dispatch `.catch` swallows the error without any bound on how many consecutive ticks a given repo can fail dispatch before some higher-severity alert fires — worth a follow-up if repo-specific dispatch failures turn out to be non-transient.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

Signal	Result	Evidence
Code review	✅ No blockers	1 reviewer
Linked issue	✅ Linked	#3807
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 54 registered-repo PR(s), 46 merged, 399 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 54 PR(s), 399 issue(s).
Gate result	✅ Passing	No configured blocker found.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: Python, TypeScript, JavaScript, Ruby, Go, Kotlin, MDX, Shell
• Official Gittensor activity: 54 PR(s), 399 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[loopover-orb]

Gittensory approves — the gate is satisfied and CI is green.