fix(orb): back off per-failure relay retries to avoid a fleet-wide storm (#1950)

[JSONbored]

Summary

retryFailedRelays (the retry-orb-relay cron, ~every 2 min) already bounds its per-tick work (RELAY_RETRY_BATCH_SIZE) and per-row attempts (5 / 1h TTL), but a row was eligible on every tick regardless of when it last failed. During a sustained outage of a brokered container, every self-host in the fleet re-POSTs its whole failed-relay backlog every 2 minutes — a synchronized outbound storm against the central Orb exactly when it is already degraded.

This adds a per-failure backoff: a row whose last_attempt_at is under RELAY_RETRY_BACKOFF_MINUTES (5) old is skipped until a later tick, so a down container is re-attempted at most ~every 5 min instead of every 2. Never-attempted rows (last_attempt_at IS NULL) stay immediately eligible, so a transient blip still recovers on the very next tick. No schema change — the store already has attempts + last_attempt_at.

Portability: the backoff bound is a modifier parameter (datetime('now', ?)), which the self-host PG dialect rewrites to now() + (?)::interval (line src/selfhost/pg-dialect.ts:44), so the placeholder count/order is unchanged across SQLite (D1) and Postgres.

Advances #1950 (rec #15 from the #1936 audit) and meets its acceptance criteria: per-instance POST rate decays during a sustained outage, and the backlog still drains within a bounded number of ticks after recovery.

Scope

• Conventional Commit title.
• Focused, backend-only change (one file + tests).
• Follows CONTRIBUTING.md; no site//CNAME.
• Linked issue (advances fix(orb): backoff + per-tick cap on retry-orb-relay fleet drains #1950).

Validation

• git diff --check
• npm run typecheck
• npm run test:coverage — the changed retryFailedRelays select is exercised by the existing orb-relay integration suite; added two regression tests covering both sides of the backoff (a row still inside the window is skipped and not re-POSTed; a row past the window is retried and its attempts increment). The last_attempt_at IS NULL immediate-eligibility path stays covered by the existing store→retry tests.
• npm run test:ci
• npm audit --audit-level=moderate

If any required check was skipped, explain why:

• No ui:openapi / cf-typegen / migration regen: reuses the existing orb_relay_failures.last_attempt_at column; no API/schema/binding/DB change.

Safety

• No secrets/wallets/hotkeys/trust-scores/etc.
• No public GitHub text change.
• No auth/CORS/session change.
• No API/OpenAPI/MCP change. No UI change.

Notes

• Chose a fixed 5-minute backoff over per-attempt exponential to keep the SQL portable (a per-row exponential interval would need (1 << attempts) inside the datetime() modifier, whose parentheses break the pg-dialect rewrite regex). Fixed backoff already satisfies the "rate decays" acceptance criterion; exponential can be a follow-up if the fleet needs a longer tail.

[loopover-orb]

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

_{Review updated: 2026-07-01 04:30:22 UTC}

2 files · 1 AI reviewer · no blockers · readiness 86/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
The change adds a last-attempt cutoff to `retryFailedRelays`, preserving immediate eligibility for never-attempted rows while spacing repeated failures by five minutes. The query, binding order, and retry update path line up with the existing schema fields, and the added tests cover both the skipped and elapsed backoff cases. I do not see a reachable correctness break in the provided diff.

Nits — 4 non-blocking

• nit: `src/orb/relay.ts:162` and `src/orb/relay.ts:281` repeat a lot of incident narrative in code comments; trim these to the invariant and operational reason so future readers do not have to re-parse the PR context.
• nit: `test/integration/orb-relay.test.ts` elapsed-backoff coverage only asserts `attempts` increments; also count the fake fetch call so the test proves the row was actually re-posted rather than only mutated through the failure path.
• In `test/integration/orb-relay.test.ts`, mirror the recent-backoff test by incrementing a `calls` counter in the elapsed-backoff fake fetch and asserting `expect(calls).toBe(1)`.
• In `src/orb/relay.ts:281`, shorten the comment to the durable rule, for example: rows with a non-null `last_attempt_at` remain ineligible until the backoff window elapses, while never-attempted rows stay eligible.

Signal	Result	Evidence
Code review	✅ No blockers	1 reviewer
Linked issue	⚠️ Missing	No linked issue or no-issue rationale found.
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (size label size:S; no linked issue context).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 68 registered-repo PR(s), 58 merged, 274 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 68 PR(s), 274 issue(s).
Gate result	✅ Passing	No configured blocker found.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository registration is not available in the local Gittensory cache.
• Public profile languages: not available
• Official Gittensor activity: 68 PR(s), 274 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Explain no-issue PR.
• Triage stale or unlinked PRs.
• No action.
• Link the issue being solved, or explicitly explain why this is a no-issue PR.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.57%. Comparing base (9e2f533) to head (9ef65dd).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1977   +/-   ##
=======================================
  Coverage   95.57%   95.57%           
=======================================
  Files         218      218           
  Lines       24257    24258    +1     
  Branches     8795     8795           
=======================================
+ Hits        23184    23185    +1     
  Misses        436      436           
  Partials      637      637           

Files with missing lines	Coverage Δ
src/orb/relay.ts	100.00% <100.00%> (ø)

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.