Parent: #1936
Problem
retry-orb-relay is enqueued every ~2-min sweep cycle when the broker is enabled (src/index.ts:133) and drains all failed relay POSTs (retryFailedRelays → src/orb/webhook.ts:94-109). During a partial Orb/network outage, every brokered self-host in the fleet retries its full failed-relay backlog on the same 2-min cadence with no per-failure backoff and no per-tick cap — a synchronized outbound POST storm against the central Orb exactly when it is already degraded (a thundering-herd amplifier, container/network resource — not GitHub budget).
Requirements
- Per-failure exponential backoff (a relay event that keeps failing is retried less often, not every tick).
- A per-tick cap on how many relay POSTs a single drain attempts.
- Preserve delivery: a recovered Orb still drains the backlog promptly, just not all-at-once fleet-wide.
Deliverables
- Add
next_attempt_at / attempt-count backoff to the failed-relay store and honor it in retryFailedRelays.
- Bound the per-drain batch size; carry the remainder to the next tick.
- Tests for backoff scheduling, per-tick cap, and post-recovery drain.
Acceptance criteria
- During a sustained Orb outage, per-instance relay POST rate decays instead of firing the whole backlog every 2 min.
- After recovery, the backlog still drains within a bounded number of ticks.
Expected outcome
A partial Orb outage no longer triggers a fleet-wide retry storm on the central broker. (Net-new from the #1936 rate-limit/resource audit, rank 15.)
Parent: #1936
Problem
retry-orb-relayis enqueued every ~2-min sweep cycle when the broker is enabled (src/index.ts:133) and drains all failed relay POSTs (retryFailedRelays→src/orb/webhook.ts:94-109). During a partial Orb/network outage, every brokered self-host in the fleet retries its full failed-relay backlog on the same 2-min cadence with no per-failure backoff and no per-tick cap — a synchronized outbound POST storm against the central Orb exactly when it is already degraded (a thundering-herd amplifier, container/network resource — not GitHub budget).Requirements
Deliverables
next_attempt_at/ attempt-count backoff to the failed-relay store and honor it inretryFailedRelays.Acceptance criteria
Expected outcome
A partial Orb outage no longer triggers a fleet-wide retry storm on the central broker. (Net-new from the #1936 rate-limit/resource audit, rank 15.)