Skip to content

fix(orb): backoff + per-tick cap on retry-orb-relay fleet drains #1950

Description

@JSONbored

Parent: #1936

Problem

retry-orb-relay is enqueued every ~2-min sweep cycle when the broker is enabled (src/index.ts:133) and drains all failed relay POSTs (retryFailedRelayssrc/orb/webhook.ts:94-109). During a partial Orb/network outage, every brokered self-host in the fleet retries its full failed-relay backlog on the same 2-min cadence with no per-failure backoff and no per-tick cap — a synchronized outbound POST storm against the central Orb exactly when it is already degraded (a thundering-herd amplifier, container/network resource — not GitHub budget).

Requirements

  • Per-failure exponential backoff (a relay event that keeps failing is retried less often, not every tick).
  • A per-tick cap on how many relay POSTs a single drain attempts.
  • Preserve delivery: a recovered Orb still drains the backlog promptly, just not all-at-once fleet-wide.

Deliverables

  • Add next_attempt_at / attempt-count backoff to the failed-relay store and honor it in retryFailedRelays.
  • Bound the per-drain batch size; carry the remainder to the next tick.
  • Tests for backoff scheduling, per-tick cap, and post-recovery drain.

Acceptance criteria

  • During a sustained Orb outage, per-instance relay POST rate decays instead of firing the whole backlog every 2 min.
  • After recovery, the backlog still drains within a bounded number of ticks.

Expected outcome

A partial Orb outage no longer triggers a fleet-wide retry storm on the central broker. (Net-new from the #1936 rate-limit/resource audit, rank 15.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions