Repository navigation
provider-retry never retries Account Pooler 429/502 failures #4839
Description
Summary
When a turn fails because the Account Pooler returned 429 (no eligible account) or 502 (could not reach the upstream), provider-retry declines to retry it. The thread stays in error until someone sends a new message, even after the pooler has recovered.
Versions and environment
- bb 0.44.0 and 0.45.0 (desktop app), macOS
- Claude Code and Codex threads routed through the Account Pooler
- provider-retry 0.1.0 (builtin), default settings
Steps to reproduce
- Run a Claude Code or Codex thread through the Account Pooler.
- During a turn, the pooler answers 429 (no eligible account) or 502 (upstream unreachable).
- The turn fails and the thread stays in
error; no retry is queued.
Observed in production, not reproduced on demand.
Expected vs actual
Actual:
- Claude Code: API Error: Request rejected (429) · No Account Pooler account is currently eligible. is classified rate-limit, but no rate-limit state is recorded for the thread, so the retry decision ends in no-rate-limit-state. The pooler's own 429 carries a retry-after with the earliest reset; nothing reads it.
- Codex: exceeded retry limit, last status: 429 Too Many Requests is classified too-many-failed-attempts and treated as not retryable.
- Account Pooler could not reach Anthropic. / Account Pooler could not reach ChatGPT. (502) ends as internal (Claude Code) or stream-disconnected (Codex), also not retryable.
Expected:
Failures the Account Pooler itself marks as temporary are retried by provider-retry: a pool 429 at the pooler's retry-after (within maximumWait), a pool 502/503 with the existing backoff. An invalid pooler bearer token (401) stays non-retryable.
Evidence
Impact in our workspace (Sept 20 to Oct 3): one 429 wave stopped seven threads at the same moment for 1 h 50 min; another left eight Codex threads idle for almost 10 h overnight; a roughly 30-minute 502 outage left a thread idle for 5 h 15 min. A manual resume after recovery (a new message or bb thread retry) worked every time.
provider-retry was disabled on this machine during these incidents (it is enabled now). The decline reasons above come from decideRetry in the 0.45.0 bundle applied to the errorInfo recorded for these failures.
Related: #4166 and #4332 (automations after transient provider failures), #4131 (provider-retry scope).
I can put up a PR with a regression test if this direction works for you.
What you ruled out
No response
Suggested priority and effort (optional)
No response
Checks
- I reproduced this on the latest release or on
main, or I say above that I could not. - I searched open and closed issues for the same problem.
- If an agent wrote this, the body ends with
> AGENT GENERATEDand links the thread or report.
AGENT GENERATED. The underlying incident report is internal and not linkable.
Activity
bb-slop-cop commentedon Oct 4, 2026
🚨 SLOP COP 🚨 ·
new-issue-autopilotproviders,provider-retry.confirmed-repro.4d15c1da0, both clean-checkout runs show five missing retry requests; the HTTP 401 and overload controls pass. Existing retry tests (15) and targeted pool tests (3) pass. This verifies enabled-plugin scheduling, not the reported live outages.Retry-Afterdeadline.brsbl commentedon Oct 4, 2026
Adding the requested user behavior for the all-accounts-exhausted case: keep the message queued and automatically retry when the first pooled account becomes usable, using the reset information already shown in Account Pooler settings. Users should not have to watch the settings page and manually resend after quota recovers.
Acceptance criteria:
Source review at
75308f407660efe0f93ffb9626936e5067b5bb03confirms that the pool already calculates the earliest recovery deadline and returns it inretry-after. The settings UI uses the same blocking-reset calculation.The comparison with direct requests is supported by the existing provider-retry policy: it schedules resettable subscription-limit failures when structured limit state is available and the reset is within the configured maximum wait. The failed-turn handler queues that retry. That behavior should also work when the reset information comes from the pool.
This follow-up is a user requirement plus source inspection, not a new live reproduction. No accounts or routing settings were changed, and no local tests were run. CLI version:
0.45.1-nightly.37130000802.1. Originating bb thread:thr_zwnrc88xqv.