Skip to content

fix(model): bound provider-response retries #2845

Open

Description

@giuliolovisotto

Priority

Wave 0 — urgent independent correctness. Bound provider retries immediately; do not wait for #3159.

Problem

Internal model-provider 429 and selected 5xx responses extend the retry limit while using fixed delays. Persistent saturation can therefore retry indefinitely, synchronize concurrent callers, amplify load, and retain collector capacity without producing an outcome.

Essential scope

  • Finite total attempts and elapsed retry time for internal provider responses.
  • One counter across changing 429 and transient 5xx status sequences.
  • Capped exponential backoff with jitter.
  • Honor valid Retry-After only within the remaining bound.
  • Treat input, authentication, authorization, configuration, and propagated context-overflow errors as terminal.
  • Preserve the already-read final status and a bounded response-body prefix.
  • Keep internal and external ownership explicit at the model-server boundary.
  • Focused fake-clock, mixed-status, cancellation, and concurrent-caller tests.

Later integration

Adapt the provider policy to a shared #3159 logical request context only if that interface lands and provides value. Independent finite transport and provider bounds are sufficient for the immediate safety fix.

Full exported telemetry belongs to #3182; this issue provides basic attempt, delay, and exhaustion logs.

Implementation candidate

#2527 contains useful elapsed-budget and Retry-After work. It must cover internal traffic, add jitter, preserve the final consumed response body, and retain external/internal policy ownership.

Acceptance criteria

  • Every internal provider retry sequence terminates by attempt or elapsed bound.
  • Changing retryable statuses never resets accounting.
  • Backoff is capped, jittered, and honors bounded Retry-After.
  • Terminal 4xx and context errors do not retry.
  • Exhaustion retains the final status and bounded response context.
  • Cancellation and concurrent callers do not synchronize into fixed-interval retry amplification.

Activity

  1. changed the title [-]Unbounded fixed-interval (0.5s) retries on 429/5xx amplify endpoint saturation — use exponential backoff + jitter + cap[/-] [+]fix(model): bound provider-response retries[/+] on Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions