[pplattner]

Summary

• cap model-request retries at 120 seconds
• use exponential backoff and honor Retry-After delay-seconds and HTTP-date values

Context

During a five-shard benchmark evaluation against a hosted endpoint, the model endpoint began returning HTTP 429 responses. Each shard ran with concurrency 10, allowing roughly 50 concurrent agent slots for this evaluation while other hosted-model evaluations were also active.

The retry loop started with MAX_NUM_TRIES = 3, but increased max_num_tries for every rate-limit response. Because both the attempt counter and the maximum increased together, a request could never exhaust its retry budget while the endpoint continued returning 429. Every failed request was then retried after a fixed 0.5-second sleep.

The affected evaluation recorded approximately 6.17 million retry attempts over about 22.5 hours. Individual logical requests reached more than 10,000 attempts, with the longest observed chain reaching 10,895. This produced a sustained lower-bound average of roughly 76 retry requests per second, in addition to normal model calls. Although the workload was a legitimate benchmark, this retry amplification made its traffic resemble a DDoS to the hosted endpoint.

This change adds a hard 120-second wall-clock retry budget so rate-limit responses cannot extend retries indefinitely. It also replaces the fixed delay with exponential backoff and honors server-provided Retry-After guidance in both delay-seconds and HTTP-date formats.

Tests

• ruff format --check nemo_gym/openai_utils.py nemo_gym/server_utils.py tests/unit_tests/test_openai_utils.py
• ruff check nemo_gym/openai_utils.py nemo_gym/server_utils.py tests/unit_tests/test_openai_utils.py
• pytest tests/unit_tests/test_openai_utils.py -q — 128 passed

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ananthsub]

This remains the provider-response implementation candidate for #2845. Before landing, it should consume the single logical request budget defined by #3159, preserve the external-vs-internal ownership added by #3079, honor Retry-After within the remaining budget, and treat terminal input errors separately. A transport retry or changing sequence of status codes must not reset accounting.