Skip to content

feat(core): add model retry controls and Gymnasium rollout options - #3415

Merged
yuhezhang-ai merged 14 commits into
mainfrom
mfathi/halluhard-core-improvements
Oct 6, 2026
Merged

yuhezhang-ai merged 14 commits into
mainfrom
mfathi/halluhard-core-improvements

Conversation

@mahanfathi

@mahanfathi mahanfathi commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds opt-in controls for multi-turn model rollouts. With default settings, existing configurations behave as on main (see Compatibility).

  • Model-server retries and timeouts (openai_model, litellm_model). Optional per-attempt request and connect timeouts, a bounded wait for a concurrency slot, and a model-server retry policy with configurable backoff (upstream_* options). upstream_max_num_tries=1 (and NeMoGymAsyncOpenAI(max_num_tries=1)) makes each provider call a single HTTP attempt, so the policy owns the whole retry schedule. A key that the client recognizes as spent or invalid stops the policy immediately. Exhausted retries raise UpstreamRetriesExhaustedError, a RuntimeError (HTTP 500 for non-streaming requests) that keeps the final provider status and body for model-call capture. LiteLLMModelServer applies the same settings on /v1/responses.
  • Status propagation. propagate_upstream_http_status_codes returns selected provider error statuses with the provider's error body instead of HTTP 500, including for streaming Responses requests. NeMoGymAsyncOpenAI also reads error codes from that body, so a propagated spent or invalid key stops a receiving Gym client, as a direct provider error does.
  • Transport. request() gains a trailing _max_num_tries parameter that caps a call's attempts on every exception path and replaces the default generic-error limit.
  • Gymnasium agent. Optional text-only history, a cap on concurrent model calls, and an explicit attempt count for agent-to-model calls. With text-only history, returned rollouts keep complete outputs, including empty assistant turns. Error-path environment cleanup is shielded from cancellation and bounded by environment_cleanup_timeout_seconds (default 60 s).
  • Hosted web search. web_search_call items with no action, or an action type the SDK does not know, are accepted and kept opaque. Known actions stay typed SDK models, and payloads round-trip unchanged.
  • Reasoning effort. Optional reasoning_effort_none_replacement reports a provider's reasoning.effort: "none" as another value for callers that cannot accept "none". It is off by default.

The new openai_model options are documented in its configuration reference.

No tracking issue: this is shared infrastructure for multi-turn evaluation, and each change is self-contained and described above.

Validation

Test suites and differentials are against this branch merged into main b0e3fa6cf.

  • Each code fix from the review has a test that fails when the fix is reverted. The documentation commit adds no test.
  • Server suites: openai_model 45, litellm_model 26, gymnasium_agent 28 passed. switchyard_model, vllm_model, inference_provider, azure_openai_model, local_vllm_model_proxy, and vllm_model_with_compaction match main, as do the other NeMoGymAsyncOpenAI users' suites (terminus_2_sandboxed_agent, tau2, browsecomp_agent, verifif, gdpval, aalcr).
  • Main's own versions of the changed test files pass against this branch: openai_model 10, litellm_model 24, gymnasium_agent 12, Responses streaming 170, test_openai_utils 247, test_responses_converter 133, test_server_utils 137.
  • Core unit suite with CI's extras: 7,019 passed, none failed. On main, 6,969 passed, and test_file_lineage_resolves_across_spawned_worker_processes failed; that test fails intermittently on both.
  • Default-config differential against a real local HTTP provider: openai_model and litellm_model; Responses, Chat Completions and Messages, streaming and not; capture on and off; 21 provider behaviors. All 1,028 scenarios match main in status, body, provider hits, forwarded requests, and capture records.
  • Propagated errors reaching a Gym client over real local HTTP (provider, model server, and NeMoGymAsyncOpenAI(internal=True)), 30 scenarios. A propagated insufficient_quota 429 or invalid_api_key 401 now stops the client after one model-server reply. Previously, the 429 was still being retried after 8 replies. Transient 429s still retry until they succeed. Default configurations, other propagated statuses, and direct provider calls are unchanged.
  • pre-commit run --all-files passes, and all commits are signed off.

Rollouts

Main (3ef478df1) versus this branch merged into it, same Nemotron 3 Nano vLLM server (gpt-5-nano for the openai_model row). The final head differs from the rolled-out commit in one docstring line and in reading error codes from a body's detail, which these default-configuration rollouts never receive.

Environment Model server Rows (main / branch) Failures Request parameters
blackjack (Gymnasium agent) vllm_model 40 / 40 0 / 0 identical
example_multi_turn_gymnasium vllm_model 10 / 10 0 / 0 identical
example_single_tool_call vllm_model 10 / 10 0 / 0 identical
example_single_tool_call openai_model (gpt-5-nano) 10 / 10 0 / 0 identical

Rewards match except blackjack, whose card draws vary between runs.

Compatibility and benchmark impact

Default configurations behave as on main for model-server requests and responses (see the differential above). Other effects:

  1. Serialized web_search_call actions no longer gain placeholder null keys (for example queries, sources). Shapes that were previously rejected are now accepted; on chat-converting backends (vllm_model, inference_provider) they fail the same way valid web_search_call items already do.
  2. max_concurrent_requests must be at least 1. Previously, 0 made every request wait forever.
  3. For environments that advertise supports_explicit_close, error-path cleanup now survives client disconnects and is bounded by environment_cleanup_timeout_seconds (default 60 s). On main this affects only openair_congestion.
  4. A model server whose responses() raises HTTPException during a streaming Responses request now returns that status instead of a response.failed event, like request validation. Among shipped model servers, only openai_model and litellm_model raise it, and only when propagation is configured.
  5. Out-of-tree subclasses of SimpleModelServer must keep a SimpleModelServerConfig-derived config, since the server now reads the new options.
  6. NeMoGymAsyncOpenAI reads error.code/error.type under detail when a body has no top-level error. A 429 or 401/403 whose detail carries a spent-key or invalid-key code now stops the client, as the same top-level code already does.

No benchmark results change.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification.
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
  • New source files include the required Apache-2.0 SPDX header. (N/A: no new source files.)
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:core Shared APIs, servers, telemetry, health, and registries complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Sep 15, 2026
Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Sep 16, 2026
mahanfathi and others added 2 commits October 1, 2026 19:54
Resolve conflicts with main's retry and model-server changes:

- Keep main's permanent quota and auth trips and its raise_for_status
  body handling. An explicit max_num_tries=1 still overrides
  max_http_attempts and never extends for internal rate limits.
- Keep main's connection-retry cap in the transport exception path
  alongside the explicit _max_num_tries cap, counting the same attempts
  so the lower limit also applies to generic errors.
- Keep main's Ray opt-out and provider-extension test in openai_model
  alongside this branch's upstream retry controls and nullable
  reasoning_effort_none_replacement.
- Accept main's raise_for_status(response, content) in the explicit-cap
  test double.

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Validate known web_search_call actions to the SDK's typed models again, so
attribute access, SDK-object construction, and from_attributes keep working,
while unknown or missing actions stay opaque and every payload still
round-trips unchanged. Guard openai_model's response rewrite against
non-dict client results, and append _max_num_tries after request()'s
existing parameters so positional callers keep their meaning.

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main requests are to stop backoff after a permanent provider failure, preserve error information needed by callers and model-call capture, and guard the retry settings that LiteLLM currently bypasses. I verified these with focused tests and injected failure cases; I did not repeat the live-model rollouts. The reasoning-default and documentation suggestions are nonblocking.

A spent or unauthorized key trips NeMoGymAsyncOpenAI, after which every
call fails without reaching the provider. The upstream retry policy still
treated that PermanentEndpointError as retryable and waited through its
whole backoff schedule on every request. Re-raise it immediately.

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
A propagated upstream status now carries the provider's error body,
parsed as JSON when possible, instead of a generic message, so callers
that look for codes such as context_length_exceeded still find them.
Exhausted retries still fail with HTTP 500, but now raise
UpstreamRetriesExhaustedError, a RuntimeError that keeps the final
provider status and body for model-call capture. Both endpoints share
one helper for the retry policy and status propagation.

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
LiteLLMModelServer inherits the upstream retry, pool-timeout, and status
propagation settings, but its own /v1/responses handler called the client
directly, so an accepted retry policy silently reduced it to one attempt.
Route the handler through the shared helper. With default settings it
still makes one call under the same concurrency limit.

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
The pinned OpenAI SDK and NeMoGymResponse both accept reasoning.effort
'none', so rewriting it to 'minimal' by default changed the reported
inference setting, even when a caller requested 'none'. Default
reasoning_effort_none_replacement to null and keep the rewrite available
for configurations whose callers cannot accept 'none'.

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
Replace the stale client-disconnect rationale on the provider slot, and
state in the config descriptions, request() docstring, and openai_model
reference that request timeouts apply per attempt, connect timeouts
include waiting for an aiohttp connection-pool slot, and an explicit
attempt count replaces the transport's generic-error retry limit. List
the new openai_model options in its configuration reference.

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
The shared Responses streaming wrapper converted every responses()
failure into an HTTP 200 response.failed event, so a status that
openai_model was configured to propagate never reached streaming
clients. Re-raise HTTPException there: like request validation, it fails
before the stream is committed. Other failures still end the stream with
response.failed.

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
@mahanfathi mahanfathi changed the title Improve model retry controls and Gymnasium rollout handling feat(core): add model retry controls and Gymnasium rollout options Oct 4, 2026
mahanfathi and others added 2 commits October 5, 2026 10:53
With propagate_upstream_http_status_codes, openai_model and
litellm_model return the provider's error body under FastAPI's `detail`.
NeMoGymAsyncOpenAI read only a top-level error.code/type, so an internal
client that received a propagated 429 insufficient_quota treated it as a
transient rate limit and kept extending its retries. When a body has no
top-level `error`, read the error object under `detail`, so a spent or
invalid key stops a receiving Gym client as it does when that client
calls the provider directly.

Signed-off-by: Mahan Fathi <mfathi@nvidia.com>
Merge main at 242d468. Verified 835 focused tests, the propagated-quota regression probe, and pre-commit run --all-files.

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
@yuhezhang-ai

Copy link
Copy Markdown
Contributor

/ok to test 1cb1e69

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All review findings are addressed, including the propagated permanent-quota error reaching the receiving Gym client.

Reviewed the clean merge of main at 242d468 into this PR. On 1cb1e69, all 835 focused tests, the bounded quota-error regression probe, and pre-commit run --all-files passed. Permanent quota errors stop after one reply; temporary rate limits still retry and recover.

Approving the code. Full repository CI has been requested and should pass before merge.

@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 6, 2026 00:18
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 6, 2026 00:18
@yuhezhang-ai
yuhezhang-ai enabled auto-merge October 6, 2026 00:18
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37394585251

Merged via the queue into main with commit a7ca33b Oct 6, 2026
43 checks passed
@yuhezhang-ai
yuhezhang-ai deleted the mfathi/halluhard-core-improvements branch October 6, 2026 00:46
ehosseiniasl pushed a commit to ehosseiniasl/Gym that referenced this pull request Oct 9, 2026
…o#4285)

Pi's SDK cannot recognize context overflow when an explicitly propagated
provider error is wrapped as `{"detail":{"error":...}}`: it reports “400
status code (no body)” instead of the provider message. Its HTTP 400/503
integration tests also omitted the propagation setting required by the
current OpenAI Model Server.

Remove the adapter's extra HTTPException conversion and use the existing
shared Model Server error handler. Selected statuses now retain the
original provider body, including top-level `error.message` and
`error.code`. LiteLLM inherits the same correction. After Chat streaming
sends a keepalive, the terminal SSE error also retains the selected
provider explanation; unselected provider errors remain generic HTTP 500
messages in the stream. The Pi fixture explicitly enables the statuses
it exercises.

The default propagation policy remains opt-in; unselected errors and
exhausted provider retries retain their HTTP 500 behavior.
**Compatibility:** consumers of explicitly propagated OpenAI/LiteLLM
errors must read the original provider body rather than the former
`detail` wrapper. Capture retains that same served body.

Follow-up to NVIDIA-NeMo#3415 and NVIDIA-NeMo#3626; related tracking:
[FEA-461](https://linear.app/nvidia/issue/FEA-461). This addresses the
separate Pi CI failure exposed while validating NVIDIA-NeMo#4251.

Validation on this branch:
- **507 passed**: OpenAI and LiteLLM adapter tests, base Model
Server/Chat and Responses streaming tests, and pinned Pi 0.80.2 SDK
outcome tests. Covers opt-in and default status handling,
Chat/Responses, streaming/nonstreaming, capture, exhausted retries, and
permanent-error handling. A delayed-error regression fails before the
fix; the pinned Pi SDK recognizes HTTP 400 context overflow and HTTP 503
both before and after the first keepalive.
- **Two real local HTTP smokes passed:** Pi SDK → Gym OpenAI Model
Server → fault-injection provider, HTTP 400 context overflow and HTTP
503 service unavailable. Exactly one upstream request per case; Pi
retains each error message and correctly distinguishes context overflow.
No generative-model benchmark run: this is an error-response wire-format
change.
- Full `pre-commit run --all-files` and `git diff --check` passed;
DCO-signed.

```bash
python -m pytest -o addopts= --import-mode=importlib -q \
  responses_api_agents/pi_agent/tests/test_outcome.py \
  responses_api_models/openai_model/tests/test_app.py \
  responses_api_models/litellm_model/tests/test_app.py \
  tests/unit_tests/test_base_responses_api_model.py \
  tests/unit_tests/test_responses_api_model_streaming.py \
  tests/unit_tests/test_chat_completions_streaming.py
```

---------

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
khazic pushed a commit to khazic/Gym-lao that referenced this pull request Oct 13, 2026
## What does this PR do?

Follow-up to NVIDIA-NeMo#3632, split out as requested in
NVIDIA-NeMo#3632 (comment).

In `_request_with_retries()`, the generic `except Exception` branch
handles every error except `ServerDisconnectedError` and
`ClientOSError`, which have their own branches. This includes aiohttp
timeouts. When the caller does not pass `_max_num_tries`, this branch
counted tries only for external callers. So an internal caller
(`_internal=True`) that set `_max_connection_retries` to 2 or more never
reached the cap and retried forever.

NVIDIA-NeMo#3415 already counts tries for callers that pass `_max_num_tries`. This
PR fixes the remaining path, where `_max_num_tries` is not passed.

- Count tries for internal callers too, so `_max_connection_retries`
stops them.
- Keep everything else the same:
- internal callers with no cap still retry with no limit and print no
retry message, as NVIDIA-NeMo#90 intended;
  - external callers still stop at `MAX_NUM_TRIES`;
  - callers that pass `_max_num_tries` are unchanged.
- Update the comment that said internal calls are not counted. It keeps
NVIDIA-NeMo#90's reason for not limiting internal calls.
- Add three tests:
  - the cap stops internal callers, with tracing off and on;
  - internal callers with no cap retry silently past `MAX_NUM_TRIES`;
- external callers stop at `MAX_NUM_TRIES` and print one retry message
per try.
- The cap test, `test_request_retry_cap_bounds_internal_callers`,
replaces `test_retry_cap_bounds_internal_callers`, which was removed
from NVIDIA-NeMo#3632.
- Make the retry tests fail instead of hang when the loop never stops:
- The test helper `_mock_global_client` (added in NVIDIA-NeMo#2383) now takes the
error to raise.
- Its fake `request()` calls `pytest.fail()` once it is called more
often than any test needs. `pytest.fail()` raises a `BaseException`,
which the loop's `except Exception` does not catch. So the test fails at
once, even if the loop never sleeps.
- Two NVIDIA-NeMo#3415 tests now use this helper instead of their own mocks:
`test_explicit_transport_attempt_cap_disables_request_retry` and
`test_request_caps_generic_errors_at_the_lower_attempt_limit`. Before,
if the generic branch ignored `_max_num_tries`, the first test hung,
sleeping a real 0.5 s per retry.

No caller in this repo sets both `_internal=True` and
`_max_connection_retries` today:
- The internal callers pass no cap: `ServerClient`, the simple agent's
direct tool calls to Resources servers (`tool_access`), the Terminus 2
sandboxed agent's model client (`NeMoGymAsyncOpenAI(internal=True)`),
the OpenHands capture route, and the LiteLLM version check.
- The callers with a cap are not internal: the remote agent and sandbox
transport use 1, Tavily search uses 0, and vLLM with `endpoint_file`
uses a default of 8.
- The OpenAI model server and the Gymnasium agent pass `_max_num_tries`
only when `upstream_max_num_tries` or
`model_server_transport_max_attempts` is set. Neither passes
`_max_connection_retries`, so neither changes.

Related:
- NVIDIA-NeMo#3553 added the cap check to this branch without counting internal
tries.
- NVIDIA-NeMo#3415 added `_max_num_tries` and counts tries on that path.
- NVIDIA-NeMo#3159 tracks bounding every retry by default, including internal
callers with no cap. This PR only makes an explicit cap work for them.
- NVIDIA-NeMo#2216 tracks the `ServerDisconnectedError` and `ClientOSError`
branches, which this PR does not change.
- Open PR NVIDIA-NeMo#2935 adds a timeout branch that copies the old counting, so
internal callers there still do not count tries. Whichever PR merges
later must count tries for every caller.

## Validation

All runs used an activated venv from `uv sync --extra dev --extra
sandbox --extra openshell --extra telemetry --extra mlflow --extra
wandb` (the extras in `full-test-suite.yml`). Every installed package
matches `uv.lock` (192 packages, including aiohttp 3.14.4, nemo-lens
0.2.0+b0f977d, and pytest 9.0.3).

- `pytest tests/unit_tests/test_server_utils.py
responses_api_agents/remote_agent/tests/test_app.py
tests/unit_tests/telemetry/`: 465 passed.
- With the old code restored,
`test_request_retry_cap_bounds_internal_callers` fails in under a second
with `Failed: retry loop did not stop` instead of hanging. It fails the
same way when the retry sleep is also removed.
- Each change below makes at least one test fail. Each run of
`test_server_utils.py` took about 3 s, and none hung:
  - the cap is ignored;
  - internal callers are limited by `MAX_NUM_TRIES`;
  - internal callers print;
  - external tries are double-counted;
  - external callers stop printing the retry message;
  - the `MAX_NUM_TRIES` raise is removed;
  - the retry sleep after a generic error is skipped;
  - the cap check ignores `_max_num_tries` attempts;
  - the generic branch ignores `_max_num_tries`;
  - the traced path drops `_internal` or `_max_connection_retries`.
- `pytest -n 16 tests/unit_tests
responses_api_agents/remote_agent/tests`: 8089 passed, 5 skipped. The
per-test results match `main` at a3cf403 (8085 passed, 5 skipped). The
4 extra results are the new tests, with the cap test run twice (tracing
off and on).
- The same focused tests pass on a local merge with the latest `main`
(NVIDIA-NeMo#4300), which changes no package versions.
- `pre-commit run --all-files`: passed.

## Rollouts

N/A. This changes a retry counter in the shared HTTP helper. No caller
in this repo changes behavior, and no model, agent, environment, or
verifier logic is involved.

## Compatibility and benchmark impact

None for callers in this repo. A caller that sets `_internal=True` and
`_max_connection_retries` to 2 or more, without `_max_num_tries`, now
stops after that many tries on errors that reach the generic `except
Exception` branch, including timeouts, instead of retrying forever. This
includes code outside this repo that uses
`NeMoGymAsyncOpenAI(internal=True, max_connection_retries=N)`.
`ServerDisconnectedError` and `ClientOSError` keep their own count, as
before.

## Checklist

- [x] I have read the [contributing
guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md).
- [x] The change is focused; unrelated "drive-by" edits are tracked as
separate issues/PRs.
- [x] Tests added or updated and pass locally, or N/A for docs-only /
non-code changes (so CI unit/server checks pass when applicable).
- [x] Documentation added or updated, or N/A with justification. N/A:
internal helper behavior with no user-facing docs.
- [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so
CI lint/format checks pass).
- [x] New source files include the required Apache-2.0 SPDX header. N/A:
no new files.
- [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check
passes).

---------

Signed-off-by: Abhishree <athittenaman@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — 1cb1e692 Deployed Oct 6, 2026 by copy-pr-bot[bot] via release / finalize / notify #4373
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants