[giuliolovisotto]

What does this PR do?

Simple Agent dropped tool observations when closing an agent session and let model HTTP errors escape without a structured agent outcome. It also could not recover from the transient failures exercised by the P0 conformance suite.

Retain tool observations at session close, correlate every model attempt with its invocation, and retry transient model HTTP errors twice by default. Exhausted session calls return the cause and partial work for the Environment Server to handle. Retries reuse the current request without replaying completed tools, and remain inside the existing checkpoint-aware policy boundary.

Register Simple Agent in the existing probe and CI matrix, using a direct HTTP shell fixture and the same seven P0 scenarios and expectations. A separate public issue is unnecessary for this focused repair of existing conformance behavior.

Validation

• .venv/bin/python -m pytest responses_api_models/vllm_model/tests responses_api_agents/simple_agent/tests/test_app.py -q --tb=short — 959 passed, including provider cancellation recovery, retry exhaustion, and retries disabled.
• .venv/bin/python -m pytest responses_api_agents/simple_agent/tests/test_app.py scripts/harness_conformance/tests tests/unit_tests/harness_capabilities tests/unit_tests/test_harness_conformance_ci.py -q --tb=short — 505 passed, 2 skipped.
• .venv/bin/python -m pytest tests/unit_tests/test_checkpoint_agent.py -q --tb=short — 41 passed.
• .venv/bin/python scripts/run_harness_conformance.py --harness simple --output ../simple-final — all seven scenarios fulfilled P0 requirements; exit 0.
• .venv/bin/pre-commit run --all-files — passed.
• .venv/bin/python tests/unit_tests/test_fern_docs_links.py — 29 passed.
• cd fern && npm run check — zero errors; redirect checking skipped because Fern authentication was unavailable.

CI on 984f47727 passes all eight server shards, session-routing integration tests, core tests and the 95% coverage gate (95.47%). Core tests: 12,025 passed, 99 skipped; sandbox tests: 1,045 passed, 6 skipped. CI run.

Simple Agent passes all seven P0 scenarios on this head (probe job). The remaining red checks are the non-blocking Kilo Code conformance result and its summaries. Its terminal-model-error failure also occurs on the base main run and is addressed separately by #5088.

Rollouts

One live GPT-4.1-mini rollout through the standard Environment Server collector calculated 17 × 23, wrote and read back 391, and returned the requested final answer. Inspected the model/tool conversation, executor observations, capture, and verifier: reward 1.0, two model calls, two tool calls, 293 total tokens, and no failure sidecar rows.

The controlled suite separately verified tool success/failure, omitted usage, HTTP 429/500 recovery, terminal model errors, and verifier failure. Regression tests cover retry exhaustion after successful tool work and preserve unknown token usage.

Compatibility and benchmark impact

Simple Agent now retries HTTP 408/425/429/5xx twice, with exponential backoff starting at 0.5 seconds and capped at 8 seconds. max_model_retries: 0 disables the additional attempts. This can recover episodes that previously failed and increases latency when transient errors persist. Terminal errors in the legacy interface still propagate after the configured attempts; agent sessions return structured failures. No benchmark accuracy claim or change to the existing conformance gate policy is made.

Checklist

• I have read the contributing guidelines.
• The change is focused.
• Tests added or updated and pass locally.
• Documentation added or updated.
• Pre-commit checks pass locally.
• New source file headers: N/A; no source files added.
• The commit has DCO sign-off and an SSH signature.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5117-984f477276b4.docs.buildwithfern.com/nemo/gym

[svcnvidia-nemo-ci]

SHIP — LGTM — no concerns found.

Reviewed all nine changed files and traced the Simple Agent retry, failure, and session paths through checkpoint handling, observation joins, and Environment Server outcome policy. Also checked the conformance integration, CI registration, regression tests, and documentation.

Static review only; I did not execute tests or independently reproduce the reported rollouts.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40655888803