[gwarmstrong]

What does this PR do?

Benchmarks that grade a harness started inside the task container, as Harbor does, need Codex in a sandbox session to send the requests a plain codex exec sends there and to be graded the way Harbor grades it. Sandbox sessions refuse extra_config, so this adds explicit settings to codex_agent, all off by default:

• Launch: model_server_forwards_reasoning_effort lets a configured reasoning_effort through once the operator has checked that the model server forwards it; inherit_sandbox_environment gives Codex the task image's environment and HOME; features leaves Codex's own feature defaults in place; request_max_retries and stream_max_retries set the provider's retries.
• Grading: verify_after_agent_error returns a run that failed while its sandbox still runs and answers as a gradable failed response with no classified cause, so the Environment Server's outcome policy verifies it, as Harbor grades any agent exit. A lost sandbox, unconfirmed cleanup, a run that failed before its first turn or a model-endpoint error (Codex's 429/5xx, capacity, request-timeout, retry-limit and stream-disconnect messages, read from the failure that ended the run) stays an execution error and is retried. A provider failure that Codex reports itself (exit 1 on turn.failed) keeps the classified cause from fix(codex): retain session evidence and report model errors #4740, so the outcome policy decides it. A graded run reports Codex's native cumulative usage, or none when work happened that no usage record covers. Its observations are stored before the sandbox liveness check, so a session closed during the check keeps them. raise_oom_score_adj makes Codex and its tools the OOM killer's preferred victims.
• Logs: sandbox_logs_dir copies Codex's session rollouts (per-call token counts and timestamps) to a directory the resources server collects. The copy runs after the run's events are read and within the capture time left, so a slow copy is cut short instead of failing the run.
• Fix: a task image without WORKDIR lends / as its workdir, which sessions refused with HTTP 422. It is now accepted; the runtime and session files stay under /tmp, and any other workdir must still be disjoint from them.

No issue: opt-in settings on the existing sandbox sessions, plus a fix found in Terminal-Bench-Science runs.

Validation

• codex_agent: 463 passed (7 skipped); pre-commit clean.
• Earlier runs on an older main: Codex 0.154.0 with self-hosted DeepSeek V4.1 Flash on Terminal-Bench-Science, 70 tasks × 3 trials, 8 h, scored 31/210 against the public leaderboard's 33/210 for the same model and harness; Codex with GPT-6 Astra on 10 tasks scored 7/10 against stock Harbor's 7/9 on the same tasks.

Rollouts

N/A: agent settings. Profiles that use them come with the Terminal-Bench-Science harness profiles.

Compatibility and benchmark impact

Defaults unchanged, except that a workdir of / is now accepted instead of refused.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[gwarmstrong]

/ok to test 74d7a2b

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5156-4601f68db032.docs.buildwithfern.com/nemo/gym

[svcnvidia-nemo-ci]

HOLD

Found one BLOCKER and three RISK issues in the opt-in failure-grading and log-capture paths. Details and acceptance checks are inline.

Reviewed all eight changed files and their directly affected session-lifecycle, outcome-policy, verification, and usage-reporting consumers. Static inspection only; no tests or sandbox/model runs were executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

---

Inline comments (escaped v1):

• responses_api_agents/codex_agent/app.py line 107: BLOCKER — Preserve retry handling for the supported CLI's timeout and capacity errors

  Failure: Enable verify_after_agent_error, replay turn.started followed by turn.failed with error.message="request timed out", and supply confirmed cleanup with return_code=-9, timed_out=False, and no cleanup error. On a live sandbox, _terminal_model_error declines the non-1 exit and _model_endpoint_error finds no marker, so the run becomes an unclassified HTTP-200 failed response. It is then verified rather than retried. The same occurs for Selected model is at capacity. Please try a different model. Both are existing supported endpoint failures in the native-error regression cases. A runner killed after the reported failure therefore lets an endpoint outage enter benchmark scores because unclassified failures verify.

  Correction: Include these native messages in _MODEL_ENDPOINT_ERRORS. At this line, the small replacement is:

      "connection timed out",
      "request timed out",
      "selected model is at capacity",
  

  The invariant is that these recognized terminal endpoint failures cannot pass the grading liveness gate. This suggestion is source-traced, not executed; please verify before adopting.

  Preservation / proposed acceptance: For each message, test the transcript and cleanup above: expect HTTP 502 and no liveness probe. Keep exit-1 reported failures classified with outcome-policy RETRY. Keep recovered retry → model output → unrelated kill gradable; HTTP-400 errors must remain non-endpoint errors, and the output-cap case must remain incomplete/max_output_tokens. The disabled opt-in must retain its existing behavior.

• responses_api_agents/codex_agent/app.py line 1117: RISK — Apply conservative usage recovery to newly gradable failures

  Failure: With verify_after_agent_error=True, complete a model/tool call and kill Codex before turn.completed. Even if the native transcript contains cumulative token_count totals of input 20/output 8/total 28, the new scored response reports 0/0/0: the CLI parser starts at zero and this fallback runs only for reported_model_error. Reward profiling consumes those numeric usage values directly, understating failed-trial costs despite available measurements.

  Correction: Extend this condition while retaining its existing body:

          if (reported_model_error or graded_error is not None) and not completed:
  

  This reuses native cumulative usage, or marks usage unavailable when prior work is possible instead of publishing a fabricated zero. Downstream detail accesses already guard response_usage is None. Also update the new graded-failure integration assertions that currently require zero usage. Please verify this source-traced suggestion before adopting.

  Preservation / proposed acceptance: Combine the killed-agent setup with the existing native-usage fixture: expect failed status, null error, no classified outcome, and usage 20/8/28. With the same prior work but no usable native usage, expect usage=None; the new graded-failure environment integration should likewise expect unavailable usage. Preserve the existing provider-failure native/CLI/missing-usage assertions, known-zero no-work provider failures, completed-run total 22, and ordinary timeout/output-cap behavior. Tests were not executed.

• responses_api_agents/codex_agent/app.py line 1105: RISK — Store captured observations before awaiting sandbox liveness

  Failure: Enable the opt-in, let a started failed run finish artifact collection, then block sandbox.status() or the true probe and close the session concurrently. This new await precedes assignment to state.observations. Session close removes the session files and cancels the activation, so the close response contains only the fallback interrupted gap rather than the already-collected conversation/tool evidence. This loses diagnostics for cancelled failed trials. The neighboring startup-stderr path deliberately stores observations before its await.

  Correction: Build and store a provisional failed/incomplete observation bundle before the grading probe; after the probe resolves, update the invocation to the final graded status. Preserve session-close ownership of cancellation—do not keep the sandbox open or extend close deadlines just to finish probing. Exact placement must also preserve native-usage recovery, so a move-only patch is not sufficiently certain to suggest here. The invariant is that cancellation during optional classification cannot discard captured observations.

  Proposed acceptance: Start a failed activation containing a partial tool/message transcript. Have status() signal entry and wait; close the session before releasing it. Expect activation cancellation, borrowed-sandbox disconnect without destruction, and close observations retaining the failed invocation and partial tool/message evidence. Preserve normal live-sandbox grading, stopped/unresponsive-sandbox execution errors, and the existing close-during-startup-stderr assertion that the failed invocation survives cancellation. These checks were not executed.

• responses_api_agents/codex_agent/sandbox.py line 125: RISK — Keep optional archival within a separate capture sub-budget

  Failure: With sandbox_logs_dir enabled, a successful run becomes an execution failure if copying takes longer than session_close_timeout_seconds (default 60 seconds). The copy allows 120 seconds, but SandboxSession._finalize wraps this entire collector in the shorter close timeout. Cancellation escapes the copy's except Exception; events have not been read, capture records TimeoutError, and execution raises instead of returning the otherwise valid response. Verification is skipped, and close can delete the source without retrying capture. This contradicts the best-effort archival contract for large logs or slow mounts.

  Correction: Thread the actual capture budget/deadline from CodexSandboxSession.execute into the collector and archive operation. Enforce and catch a local archive timeout that expires before the enclosing deadline, reserving time for required event/metadata capture. The invariant is that optional copy failure or timeout cannot discard a readable transcript or change grading disposition. Budget allocation needs implementation verification; merely moving the copy after reads or catching outer cancellation still allows collection to abort before returning.

  Preservation / proposed acceptance: Set a short close budget and make only the archive exec wait past it while events and cleanup remain valid. Expect a completed response with 22 tokens, retained conversation observations, and successful close; archival may be omitted with a warning. Retain byte-for-byte copying in the existing archive test, no-op behavior without a logs/source directory, and successful response capture when copying returns nonzero or raises. Timeout/close must still preserve available partial logs only after confirmed termination; unconfirmed cleanup must continue to prevent capture and grading. Checks were not executed.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[gwarmstrong]

/ok to test 4601f68

[gwarmstrong]

All four findings reproduced against the code, and each now has a test that fails on the previous head.

1. Capacity and request-timeout errors (BLOCKER). Added selected model is at capacity and request timed out to _MODEL_ENDPOINT_ERRORS (8ae8bf5). test_native_endpoint_failures_are_retried_not_graded covers both messages two ways: a runner killed after Codex reported the failure is an execution error (502) and the sandbox is never probed, and the exit-1 reported failure still keeps its classified cause and is retried by the outcome policy. Both messages are also added to the recovered-retry transcript in test_recovered_codex_0154_retries_then_an_agent_error_is_graded, which stays gradable.

2. Usage of a graded failure. A failed run that verify_after_agent_error may grade now takes the same native-usage recovery as a reported provider failure (e18c8cb): Codex's native cumulative usage when present, otherwise None when there was prior work, never the CLI parser's zero. test_graded_failure_reports_native_usage_or_none checks 20/8/28 from the native transcript, and None without it. The graded-failure environment integration now expects unavailable usage.

3. Observations before the liveness probe. The probe now runs after the observation bundle is stored (e18c8cb). If grading changes the status (a run that also hit its time limit becomes incomplete), the stored invocation is updated in place. Session close keeps ownership of cancellation. test_close_during_the_grading_probe_keeps_the_observations holds status(), closes the session, and checks the activation is cancelled, the borrowed sandbox is disconnected and not stopped, and the close observations keep the failed invocation with its message and tool call.

4. Archive copy and the capture budget. The session-rollout copy now runs after events, runtime, the native snapshot and model attempts are read, and under its own timeout that ends one second before the capture budget does (SandboxSession's close timeout, passed in from execute and close) (4601f68). A timeout is logged and the copy skipped. test_a_slow_session_rollout_copy_does_not_cost_the_run sets a 3 s close budget and makes only the copy hang: the run returns completed with 22 tokens and its observations, and close succeeds. The byte-for-byte copy test and the no-op cases are unchanged; the ordering test now expects the copy last.

codex_agent: 465 passed (7 skipped); supervisor and sandbox-session units 87; pre-commit and detect-secrets clean.

[yaoyu-33]

@gwarmstrong this branch now conflicts with main. Could you merge main in? We'll approve once it merges cleanly.

[yaoyu-33]

@gwarmstrong the inline note is non-blocking; the only thing left is that the branch now conflicts with main after #5151, in responses_api_agents/codex_agent/sandbox_runner.py and tests/test_sandbox_runner.py. In the runner, #5151 added NEMO_GYM_SUPERVISOR to the default environment allowlist so the supervisor can find Codex's tools after they leave its process tree (sandbox_runner.py#L63-L64); please keep it in the non-inherit_env branch when you merge main.

[yaoyu-33]

Approving at 4601f68; the inline note is non-blocking. Please merge main (conflict with #5151 in the Codex sandbox runner) before landing.

[gwarmstrong]

/ok to test 04ed490

[gwarmstrong]

@yaoyu-33 thanks, main is merged in (04ed490). The only conflicts were with #5151 in sandbox_runner.py and its test. In the runner, NEMO_GYM_SUPERVISOR is now in the default environment allowlist, the one used when inherit_env is off; with inherit_env on, Codex already gets the whole task environment, which includes it. In the test file I kept this PR's inherit_env test and dropped test_lost_spawn_handle_requires_sandbox_teardown, which #5151 removed along with the Python supervisor.

Nothing else needed adapting: the Codex runner still runs as a standard-library Python runner under the bash supervisor. codex_agent: 463 passed (7 skipped; Python 3.8 and the pinned Codex CLI aren't installed here); test_process_supervisor.py and test_sandbox_session.py: 93 passed; pre-commit clean.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40739984627

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40745253835

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40750164067

[gwarmstrong]

Smoke on this head (04ed490, with PR 6 66d4c71 and the Terminal-Bench-Science Codex profile, on main c155b77): Codex 0.154.0 in two Terminal-Bench-Science task sandboxes on OpenSandbox with verify_after_agent_error on, against a self-hosted DeepSeek V4.1 Flash replica, 30 minutes per trial.

geometric-pharmacophore-alignment, whose image has no WORKDIR, ran with Codex's working directory at / and was stopped at the cap and graded (0/7). ankle-mri-findings ran in /app and was graded (1/5). 221 model calls, all 200 apart from the one in flight at the cap. No agent error occurred in these two trials, so the verify-after-error path wasn't exercised; its unit tests cover it.