[athitten]

What does this PR do?

Codex sandbox sessions now accept MCP tool grants. Before this PR, a session with any required tool grant was rejected with a 422, so a benchmark whose tools live on its resources server could not run with Codex.

• Grants become Codex MCP servers. Each MCP grant becomes an mcp_servers entry in the session's config.toml: the grant URL, the session token as http_headers, and required = true for a required grant. The non-session path already writes this entry from /seed_session metadata; both paths now share one helper.
• Required server fails early. A required server that does not connect stops Codex before the first model call, so the activation fails instead of running without its tools.
• Startup errors carry the cause. When Codex fails before starting a turn, the error now carries the end of Codex's stderr, where Codex reports such failures. Before, it said only "Codex exited with code 1".
  • Grant header values are replaced by [redacted]. Each value is matched as written and JSON-escaped. A value that Codex cut off inside a long error body is replaced through its last echoed character, so no part of a multiword value stays visible. All values are matched against the original text and replaced in one pass.
  • Long lines are shortened, and at most 2,000 characters are kept.
  • The same text is kept as an agent_startup_failed observation gap. Environment Servers keep the observations, not the 502 body, in their failure records. If the session is closed during the stderr read, the observations are still returned, without this gap.
  • The stderr read is bounded by session_close_timeout_seconds.
• Grants Codex cannot use.
  • Required direct_http grants are still rejected with a 422.
  • Codex starts only MCP servers whose names use letters, digits, - and _. A required grant with another name is rejected at seed time. An optional one is left out and recorded as an mcp_server_skipped observation gap.
• New optional field mcp_tool_timeout_seconds. It is written as tool_timeout_sec and must not exceed timeout. Unset keeps the pinned CLI's default of 300 seconds. The README explains why it should exceed the server's slowest call: Codex runs no other tool while an MCP call is in flight, and a call that times out keeps running on the server.
• MCP tool names in observations. Close observations now name an MCP call by its tool, as the response does, instead of mcp_tool_call.
• Docs. The Codex README and the Codex rows of the two fern harness tables now describe MCP grants.

Needed to run Codex on GDPVal (#4178, merged), whose resources server serves finish, abandon_task_finish, web_search and fetch_web_page. No issue.

Related:

• feat(pi): use MCP tool grants in sandbox sessions #4587 (merged) does the same for Pi sandbox sessions.
• feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw #3902 (merged) keeps the task's HOME in Codex sessions.
• feat(gdpval): run Codex in place of Stirrup #4591 runs Codex on GDPVal with this PR.
• fix(agents): hide the sandbox supervisor and runners from pkill python #4805 fixes the pkill -f python3 runner kill seen in the rollouts below.
• Overlaps in codex_agent/app.py:
  • fix(codex): retain session evidence and report model errors #4740 conflicts at the imports, the activation's _build_config call and the error path. It rebuilds the observation bundle after that point, so whichever lands second must add the agent_startup_failed gap to the final bundle.
  • feat(codex): run Codex sessions through an optional sandbox session capture #4281 already conflicts with main and touches the same seed checks, config fields and error path.
  • fix(agent): refuse unsupported agent session seeds with a final 422 #4680 merges cleanly. If it lands first, the two new seed 422s could move to its AgentSeedRejectedError.

Validation

• Rebased onto main at c62f9cc, which adds Codex failure classification (feat(agent): report native causes for shared grading policy #4834) and builds observations from the native transcript when there is one (fix(codex): retain session evidence and report model errors #4740). A startup failure has no turn.failed, so it still ends in the 502 with the stderr tail, and the agent_startup_failed gap is added to the observations on both paths. It keeps main's "agent session" wording (refactor: describe session support without "native" #4854), its 422 message for unusable grants, and calls_model_from_sandbox.
• pytest responses_api_agents/codex_agent/tests tests/unit_tests/test_agent_outcome.py at the final head, with CODEX_TEST_BIN set to the Codex 0.144.4 binary, in a venv built with uv sync --extra dev --extra sandbox from uv.lock: 409 passed, 1 skipped (the skipped test needs Python 3.8).
• Unit tests cover:
  • grants reaching the sandbox config.toml (URL, token header, required, timeout, a hyphenated name), and an episode grant replacing a configured grant of the same name;
  • the non-session entry with tool_timeout_sec;
  • required direct HTTP grants and invalid required names rejected before the sandbox is touched;
  • an invalid optional name left out with an mcp_server_skipped gap;
  • the timeout bounds (0, negative, NaN, above timeout; equal to timeout is accepted);
  • a startup failure reporting a bounded stderr tail and an agent_startup_failed gap, while a failure after a turn started does not;
  • grant tokens redacted, including one cut off by Codex, a value with spaces, and a multiword value cut off by Codex or followed directly by a full copy, two values sharing a start, and a JSON-escaped echo;
  • close observations naming an MCP call by its tool (finish), not mcp_tool_call;
  • a stderr read that hangs does not hold the activation;
  • a close during the stderr read keeps the observations.
• Each new or extended test was checked to fail when the code it guards is broken.
• Tests with the real Codex 0.144.4 binary (skipped unless CODEX_TEST_BIN is set):
  • Codex calls a tool on a real Gym resources server exposed over MCP. The call reaches the route under the session the grant's token was minted for, and the result goes back to the model.
  • With an unreachable required server, Codex exits with an error before calling the model: no turn.started event, and "required MCP servers failed to initialize" on stderr.
  • With an unreachable optional server, the turn starts and the model is called.
• Checked by hand with the same binary:
  • Codex offers each server as the namespace mcp__<server>. It adds list_mcp_resources, list_mcp_resource_templates and read_mcp_resource once any MCP server is configured.
  • tool_timeout_sec ends a slow call with an error; without it, a call times out after 300 seconds. After a timeout, the next call to the same server waits until the first returns.
  • Against fix: decouple GDPVal and Stirrup #4178's GDPVal resources server with expose_tools_over_mcp: true, Codex sees the four task tools, its finish call is recorded for the session, and /verify persists the submitted file.
• pre-commit run --all-files: passed, with no files changed.

Rollouts

GDPVal with #4178 (before it merged) plus this PR, the HOME change from #3902 and #4591:

• Model: an internal checkpoint served with vLLM.
• Codex 0.144.4 in Apptainer task sandboxes, with the harness-neutral prompt and the task tools over MCP.
• These ran on this PR's first revision. Later commits added the opt-in mcp_tool_timeout_seconds (unset in these runs), skip optional grants whose names Codex cannot use, and otherwise changed only the error text, observation gaps, docs and tests. The entry written for GDPVal's gdpval_resources_server grant is unchanged.

Results:

• Smoke on fix: decouple GDPVal and Stirrup #4178 at bf9f8db, 3 tasks: each called finish once over MCP, and the files were saved under task_<id>/repeat_0/. 81 model calls, all HTTP 200.
• Smoke on fix: decouple GDPVal and Stirrup #4178's earlier head, 3 tasks: each called finish once over MCP, and the submitted workbooks were persisted with finish_params.json. 56 model calls, all HTTP 200.
• Full run, 220 tasks:
  • 217 runs called finish (one of them twice).
  • One ended with a summary message without calling finish, and one hit the 3,600-second Codex timeout.
  • One failed when the model ran pkill -f python3, which killed the Codex runner (fix(agents): hide the sandbox supervisor and runners from pkill python #4805).
  • 6,001 model calls, with no context-length errors.
  • Scores with the official judge panel: ELO 1184.9 and rubric 82.3 over 217 tasks, against 1102.5 and 81.2 for the earlier Codex run counted the same way. Details are in feat(gdpval): run Codex in place of Stirrup #4591.

Compatibility and benchmark impact

• Sessions without MCP grants are unchanged: no mcp_servers entry is written.
• Optional MCP grants, which sessions used to ignore, are now written to config.toml. No config in the repo creates them today.
• The non-session path writes the same entries as before unless mcp_tool_timeout_seconds is set.
• An optional server that does not connect is left out of the tool list without an observation gap; only Codex's stderr shows it.
• With grants, the model also sees Codex's three MCP resource helper tools.
• The session token is readable inside the sandbox; it gives access only to that episode's tools.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Documentation added or updated, or N/A with justification.
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
• New source files include the required Apache-2.0 SPDX header. N/A: no new files.
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[athitten]

/ok to test ef74ead

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4590-452cfc2d2b9d.docs.buildwithfern.com/nemo/gym

[svcnvidia-nemo-ci]

HOLD — 1 BLOCKER and 1 NOTE.

MCP grant wiring and required-server failure handling are covered by focused tests, but truncated multiword header values can still expose secrets in startup diagnostics. The newly enabled session path also records MCP event categories instead of tool names in close observations.

Completed a source-only review of all seven changed files and their directly affected grant, configuration, observation, and session-close contracts. Tests were inspected, not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[athitten]

/review reset=true

[athitten]

/ok to test 55ba5a6

[svcnvidia-nemo-ci]

HOLD — 2 BLOCKERs.

The prior truncated-multiword counterexample and MCP observation-name mismatch are fixed in the current source. Two distinct redaction failures remain: headers sharing a prefix can interfere with one another, and JSON-escaped header echoes bypass matching. Both can leave secret fields in activation errors and persisted startup-failure gaps.

Completed a source-only review of all seven changed files and the directly affected grant, configuration, observation, and session-close contracts. Tests were inspected, not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[athitten]

/review reset=true

[athitten]

/ok to test 0f7aa16

[athitten]

Rebased onto main c62f9cc (#4740) as 452cfc2; the PR's changes are unchanged apart from following #4740's reworked error and observation code.

[athitten]

/ok to test 452cfc2

[yaoyu-33]

Thanks for the rebase. Re-reviewed at 452cfc2.

• The rebase onto #4740 and #4854 carries every change in this PR over to main's new activation code exactly once:
  • grants written to config.toml;
  • the seed refusals, with main's "Codex in the task sandbox" wording;
  • required servers stopping Codex before the first model call;
  • the redacted stderr tail in the 502 and in the agent_startup_failed gap;
  • the mcp_server_skipped gaps;
  • observations kept when close interrupts the stderr read.
• calls_model_from_sandbox and main's fern wording survive. Comparing the PR's tests before and after the rebase, no assertion was weakened; the only change is the switch to main's identical pinned_cli() helper.
• pytest responses_api_agents/codex_agent/tests at this head: 318 passed, 51 skipped (Linux-only and CODEX_TEST_BIN cases), and 1 failure from the OpenSandbox SDK missing locally, unrelated.
• Optional: the startup-failure tests run only on the CLI-events fallback path, because the fake sandbox never publishes native_session.jsonl. We checked that the native-transcript path also gets the stderr tail and exactly one gap. A parametrized case with a minimal native transcript would keep that from regressing.

No further comments from us.

[yaoyu-33]

Approving at 452cfc2. The rebase onto #4740 and #4854 keeps every MCP-grant behavior once, with main's wording and calls_model_from_sandbox, and the codex_agent tests pass.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39949344842