[athitten]

What does this PR do?

Adds benchmarks/gdpval/codex.yaml, which runs GDPVal with Codex in place of Stirrup, using the same resources server, judges and multi-stage driver, so Codex can be compared with Stirrup and Pi on the same tasks and scoring:

export PERSIST_DELIVERABLES_DIR=$PWD/output/gdpval_codex/deliverables
gym eval prepare --benchmark gdpval/codex
gym eval run --model-type openai_model --benchmark gdpval/codex --split benchmark --output results/gdpval_codex.jsonl \
    --model-url <vllm_base_url> --model-api-key <vllm_api_key> --model <served_model_name> \
    '++policy_model.responses_api_models.openai_model.extra_body={temperature: 1.0, top_p: 0.95, max_output_tokens: 32768}' \
    '++policy_model.responses_api_models.openai_model.propagate_upstream_http_status_codes=[400]'

• Codex runs in the task sandbox the resources server lends it, with /root as its working directory.
• It reaches finish, abandon_task_finish, web_search and fetch_web_page over MCP: expose_tools_over_mcp: true on the resources server and resources_tool_transports: [mcp] on the environment server. The model sees them as mcp__gdpval_resources_server__<tool>.
• codex.yaml includes config.yaml, removes the Stirrup agent with gdpval_stirrup_agent: null and points gdpval_environment_server at Codex, as pi.yaml does.
• Codex settings:
  • version 0.144.4 with a 262,144-token context;
  • concurrency: 32, as pi.yaml sets it: at least the environment server's max_concurrent_episodes;
  • compaction at 196,608 tokens. Codex checks after each reply that calls a tool, so the context can pass the limit by about a reply and its tool output, and the compaction call reserves 32,768 tokens for its own reply. A near-full reply with a large tool result can still overflow the compaction call; in the run below, 7 runs compacted and none overflowed;
  • timeout: 10800 for the Codex process, inside the environment server's 18,000 s episode timeout;
  • mcp_tool_timeout_seconds: 900. A Codex MCP call that times out keeps running on the server and holds back the next call to it (such as finish), so the timeout stays well above a slow web call.
• Codex sends no sampling or per-call output limit to a Gym model server, and the Codex agent rejects them in the request. So the command sets them in openai_model's extra_body, as in the run below. It also passes vLLM's context-length error to Codex as an HTTP 400, so such a run fails within seconds with vLLM's message; Codex does not compact on it.
• finish does not end a Codex run. Codex continues until the model replies without a tool call; a later finish or abandon_task_finish replaces the recorded call, and /verify collects the files at the end.
• The README's "Run another harness" section gets the Codex command and settings.

It builds on #4589 and #4590, both now on main: #4589 adds the harness-neutral rows, the harness config test and the README section, and #4590 lets Codex sandbox sessions use MCP tool grants (without it, every gdpval/codex episode would fail at agent setup with HTTP 422). This PR's commit (5916dea77) also adds codex to that config test, which builds each agent's config class, so a misspelled key or a value that breaks a validator (for example mcp_tool_timeout_seconds above timeout) fails the test.

It also relies on #3902, now on main, which keeps the task's HOME, so a file the model saves under ~ is still in the sandbox at /verify.

No issue: this ports athitten#1 (Codex × GDPval on the yaoyu-33#8 prototype) to the merged #4178 design.

Validation

At 5916dea, rebased onto main ab25f45 after #4589 and #4590 merged. git range-diff shows the commit unchanged from cc82ea2, the approved head.

• pytest resources_servers/gdpval/tests in a fresh venv from the server's requirements.txt, with cryptography installed: 864 passed, 1 skipped (it needs Stirrup installed). A misspelled key or a mcp_tool_timeout_seconds above timeout in codex.yaml fails the harness config test.
• pytest tests/unit_tests/test_benchmarks.py tests/unit_tests/test_discovery.py tests/unit_tests/test_registry.py tests/unit_tests/test_cli.py: 287 passed, 1 failed. The failure is a workstation issue: its hostname resolves to 127.0.1.1, so four configs this PR does not touch (apex_shortlist and hle) are skipped by main's sandbox-host check.
• pytest responses_api_agents/codex_agent/tests tests/unit_tests/test_agent_outcome.py with CODEX_TEST_BIN set to Codex 0.144.4: 409 passed, 1 skipped (it needs Python 3.8).
• The checks below ran at an earlier head, ff00cd9. Since then, only the base and the config test changed.
• The README command, through gym eval run's argument handling, gives openai_model the model flags, the extra_body and the 400 setting, and resolves with no warnings. gym eval prepare --benchmark gdpval/codex writes the 220 harness-neutral rows.
• End to end on this commit: real Codex 0.144.4 in the Codex agent's sandbox session, main's GDPVal resources server over MCP with Gym's local sandbox provider, and the real openai_model server configured by the README command, in front of a scripted Responses engine. The host's Node and Codex stood in for the in-sandbox install.
  • The model saw the four tools as mcp__gdpval_resources_server__<tool>.
  • finish, abandon_task_finish, and finish with web_search in one turn each ended after one closing model call, and /verify saved finish_params.json and the files. A later finish or abandon_task_finish replaced the earlier one.
  • Every upstream call had temperature 1.0, top_p 0.95 and max_output_tokens 32,768; without the overrides, none did.
  • vLLM's context-length 400 ended the run at once with vLLM's message, and /verify was not called; with the default HTTP 500, Codex retried the request. A run that timed out after finish reached /verify, which saved its submission.
• pre-commit run --all-files: passed, with no changes.

Rollouts

Smoke of the previous head ff00cd9 (main 4c66403 with #4589, #4590 at cd4ace0 and this PR's commit), on a GPU cluster with the same model as the runs below. It used the README's gym eval prepare and gym eval run commands with the placeholders filled in, --limit 3, and its own output, log and deliverables paths. The cluster has no OpenSandbox, LibreOffice or Tavily key, so a local env.yaml ran the task sandboxes under Apptainer with the GDPVal image and turned judging off, and TAVILY_API_KEY was a placeholder (no run called web_search).

• No failure rows; all 83 model calls returned HTTP 200, and none hit the output limit or needed compaction.
• Each run called finish once over MCP, and /verify saved the files and finish_params.json under task_<id>/repeat_0/.
• After finish, Codex made one more model call in each run: a closing message with no tool call.

The earlier smoke and the 220-task run below used #4178 before it merged. Smoke on #4178 at bf9f8db, merged with main, an earlier revision of #4590 and #4586 (LibreOffice command check), 3 tasks with the same model and sandbox as the full run below and this config's defaults:

• Each run called finish once over MCP, and /verify saved the files, finish_params.json and the reference files under task_<id>/repeat_0/.
• After finish, Codex made one more model call in each run: a closing message with no tool call.
• 81 model calls, all HTTP 200; no failure rows.

220 GDPVal tasks with Codex 0.144.4, from #4178 at fd1cb9e merged with main plus an earlier revision of #4590, the HOME change from #3902, and #4586:

• Model: an internal checkpoint served with vLLM, 262,144-token context, reached through openai_model with the README command's extra_body (temperature 1.0, top_p 0.95, 32,768 output tokens per call) and propagate_upstream_http_status_codes: [400].
• Codex process timeout: 3,600 s, as in the earlier Codex run. This config's default is 10,800 s, as in pi.yaml.
• The run predates openai_model's 1,800 s limit per model call (fix(openai): jittered backoff, Retry-After and a default provider timeout #4786); its slowest call took 335 s.
• Sandbox: Apptainer with the GDPVal image.
• Judging: generation with judge_deliverables: false, then the GPT-5.5 / Gemini 3.1 Pro / Opus 4.8 panel, with:
  • multistage ELO with 18 references: 45 tasks against all 18, then all 220 tasks against 4;
  • a rubric re-judge with gym eval reverify.

Scores, compared with the earlier Codex run (athitten#1) counted the same way:

	This PR	athitten#1
ELO, a task without a deliverable counted as lost	1184.9	1102.5
ELO as reported (comparison/eval_elo)	1186.1	1104.2
Rubric over the same 217 tasks, no submission = 0	82.3	81.3

• ELO final stage:
  • 216 of 220 tasks judged; accepted at 98.2% coverage, not degraded, no imputed losses.
  • Left out: 3 tasks whose files are too large for every judge (the same 3 as before) and the run whose Codex runner was killed, which never reached /verify. The 2 runs that did not call finish were judged as empty submissions.
• Rubric:
  • Left out: 2 tasks over a judge's input limit (the same 2 as before) and 1 task that GPT-5.5 rejected three times.
  • The 3 runs that did not submit count as 0. Over all 220 tasks, with unjudged tasks as 0: 81.1 against 80.5.
• Generation:
  • 217 runs called finish (one of them twice).
  • One ended with a summary message without calling finish, and one hit the 3,600 s timeout.
  • One failed when the model ran pkill -f python3, which killed the Codex runner.
  • 6,001 model calls, with no context-length errors. 7 runs compacted their context (8 compaction calls).
  • 28 calls hit the output limit, in 15 episodes; Codex continued after each.
• No web_search call worked, because the run used a placeholder Tavily key. fetch_web_page worked.

The setup is not identical to the earlier run: harness-neutral prompt, finish over MCP, a page-fetch tool, compaction at 196,608 tokens. So part of the difference comes from these setup changes.

Compatibility and benchmark impact

• New config only: config.yaml and the Stirrup path are unchanged.
• Codex scores use the harness-neutral prompt, so they are not AA v2 scores.
• Codex keeps about 12 KB of each tool result, while GDPVal's web tools return up to 40,000 characters, so Codex sees only the start and end of a long page.
• A Codex run that reaches timeout is judged on what it submitted before the timeout (nothing if it had not called finish) and is not retried. Stirrup reports the same event with stop_reason: wall_time, which GDPVal masks and retries. One run above timed out.
• web_search needs a valid TAVILY_API_KEY: gym eval run stops if it is unset, and with an invalid key each search returns an error to the model, as in the run above.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Documentation added or updated, or N/A with justification.
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yuhezhang-ai]

Just curious while scanning through some new PRs: I noticed #4587 has Pi stop after a successful finish, while Codex here can keep working afterward. Is that intentional, or will be unified later?

[athitten]

@yuhezhang-ai Intentional for now. Pi can end a run when a tool result sets terminate: true, so #4587 uses that for finish. We found no way for a tool call to end a Codex 0.144.4 run, so Codex keeps going until the model stops; /verify keeps the last finish call and collects the files at the end. In our 220-task run, 216 of the 217 Codex runs that called finish called it once. We can unify this if Codex gets a way to end a run on a tool, or if the environment server ends the episode on finish.

[athitten]

/ok to test 2eb11d4

[svcnvidia-nemo-ci]

HOLD — 1 RISK: the new Codex preset cannot run against the reviewed head because its declared session-MCP prerequisite is absent.

Reviewed all seven changed files and their directly affected preparation, agent-session, MCP, and verification paths. The harness-neutral preparation preserves the Stirrup row contract, and Pi’s configured finish behavior matches its adapter. Validation reported against a tree merged with #4590 does not establish compatibility of this immutable head.

Static review only; tests and rollouts were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[svcnvidia-nemo-ci]

RISK — Land the native Codex MCP prerequisite before enabling this preset.

Failure / blast radius: With otherwise valid sandbox/model settings, every gdpval/codex episode fails during agent setup. This transport makes the environment construct a required MCPToolAccess, but CodexAgent._seed_agent_session_state rejects every required grant with HTTP 422. Thus none of the 220 tasks reaches policy execution or verification. The description acknowledges #4590, but that implementation is absent here.

Correction: Rebase onto the complete prerequisite before merging, or include that implementation. Do not simply remove the rejection or make the grant optional: the native activation also calls _build_config without MCP servers, so that alone would run without finish and the web tools. The required invariant is that supported required MCP grants are accepted and configured with their authenticated URLs/headers before inference, while unsupported required HTTP grants still fail closed. The prerequisite must also declare and apply this preset’s mcp_tool_timeout_seconds; CodexAgentConfig currently has no such field. This needs the complete adapter change, not a YAML-only patch.

Preservation / acceptance — proposed checks, not executed: Add a test using this resolved preset and the environment-generated GDPVal grant: with sandbox installation mocked, session seeding must return 200, and generated TOML must contain the GDPVal MCP URL, session-token headers, required status, and 900-second tool timeout. A scripted episode must discover all four tools, call finish, close normally, and persist its submitted file through /verify; preserve the documented Codex behavior that finish itself does not terminate the run. Retain the existing no-grant borrowed-sandbox case’s 200 response and disconnect-without-stop assertions, plus the required-direct-HTTP case’s 422-before-connection assertion in the Codex session tests. Keep Pi’s terminal-finish behavior and Stirrup’s request tool list unchanged.

[athitten]

Fixed in 79076a4e5: this branch now carries #4590 at cd4ace0 as one commit, so seeding accepts the required GDPVal MCP grant and writes it to Codex's config with its URL, session-token header, required = true and a 900 s tool timeout. Checked end to end with Codex 0.144.4, and on 3 GDPVal tasks with a real model. I'll drop that commit by rebasing onto main after #4590 merges.

[athitten]

/ok to test ff00cd9

[athitten]

/review reset=true

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4591-cc82ea22bf03.docs.buildwithfern.com/nemo/gym

[athitten]

Restacked on main c62f9cc: #4589 (f0c065e), #4590 squashed at 452cfc2, then the Codex commit, unchanged.

[athitten]

/ok to test cc82ea2

[yaoyu-33]

Re-reviewed at cc82ea2, the Codex commit on top of #4589 (f0c065e) and #4590 (452cfc2).

• The config test now maps codex to CodexAgentConfig and builds each config, so the validators run. With mcp_tool_timeout_seconds: 20000 in codex.yaml, the codex case fails as it should. test_harness_configs.py and test_prepare.py: 11 passed.
• The branch merges cleanly now. The plan to rebase to the single Codex commit after #4589 and #4590 merge sounds right.

No further comments from us.

[yaoyu-33]

Approving at cc82ea2, the Codex commit. codex.yaml resolves, validates against CodexAgentConfig, and fits the episode timeout. The README's Codex claims we checked against Codex rust-v0.144.4 hold, and the config test now covers codex. The branch still carries #4589 and #4590, which are approved separately; please merge after them and rebase to this commit as planned.

Two non-blocking README corrections are inline, both about how Codex runs are scored, since readers will compare these numbers with Pi's and Stirrup's.

[yaoyu-33]

benchmarks/gdpval/README.md:489

Non-blocking: with propagate_upstream_http_status_codes: [400], a context-length failure after finish is judged, not skipped. Codex classifies the upstream 400 as model_request_error with status_code: 400 (codex_agent/app.py:567-575), and the environment server's default outcome policy sends 400 to verify (single_agent_turn/app.py:121-125). So the run is judged on the recorded finish, or as an empty submission without one. Could this sentence say that? Or, if skipping these runs is the intent, codex.yaml would need an agent_outcome_policy override.

[athitten]

Thanks @yaoyu-33, will add in the follow-up PR along with others

[athitten]

Addressed in #5076: README.md:490-495. With [400], the run is now described as judged on its last finish or abandon_task_finish, or as an empty submission, and the default HTTP 500 case as a failure that --resume reruns.

[yaoyu-33]

benchmarks/gdpval/README.md:473

Non-blocking: could the Codex bullets say how a timeout is scored, as the Pi section now does? Codex sets no stop_reason: wall_time, so GDPVal's agent_timeout mask never applies. A Codex run that reaches timeout is judged on its last finish, with the files as they are when it stops, or as an empty submission without one, and it is not retried. The PR body says this, but the README doesn't.

[athitten]

Will add in the follow-up PR along with others

[athitten]

Addressed in #5076: README.md:499-504. The Codex bullets now say how a timeout is scored. One change from your note: a timed-out run is rerun when Codex also reported an error during the run. A Pi run that times out while still working is now masked, as Stirrup's is, so the bullet ends with that difference.

[athitten]

/ok to test 5916dea

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39997461092