[yaoyu-33]

Integration with merged Codex support — October 8

Head 3f68583a8a1e4794cf82122576d94294b74b38af merges main acd53a9249b40938b2a3be01c07588ecf0308197 after #3640 landed. SWE-Pro permits both pi_agent and codex_agent, preserving every existing allowed agent. Both native-default composition regressions remain intact. The migration assertions also retain #4141's Environment Server name-preservation semantics.

Validation on this integrated source: 977 passed, 12 Linux-only skips on macOS/Python 3.13.14, with verified source-bound imports. The selection covers migration, global configuration, EnvironmentServer, base model capture/HTTP behavior, streaming, and both Pi and Codex app/native-session tests. Full pre-commit run --all-files and the diff whitespace check against main passed; the merge commit has DCO sign-off.

python -S -m pytest tests/unit_tests/test_add_legacy_agent_environment_servers.py tests/unit_tests/test_global_config.py tests/unit_tests/test_environment_server.py tests/unit_tests/test_base_responses_api_model.py tests/unit_tests/test_responses_api_model_streaming.py responses_api_agents/pi_agent/tests/test_app.py responses_api_agents/pi_agent/tests/test_sandbox.py responses_api_agents/codex_agent/tests/test_app.py responses_api_agents/codex_agent/tests/test_native_sessions.py -q

PYTHONPATH was set to this isolated worktree and the dependency-only Python 3.13 site-packages. Pi runtime files are unchanged from 85d2b4629; Codex, shared runtime code, and the launcher race fix match merged main. Fresh model rollout is N/A for the configuration/test conflict resolution; prior real-model and native SWE-Pro evidence remains pinned to its recorded revisions.

Required CI on 3f68583a8a1e4794cf82122576d94294b74b38af has passed: Nemo_CICD_Test, including core/sandbox tests and all eight server shards, plus lint and DCO. The overall workflow is marked failed because the optional harness-conformance diagnostics failed; these are non-blocking and have not been suppressed or changed. Fresh approval and normal merge-queue validation are still required. Earlier successful CI on 85d2b4629 remains historical evidence only.

Review compatibility follow-up — October 7

Head 520b851fa85a9ed4f09335e35f0ced660893a3bf addresses the existing-consumer review request. Native sandbox sessions retain strict request validation. Calls without a native session preserve the existing AnySWE, AnyTerminal, and HarnessAgent contract: forwarded model/sampling/token fields no longer raise HTTP 422; configured Pi controls remain authoritative. The README explicitly describes this compatibility behavior. The separate pi_sandboxed_agent introduced by merged #3575 remains supported.

Validation on the committed source (macOS arm64, Python 3.13.14):

• Two new regression cases failed before the fix and pass afterward, covering real ASGI requests and HarnessAgent's scope-less request shape, response/usage preservation, configuration ownership, and request immutability. Native unsupported-control rejection remains tested.
• Pi app/session tests: 186 passed, 6 Linux-only skips.
• Pi, PiSandboxed, AnySWE, AnyTerminal, HarnessAgent, and migration tests: 405 passed, 28 skipped, 3 local failures. All three failures are unchanged AnyTerminal Docker-provider constructor tests because this host has no Docker binary; full Linux CI remains required.
• Source-bound test command: python -S -m pytest responses_api_agents/pi_agent/tests responses_api_agents/pi_sandboxed_agent/tests responses_api_agents/anyswe_agent/tests responses_api_agents/anyterminal_agent/tests responses_api_agents/harness_agent/tests tests/unit_tests/test_add_legacy_agent_environment_servers.py --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short with this checkout and dependency-only site-packages on PYTHONPATH.
• python -S -m pre_commit run --all-files and git diff --check passed; commit is DCO-signed.
• Fresh real-model compatibility smoke: 2/2 passed, stock math verifier reward 1 each, Pi 0.80.2, NIM nvidia/nemotron-3-super-120b-a12b. Both request shapes forwarded the previously rejected fields, executed actual Bash/file tools, returned nonzero usage, preserved caller inputs, and removed their temporary Pi workspaces. Verified the tested runtime hash matches the committed file. These are local compatibility-path smokes, not new native sandbox lifecycle or SWE-Pro accuracy results.

Full current-head CICD 37659390982 passed on 520b851fa85a9ed4f09335e35f0ced660893a3bf, including core/sandbox tests, all eight server shards, and the aggregate gate. Wheel validation, lint, copyright, secret scanning, and DCO also pass. Final review recheck found no new comments or inline threads. CI was started with /ok to test for this exact commit.

---

Shared contracts — October 6

Head 730765c70. Aligned the common implementation across #3626, #3640, #3637 and #3639:

• One upstream HTTP-error handler in SimpleResponsesAPIModel; preserve status/body for OpenAI and vLLM, including errors before buffered Responses streaming starts. No adapter-specific duplicate.
• Identical native-template migration rules and shared regressions. Explicit-null native templates stay native; bound/inherited Resources still use compatibility routing.
• Shared sandbox lifecycle helpers still match main. Harness runtimes, request policies, benchmark settings and scoring are unchanged; Codex-specific Responses compatibility stays scoped to Codex.

Validation: macOS/Python 3.13.14 selected suites 1,741 passed / 36 skipped; six final HTTP-error regressions and all-file pre-commit pass. Skips include Linux and optional runtime tests. No fresh Linux/model rollout or full-repository CI result is claimed for this update. Exact scope, commands and evidence.

---

Main sync — October 6

Synced with main 045dae678 after #3961; head 5c3f3fcf4. Shared agent_utils and Hermes files now match main and are absent from this PR's diff.

• Preserve main's failed-attempt ownership capture and this PR's HTTP-error forwarding. Exempt the explicit-null Pi native template from legacy migration, with bound/inherited-resource and real-template composition regressions.
• Final-source macOS/Python 3.13.14: 1,589 passed, 47 skipped; 17 subtests passed. All-file pre-commit passes; Fern check: 0 errors, 1 warning.
• No fresh Linux or real-model run for this synchronization. Earlier rollout evidence below applies only to its recorded source pins. Full mirrored repository CI is separate; no trust-gate approval was posted.

---

Pi runs inside a borrowed task sandbox, or creates one using its configured provider when no sandbox is supplied. EnvironmentServer closes the agent before verification.

Implements the native sandboxed-harness session lifecycle. Targets main; shared helpers/tests match #3961 at 2093960be.

Added / expected

• Use nemo_gym/agent_utils and the updated SandboxSession contract: session_dir, stage_activation, stop_harness.
• The session uploads the supervisor, owns control files/deadlines and exposes diagnostic logs. Adapters keep installation, argv, runtime metadata and output parsing.
• Preserve retryable cleanup, delayed-launch fencing and partial output before release. Missing optional diagnostics do not erase valid output; positive cleanup confirmation remains required.
• Update existing Hermes supervisor imports without taking its separate config refactor. Merge main at d04762236.

Validation — d91469b

• macOS arm64 / Python 3.13.14: 1,137 passed, 45 skipped, covering the harness, shared lifecycle/provider/session, both Hermes adapters, SWE-Pro, EnvironmentServers and model components.
• Full pre-commit run --all-files, focused shared-module mypy and whitespace checks pass; all new commits have DCO sign-off.
• Offline wheel build passes; packaged helpers match source byte-for-byte, with no obsolete helper paths.
• Shared implementation and its three test files match refactor(hermes): decouple configuration and share sandbox lifecycle #3961 byte-for-byte. Added checks for activation-time upload and shared-log diagnostics when output is missing.
• Fresh Linux/model smoke is pending workstation approval. Earlier real-model and benchmark results below apply only to their stated commits.
• Commands, evidence and handoff.

Concerns / limits

Full current-head CI and SWE-Pro accuracy have not been rerun. Skips include Linux-only process tests and optional runtime checks. Pi's existing legacy request-control compatibility work remains separate. This update does not change model settings, rewards, benchmark recipes or TTL defaults. No README command/config change is required; internal API migration is documented here and in the shared package docstrings.

Earlier revision-pinned implementation and validation

The following previous description is historical, not evidence for this head.

Pi runs inside a borrowed task sandbox, or creates one using its configured provider when no sandbox is supplied. EnvironmentServer closes the agent before verification.

Implements the native sandboxed-harness session lifecycle. Targets main; adopts the shared lifecycle from #3961 at 614a08b02.

Added / expected

• Reuse refactor(hermes): decouple configuration and share sandbox lifecycle #3961's SandboxSession for execute/close, cleanup confirmation and owned/borrowed release. Harness adapters retain runtime installation, commands and output parsing.

• Keep positive cleanup proof mandatory. Malformed optional runtime/exit diagnostics become explicit gaps; valid harness output is preserved, and an unknown exit code is never fabricated as zero.

• Capture partial output before releasing the sandbox. Merge current main and preserve the model bridge's upstream-error, streaming and input-immutability contracts.

• Pi's existing request-control policy is unchanged: request-level max_output_tokens is rejected; configured max_output_tokens remains a per-model-call cap. The older AnySWE/HarnessAgent compatibility request remains separate.

Validation — 6140623

• macOS/Python 3.13.14: 491 passed, 39 skipped.
• Linux x86_64/Python 3.13.14: 519 passed, 11 skipped, including process-supervision, cancellation and cleanup-retry tests.
• Full pre-commit run --all-files, focused shared-module mypy and git diff --check pass; commits have DCO sign-off.
• Fresh real-model smoke: 2/2 pass, reward 1 each, one owned and one borrowed sandbox. Pi 0.80.2, private Node 22.19.0, Docker and NIM Nemotron-3-Super-120B-A12B. Checked actual tool execution, file output, nonzero usage, cleanup receipts and close-before-verify ordering. No task containers remain.
• Exact source hashes were verified before/after execution. Commands, evidence and handoff.

Concerns / limits

The new smoke uses a small coding task and the stock math verifier; it is not a new SWE-Pro accuracy result. Full-dataset scores below apply only to their recorded revisions. Eleven Linux optional Node/extension tests skip; pinned Pi itself ran in the model smoke. Full repository CI is separate: currently reported lint, copyright, secret and DCO checks pass; no current-head full-CI result is claimed.

Historical revision-pinned validation and implementation notes

The following is the previous description, preserved for evidence. References to old bases, draft status, strict metadata validation and pending Linux/model checks describe those older revisions, not the current head.

Pi runs inside the task sandbox supplied by the benchmark's Resources Server. The Environment Server owns setup, activation, verification, and cleanup; Pi borrows the sandbox and runs its own model/tool loop.

Load responses_api_agents/pi_agent/configs/pi_agent.yaml for the standard path. SWE-bench Pro remains independent in resources_servers/swebench_pro/configs/swebench_pro.yaml. The run binds pi_agent and swebench_pro_resources_server through the Environment Server; there is no Pi-specific benchmark recipe or combined deployment name. The README uses prepared flat rows through single_agent_turn_legacy, without a separate materialization script. Model selection defaults to ${policy_model_name}.

Implements MB-1694; evaluation is tracked in MB-1712. Targets main after #3554 merged.

• Reuse shared agent-session seed/close and receipt retention. Identical activation retries join or replay one invocation; losing an HTTP waiter does not cancel it. The Environment Server and provider TTL own cleanup/expiry.

• Reuse shared Linux process supervision. An atomic launch claim fences delayed starts, and verification requires confirmed descendant cleanup. Pi never destroys its borrowed sandbox. Include a separate shared-supervisor fix for a child created before Popen loses its handle (reported on fix(hermes): harden sandbox sessions #3554).

• Compatibility change: local and sandbox calls now concatenate the configured system prompt, request instructions, and input system message in that order. Unsupported non-default request controls, including developer messages, are rejected instead of silently ignored. Request-level max_output_tokens is rejected because total-response budgeting is unsupported; the configured Pi max_output_tokens remains a per-model-call cap.

• Preserve upstream HTTP status and error body in the shared OpenAI Model Server so SDKs can recognize context-limit errors instead of seeing a generic 500.

• Provider/API and runtime errors propagate without benchmark verification. Model limits/timeouts preserve gradable partial work. Context overflow uses pinned Pi's own classifier; observations survive failed activations.

• Install private Node and pinned Pi outside the repository, supporting Python 3.8+, glibc x86_64/aarch64, and musl x86_64. Preserve task runtimes, installer stdout/stderr, npm script restrictions, bash deadlines, and model HTTP deadlines. The pinned Node build does not support arm64 musl.

• Confirm supervisor cleanup before cancelling provider execution; check remote receipts before signaling stored PIDs; reject non-finite or nonpositive deadlines and access to partially seeded sessions. These focused follow-ups address the latest fix(hermes): harden sandbox sessions #3554 review comments.

Validation:

• Final Pi head ca7d42212: 642 tests passed across Pi, Hermes session/runner integration, shared sessions/process supervision, Environment Server, and Model Server capture/streaming/messages. Tests include real Linux process-group cancellation, stale-PID protection, delayed launches, Python 3.8, Node extensions, and Pi 0.80.2 SDK behavior. Coverage is 96.07% across six affected Python runtime modules. Full pre-commit run --all-files passed; DCO sign-offs present.
• README configuration resolves independent benchmark/agent references and inherits the model ID. No suffixed Pi configuration or combined benchmark/agent recipe remains. Reviewed all 44 inline comments and 15 reviews on fix(hermes): harden sandbox sessions #3554 through its October 1 review, plus issue discussion; follow-up findings.
• Completed HSG full evaluation: 358/731 (48.97%), 731 graded, 0 ungraded, 0 agent/verifier errors. Repaired historical baseline: 359/731 (49.11%), 730 graded, 1 ungraded. Difference: one solved task / −0.137 percentage points; 39 pass→fail, 38 fail→pass. Changed failures were inspected against raw test logs and show candidate assertion/build/import failures, not new harness or grading infrastructure failures. One stochastic run cannot establish identical outcomes or prove absence of all regression.
• Independently confirmed complete required-test evidence for all 358 reported passes; matched all 731 original task inputs and immutable image digests. Audited 36,953 model calls, 46,903 tool calls, and 1,462 successful agent/resources close receipts. A real upstream HTTP 400 context limit was preserved, compacted by Pi, and followed by continued model calls and normal grading. One task hit its 32,768-token output cap and was normally graded zero.
• Same N35 Super GA checkpoint and serving setup as the repaired baseline: TP4, four GB200 GPUs, 262,144 context, Pi 0.80.2 with high thinking, temperature 1 / top-p .95, 32,768 output tokens per call, concurrency 32. Full evaluation source is frozen at 28df43c18, consisting of Pi runtime 0c13326ce plus the separately disclosed, hash-pinned historical benchmark parser repairs. Later test-only commit 8300a1cef had byte-identical runtime. The subsequent guard changes in ca7d42212 are not included in that full result.
• The first 373 original attempts and remaining 358 original attempts cover disjoint original IDs with no retries or replacement. The first controller stopped dispatch during a roughly seven-minute model API stall, so its W&B run is marked failed although all 373 started episodes returned normally. The same model allocation recovered without restart and served the continuation. Both sandbox cleanup inventories are empty, and the owned model was cancelled.
• Before that full run, HSG setup passed 338 tests (10 optional checks skipped on the CPU controller and covered locally), a real two-turn model tool preflight passed, and the 8-task smoke produced 8 graded / 6 solved, zero agent/verifier errors, 411 model calls, 522 tool calls, and 16 close receipts.
• Final guard-version HSG validation completed: 360 setup tests passed (10 optional runtime checks skipped and covered locally), the real two-turn tool preflight passed, and the 8-task smoke returned 8 graded / 5 solved / 0 agent or verifier errors. Snapshot dc1550231 contains Pi ca7d42212 plus the same benchmark repairs. Audited all 406 model calls, 522 tool calls, 406 available model references, and 16 successful close receipts; all five passes have complete required-test evidence. Zero results: a NodeBB assertion failure, ProtonMail UI assertion failures, and the previously observed Flipt Secure-field compilation failure. All 6,134 source entries match after execution; the run is finished, sandbox inventories are empty, and the owned model was cancelled. This validates the guard changes with real rollouts; it is separate from the earlier 731-task result.

The upstream target mirror NVIDIA-NeMo/Gym:codex/hermes-swe-pro-validation is synchronized with #3554 at d5f13f54cf762c7e687dd2d50d2583efe2c9b717. The mirror advanced by 11 commits using a fast-forward. Pi head ca7d42212 is unchanged, so the recorded validation still applies; the diff against the synchronized base contains 23 changed files.

The inherited SWE Pro response still reports incomplete verification separately via evaluation_completed=false; changing its default reward/mask contract is flagged for the shared benchmark rather than silently changed here. Full-run accuracy and grading coverage are reported separately.

Shared runner integration (2026-10-02)

Integrated the approved split-contract refactor in 5741f9f4fd816f6c4e79ac1002793adb1db6c3af from draft #10 / draft #11. Cleanup uses the existing supervisor CleanupReceipt; worker identity uses RunnerRuntimeInfo; harness output stays separate. No combined SandboxRunnerResult.

Only the Pi callers, shared helper, and relevant tests were integrated here. Hermes-specific callers and regressions are routed to #3961. The shared helper and its tests are identical across all three PRs. Remote supervisor/worker files, model requests, sampling and scoring logic are unchanged; malformed cleanup/runtime metadata is now rejected. A fenced, unstarted worker retains an unknown exit code, and runtime failure does not discard confirmed cleanup.

Validation on this integrated head (macOS, Python 3.13.14; pinned Hermes source and existing dependency environments; localhost fake model/MCP servers): 448 passed, 37 skipped. Skips include Linux-only process-supervision and optional runtime/CLI checks. Full pre-commit run --all-files and git diff --check passed; commit is DCO-signed.

python -m pytest tests/unit_tests/test_sandbox_runner_transport.py \
  tests/unit_tests/test_agent_sessions.py tests/unit_tests/test_agent_session_retries.py \
  tests/unit_tests/test_process_supervisor.py \
  responses_api_agents/pi_agent/tests responses_api_agents/hermes_agent/tests \
  --import-mode=importlib -q -o addopts='' -o log_cli=false

CI classification: full (shared core changes). No fresh real-model rollout or Linux rerun was performed for this integration; prior rollout evidence above remains pinned to its stated revision and is not evidence for this new head. Existing PR readiness/labels are unchanged.

Rebase to main — October 2

Targets main; #3554 is merged. Current-head macOS/Python 3.13.14 checks: 337 passed, 26 skipped, full pre-commit and whitespace checks passed. Linux-only/optional tests remain skipped; no fresh real-model run was performed. Earlier rollout evidence remains tied to its recorded commits.

HSG Linux test follow-up (2026-10-03)

Published test-only commit 9f28b7278dfc53a292a128587520b92af12e0fdf. Isolate the stdlib event-producer subprocess with -I -S and include cleanup/runtime/event logs before decoding failed output. The original intermittent empty capture remains unexplained; a controlled slow site hook fails before the patch and passes afterward under the same deadline.

HSG Linux aarch64/Python 3.13.14: 366 passed / 13 skipped, plus 30/30 distinct executions of the affected test. Full local pre-commit run --all-files and whitespace checks pass; commit has DCO sign-off. The listed GitHub lint, copyright, secret and DCO checks also pass. Full repository/component CI remains separate.

python -m pytest responses_api_agents/pi_agent/tests \
  tests/unit_tests/test_base_responses_api_agent.py \
  tests/unit_tests/test_agent_sessions.py tests/unit_tests/test_agent_session_retries.py \
  tests/unit_tests/test_episode_types.py environment_servers/single_agent_turn/tests \
  tests/unit_tests/test_sandbox_runner_transport.py tests/unit_tests/test_process_supervisor.py responses_api_agents/pi_sandboxed_agent/tests responses_api_models/openai_model/tests \
  --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short

Executed with source-bound imports and compatible dependency-only site-packages; shared-board evidence, exact drivers, pins, jobs, skips and limitations. All four selected HSG suites pass 1,860 tests with 20 skips. Prior owned/borrowed real-model smokes remain pinned to their original revisions; these later test-only edits do not change production behavior and no model rerun is claimed.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-codex-pi-sandbox-native-session.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/agent-server/pi-sandbox-sessions

[cmunley1]

can you please check that this does not break consumers of this, such as anyswe_agent, anyterminal_agent, and harness_agent based environments? perhaps it can be pi_sandboxed_agent similar to opencode_sandboxed_agent? Thanks!

[cmunley1]

please also see #3575 which adds pi_sandboxed_agent

[yaoyu-33]

/ok to test 3f68583

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37833861517