[yaoyu-33]

Launcher shutdown race — October 8

Head 32b7b230bb2a78300c2d0cd65a6db089211065cd fixes the recurring core CI shutdown failure. A TERM signal can arrive after a TP1 engine starts but before its PID is appended to worker_pids; cleanup then misses that process. Cleanup now uses the serving shell's job table, which already contains the child, and preserves the shutdown exit status. This also covers the router/Mooncake children owned by that shell. Model arguments, inference behavior and agent/verifier code are unchanged.

Validation: a deterministic reproduction of that exact launch/registration window failed before the fix with the same missing fourth-engine stop seen in CI, and passes after it. Added TERM and INT regressions requiring all four engines plus the router to stop; the five-second timeout and existing assertions remain unchanged. Source-bound launcher suite: 35 passed, 6 skipped, 199 subtests passed on macOS/Bash 3.2. The six existing skips require Bash 5.1+; Linux CI remains required. Full pre-commit run --all-files and git diff --check passed; commit is DCO-signed. Fresh real-model rollout: N/A for this launcher process-bookkeeping fix, validated with actual isolated child processes and signals; prior agent/model rollout evidence remains pinned to its revisions.

Full CI on head 32b7b230bb2a78300c2d0cd65a6db089211065cd passed, including Linux core/sandbox tests, all eight server shards and aggregate gates. All 31 current passing checks and 11 legitimate skips are complete; fresh reviewer approval remains required. This repairs the CI blocker directly; it does not treat the recurring shutdown failure as a transient runner issue.

Merge queue compatibility — October 8

Head cc532c42d46e289a6b931ba7b0b10c5e7c4470d7 merges current main and aligns the native-capable migration regression with #4141: an agent rename preserves the existing Environment Server's name. The test also verifies the rename overlay stays untouched, the source receives exactly one server, and the renamed agent retains its Resources binding.

The queue run failed in those five parameterized cases. The new commit differs from that exact queue tree only in the migration test; it applies the same correction validated for #3626.

Validation on this head: 267 passed using verified source-bound imports for tests/unit_tests/test_add_legacy_agent_environment_servers.py, tests/unit_tests/test_global_config.py, and tests/unit_tests/test_environment_server.py. Full pre-commit run --all-files and whitespace checks against main and the prior queue tree passed. Commit has DCO sign-off. Full new-head CI was requested with /ok to test cc532c42d46e289a6b931ba7b0b10c5e7c4470d7 and remains pending.

Fresh model rollout: N/A for this test-only correction relative to the queue tree; prior model evidence remains pinned to its recorded revisions.

Shared contracts — October 6

Head 5a2e582dc. Aligned the common implementation across #3626, #3640, #3637 and #3639:

• One upstream HTTP-error handler in SimpleResponsesAPIModel; preserve status/body for OpenAI and vLLM, including errors before buffered Responses streaming starts. No adapter-specific duplicate.
• Identical native-template migration rules and shared regressions. Explicit-null native templates stay native; bound/inherited Resources still use compatibility routing.
• Shared sandbox lifecycle helpers still match main. Harness runtimes, request policies, benchmark settings and scoring are unchanged; Codex-specific Responses compatibility stays scoped to Codex.

Validation: macOS/Python 3.13.14 selected suites 1,779 passed / 53 skipped; six final HTTP-error regressions and all-file pre-commit pass. Skips include Linux and optional runtime tests. No fresh Linux/model rollout or full-repository CI result is claimed for this update. Exact scope, commands and evidence.

---

Main sync — October 6

Synced with main 045dae678 after #3961; head e4bf3d088. Shared agent_utils and Hermes files now match main and are absent from this PR's diff.

• Preserve the Responses bridge and native adapter. Exempt the explicit-null Codex template from legacy migration, with inherited-resource and real-template regressions. Remove the redundant public seed override; invalid multi-worker configuration now follows the base route's ValueError contract (like Hermes), rather than adapter-specific HTTP 422, and still fails before sandbox setup.
• Final-source macOS/Python 3.13.14 serial suite: 2,054 passed, 64 skipped; 17 subtests passed. All-file pre-commit passes; Fern check: 0 errors, 1 warning. One unchanged SWE-Pro Xvfb test failed during parallel validation; all 31 verifier tests and the full selection passed when run separately. No test was disabled or weakened.
• No fresh Linux or real-model run for this synchronization. Earlier rollout evidence below applies only to its recorded source pins. Full mirrored repository CI is separate; no trust-gate approval was posted.

---

Codex runs inside a borrowed task sandbox, or creates one using its configured provider when no sandbox is supplied. EnvironmentServer closes the agent before verification.

Implements the native sandboxed-harness session lifecycle. Targets main; shared helpers/tests match #3961 at 2093960be.

Added / expected

• Use nemo_gym/agent_utils and the updated SandboxSession contract: session_dir, stage_activation, stop_harness.
• The session uploads the supervisor, owns control files/deadlines and exposes diagnostic logs. Adapters keep installation, argv, runtime metadata and output parsing.
• Preserve retryable cleanup, delayed-launch fencing and partial output before release. Missing optional diagnostics do not erase valid output; positive cleanup confirmation remains required.
• Update existing Hermes supervisor imports without taking its separate config refactor. Merge main at d04762236.
• Correct the stale vLLM test to assert the existing upstream HTTP status/body passthrough. Runtime behavior is unchanged.

Validation — 5151a3a

• macOS arm64 / Python 3.13.14: 1,605 passed, 62 skipped, covering the harness, shared lifecycle/provider/session, both Hermes adapters, SWE-Pro, EnvironmentServers and model components, plus Responses bridge/converter/streaming.
• Full pre-commit run --all-files, focused shared-module mypy and whitespace checks pass; all new commits have DCO sign-off.
• Offline wheel build passes; packaged helpers match source byte-for-byte, with no obsolete helper paths.
• Shared implementation and its three test files match refactor(hermes): decouple configuration and share sandbox lifecycle #3961 byte-for-byte. Added checks for activation-time upload and shared-log diagnostics when output is missing.
• Fresh Linux/model smoke is pending workstation approval. Earlier real-model and benchmark results below apply only to their stated commits.
• Commands, evidence and handoff.

Concerns / limits

Full current-head CI and SWE-Pro accuracy have not been rerun. Skips include Linux-only process tests and optional runtime checks. The pre-existing minimal-image Bash bootstrap follow-up remains separate. This update does not change model settings, rewards, benchmark recipes or TTL defaults. No README command/config change is required; internal API migration is documented here and in the shared package docstrings.

Earlier revision-pinned implementation and validation

The following previous description is historical, not evidence for this head.

Codex runs inside a borrowed task sandbox, or creates one using its configured provider when no sandbox is supplied. EnvironmentServer closes the agent before verification.

Implements the native sandboxed-harness session lifecycle. Targets main; adopts the shared lifecycle from #3961 at 614a08b02.

Added / expected

• Reuse refactor(hermes): decouple configuration and share sandbox lifecycle #3961's SandboxSession for execute/close, cleanup confirmation and owned/borrowed release. Harness adapters retain runtime installation, commands and output parsing.

• Keep positive cleanup proof mandatory. Malformed optional runtime/exit diagnostics become explicit gaps; valid harness output is preserved, and an unknown exit code is never fabricated as zero.

• Capture partial output before releasing the sandbox. Merge current main and preserve the model bridge's upstream-error, streaming and input-immutability contracts.

• Keep the Codex Responses-to-Chat-Completions bridge. Fix legacy HarnessAgent requests without an ASGI scope.

Validation — 65186e3

• macOS/Python 3.13.14: 945 passed, 56 skipped.
• Linux x86_64/Python 3.13.14: 999 passed, 2 skipped, including process-supervision, cancellation and cleanup-retry tests.
• Full pre-commit run --all-files, focused shared-module mypy and git diff --check pass; commits have DCO sign-off.
• Fresh real-model smoke: 2/2 pass, reward 1 each, one owned and one borrowed sandbox. Codex 0.144.4, private Node 22.19.0, Docker and NIM Nemotron-3-Super-120B-A12B. Checked actual tool execution, file output, nonzero usage, cleanup receipts and close-before-verify ordering. No task containers remain.
• Exact source hashes were verified before/after execution. Commands, evidence and handoff.

Concerns / limits

The new smoke uses a small coding task and the stock math verifier; it is not a new SWE-Pro accuracy result. Full-dataset scores below apply only to their recorded revisions. Two opt-in Linux CLI boundary tests skip; pinned Codex itself ran in the model smoke. Full repository CI is separate: currently reported lint, copyright, secret and DCO checks pass; no current-head full-CI result is claimed.

Historical revision-pinned validation and implementation notes

The following is the previous description, preserved for evidence. References to old bases, draft status, strict metadata validation and pending Linux/model checks describe those older revisions, not the current head.

Codex runs its CLI and tools inside the benchmark's sandbox using the shared agent-session contract. Benchmark and harness launch independently; EnvironmentServer binds them, closes the agent before verification, and propagates provider/runtime failures without scoring them as zeros.

Implements the updated MB-1694 contract. Stacked on #3554, integrated through Hermes 95a2b6481. Remains draft while broader validation is incomplete.

• Replace Codex-specific session bookkeeping with shared immutable bindings, cleanup-only setup errors, bounded close receipts, and identical-activation join/replay. Disconnected waiters do not cancel execution.
• Reuse the shared Linux supervisor. Fence delayed launches, confirm descendant cleanup before cancelling provider execution, check receipts before signaling PIDs, and remove only adapter-owned files.
• Remove benchmarks/swebench/pro/codex_native.yaml. Compose independent codex_agent.yaml, SWE-Pro resources, and single_agent_turn_legacy; prepared rows need no additional materializer. The agent README contains the default command.
• Retain pinned Python 3.8/glibc/musl runtime compatibility from the preceding revision. Task interpreters and libraries remain unchanged.
• Preserve model HTTP status/body, reasoning/tool history, partial output, and explicit usage gaps. Model-output/wall-time limits remain incomplete and potentially gradable; known CLI advisories do not turn those limits into execution errors. Provider/runtime errors bypass verification.

Validation at c1e419ef7bf7940666749a755184200191b8e0e4:

• Linux/Python 3.13.14: 728 passed, including the actual Codex 0.144.4 CLI boundary tests. Suites: Codex, OpenAI Model Server, SWE-Pro, both single-agent EnvironmentServers, shared session retries/supervisor, Responses conversion/streaming. Real subprocess tests cover detached descendants, cancellation, delayed launches, lost handles, cleanup retries, and Python 3.8 worker execution.
• python -m pre_commit run --all-files and git diff --check passed; all new commits have DCO sign-off.
• Additional external vLLM capture suite: 117 passed. HSG/Python 3.13.14 controller suites: 386 passed, 3 skipped (opt-in local CLI and Python 3.8 binaries).
• HSG real-model smoke: 3/3 graded, 2/3 solved, all three agents completed; zero setup/agent/verifier errors or masked rows. NodeBB: 290/291 required tests passed, reward 0; Ansible/Python 3.8: 8/8, reward 1; Teleport/Alpine: 4/4, reward 1. NodeBB's zero is a real canSendValidation assertion failure at test/user/emails.js:136, confirmed in raw output. Teleport also reports an unrelated TestPasswordTimingAttack failure outside its four benchmark-required tests; it is retained in the artifacts.
• Smoke controls: Codex 0.144.4, private Node 22.19.0, N35 Super GA, four GB200 / TP4+EP / no MTP, context 262144, compaction 229376, concurrency 3, one original attempt per task, 900s agent bound, and the stock verifier. No benchmark/parser patches or score-based retries. A run-local HTTP controller loads independent configs and posts standard prepared rows to EnvironmentServer.
• Evidence audit passed: 194 captured model calls, all HTTP 200, with effective output cap 32768 / temperature 1.0 / top-p 0.95; aggregate CLI input/output/total usage matches captures for each task. 219 tool records. Exact per-call join gaps remain explicit.
• All three agent closes preceded verification. Remote checks found the runtime process and session files gone, with only the sandbox keep-alive process remaining; the host runtime installer was never started. Six successful agent/resources close receipts; post-run inventory found no owned sandboxes, and the model allocation is released. Normal agent close took 0.87–2.57s; this is not a concurrency/overhead benchmark.
• Source checks before and after execution verified all 6,125 entries against the frozen commit with no mismatches. HSG run root: /lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_nemogym/users/yuya/pr3640-contract-n35ga-20261001. Setup/model/eval/cleanup jobs: 7603886 / 7603926 / 7603927 / 7603928; independent audit: 7604677.

Limitations: no full-dataset accuracy claim, controlled same-harness before/after comparison, second real pairing, or measured supervisor overhead at intended concurrency. Exact model-call join IDs and some CLI usage details remain unavailable and are marked as observation gaps. Shared failed-episode responses still omit close observations; the smoke retains raw close receipts separately. Inference support does not establish training token-ID/logprob support.

Shared runner integration (2026-10-02)

Integrated the approved split-contract refactor in 3fd83ead72d7ddf20c7f2fad2e6eca691678046a from draft #10 / draft #11. Cleanup uses the existing supervisor CleanupReceipt; worker identity uses RunnerRuntimeInfo; harness output stays separate. No combined SandboxRunnerResult.

Only the Codex callers, shared helper, and relevant tests were integrated here. Hermes-specific callers and regressions are routed to #3961. The shared helper and its tests are identical across all three PRs. Remote supervisor/worker files, model requests, sampling and scoring logic are unchanged; malformed cleanup/runtime metadata is now rejected. A fenced, unstarted worker retains an unknown exit code, and runtime failure does not discard confirmed cleanup.

Validation on this integrated head (macOS, Python 3.13.14; pinned Hermes source and existing dependency environments; localhost fake model/MCP servers): 459 passed, 50 skipped. Skips include Linux-only process-supervision and optional runtime/CLI checks. Full pre-commit run --all-files and git diff --check passed; commit is DCO-signed.

python -m pytest tests/unit_tests/test_sandbox_runner_transport.py \
  tests/unit_tests/test_agent_sessions.py tests/unit_tests/test_agent_session_retries.py \
  tests/unit_tests/test_process_supervisor.py \
  responses_api_agents/codex_agent/tests responses_api_agents/hermes_agent/tests \
  --import-mode=importlib -q -o addopts='' -o log_cli=false

CI classification: full (shared core changes). No fresh real-model rollout or Linux rerun was performed for this integration; prior rollout evidence above remains pinned to its stated revision and is not evidence for this new head. Existing PR readiness/labels are unchanged.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yaoyu-33]

/ok to test 32b7b23

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37815214869