Repository navigation
VLLMModel propogates token IDs - #11
Merged
Merged
Conversation
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
bxyu-nvidia
marked this pull request as ready for review
September 8, 2025 17:51
Signed-off-by: Brian Yu <bxyu@nvidia.com>
soares-f
pushed a commit
that referenced
this pull request
Sep 15, 2025
Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: soares-f <soarescmsa@gmail.com>
abhibha-nvidia
pushed a commit
that referenced
this pull request
Sep 28, 2025
Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Abhibha Gupta <abhibhag@nvidia.com>
abhibha-nvidia
pushed a commit
that referenced
this pull request
Sep 29, 2025
Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: Abhibha Gupta <abhibhag@nvidia.com>
abubakaria56
pushed a commit
to abubakaria56/Gym
that referenced
this pull request
Mar 2, 2026
Signed-off-by: Brian Yu <bxyu@nvidia.com>
abubakaria56
pushed a commit
to abubakaria56/Gym
that referenced
this pull request
Mar 2, 2026
Signed-off-by: Brian Yu <bxyu@nvidia.com>
5 tasks
copy-pr-bot Bot
pushed a commit
that referenced
this pull request
Oct 6, 2026
Integrate Hermes-specific changes from yaoyu-33#10 and #11 into #3961. Keep harness output separate from cleanup and worker runtime metadata. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
nemo-autobot-origin Bot
pushed a commit
to svcnemo-autobot/Gym
that referenced
this pull request
Oct 8, 2026
### Launcher shutdown race — October 8 Head `32b7b230bb2a78300c2d0cd65a6db089211065cd` fixes the recurring [core CI shutdown failure](https://github.com/NVIDIA-NeMo/Gym/actions/runs/37806810516/job/113413690379). A TERM signal can arrive after a TP1 engine starts but before its PID is appended to `worker_pids`; cleanup then misses that process. Cleanup now uses the serving shell's job table, which already contains the child, and preserves the shutdown exit status. This also covers the router/Mooncake children owned by that shell. Model arguments, inference behavior and agent/verifier code are unchanged. Validation: a deterministic reproduction of that exact launch/registration window failed before the fix with the same missing fourth-engine stop seen in CI, and passes after it. Added TERM and INT regressions requiring all four engines plus the router to stop; the five-second timeout and existing assertions remain unchanged. Source-bound launcher suite: **35 passed, 6 skipped, 199 subtests passed** on macOS/Bash 3.2. The six existing skips require Bash 5.1+; Linux CI remains required. Full `pre-commit run --all-files` and `git diff --check` passed; commit is DCO-signed. Fresh real-model rollout: N/A for this launcher process-bookkeeping fix, validated with actual isolated child processes and signals; prior agent/model rollout evidence remains pinned to its revisions. [Full CI on head `32b7b230bb2a78300c2d0cd65a6db089211065cd` passed](https://github.com/NVIDIA-NeMo/Gym/actions/runs/37809392943), including Linux core/sandbox tests, all eight server shards and aggregate gates. All 31 current passing checks and 11 legitimate skips are complete; fresh reviewer approval remains required. This repairs the CI blocker directly; it does not treat the recurring shutdown failure as a transient runner issue. ### Merge queue compatibility — October 8 Head `cc532c42d46e289a6b931ba7b0b10c5e7c4470d7` merges current main and aligns the native-capable migration regression with [NVIDIA-NeMo#4141](NVIDIA-NeMo#4141): an agent rename preserves the existing Environment Server's name. The test also verifies the rename overlay stays untouched, the source receives exactly one server, and the renamed agent retains its Resources binding. The [queue run](https://github.com/NVIDIA-NeMo/Gym/actions/runs/37803971900) failed in those five parameterized cases. The new commit differs from that exact queue tree only in the migration test; it applies the same correction validated for NVIDIA-NeMo#3626. Validation on this head: **267 passed** using verified source-bound imports for `tests/unit_tests/test_add_legacy_agent_environment_servers.py`, `tests/unit_tests/test_global_config.py`, and `tests/unit_tests/test_environment_server.py`. Full `pre-commit run --all-files` and whitespace checks against main and the prior queue tree passed. Commit has DCO sign-off. Full new-head CI was requested with `/ok to test cc532c4` and remains pending. Fresh model rollout: N/A for this test-only correction relative to the queue tree; prior model evidence remains pinned to its recorded revisions. ### Shared contracts — October 6 Head `5a2e582dc`. Aligned the common implementation across NVIDIA-NeMo#3626, NVIDIA-NeMo#3640, NVIDIA-NeMo#3637 and NVIDIA-NeMo#3639: - One upstream HTTP-error handler in `SimpleResponsesAPIModel`; preserve status/body for OpenAI and vLLM, including errors before buffered Responses streaming starts. No adapter-specific duplicate. - Identical native-template migration rules and shared regressions. Explicit-null native templates stay native; bound/inherited Resources still use compatibility routing. - Shared sandbox lifecycle helpers still match main. Harness runtimes, request policies, benchmark settings and scoring are unchanged; Codex-specific Responses compatibility stays scoped to Codex. Validation: macOS/Python 3.13.14 selected suites **1,779 passed / 53 skipped**; six final HTTP-error regressions and all-file pre-commit pass. Skips include Linux and optional runtime tests. No fresh Linux/model rollout or full-repository CI result is claimed for this update. [Exact scope, commands and evidence](https://gitlab-master.nvidia.com/yuya/how-to-run/-/blob/main/issues/2026-10-06-four-harness-common-contracts.md). --- ### Main sync — October 6 Synced with main `045dae678` after NVIDIA-NeMo#3961; head `e4bf3d088`. Shared `agent_utils` and Hermes files now match main and are absent from this PR's diff. - Preserve the Responses bridge and native adapter. Exempt the explicit-null Codex template from legacy migration, with inherited-resource and real-template regressions. Remove the redundant public seed override; invalid multi-worker configuration now follows the base route's `ValueError` contract (like Hermes), rather than adapter-specific HTTP 422, and still fails before sandbox setup. - Final-source macOS/Python 3.13.14 serial suite: **2,054 passed, 64 skipped; 17 subtests passed**. All-file pre-commit passes; Fern check: 0 errors, 1 warning. One unchanged SWE-Pro Xvfb test failed during parallel validation; all 31 verifier tests and the full selection passed when run separately. No test was disabled or weakened. - No fresh Linux or real-model run for this synchronization. Earlier rollout evidence below applies only to its recorded source pins. Full mirrored repository CI is separate; no trust-gate approval was posted. --- Codex runs inside a borrowed task sandbox, or creates one using its configured provider when no sandbox is supplied. EnvironmentServer closes the agent before verification. Implements [GYMINT-5](https://linear.app/nvidia/issue/GYMINT-5/runbook-onboard-sandboxed-harnesses-to-native-environmentserver). Targets `main`; shared helpers/tests match NVIDIA-NeMo#3961 at `2093960be`. ## Added / expected - Use `nemo_gym/agent_utils` and the updated `SandboxSession` contract: `session_dir`, `stage_activation`, `stop_harness`. - The session uploads the supervisor, owns control files/deadlines and exposes diagnostic logs. Adapters keep installation, argv, runtime metadata and output parsing. - Preserve retryable cleanup, delayed-launch fencing and partial output before release. Missing optional diagnostics do not erase valid output; positive cleanup confirmation remains required. - Update existing Hermes supervisor imports without taking its separate config refactor. Merge main at `d04762236`. - Correct the stale vLLM test to assert the existing upstream HTTP status/body passthrough. Runtime behavior is unchanged. ## Validation — 5151a3a - macOS arm64 / Python 3.13.14: **1,605 passed, 62 skipped**, covering the harness, shared lifecycle/provider/session, both Hermes adapters, SWE-Pro, EnvironmentServers and model components, plus Responses bridge/converter/streaming. - Full `pre-commit run --all-files`, focused shared-module mypy and whitespace checks pass; all new commits have DCO sign-off. - Offline wheel build passes; packaged helpers match source byte-for-byte, with no obsolete helper paths. - Shared implementation and its three test files match NVIDIA-NeMo#3961 byte-for-byte. Added checks for activation-time upload and shared-log diagnostics when output is missing. - **Fresh Linux/model smoke is pending workstation approval.** Earlier real-model and benchmark results below apply only to their stated commits. - [Commands, evidence and handoff](https://gitlab-master.nvidia.com/yuya/how-to-run/-/blob/main/issues/2026-10-05-pi-codex-shared-session-parents.md). ## Concerns / limits Full current-head CI and SWE-Pro accuracy have not been rerun. Skips include Linux-only process tests and optional runtime checks. The pre-existing minimal-image Bash bootstrap follow-up remains separate. This update does not change model settings, rewards, benchmark recipes or TTL defaults. No README command/config change is required; internal API migration is documented here and in the shared package docstrings. <details> <summary>Earlier revision-pinned implementation and validation</summary> The following previous description is historical, not evidence for this head. Codex runs inside a borrowed task sandbox, or creates one using its configured provider when no sandbox is supplied. EnvironmentServer closes the agent before verification. Implements [GYMINT-5](https://linear.app/nvidia/issue/GYMINT-5/runbook-onboard-sandboxed-harnesses-to-native-environmentserver). Targets `main`; adopts the shared lifecycle from NVIDIA-NeMo#3961 at `614a08b02`. ## Added / expected - Reuse NVIDIA-NeMo#3961's `SandboxSession` for execute/close, cleanup confirmation and owned/borrowed release. Harness adapters retain runtime installation, commands and output parsing. - Keep positive cleanup proof mandatory. Malformed optional runtime/exit diagnostics become explicit gaps; valid harness output is preserved, and an unknown exit code is never fabricated as zero. - Capture partial output before releasing the sandbox. Merge current `main` and preserve the model bridge's upstream-error, streaming and input-immutability contracts. - Keep the Codex Responses-to-Chat-Completions bridge. Fix legacy HarnessAgent requests without an ASGI `scope`. ## Validation — 65186e3 - macOS/Python 3.13.14: **945 passed, 56 skipped**. - Linux x86_64/Python 3.13.14: **999 passed, 2 skipped**, including process-supervision, cancellation and cleanup-retry tests. - Full `pre-commit run --all-files`, focused shared-module mypy and `git diff --check` pass; commits have DCO sign-off. - Fresh real-model smoke: **2/2 pass, reward 1 each**, one owned and one borrowed sandbox. Codex 0.144.4, private Node 22.19.0, Docker and NIM Nemotron-3-Super-120B-A12B. Checked actual tool execution, file output, nonzero usage, cleanup receipts and close-before-verify ordering. No task containers remain. - Exact source hashes were verified before/after execution. [Commands, evidence and handoff](https://gitlab-master.nvidia.com/yuya/how-to-run/-/blob/main/issues/2026-10-05-pi-codex-shared-session-parents.md). ## Concerns / limits The new smoke uses a small coding task and the stock math verifier; it is **not a new SWE-Pro accuracy result**. Full-dataset scores below apply only to their recorded revisions. Two opt-in Linux CLI boundary tests skip; pinned Codex itself ran in the model smoke. Full repository CI is separate: currently reported lint, copyright, secret and DCO checks pass; no current-head full-CI result is claimed. <details> <summary>Historical revision-pinned validation and implementation notes</summary> The following is the previous description, preserved for evidence. References to old bases, draft status, strict metadata validation and pending Linux/model checks describe those older revisions, not the current head. Codex runs its CLI and tools inside the benchmark's sandbox using the shared agent-session contract. Benchmark and harness launch independently; EnvironmentServer binds them, closes the agent before verification, and propagates provider/runtime failures without scoring them as zeros. Implements the updated [MB-1694 contract](https://linear.app/nvidia/issue/MB-1694/runbook-onboard-sandboxed-harnesses-to-native-environmentserver). Stacked on NVIDIA-NeMo#3554, integrated through Hermes `95a2b6481`. Remains draft while broader validation is incomplete. - Replace Codex-specific session bookkeeping with shared immutable bindings, cleanup-only setup errors, bounded close receipts, and identical-activation join/replay. Disconnected waiters do not cancel execution. - Reuse the shared Linux supervisor. Fence delayed launches, confirm descendant cleanup before cancelling provider execution, check receipts before signaling PIDs, and remove only adapter-owned files. - Remove `benchmarks/swebench/pro/codex_native.yaml`. Compose independent `codex_agent.yaml`, SWE-Pro resources, and `single_agent_turn_legacy`; prepared rows need no additional materializer. The agent README contains the default command. - Retain pinned Python 3.8/glibc/musl runtime compatibility from the preceding revision. Task interpreters and libraries remain unchanged. - Preserve model HTTP status/body, reasoning/tool history, partial output, and explicit usage gaps. Model-output/wall-time limits remain incomplete and potentially gradable; known CLI advisories do not turn those limits into execution errors. Provider/runtime errors bypass verification. Validation at `c1e419ef7bf7940666749a755184200191b8e0e4`: - Linux/Python 3.13.14: **728 passed**, including the actual Codex **0.144.4** CLI boundary tests. Suites: Codex, OpenAI Model Server, SWE-Pro, both single-agent EnvironmentServers, shared session retries/supervisor, Responses conversion/streaming. Real subprocess tests cover detached descendants, cancellation, delayed launches, lost handles, cleanup retries, and Python 3.8 worker execution. - `python -m pre_commit run --all-files` and `git diff --check` passed; all new commits have DCO sign-off. - Additional external vLLM capture suite: **117 passed**. HSG/Python 3.13.14 controller suites: **386 passed, 3 skipped** (opt-in local CLI and Python 3.8 binaries). - [HSG real-model smoke](https://wandb.ai/nvidia-nemo-framework/yuyao-nemo-gym-exps/runs/pr3640-contract-n35ga-20261001-smoke): **3/3 graded, 2/3 solved**, all three agents completed; zero setup/agent/verifier errors or masked rows. NodeBB: 290/291 required tests passed, reward 0; Ansible/Python 3.8: 8/8, reward 1; Teleport/Alpine: 4/4, reward 1. NodeBB's zero is a real `canSendValidation` assertion failure at `test/user/emails.js:136`, confirmed in raw output. Teleport also reports an unrelated `TestPasswordTimingAttack` failure outside its four benchmark-required tests; it is retained in the artifacts. - Smoke controls: Codex 0.144.4, private Node 22.19.0, N35 Super GA, four GB200 / TP4+EP / no MTP, context 262144, compaction 229376, concurrency 3, one original attempt per task, **900s agent bound**, and the stock verifier. No benchmark/parser patches or score-based retries. A run-local HTTP controller loads independent configs and posts standard prepared rows to EnvironmentServer. - Evidence audit passed: **194 captured model calls**, all HTTP 200, with effective output cap 32768 / temperature 1.0 / top-p 0.95; aggregate CLI input/output/total usage matches captures for each task. **219 tool records**. Exact per-call join gaps remain explicit. - All three agent closes preceded verification. Remote checks found the runtime process and session files gone, with only the sandbox keep-alive process remaining; the host runtime installer was never started. Six successful agent/resources close receipts; post-run inventory found no owned sandboxes, and the model allocation is released. Normal agent close took 0.87–2.57s; this is not a concurrency/overhead benchmark. - Source checks before and after execution verified all **6,125 entries** against the frozen commit with no mismatches. HSG run root: `/lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_nemogym/users/yuya/pr3640-contract-n35ga-20261001`. Setup/model/eval/cleanup jobs: 7603886 / 7603926 / 7603927 / 7603928; independent audit: 7604677. Limitations: no full-dataset accuracy claim, controlled same-harness before/after comparison, second real pairing, or measured supervisor overhead at intended concurrency. Exact model-call join IDs and some CLI usage details remain unavailable and are marked as observation gaps. Shared failed-episode responses still omit close observations; the smoke retains raw close receipts separately. Inference support does not establish training token-ID/logprob support. ## Shared runner integration (2026-10-02) Integrated the approved split-contract refactor in `3fd83ead72d7ddf20c7f2fad2e6eca691678046a` from [draft NVIDIA-NeMo#10](yaoyu-33#10) / [draft NVIDIA-NeMo#11](yaoyu-33#11). Cleanup uses the existing supervisor `CleanupReceipt`; worker identity uses `RunnerRuntimeInfo`; harness output stays separate. No combined `SandboxRunnerResult`. Only the Codex callers, shared helper, and relevant tests were integrated here. Hermes-specific callers and regressions are routed to NVIDIA-NeMo#3961. The shared helper and its tests are identical across all three PRs. Remote supervisor/worker files, model requests, sampling and scoring logic are unchanged; malformed cleanup/runtime metadata is now rejected. A fenced, unstarted worker retains an unknown exit code, and runtime failure does not discard confirmed cleanup. Validation on this integrated head (macOS, Python 3.13.14; pinned Hermes source and existing dependency environments; localhost fake model/MCP servers): **459 passed, 50 skipped**. Skips include Linux-only process-supervision and optional runtime/CLI checks. Full `pre-commit run --all-files` and `git diff --check` passed; commit is DCO-signed. ```sh python -m pytest tests/unit_tests/test_sandbox_runner_transport.py \ tests/unit_tests/test_agent_sessions.py tests/unit_tests/test_agent_session_retries.py \ tests/unit_tests/test_process_supervisor.py \ responses_api_agents/codex_agent/tests responses_api_agents/hermes_agent/tests \ --import-mode=importlib -q -o addopts='' -o log_cli=false ``` CI classification: full (shared core changes). No fresh real-model rollout or Linux rerun was performed for this integration; prior rollout evidence above remains pinned to its stated revision and is not evidence for this new head. Existing PR readiness/labels are unchanged. </details> </details> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
ananthsub
pushed a commit
to jiwasawa/Gym
that referenced
this pull request
Oct 8, 2026
### Integration with merged Codex support — October 8 Head `3f68583a8a1e4794cf82122576d94294b74b38af` merges main `acd53a9249b40938b2a3be01c07588ecf0308197` after [NVIDIA-NeMo#3640](NVIDIA-NeMo#3640) landed. SWE-Pro permits both `pi_agent` and `codex_agent`, preserving every existing allowed agent. Both native-default composition regressions remain intact. The migration assertions also retain [NVIDIA-NeMo#4141](NVIDIA-NeMo#4141 Environment Server name-preservation semantics. Validation on this integrated source: **977 passed, 12 Linux-only skips** on macOS/Python 3.13.14, with verified source-bound imports. The selection covers migration, global configuration, EnvironmentServer, base model capture/HTTP behavior, streaming, and both Pi and Codex app/native-session tests. Full `pre-commit run --all-files` and the diff whitespace check against main passed; the merge commit has DCO sign-off. ```text python -S -m pytest tests/unit_tests/test_add_legacy_agent_environment_servers.py tests/unit_tests/test_global_config.py tests/unit_tests/test_environment_server.py tests/unit_tests/test_base_responses_api_model.py tests/unit_tests/test_responses_api_model_streaming.py responses_api_agents/pi_agent/tests/test_app.py responses_api_agents/pi_agent/tests/test_sandbox.py responses_api_agents/codex_agent/tests/test_app.py responses_api_agents/codex_agent/tests/test_native_sessions.py -q ``` `PYTHONPATH` was set to this isolated worktree and the dependency-only Python 3.13 site-packages. Pi runtime files are unchanged from `85d2b4629`; Codex, shared runtime code, and the launcher race fix match merged main. Fresh model rollout is N/A for the configuration/test conflict resolution; [prior real-model and native SWE-Pro evidence](https://gitlab-master.nvidia.com/yuya/how-to-run/-/blob/main/issues/2026-10-07-pi-codex-review-comments.md) remains pinned to its recorded revisions. Required CI on `3f68583a8a1e4794cf82122576d94294b74b38af` has passed: [Nemo_CICD_Test](https://github.com/NVIDIA-NeMo/Gym/actions/runs/37818113383/job/113459296541), including core/sandbox tests and all eight server shards, plus lint and DCO. The overall workflow is marked failed because the optional harness-conformance diagnostics failed; these are non-blocking and have not been suppressed or changed. Fresh approval and normal merge-queue validation are still required. Earlier [successful CI on `85d2b4629`](https://github.com/NVIDIA-NeMo/Gym/actions/runs/37802918495/attempts/2) remains historical evidence only. ### Review compatibility follow-up — October 7 Head `520b851fa85a9ed4f09335e35f0ced660893a3bf` addresses [the existing-consumer review request](NVIDIA-NeMo#3626 (comment)). Native sandbox sessions retain strict request validation. Calls without a native session preserve the existing AnySWE, AnyTerminal, and HarnessAgent contract: forwarded model/sampling/token fields no longer raise HTTP 422; configured Pi controls remain authoritative. The README explicitly describes this compatibility behavior. The separate `pi_sandboxed_agent` introduced by merged NVIDIA-NeMo#3575 remains supported. Validation on the committed source (macOS arm64, Python 3.13.14): - Two new regression cases failed before the fix and pass afterward, covering real ASGI requests and HarnessAgent's scope-less request shape, response/usage preservation, configuration ownership, and request immutability. Native unsupported-control rejection remains tested. - Pi app/session tests: **186 passed, 6 Linux-only skips**. - Pi, PiSandboxed, AnySWE, AnyTerminal, HarnessAgent, and migration tests: **405 passed, 28 skipped, 3 local failures**. All three failures are unchanged AnyTerminal Docker-provider constructor tests because this host has no Docker binary; full Linux CI remains required. - Source-bound test command: `python -S -m pytest responses_api_agents/pi_agent/tests responses_api_agents/pi_sandboxed_agent/tests responses_api_agents/anyswe_agent/tests responses_api_agents/anyterminal_agent/tests responses_api_agents/harness_agent/tests tests/unit_tests/test_add_legacy_agent_environment_servers.py --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short` with this checkout and dependency-only site-packages on `PYTHONPATH`. - `python -S -m pre_commit run --all-files` and `git diff --check` passed; commit is DCO-signed. - Fresh real-model compatibility smoke: **2/2 passed, stock math verifier reward 1 each**, Pi **0.80.2**, NIM `nvidia/nemotron-3-super-120b-a12b`. Both request shapes forwarded the previously rejected fields, executed actual Bash/file tools, returned nonzero usage, preserved caller inputs, and removed their temporary Pi workspaces. Verified the tested runtime hash matches the committed file. These are local compatibility-path smokes, not new native sandbox lifecycle or SWE-Pro accuracy results. Full current-head [CICD 37659390982](https://github.com/NVIDIA-NeMo/Gym/actions/runs/37659390982) passed on `520b851fa85a9ed4f09335e35f0ced660893a3bf`, including core/sandbox tests, all eight server shards, and the aggregate gate. Wheel validation, lint, copyright, secret scanning, and DCO also pass. Final review recheck found no new comments or inline threads. CI was started with `/ok to test` for this exact commit. --- ### Shared contracts — October 6 Head `730765c70`. Aligned the common implementation across NVIDIA-NeMo#3626, NVIDIA-NeMo#3640, NVIDIA-NeMo#3637 and NVIDIA-NeMo#3639: - One upstream HTTP-error handler in `SimpleResponsesAPIModel`; preserve status/body for OpenAI and vLLM, including errors before buffered Responses streaming starts. No adapter-specific duplicate. - Identical native-template migration rules and shared regressions. Explicit-null native templates stay native; bound/inherited Resources still use compatibility routing. - Shared sandbox lifecycle helpers still match main. Harness runtimes, request policies, benchmark settings and scoring are unchanged; Codex-specific Responses compatibility stays scoped to Codex. Validation: macOS/Python 3.13.14 selected suites **1,741 passed / 36 skipped**; six final HTTP-error regressions and all-file pre-commit pass. Skips include Linux and optional runtime tests. No fresh Linux/model rollout or full-repository CI result is claimed for this update. [Exact scope, commands and evidence](https://gitlab-master.nvidia.com/yuya/how-to-run/-/blob/main/issues/2026-10-06-four-harness-common-contracts.md). --- ### Main sync — October 6 Synced with main `045dae678` after NVIDIA-NeMo#3961; head `5c3f3fcf4`. Shared `agent_utils` and Hermes files now match main and are absent from this PR's diff. - Preserve main's failed-attempt ownership capture and this PR's HTTP-error forwarding. Exempt the explicit-null Pi native template from legacy migration, with bound/inherited-resource and real-template composition regressions. - Final-source macOS/Python 3.13.14: **1,589 passed, 47 skipped; 17 subtests passed**. All-file pre-commit passes; Fern check: 0 errors, 1 warning. - No fresh Linux or real-model run for this synchronization. Earlier rollout evidence below applies only to its recorded source pins. Full mirrored repository CI is separate; no trust-gate approval was posted. --- Pi runs inside a borrowed task sandbox, or creates one using its configured provider when no sandbox is supplied. EnvironmentServer closes the agent before verification. Implements [GYMINT-5](https://linear.app/nvidia/issue/GYMINT-5/runbook-onboard-sandboxed-harnesses-to-native-environmentserver). Targets `main`; shared helpers/tests match NVIDIA-NeMo#3961 at `2093960be`. ## Added / expected - Use `nemo_gym/agent_utils` and the updated `SandboxSession` contract: `session_dir`, `stage_activation`, `stop_harness`. - The session uploads the supervisor, owns control files/deadlines and exposes diagnostic logs. Adapters keep installation, argv, runtime metadata and output parsing. - Preserve retryable cleanup, delayed-launch fencing and partial output before release. Missing optional diagnostics do not erase valid output; positive cleanup confirmation remains required. - Update existing Hermes supervisor imports without taking its separate config refactor. Merge main at `d04762236`. ## Validation — d91469b - macOS arm64 / Python 3.13.14: **1,137 passed, 45 skipped**, covering the harness, shared lifecycle/provider/session, both Hermes adapters, SWE-Pro, EnvironmentServers and model components. - Full `pre-commit run --all-files`, focused shared-module mypy and whitespace checks pass; all new commits have DCO sign-off. - Offline wheel build passes; packaged helpers match source byte-for-byte, with no obsolete helper paths. - Shared implementation and its three test files match NVIDIA-NeMo#3961 byte-for-byte. Added checks for activation-time upload and shared-log diagnostics when output is missing. - **Fresh Linux/model smoke is pending workstation approval.** Earlier real-model and benchmark results below apply only to their stated commits. - [Commands, evidence and handoff](https://gitlab-master.nvidia.com/yuya/how-to-run/-/blob/main/issues/2026-10-05-pi-codex-shared-session-parents.md). ## Concerns / limits Full current-head CI and SWE-Pro accuracy have not been rerun. Skips include Linux-only process tests and optional runtime checks. Pi's existing legacy request-control compatibility work remains separate. This update does not change model settings, rewards, benchmark recipes or TTL defaults. No README command/config change is required; internal API migration is documented here and in the shared package docstrings. <details> <summary>Earlier revision-pinned implementation and validation</summary> The following previous description is historical, not evidence for this head. Pi runs inside a borrowed task sandbox, or creates one using its configured provider when no sandbox is supplied. EnvironmentServer closes the agent before verification. Implements [GYMINT-5](https://linear.app/nvidia/issue/GYMINT-5/runbook-onboard-sandboxed-harnesses-to-native-environmentserver). Targets `main`; adopts the shared lifecycle from NVIDIA-NeMo#3961 at `614a08b02`. ## Added / expected - Reuse NVIDIA-NeMo#3961's `SandboxSession` for execute/close, cleanup confirmation and owned/borrowed release. Harness adapters retain runtime installation, commands and output parsing. - Keep positive cleanup proof mandatory. Malformed optional runtime/exit diagnostics become explicit gaps; valid harness output is preserved, and an unknown exit code is never fabricated as zero. - Capture partial output before releasing the sandbox. Merge current `main` and preserve the model bridge's upstream-error, streaming and input-immutability contracts. - Pi's existing request-control policy is unchanged: request-level `max_output_tokens` is rejected; configured `max_output_tokens` remains a per-model-call cap. The older AnySWE/HarnessAgent compatibility request remains separate. ## Validation — 6140623 - macOS/Python 3.13.14: **491 passed, 39 skipped**. - Linux x86_64/Python 3.13.14: **519 passed, 11 skipped**, including process-supervision, cancellation and cleanup-retry tests. - Full `pre-commit run --all-files`, focused shared-module mypy and `git diff --check` pass; commits have DCO sign-off. - Fresh real-model smoke: **2/2 pass, reward 1 each**, one owned and one borrowed sandbox. Pi 0.80.2, private Node 22.19.0, Docker and NIM Nemotron-3-Super-120B-A12B. Checked actual tool execution, file output, nonzero usage, cleanup receipts and close-before-verify ordering. No task containers remain. - Exact source hashes were verified before/after execution. [Commands, evidence and handoff](https://gitlab-master.nvidia.com/yuya/how-to-run/-/blob/main/issues/2026-10-05-pi-codex-shared-session-parents.md). ## Concerns / limits The new smoke uses a small coding task and the stock math verifier; it is **not a new SWE-Pro accuracy result**. Full-dataset scores below apply only to their recorded revisions. Eleven Linux optional Node/extension tests skip; pinned Pi itself ran in the model smoke. Full repository CI is separate: currently reported lint, copyright, secret and DCO checks pass; no current-head full-CI result is claimed. <details> <summary>Historical revision-pinned validation and implementation notes</summary> The following is the previous description, preserved for evidence. References to old bases, draft status, strict metadata validation and pending Linux/model checks describe those older revisions, not the current head. Pi runs inside the task sandbox supplied by the benchmark's Resources Server. The Environment Server owns setup, activation, verification, and cleanup; Pi borrows the sandbox and runs its own model/tool loop. Load `responses_api_agents/pi_agent/configs/pi_agent.yaml` for the standard path. SWE-bench Pro remains independent in `resources_servers/swebench_pro/configs/swebench_pro.yaml`. The run binds `pi_agent` and `swebench_pro_resources_server` through the Environment Server; there is no Pi-specific benchmark recipe or combined deployment name. The README uses prepared flat rows through `single_agent_turn_legacy`, without a separate materialization script. Model selection defaults to `${policy_model_name}`. Implements [MB-1694](https://linear.app/nvidia/issue/MB-1694); evaluation is tracked in [MB-1712](https://linear.app/nvidia/issue/MB-1712). Targets `main` after NVIDIA-NeMo#3554 merged. - Reuse shared agent-session seed/close and receipt retention. Identical activation retries join or replay one invocation; losing an HTTP waiter does not cancel it. The Environment Server and provider TTL own cleanup/expiry. - Reuse shared Linux process supervision. An atomic launch claim fences delayed starts, and verification requires confirmed descendant cleanup. Pi never destroys its borrowed sandbox. Include a separate shared-supervisor fix for a child created before `Popen` loses its handle (reported on NVIDIA-NeMo#3554). - Compatibility change: local and sandbox calls now concatenate the configured system prompt, request instructions, and input system message in that order. Unsupported non-default request controls, including developer messages, are rejected instead of silently ignored. Request-level `max_output_tokens` is rejected because total-response budgeting is unsupported; the configured Pi `max_output_tokens` remains a per-model-call cap. - Preserve upstream HTTP status and error body in the shared OpenAI Model Server so SDKs can recognize context-limit errors instead of seeing a generic 500. - Provider/API and runtime errors propagate without benchmark verification. Model limits/timeouts preserve gradable partial work. Context overflow uses pinned Pi's own classifier; observations survive failed activations. - Install private Node and pinned Pi outside the repository, supporting Python 3.8+, glibc x86_64/aarch64, and musl x86_64. Preserve task runtimes, installer stdout/stderr, npm script restrictions, bash deadlines, and model HTTP deadlines. The pinned Node build does not support arm64 musl. - Confirm supervisor cleanup before cancelling provider execution; check remote receipts before signaling stored PIDs; reject non-finite or nonpositive deadlines and access to partially seeded sessions. These focused follow-ups address the latest NVIDIA-NeMo#3554 review comments. Validation: - Final Pi head `ca7d42212`: 642 tests passed across Pi, Hermes session/runner integration, shared sessions/process supervision, Environment Server, and Model Server capture/streaming/messages. Tests include real Linux process-group cancellation, stale-PID protection, delayed launches, Python 3.8, Node extensions, and Pi 0.80.2 SDK behavior. Coverage is 96.07% across six affected Python runtime modules. Full `pre-commit run --all-files` passed; DCO sign-offs present. - README configuration resolves independent benchmark/agent references and inherits the model ID. No suffixed Pi configuration or combined benchmark/agent recipe remains. Reviewed all 44 inline comments and 15 reviews on NVIDIA-NeMo#3554 through its October 1 review, plus issue discussion; [follow-up findings](NVIDIA-NeMo#3554 (comment)). - Completed HSG full evaluation: **358/731 (48.97%), 731 graded, 0 ungraded, 0 agent/verifier errors**. Repaired historical baseline: **359/731 (49.11%), 730 graded, 1 ungraded**. Difference: one solved task / −0.137 percentage points; 39 pass→fail, 38 fail→pass. Changed failures were inspected against raw test logs and show candidate assertion/build/import failures, not new harness or grading infrastructure failures. One stochastic run cannot establish identical outcomes or prove absence of all regression. - Independently confirmed complete required-test evidence for all 358 reported passes; matched all 731 original task inputs and immutable image digests. Audited 36,953 model calls, 46,903 tool calls, and 1,462 successful agent/resources close receipts. A real upstream HTTP 400 context limit was preserved, compacted by Pi, and followed by continued model calls and normal grading. One task hit its 32,768-token output cap and was normally graded zero. - Same N35 Super GA checkpoint and serving setup as the repaired baseline: TP4, four GB200 GPUs, 262,144 context, Pi 0.80.2 with high thinking, temperature 1 / top-p .95, 32,768 output tokens per call, concurrency 32. Full evaluation source is frozen at `28df43c18`, consisting of Pi runtime `0c13326ce` plus the separately disclosed, hash-pinned historical benchmark parser repairs. Later test-only commit `8300a1cef` had byte-identical runtime. **The subsequent guard changes in `ca7d42212` are not included in that full result.** - The [first 373 original attempts](https://wandb.ai/nvidia-nemo-framework/yuyao-nemo-gym-exps/runs/pr3626-n35ga-consistency-20261001-v2-full) and [remaining 358 original attempts](https://wandb.ai/nvidia-nemo-framework/yuyao-nemo-gym-exps/runs/pr3626-n35ga-consistency-20261001-v2-c1-full) cover disjoint original IDs with no retries or replacement. The first controller stopped dispatch during a roughly seven-minute model API stall, so its W&B run is marked failed although all 373 started episodes returned normally. The same model allocation recovered without restart and served the continuation. Both sandbox cleanup inventories are empty, and the owned model was cancelled. - Before that full run, HSG setup passed 338 tests (10 optional checks skipped on the CPU controller and covered locally), a real two-turn model tool preflight passed, and the [8-task smoke](https://wandb.ai/nvidia-nemo-framework/yuyao-nemo-gym-exps/runs/pr3626-n35ga-consistency-20261001-v2-smoke) produced 8 graded / 6 solved, zero agent/verifier errors, 411 model calls, 522 tool calls, and 16 close receipts. - Final guard-version HSG validation completed: 360 setup tests passed (10 optional runtime checks skipped and covered locally), the real two-turn tool preflight passed, and the [8-task smoke](https://wandb.ai/nvidia-nemo-framework/yuyao-nemo-gym-exps/runs/pr3626-n35ga-review-guards-20261001-smoke) returned **8 graded / 5 solved / 0 agent or verifier errors**. Snapshot `dc1550231` contains Pi `ca7d42212` plus the same benchmark repairs. Audited all 406 model calls, 522 tool calls, 406 available model references, and 16 successful close receipts; all five passes have complete required-test evidence. Zero results: a NodeBB assertion failure, ProtonMail UI assertion failures, and the previously observed Flipt `Secure`-field compilation failure. All 6,134 source entries match after execution; the run is finished, sandbox inventories are empty, and the owned model was cancelled. This validates the guard changes with real rollouts; it is separate from the earlier 731-task result. The upstream target mirror `NVIDIA-NeMo/Gym:codex/hermes-swe-pro-validation` is synchronized with NVIDIA-NeMo#3554 at `d5f13f54cf762c7e687dd2d50d2583efe2c9b717`. The mirror advanced by 11 commits using a fast-forward. Pi head `ca7d42212` is unchanged, so the recorded validation still applies; the diff against the synchronized base contains 23 changed files. The inherited SWE Pro response still reports incomplete verification separately via `evaluation_completed=false`; changing its default reward/mask contract is flagged for the shared benchmark rather than silently changed here. Full-run accuracy and grading coverage are reported separately. ## Shared runner integration (2026-10-02) Integrated the approved split-contract refactor in `5741f9f4fd816f6c4e79ac1002793adb1db6c3af` from [draft NVIDIA-NeMo#10](yaoyu-33#10) / [draft NVIDIA-NeMo#11](yaoyu-33#11). Cleanup uses the existing supervisor `CleanupReceipt`; worker identity uses `RunnerRuntimeInfo`; harness output stays separate. No combined `SandboxRunnerResult`. Only the Pi callers, shared helper, and relevant tests were integrated here. Hermes-specific callers and regressions are routed to NVIDIA-NeMo#3961. The shared helper and its tests are identical across all three PRs. Remote supervisor/worker files, model requests, sampling and scoring logic are unchanged; malformed cleanup/runtime metadata is now rejected. A fenced, unstarted worker retains an unknown exit code, and runtime failure does not discard confirmed cleanup. Validation on this integrated head (macOS, Python 3.13.14; pinned Hermes source and existing dependency environments; localhost fake model/MCP servers): **448 passed, 37 skipped**. Skips include Linux-only process-supervision and optional runtime/CLI checks. Full `pre-commit run --all-files` and `git diff --check` passed; commit is DCO-signed. ```sh python -m pytest tests/unit_tests/test_sandbox_runner_transport.py \ tests/unit_tests/test_agent_sessions.py tests/unit_tests/test_agent_session_retries.py \ tests/unit_tests/test_process_supervisor.py \ responses_api_agents/pi_agent/tests responses_api_agents/hermes_agent/tests \ --import-mode=importlib -q -o addopts='' -o log_cli=false ``` CI classification: full (shared core changes). No fresh real-model rollout or Linux rerun was performed for this integration; prior rollout evidence above remains pinned to its stated revision and is not evidence for this new head. Existing PR readiness/labels are unchanged. ## Rebase to main — October 2 Targets `main`; NVIDIA-NeMo#3554 is merged. Current-head macOS/Python 3.13.14 checks: **337 passed, 26 skipped**, full pre-commit and whitespace checks passed. Linux-only/optional tests remain skipped; no fresh real-model run was performed. Earlier rollout evidence remains tied to its recorded commits. ## HSG Linux test follow-up (2026-10-03) Published test-only commit `9f28b7278dfc53a292a128587520b92af12e0fdf`. Isolate the stdlib event-producer subprocess with `-I -S` and include cleanup/runtime/event logs before decoding failed output. The original intermittent empty capture remains unexplained; a controlled slow site hook fails before the patch and passes afterward under the same deadline. HSG Linux aarch64/Python 3.13.14: **366 passed / 13 skipped**, plus **30/30** distinct executions of the affected test. Full local `pre-commit run --all-files` and whitespace checks pass; commit has DCO sign-off. The listed GitHub lint, copyright, secret and DCO checks also pass. Full repository/component CI remains separate. ```bash python -m pytest responses_api_agents/pi_agent/tests \ tests/unit_tests/test_base_responses_api_agent.py \ tests/unit_tests/test_agent_sessions.py tests/unit_tests/test_agent_session_retries.py \ tests/unit_tests/test_episode_types.py environment_servers/single_agent_turn/tests \ tests/unit_tests/test_sandbox_runner_transport.py tests/unit_tests/test_process_supervisor.py responses_api_agents/pi_sandboxed_agent/tests responses_api_models/openai_model/tests \ --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short ``` Executed with source-bound imports and compatible dependency-only site-packages; [shared-board evidence, exact drivers, pins, jobs, skips and limitations](https://gitlab-master.nvidia.com/yuya/how-to-run/-/blob/main/fixes/harness-linux-test-stability.md). All four selected HSG suites pass 1,860 tests with 20 skips. Prior owned/borrowed real-model smokes remain pinned to their original revisions; these later test-only edits do not change production behavior and no model rerun is claimed. </details> </details> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.