fix(agent): restore main harness conformance evidence by yaoyu-33 · Pull Request #4450 · NVIDIA-NeMo/Gym

yaoyu-33 · 2026-10-10T19:38:26Z

The main/merge-queue run 38069955548 completed all probe jobs but failed the P0 measurement: Codex 0/7, OpenCode 5/7, Pi 6/7, Hermes 7/7. The reporter is correctly exposing retained-evidence failures.

This integrates the applicable existing fixes with the current native-session adapters:

Port fix(codex): retain native rollout evidence and retry rate limits #4047's temporary Codex session reader and bounded 429 forwarding retries, including its timeout result preservation. Keep invocation ownership, original tool IDs/arguments/results and shell exit status before removing the temporary session. Native sandbox-session execution remains as implemented on main.
Port the shell-outcome portion of fix(opencode): preserve overrides and sandbox tool outcomes #4046 into the shared OpenCode artifact parser. Nonzero exits fail; absent/null/invalid exits remain unknown. Preserve native turn semantics, per that PR's review discussion.
Require one terminal-400 token reconciliation outcome: healthy, with a known zero aggregate for zero finished responses. Preserve Hermes' actual runtime counters when early error returns omit them. Failed model termination still must be unhealthy; missing aggregate usage and incorrect nonzero totals fail. The health engine is unchanged.

The token-total rule follows merged #4287. This deliberately does not revive closed #4286's proposal to accept either healthy or unobserved. No separate issue is necessary: the linked run is the reproduction, and #4046/#4047 plus FEA-461 track the underlying gaps. Existing PRs are left intact for author/maintainer coordination. Codex work is attributed to Frederic Lafrance and Giulio Lovisotto; OpenCode's nonzero-exit correction derives from Frederic's patch.

Validation on macOS arm64 / Python 3.13.14, OpenAI SDK 2.44.0; base 728ac3395d1bc26b995b03ffb9aa6f0362b768ea:

2,050 passed, 113 skipped in the final selected suite; skips are platform/optional-runtime cases.
Full pre-commit run --all-files and staged/working-tree git diff --check pass. Commit has DCO sign-off.
28/28 conformance scenarios pass across all four pinned harnesses; final Codex-only rerun is also 7/7.
Real nvidia/nemotron-3-super-120b-a12b model smoke: Codex, OpenCode and Hermes each have four captured model calls, one failed tool followed by two successful tools, and reward 1.0. A local verifier independently checked the file contains 391 and the final answer contains CONFORMANCE_DONE. OpenCode's initial attempt was denied by its native external-directory permission; the rerun explicitly granted only the smoke's temporary artifact directory. Both attempts are retained.

Exact test selection:

python -m pytest --import-mode=importlib -q -o addopts='' -o log_cli=false \
  scripts/harness_conformance/tests tests/unit_tests/harness_capabilities \
  tests/unit_tests/test_harness_conformance_ci.py tests/unit_tests/test_harness_conformance_checks.py \
  tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_rollout_health.py \
  responses_api_agents/codex_agent/tests responses_api_agents/opencode_agent/tests \
  responses_api_agents/opencode_sandboxed_agent/tests responses_api_agents/pi_agent/tests \
  responses_api_agents/hermes_agent/tests
python scripts/run_harness_conformance.py --harness codex --harness opencode --harness pi --harness hermes --output <NEW_DIRECTORY>

Runtime pins: Codex 0.144.4, OpenCode 1.17.11, Pi 0.80.2, Hermes 26bb847a. Source imports were verified against the repair worktree; an early mixed-source/SDK setup attempt was discarded. The final Codex suite was rerun after adding malformed-record protection.

Compatibility: correlated local Codex runs temporarily persist a session and can retry 429 twice; OpenCode shell-status evidence becomes accurate; terminal-rejection aggregate usage is strict. No dataset, reward policy, model settings, or benchmark-score claim changes. These probes qualify local adapter evidence; remote sandbox lifecycle qualification, Linux-only skipped tests and full repository CI remain separate. CI classification is full, because runner/shared test paths changed. No CI trust-gate comment was posted.

Follow-up validation (daab232a8): Codex returns observations and timeout state directly instead of encoding synthetic stdout events. All 227 Codex tests passed (45 skipped), all-file pre-commit passed, and all seven fresh Codex conformance scenarios passed. Retry behavior is unchanged.

Port the applicable local Codex evidence and bounded retry implementation from NVIDIA-NeMo#4047 and shell outcome classification from NVIDIA-NeMo#4046 onto the current session adapters. Preserve unknown shell outcomes and retain Hermes native usage counters on early errors. Enforce one terminal-rejection token reconciliation outcome without relaxing missing or contradictory evidence checks. Co-authored-by: Frederic Lafrance <flafrance@nvidia.com> Co-authored-by: Giulio Lovisotto <glovisotto@nvidia.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

copy-pr-bot · 2026-10-10T19:38:30Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Respect explicit interleaved settings and apply the shell-outcome rules from #4450 to the separate sandboxed adapter. Co-authored-by: Frederic Lafrance <flafrance@nvidia.com> Co-authored-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

giuliolovisotto · 2026-10-12T09:57:28Z

/ok to test daab232

github-actions · 2026-10-12T10:12:52Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38234888727

Respect explicit interleaved settings and apply the shell-outcome rules from #4450 to the separate sandboxed adapter. Co-authored-by: Frederic Lafrance <flafrance@nvidia.com> Co-authored-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

…eMo#4046) After NVIDIA-NeMo#4450, two OpenCode gaps remain: local configuration overwrites explicit `interleaved` settings, and the separate `opencode_sandboxed_agent` reader treats completed shell execution as successful even for nonzero exits. This preserves explicit `interleaved` settings (defaulting to `reasoning_content` only when absent) and applies the existing shared reader's shell-outcome handling to the separate sandboxed reader: nonzero integer exits are failed; missing/invalid exits are unknown with an evidence gap. The existing outcome regression runs against both readers. Native policy turns and reward computation are unchanged. Based directly on main after NVIDIA-NeMo#4450 merged. This follows up NVIDIA-NeMo#4046's original review concerns; no separate issue is needed for these two remaining fixes. Validation on rebased commit `b1735c5e2`, main `33be39484`: - Both complete OpenCode adapter test directories: **420 passed, 37 skipped** (`pytest -q -o addopts='' -o log_cli=false --import-mode=importlib responses_api_agents/opencode_agent/tests responses_api_agents/opencode_sandboxed_agent/tests`). - `pre-commit run --all-files` and `git diff --check`: passed. - Fresh `scripts/run_harness_conformance.py --harness opencode`: **7/7 scenarios passed**, OpenCode 1.17.11. - Fresh live `nvidia/nvidia/nemotron-3-nano-30b-a3b` smoke through Gym collection: **4 model calls, reward 1, evaluation_completed=true**. An independent verifier checked the computed file value (391) and final answer. Both production readers agreed on one failed shell tool and two successful tools from the saved native SQLite artifact. This validates the separate sandboxed reader, not a full container execution; Docker-backed conformance is separate work. No dataset or scoring-policy changes. Saved sandbox tool outcomes become more accurate. Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> Co-authored-by: Giulio Lovisotto <glovisotto@nvidia.com> Co-authored-by: yaoyu-33 <yaoyu.094@gmail.com>

…Mo#4473) The terminal HTTP 400 scenario currently excludes verifier evidence and skips the verifier outcome/reward checks. A harness can therefore pass while skipping grading or masking a graded rollout. Require verifier evidence for this scenario, reuse the existing observed-outcome and saved-reward checks, and add one P0 behavioral check requiring `evaluation_completed=true` and `mask_sample=false`. Final-answer checking remains inapplicable when no answer was generated. The same saved-outcome check applies to the other controlled scenarios, whose verifier also completes normally. Follow-up to NVIDIA-NeMo#4450. No separate GitHub issue is needed for this focused gap in the existing scenario. No producer or health-engine changes; stricter conformance can expose previously hidden producer failures. Keep draft pending the Hermes masking follow-up. Validation: - `python -m pytest -o addopts= --import-mode=importlib -q scripts/harness_conformance/tests tests/unit_tests/harness_capabilities tests/unit_tests/test_harness_conformance_checks.py tests/unit_tests/test_harness_conformance_ci.py`: 379 passed, 2 skipped. - `pre-commit run --all-files`: passed. - Fresh local scripted conformance on this change: | Harness | Passing scenarios | Remaining failure | | --- | --- | --- | | `opencode_agent` | 7/7 | None | | `codex_agent` | 7/7 | None | | `pi_agent` | 7/7 | None | | `hermes_agent` | 6/7 | `model_error`: `verifier.saved_outcome` | Hermes independently calls the verifier and saves reward `0.0` and `evaluation_completed=true`, then saves `mask_sample=true`. This is the previously hidden producer gap exposed by the new check. No expectation is weakened to accept it. Pi was rerun from scratch with its installed 0.80.2 binary on PATH after the first launch could not find it. - Regression coverage: skipped verifier, changed reward, masked/incomplete result, missing completion flag, and missing rollout delivery. - External live-model smoke: N/A; this changes the conformance checker only. The scripted suite executes the real harness processes and collects fresh rollouts. GitHub lint, copyright, secrets and DCO checks passed. No inline review comments at inspection. Full conformance CI is not automatically dispatched for a draft PR; the copy-PR bot reports that restriction. Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

yaoyu-33 added bug Something isn't working area:agent Agent harnesses and Responses API agent behavior needs-review PR is ready for code review and waiting on a reviewer labels Oct 10, 2026

yaoyu-33 added the complexity:high Cross-component or shared-contract change with a large review surface label Oct 10, 2026

giuliolovisotto mentioned this pull request Oct 11, 2026

fix(nooa): retain task services through grading and preserve terminal failures #4291

Closed

4 of 5 tasks

giuliolovisotto mentioned this pull request Oct 12, 2026

fix(opencode): preserve overrides and sandbox tool outcomes #4046

Merged

refactor(codex): return evidence directly

daab232

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

lafrancef mentioned this pull request Oct 12, 2026

fix(codex): retain native rollout evidence and retry rate limits #4047

Closed

giuliolovisotto self-requested a review October 12, 2026 09:55

giuliolovisotto enabled auto-merge October 12, 2026 09:55

giuliolovisotto approved these changes Oct 12, 2026

View reviewed changes

nemo-automation-bot Bot requested review from a team October 12, 2026 09:58

copy-pr-bot Bot temporarily deployed to public October 12, 2026 09:58 Inactive

copy-pr-bot Bot temporarily deployed to public October 12, 2026 10:00 Inactive

copy-pr-bot Bot temporarily deployed to public October 12, 2026 10:01 Inactive

giuliolovisotto added this pull request to the merge queue Oct 12, 2026

Merged via the queue into NVIDIA-NeMo:main with commit 33be394 Oct 12, 2026
49 checks passed

giuliolovisotto mentioned this pull request Oct 12, 2026

test(conformance): require grading after terminal HTTP 400 #4473

Merged

yuhezhang-ai mentioned this pull request Oct 12, 2026

feat(eval): persist failures and resume unfinished evaluations #3302

Merged

5 tasks done

This was referenced Oct 14, 2026

feat(token-capture): preserve finish reasons and terminal truncation #3973

Merged

fix(model): return OpenAI context-overflow error envelopes #3978

Merged

svcnemo-autobot mentioned this pull request Nov 4, 2026

Feat(indic jeebench): Add Indic JEE Bench #4465

Open

5 tasks done

Navigation Menu

Repository navigation

fix(agent): restore main harness conformance evidence - #4450

fix(agent): restore main harness conformance evidence#4450
giuliolovisotto merged 2 commits into
NVIDIA-NeMo:mainfrom
yaoyu-33:codex/fix-main-conformance

yaoyu-33 commented Oct 10, 2026 •

edited by giuliolovisotto

Loading

Uh oh!

copy-pr-bot Bot commented Oct 10, 2026

Uh oh!

giuliolovisotto commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

yaoyu-33 commented Oct 10, 2026 • edited by giuliolovisotto Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Oct 10, 2026

Uh oh!

giuliolovisotto commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yaoyu-33 commented Oct 10, 2026 •

edited by giuliolovisotto

Loading