Repository navigation
fix(ci): align harness conformance checks with capture semantics - #4251
Merged
giuliolovisotto merged 2 commits intoOct 9, 2026
Merged
Conversation
yaoyu-33
added
bug
7 tasks done
yaoyu-33
added
the
complexity:low
giuliolovisotto
force-pushed
the
codex/fix-harness-conformance
branch
from
October 9, 2026 10:06
e8b12df to
2f29cc2
Compare
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-4251-85c2af039bf2.docs.buildwithfern.com/nemo/gym |
This was referenced Oct 9, 2026
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
giuliolovisotto
force-pushed
the
codex/fix-harness-conformance
branch
from
October 9, 2026 11:18
da18e98 to
85c2af0
Compare
giuliolovisotto
approved these changes
Oct 9, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37953189183 |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37953690934 |
yaoyu-33
pushed a commit
to yaoyu-33/Gym
that referenced
this pull request
Oct 9, 2026
…#4288) ## What changes Local OpenCode lacked the correlation plugin already enabled by the sandboxed adapter, and the adapters separately selected their runtime versions. Both now use `opencode_agent/runtime.py`: one release pin, one unchanged JavaScript patch, and one `apply_observability_patch(...)` function. The sandbox only adds the necessary upload of that same file. Existing user plugins are preserved; the patch is enabled with model-call observability. The shared module is 15 lines. The plugin is moved unchanged. Both YAML version duplicates are removed, and the CI installer uses the same default. Configured versions must match the executable, including preinstalled/cached sandbox binaries. Local version mismatches now raise an actionable error instead of warning and continuing. Explicit `opencode_version` overrides remain supported in both adapters; the default is stock 1.17.11 for the relevant platform. Split from NVIDIA-NeMo#4251 at Giulio's request. That PR retains the independent checker and scripted-capture header fixes; it contains no plugin-loading change. This PR targets main independently. No reward/scoring changes or binary fork. ## Validation ```bash PYTHONPATH=$PWD python -m pytest \ responses_api_agents/opencode_agent/tests \ responses_api_agents/opencode_sandboxed_agent/tests \ tests/unit_tests/test_harness_conformance_ci.py \ scripts/harness_conformance/tests -q python -m pre_commit run --all-files git diff --check ``` **302 passed, 2 skipped**; all-files pre-commit passed. Tests cover both adapters' plugin configuration, preserved user plugins, capture disabled, the shared CI/default version, actual plugin hooks, and rejecting mismatched binaries before task execution. Fresh real-runtime smoke artifacts are in `/Users/glovisotto/repos/notes/opencode-shared-runtime-20261009/`: - Local stock OpenCode 1.17.11: 3 calls linked to 3 turns, two shell tools, reward 1 and `evaluation_completed=true`. The standalone scripted fixture supplies the normal production assistant-message capture header (the separate runner fix in NVIDIA-NeMo#4251). - Docker sandbox, stock Linux OpenCode 1.17.11: injected 503 then two successes; 3 captured attempts linked to 2 turns, tool executed and requested marker returned. Plugin comes from the adapter upload, without manual plugin configuration. - Live inference smoke in Docker: one completed call and one correctly linked turn, usage retained. The Llama-3.1-8B response did not match the requested marker, so the smoke's task assertion failed; only runtime/correlation validation passed. No SWE-Pro benchmark qualification is claimed. Source, scripts, logs and fresh artifacts are retained with the report. Full core/all-server suites were not run locally. No new dependencies. New module has the NVIDIA SPDX header; commit has DCO sign-off. --------- Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
ehosseiniasl
pushed a commit
to ehosseiniasl/Gym
that referenced
this pull request
Oct 9, 2026
…o#4285) Pi's SDK cannot recognize context overflow when an explicitly propagated provider error is wrapped as `{"detail":{"error":...}}`: it reports “400 status code (no body)” instead of the provider message. Its HTTP 400/503 integration tests also omitted the propagation setting required by the current OpenAI Model Server. Remove the adapter's extra HTTPException conversion and use the existing shared Model Server error handler. Selected statuses now retain the original provider body, including top-level `error.message` and `error.code`. LiteLLM inherits the same correction. After Chat streaming sends a keepalive, the terminal SSE error also retains the selected provider explanation; unselected provider errors remain generic HTTP 500 messages in the stream. The Pi fixture explicitly enables the statuses it exercises. The default propagation policy remains opt-in; unselected errors and exhausted provider retries retain their HTTP 500 behavior. **Compatibility:** consumers of explicitly propagated OpenAI/LiteLLM errors must read the original provider body rather than the former `detail` wrapper. Capture retains that same served body. Follow-up to NVIDIA-NeMo#3415 and NVIDIA-NeMo#3626; related tracking: [FEA-461](https://linear.app/nvidia/issue/FEA-461). This addresses the separate Pi CI failure exposed while validating NVIDIA-NeMo#4251. Validation on this branch: - **507 passed**: OpenAI and LiteLLM adapter tests, base Model Server/Chat and Responses streaming tests, and pinned Pi 0.80.2 SDK outcome tests. Covers opt-in and default status handling, Chat/Responses, streaming/nonstreaming, capture, exhausted retries, and permanent-error handling. A delayed-error regression fails before the fix; the pinned Pi SDK recognizes HTTP 400 context overflow and HTTP 503 both before and after the first keepalive. - **Two real local HTTP smokes passed:** Pi SDK → Gym OpenAI Model Server → fault-injection provider, HTTP 400 context overflow and HTTP 503 service unavailable. Exactly one upstream request per case; Pi retains each error message and correctly distinguishes context overflow. No generative-model benchmark run: this is an error-response wire-format change. - Full `pre-commit run --all-files` and `git diff --check` passed; DCO-signed. ```bash python -m pytest -o addopts= --import-mode=importlib -q \ responses_api_agents/pi_agent/tests/test_outcome.py \ responses_api_models/openai_model/tests/test_app.py \ responses_api_models/litellm_model/tests/test_app.py \ tests/unit_tests/test_base_responses_api_model.py \ tests/unit_tests/test_responses_api_model_streaming.py \ tests/unit_tests/test_chat_completions_streaming.py ``` --------- Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
7 tasks done
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Align conformance with #4045: retained HTTP errors belong to an invocation and may have zero or one step reference. Successful policy responses still require a step; compaction calls remain separate. This corrects the P0 check introduced in #4192 without inventing turns for failed HTTP exchanges.
Also pass the configured assistant-message header to the scripted Model Server's capture middleware, matching normal Model Server startup. Tests cover header retention on HTTP success/failure and reject duplicate links or missing ownership.
The OpenCode plugin-loading commit was removed and replaced by the separate shared-runtime PR #4288. This PR only changes conformance checks, probe wiring, their tests and documentation. It is rebased on main
77509af61.Validation
After removing the plugin-loading commit:
PYTHONPATH=$PWD python -m pytest \ tests/unit_tests/harness_capabilities \ scripts/harness_conformance/tests \ tests/unit_tests/test_harness_conformance_checks.py -q342 passed, 2 skipped. Full core and all-server suites were not run locally. The all-files pre-commit check passed on the rebased branch before removing the isolated plugin-loading commit.
The fresh four-harness probe run on the rebased branch, before that split, gave OpenCode 5/7, Hermes 7/7, Pi 6/7, Codex 0/7. These are not results for this final split: OpenCode additionally needs the separate plugin setup. Hermes benefits from #4048 now on main. Pi's two provider-error classification unit failures remain separate (#4285); its HTTP-400 health expectation also remains unresolved. Codex retains independent evidence gaps.
No fresh generative-model benchmark is needed for this diagnostic/probe correction. No API migration or reward/scoring change. No new source files; existing licenses retained. All commits have DCO sign-off.