fix(opencode): retain sandbox identity and report session errors by giuliolovisotto · Pull Request #4741 · NVIDIA-NeMo/Gym

giuliolovisotto · 2026-10-19T14:21:35Z

Review follow-up on `4605a7e` (2026-10-22)

Warn during agent-session setup when the configured OpenCode version differs from the tested outcome-classifier version, 1.17.11. Keep explicitly pinned alternative versions usable.

Validation: 198 OpenCode session/review tests passed, including the warning and no-warning cases; scoped pre-commit passed.

Follow-up on `30c3e0b` (2026-10-22)

252 OpenCode session, continuation and error-formatting tests passed; scoped pre-commit passed. Explicit ContextOverflowError reports use model_context_limit; HTTP 400/413 alone remains model_request_error. Context overflow still returns incomplete and retains partial work. The renamed format_error_message helper retains the same 16,000-character bound.

Rebase validation before the label/formatter follow-up: the combined tip e5240e16e passed 28/28 Linux conformance scenarios, seven per harness. The policy-only head a928f859e passed 4/7 Hermes scenarios; the dependent stack fixes its evidence gaps. Both returned seven Hermes rollout records with no episode-failure rows; these scenarios do not establish exhausted-provider retry counts. These results are pinned to those revisions, not a claim of fresh CI or a new live-model run on the latest heads.

Retain the provider-assigned sandbox ID and reported API failures in agent sessions. Classify status-less retryable API errors as transport failures; retain native context-overflow stops as incomplete with partial work, while independent worker/cleanup failures still raise. Provider error text includes only the message and status, bounded to 16,000 characters; structured classification is retained separately.

Stack: #4834 (policy) → #4742 (Hermes) → #4739 (Pi) → #4741 (OpenCode) → #4740 (Codex), registered as GitHub stack #4899. These changes must land together. Do not merge #4834 alone: the Hermes and Codex classifier fixes are in #4742 and #4740.

Scope: agent-session execution only; legacy paths are not part of this follow-up. No conformance expectations or shared policy defaults are changed.

Validation (2026-10-21)

Full OpenCode adapter suite: 426 passed, 38 skipped on macOS; scoped pre-commit passed.
Fresh Linux conformance on combined stack tip d0b6b950a34bfbc63dfbfdf23c4abbf6d2e1bf34: 28/28, including OpenCode 7/7; this is combined-stack evidence, not an isolated-base claim.
One fresh live-model task passed with reward 1, unmasked completed verification, model-call/tool evidence and no failure sidecar. The route was GPT-4.1-mini. Linux local subprocess sandbox exercised; remote providers and Slurm were not exercised.
Portable live-smoke script and commands; no author-local paths are needed to reproduce it.
Historical CI for the previous head; inspect its actual outcomes rather than treating local validation as CI success.

python -m pytest responses_api_agents/opencode_agent/tests --import-mode=importlib -q -o addopts='' -o log_cli=false
python scripts/run_harness_conformance.py --harness opencode --timeout 300 --output /tmp/fresh-opencode-conformance

Fresh CI on combined stack tip d0b6b950a: this harness passed 7/7 (download its report/tests/logs); all four stack harnesses passed, while the separate Kilocode model_error failure keeps the non-blocking aggregate red (full CI summary).

copy-pr-bot · 2026-10-19T14:21:39Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

yaoyu-33

Thanks for the OpenCode fix. The classification is conservative and the tests assert real behavior (567 passed, 57 skipped locally). Main points: failed responses reaching /verify without a masking policy to receive them, status-less provider errors still raising 502, and a stale README line. Also optional: sandbox_id can be empty for ECS handles and the sync Sandbox has no equivalent, the serialized native error is unbounded, and the classifier's rejection guards are untested.

Review context: these five PRs (#4782 contract; #4739 Pi, #4740 Codex, #4741 OpenCode, #4742 Hermes) are a split of one change. This PR's diff vs main also contains #4782's conformance files, so please retarget/rebase once #4782 merges. Note that harness_conformance runs all four probes for any shared-file change, so model_error should fail for every harness until its fix lands; if that check is required, #4782 and the siblings need to land together.

PR hygiene: the validation commands and live-smoke artifacts reference an author-local path, and current-head CI isn't in the evidence (the combined 28/28 run is for 118f1de37). Could you attach the exact commands and the CI run for this head?

Generated by Claude Code

yaoyu-33 · 2026-10-20T16:28:04Z

            output=output,
            usage=usage,
            status=status,
+            error={"code": "server_error", "message": json.dumps(model_error)} if model_error else None,


app.py:1029

This now returns recognized OpenCode provider errors (any 400-599) as a failed response that goes on to /verify, which matches the #4782 contract. But I couldn't find anything on the environment-server side that masks it: single_agent_turn forwards the response unconditionally and mask_sample is only set by individual verifiers. Before this change a provider error raised a 502, which the environment server retries or drops, so a transient provider error can now become an unmasked reward-0 sample. Could you point to where that is masked, or land the masking/retry policy first (or limit this to model-caused statuses, as Hermes does for 400/413/422)?

Done—the stack includes #4834’s shared policy, which skips verification for provider execution failures and retries transient failures while leaving grading of 400/413/422 configurable.

yaoyu-33 · 2026-10-20T16:28:04Z

+            return None
+        info = assistants[-1]
+        error = info.get("error")
+        if not isinstance(error, dict) or error.get("name") != "APIError":


app.py:1058

Does OpenCode emit an APIError without a statusCode for connection resets, header timeouts and stream errors, and a separate context-overflow error? If so those still end up as a 502 here instead of a failed response, which seems at odds with "terminal provider failure => failed response" (and Hermes grades the equivalent 413 as a rejection). Could you accept status-less APIError, or note that these are intentionally still raised?

Status-less APIErrors now return classified failed responses using OpenCode’s retryable transport marker, while separate non-APIError failures, including context overflow, still use the existing exception path.

Done in 90ed544: status-less retryable API errors remain classified transport failures, while native ContextOverflowError returns incomplete with partial work retained and independent cleanup failures still raised.

yaoyu-33 · 2026-10-20T16:28:04Z

        )

+    @staticmethod
+    def _terminal_model_error(export: dict[str, Any], cleanup: dict | None) -> dict[str, Any] | None:


Docs: README.md:122

This line still says "Runtime/model errors fail before verification", but a recognized terminal provider error now returns a failed response that can reach verification. Could you update it to describe that, and that execution/cleanup failures still raise?

Done—the README now distinguishes reported API errors governed by the Environment Server policy from worker and cleanup execution failures.

lafrancef · 2026-10-22T07:21:08Z


+    @staticmethod
+    def _reported_model_error(native_result: Mapping[str, Any], cleanup: dict | None) -> bool:
+        """Recognize a completed API error or context limit; tested on OpenCode 1.17.11.


Do we expect this to hold across harness versions? If not, is there a way we could flag that this check might not give the correct result on a different version?

Added a warning in 4605a7ecd: session setup flags any configured OpenCode version other than the tested 1.17.11, explaining that error classification is untested on that version. An alternative exact version remains usable. Tests cover both warning and no-warning cases.

marta-sd · 2026-10-22T07:35:04Z

+
+        Independent worker failures, cleanup errors and timeouts are not provider reports.
+        """
+        finishes = {"APIError": None, "ContextOverflowError": "error"}


It would be good to define constants - we use these strings in several places

Done in 71f37cfb8: APIError and ContextOverflowError now have shared constants used by error recognition, classification and context-limit handling. Existing tests retain the literal upstream names to catch accidental value changes. All 254 focused OpenCode tests and scoped pre-commit checks passed.

marta-sd · 2026-10-22T08:55:16Z

                if isinstance(failure, asyncio.CancelledError):
                    record.error_type = "cancelled"
+                elif context_limit:
+                    record.error_type = "context_limit"


double checking: we don't rely on particular values anywhere, right? this is more like metadata/comment than a classification that is consumed downstream?

Yes, for this field: current Gym consumers do not branch on the "context_limit" value in AgentInvocation.error_type; it is descriptive observation metadata. Grading/retry decisions use the response status and the separate typed ng_agent_outcome.failure_kind. This context-limit path returns incomplete, so it remains gradable.

@marta-sd for now with these PRs I'm trying to add as much coverage as I can so that components can make decisions upstream on what to do with this outcome (the main use-cases now for using this info is to decide is whether to "retry", "grade even if incomplete", "exclude from scoring")

github-actions · 2026-10-22T13:19:00Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39785736592

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

giuliolovisotto · 2026-10-22T14:29:38Z

/ok to test

github-actions · 2026-10-22T14:50:47Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39799272539

giuliolovisotto added bug Something isn't working area:agent Agent harnesses and Responses API agent behavior labels Oct 19, 2026

giuliolovisotto marked this pull request as ready for review October 19, 2026 14:58

giuliolovisotto added the needs-review PR is ready for code review and waiting on a reviewer label Oct 19, 2026

nemo-automation-bot Bot requested review from a team October 19, 2026 14:59

copy-pr-bot Bot temporarily deployed to public October 19, 2026 14:59 Inactive

copy-pr-bot Bot temporarily deployed to public October 19, 2026 15:01 Inactive

copy-pr-bot Bot temporarily deployed to public October 19, 2026 15:02 Inactive

yaoyu-33 added the complexity:medium Single-domain change with interacting parts or a moderate review surface label Oct 19, 2026

giuliolovisotto mentioned this pull request Oct 20, 2026

test(conformance): check terminal errors reported to environments #4782

Merged

giuliolovisotto closed this Oct 20, 2026

giuliolovisotto mentioned this pull request Oct 20, 2026

fix(pi): report session errors and retain ownership #4739

Merged

giuliolovisotto reopened this Oct 20, 2026

giuliolovisotto changed the title ~~fix(opencode): retain agent-session conformance evidence~~ fix(opencode): retain sandbox identity and report session errors Oct 20, 2026

giuliolovisotto changed the base branch from main to glovisotto/conformance-terminal-errors October 20, 2026 09:58

giuliolovisotto force-pushed the glovisotto/opencode-session-conformance branch from f4ccfd1 to 5f19114 Compare October 20, 2026 09:58

This was referenced Oct 20, 2026

fix(hermes): retain session evidence and report provider errors #4742

Merged

fix(codex): retain session evidence and report model errors #4740

Merged

copy-pr-bot Bot temporarily deployed to public October 20, 2026 09:59 Inactive

copy-pr-bot Bot temporarily deployed to public October 20, 2026 10:01 Inactive

copy-pr-bot Bot temporarily deployed to public October 20, 2026 10:02 Inactive

Base automatically changed from glovisotto/conformance-terminal-errors to main October 20, 2026 11:41

github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 20, 2026

github-actions Bot mentioned this pull request Oct 20, 2026

[Tracker] Pull request handoff SLA #3402

Open

yaoyu-33 reviewed Oct 20, 2026

View reviewed changes

github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 20, 2026

copy-pr-bot Bot temporarily deployed to public October 22, 2026 06:17 Inactive

giuliolovisotto force-pushed the glovisotto/opencode-session-conformance branch from 4fc749f to 30c3e0b Compare October 22, 2026 06:25

copy-pr-bot Bot temporarily deployed to public October 22, 2026 06:25 Inactive

copy-pr-bot Bot temporarily deployed to public October 22, 2026 06:27 Inactive

copy-pr-bot Bot temporarily deployed to public October 22, 2026 06:28 Inactive

lafrancef previously approved these changes Oct 22, 2026

View reviewed changes

giuliolovisotto dismissed lafrancef’s stale review via 4605a7e October 22, 2026 08:36

giuliolovisotto force-pushed the glovisotto/opencode-session-conformance branch from 30c3e0b to 4605a7e Compare October 22, 2026 08:36

copy-pr-bot Bot temporarily deployed to public October 22, 2026 08:37 Inactive

copy-pr-bot Bot temporarily deployed to public October 22, 2026 08:39 Inactive

marta-sd reviewed Oct 22, 2026

View reviewed changes

marta-sd previously approved these changes Oct 22, 2026

View reviewed changes

giuliolovisotto added 9 commits October 22, 2026 13:43

fix(opencode): retain sandbox failure evidence

f2850cf

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

fix(sandbox): preserve existing identity property

01b21af

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

fix(opencode): classify native transport failures

de7dbf3

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

test(opencode): assert policy enum action

c911cd1

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

fix(opencode): retain context-limit outcomes

5a4e923

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

test(opencode): cover failures during continuation

570a97b

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

fix(opencode): distinguish model context overflow

cd25b74

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

fix(opencode): flag untested outcome versions

0ab2170

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

refactor(opencode): name provider error constants

ed00b9c

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

mlazuka approved these changes Oct 22, 2026

View reviewed changes

Conversation

giuliolovisotto commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review follow-up on 4605a7e (2026-10-22)

Follow-up on 30c3e0b (2026-10-22)

Validation (2026-10-21)

Uh oh!

copy-pr-bot Bot commented Oct 19, 2026

Uh oh!

yaoyu-33 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 22, 2026

Uh oh!

giuliolovisotto commented Oct 22, 2026

Uh oh!

github-actions Bot commented Oct 22, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

giuliolovisotto commented Oct 19, 2026 •

edited

Loading

Review follow-up on `4605a7e` (2026-10-22)

Follow-up on `30c3e0b` (2026-10-22)