Skip to content

fix(opencode): report confirmed sandbox OOMs and preserve capture failures - #5128

Merged
AdamRajfer merged 47 commits into
mainfrom
mpatelka/opencode-failure-reporting-4685
Nov 5, 2026
Merged

AdamRajfer merged 47 commits into
mainfrom
mpatelka/opencode-failure-reporting-4685

Conversation

@mpatel31415

@mpatel31415 mpatel31415 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

OpenCode preserves confirmed sandbox OOM evidence and capture failures for the owning policy, rather than deciding their score.

Targets #5127 in #5127 → #5128 → #5129 → #5605. Refs #4685.

  • Both OpenCode adapters report canonical stage="agent" for agent execution. Locally created setup failures use seed; existing event IDs, stages and causes are preserved.
  • report_agent_sandbox_oom remains opt-in. Confirmed agent-container OOM carries its original event ID and AgentOutcome; the caller applies scoring and attribution policy.
  • Missing or corrupt primary captures and uncertain command outcomes remain ungraded failures. A timeout without usable capture does not enable a zero fallback.
  • Ripgrep staging bounds queue wait, upload attempts and backoff. Ambiguous uploads retry at a fresh remote path; uncertain commands are never replayed.
  • Opt-in split execution gives setup and agent execution separate sandbox_timeout budgets. Plugin read-back, query/stdin/env support and converter behavior are retained.

Compatibility

Ungraded handling of missing/corrupt primary captures and timeouts without a launch marker is intentional and applies even when report_agent_sandbox_oom is disabled. Those cases do not establish a usable completed agent run; the flag controls confirmed-OOM reporting, not capture validity.

A sandbox provider_io_timeout while collecting the optional SQLite snapshot preserves a valid primary export, records the original failure, and reports observation gaps. OOM, unknown outcomes, other typed failures and primary-export failures still propagate.

SWE-bench Pro and Multilingual are not covered by the OOM-reporting flag.

Validation

Integrated selected tests, scoped pre-commit/import-linter and fresh model-backed scenarios passed as detailed in #5127, including exclusions and exact source pins. Independent Claude review and one fix pass completed. Commits are SSH-signed with DCO.

Both adapter paths and preserved origin stages have regression coverage; the live OpenCode → DeepSWE path exercises the actual canonical observation.

Full repository CI and representative real-task qualification are not claimed.

Latest review fixes: the final assembled selection at 6f247486a3ff58b72ca9d83933960f4c547e6c2d passed 949 tests; scoped pre-commit and import-linter passed. Individual fixes received independent Claude review and one fix pass. These checks do not assert full repository CI or benchmark-wide score invariance.

Final-source live snapshot check at 6f247486a3ff58b72ca9d83933960f4c547e6c2d: a real-model OpenCode → DeepSWE control retained reward 1, primary output/root usage and the original failure event after an injected optional snapshot download timeout. The preceding attempt stopped before injection with an unknown command exit; it is retained as a failed attempt, not counted as validation. Two calls on the successful retry plus one on the first attempt were captured, and all five disposable sandboxes were confirmed deleted. This is a synthetic plumbing check, not representative benchmark qualification.

@mpatel31415 mpatel31415 added bug Something isn't working area:agent Agent harnesses and Responses API agent behavior labels Oct 27, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 27, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Oct 27, 2026
@giuliolovisotto
giuliolovisotto force-pushed the mpatelka/opencode-failure-reporting-4685 branch from ea55d66 to 12d0b6b Compare October 28, 2026 10:36
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 28, 2026
@giuliolovisotto
giuliolovisotto removed this pull request from stack #5130 November 3, 2026 10:51
giuliolovisotto and others added 9 commits November 5, 2026 17:20
An initial inspect can run before a sandbox terminates. Allow one final inspect when lifecycle status becomes terminal, bounded by the unchanged diagnostic deadline.

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
(cherry picked from commit 78abab71650b47c38c21a2cf5576f52256320709)
…lures

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Keep failure IDs and bounded observations across cached agent replies and checkpoint restoration so verifiers receive the original provider evidence. Forward diagnostic reports without selecting a terminal failure; missing evidence remains ungradable.

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Keep ungraded setup and capture failures typed and evaluation-side. Preserve canonical provider events and strict artifact checks. Report confirmed OOM outcomes through the existing environment policy so configured fail or retry actions apply, while the default still verifies and the verifier owns scoring.

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Bound staging queue, upload attempts and backoff by the configured timeout. Preserve harness attribution and command exit facts when execution cannot be classified, and document separate setup and task budgets.

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Replace the sandbox observation phase field with canonical stage. Keep
origin stages and event identities when forwarding execution failures.

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
(cherry picked from commit 199db43397e63c1b057b2b6ec24fde699d0164b8)
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
(cherry picked from commit db417b07adabf6bd375a1707cbd6e387a9eff970)
@giuliolovisotto
giuliolovisotto force-pushed the mpatelka/opencode-failure-reporting-4685 branch from 4693b37 to 1a64a1c Compare November 5, 2026 16:29
@giuliolovisotto

Copy link
Copy Markdown
Contributor

Addressed the review in a7af4c7 and 1a64a1c:

  • Optional snapshot I/O timeout now preserves a valid primary capture and records gaps plus the original failure; primary capture, OOM and unknown-outcome failures remain failures.
  • The SWE-bench assertion compares the whole dictionary and expects stage="verification", rather than the nonexistent phase field or None. Final raw ripgrep upload timeout is typed without widening its retry budget.
  • Missing/corrupt primary captures and timeouts without a launch marker are intentionally ungraded even with the OOM flag disabled: they do not establish a usable completed run. The description states this behavior and excludes SWE-bench Pro/Multilingual from OOM-flag coverage.
  • error_message remains an optional shared observation field; the canonical event referenced by failure_id carries the reason, so this path does not duplicate it. verification_failure_fields() is consumed via response.model_copy(update=fields) when a report exists.

The assembled local selection passed 949 tests. Representative benchmark qualification is not claimed.

AdamRajfer
AdamRajfer previously approved these changes Nov 5, 2026

@AdamRajfer AdamRajfer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Snapshot failures other than a sandbox I/O timeout propagate again, the SWE-bench assertion checks the full dict, and the ripgrep exhaustion is typed. The ungraded-without-flag behaviour is called out in the description, which is enough for me. Approving.

@AdamRajfer
AdamRajfer added this pull request to the merge queue Nov 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue because a pull request earlier in the stack was removed Nov 5, 2026
@AdamRajfer
AdamRajfer added this pull request to the merge queue Nov 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 5, 2026
Base automatically changed from mpatelka/sandbox-failure-contracts-4685 to main November 5, 2026 17:52
Preparation for the base move after #5127 landed. Five files conflicted:
both sides added names to the same import lists and export map
(agent_outcome.py, core/__init__.py: union, the branch's superset kept),
both sides extended the swebench observation assertion (stage from main,
wall_time_s from the branch: both kept), the OpenCode seed gained a
sandbox_id guard on the branch and an export_dir argument on main (both
kept), and both sides appended tests at the end of the OpenCode session
test module (both kept, branch first).

One semantic conflict beyond the textual ones: main's new
test_session_exports test expected a failed import setup step to raise the
old RuntimeError with "exit=N", while the branch raises the typed
NgFailureEvent (agent_route_failed, opencode.setup, seed) carrying
"return_code=N". The test now asserts the typed form and keeps its point:
an import that never ran or was killed stays retryable, never a seed
rejection.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer

AdamRajfer commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Merged main into the branch to clear the conflicts after #5127 landed. One import test was adjusted to the typed failure event the branch raises; no functional change, suites green locally.

@AdamRajfer

Copy link
Copy Markdown
Contributor

/ok to test 7ecaeed

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42099741244

This branch was previously deployed

1 inactive deployment
public — 7ecaeed2 Deployed Nov 5, 2026 by copy-pr-bot[bot] via release / finalize / notify #10461
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants