Repository navigation
fix(opencode): report confirmed sandbox OOMs and preserve capture failures - #5128
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
8c91a82 to
ea55d66
Compare
|
🌿 Preview your docs: https://nvidia-preview-pr-5128-12d0b6b0f783.docs.buildwithfern.com/nemo/gym |
ea55d66 to
12d0b6b
Compare
An initial inspect can run before a sandbox terminates. Allow one final inspect when lifecycle status becomes terminal, bounded by the unchanged diagnostic deadline. Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> (cherry picked from commit 78abab71650b47c38c21a2cf5576f52256320709)
…lures Signed-off-by: Martyna Patelka <mpatelka@nvidia.com> Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Keep failure IDs and bounded observations across cached agent replies and checkpoint restoration so verifiers receive the original provider evidence. Forward diagnostic reports without selecting a terminal failure; missing evidence remains ungradable. Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Keep ungraded setup and capture failures typed and evaluation-side. Preserve canonical provider events and strict artifact checks. Report confirmed OOM outcomes through the existing environment policy so configured fail or retry actions apply, while the default still verifies and the verifier owns scoring. Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Bound staging queue, upload attempts and backoff by the configured timeout. Preserve harness attribution and command exit facts when execution cannot be classified, and document separate setup and task budgets. Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Replace the sandbox observation phase field with canonical stage. Keep origin stages and event identities when forwarding execution failures. Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> (cherry picked from commit 199db43397e63c1b057b2b6ec24fde699d0164b8)
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> (cherry picked from commit db417b07adabf6bd375a1707cbd6e387a9eff970)
4693b37 to
1a64a1c
Compare
|
Addressed the review in a7af4c7 and 1a64a1c:
The assembled local selection passed 949 tests. Representative benchmark qualification is not claimed. |
AdamRajfer
left a comment
There was a problem hiding this comment.
Looks good. Snapshot failures other than a sandbox I/O timeout propagate again, the SWE-bench assertion checks the full dict, and the ripgrep exhaustion is typed. The ungraded-without-flag behaviour is called out in the description, which is enough for me. Approving.
Preparation for the base move after #5127 landed. Five files conflicted: both sides added names to the same import lists and export map (agent_outcome.py, core/__init__.py: union, the branch's superset kept), both sides extended the swebench observation assertion (stage from main, wall_time_s from the branch: both kept), the OpenCode seed gained a sandbox_id guard on the branch and an export_dir argument on main (both kept), and both sides appended tests at the end of the OpenCode session test module (both kept, branch first). One semantic conflict beyond the textual ones: main's new test_session_exports test expected a failed import setup step to raise the old RuntimeError with "exit=N", while the branch raises the typed NgFailureEvent (agent_route_failed, opencode.setup, seed) carrying "return_code=N". The test now asserts the typed form and keeps its point: an import that never ran or was killed stays retryable, never a seed rejection. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
Merged main into the branch to clear the conflicts after #5127 landed. One import test was adjusted to the typed failure event the branch raises; no functional change, suites green locally. |
|
/ok to test 7ecaeed |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42099741244 |
OpenCode preserves confirmed sandbox OOM evidence and capture failures for the owning policy, rather than deciding their score.
Targets #5127 in #5127 → #5128 → #5129 → #5605. Refs #4685.
stage="agent"for agent execution. Locally created setup failures useseed; existing event IDs, stages and causes are preserved.report_agent_sandbox_oomremains opt-in. Confirmed agent-container OOM carries its original event ID and AgentOutcome; the caller applies scoring and attribution policy.sandbox_timeoutbudgets. Plugin read-back, query/stdin/env support and converter behavior are retained.Compatibility
Ungraded handling of missing/corrupt primary captures and timeouts without a launch marker is intentional and applies even when
report_agent_sandbox_oomis disabled. Those cases do not establish a usable completed agent run; the flag controls confirmed-OOM reporting, not capture validity.A sandbox
provider_io_timeoutwhile collecting the optional SQLite snapshot preserves a valid primary export, records the original failure, and reports observation gaps. OOM, unknown outcomes, other typed failures and primary-export failures still propagate.SWE-bench Pro and Multilingual are not covered by the OOM-reporting flag.
Validation
Integrated selected tests, scoped pre-commit/import-linter and fresh model-backed scenarios passed as detailed in #5127, including exclusions and exact source pins. Independent Claude review and one fix pass completed. Commits are SSH-signed with DCO.
Both adapter paths and preserved origin stages have regression coverage; the live OpenCode → DeepSWE path exercises the actual canonical observation.
Full repository CI and representative real-task qualification are not claimed.
Latest review fixes: the final assembled selection at
6f247486a3ff58b72ca9d83933960f4c547e6c2dpassed 949 tests; scoped pre-commit and import-linter passed. Individual fixes received independent Claude review and one fix pass. These checks do not assert full repository CI or benchmark-wide score invariance.Final-source live snapshot check at
6f247486a3ff58b72ca9d83933960f4c547e6c2d: a real-model OpenCode → DeepSWE control retained reward 1, primary output/root usage and the original failure event after an injected optional snapshot download timeout. The preceding attempt stopped before injection with an unknown command exit; it is retained as a failed attempt, not counted as validation. Two calls on the successful retry plus one on the first attempt were captured, and all five disposable sandboxes were confirmed deleted. This is a synthetic plumbing check, not representative benchmark qualification.