Repository navigation
Conversation
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
This was referenced Sep 23, 2026
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
yaoyu-33
merged commit Sep 23, 2026
00db9de
into
NVIDIA-NeMo:codex/hermes-swe-pro-validation
7 checks passed
This was referenced Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrated into #3554 and mirrored to
NVIDIA-NeMo/Gym:codex/hermes-swe-pro-validationat00db9de0a4159dcbba0e77f14cc6c8a658de9c96. This merged PR retains the focused review and validation record.OpenSandbox command logs can omit the final newline from
git diff, turning a valid model patch intocorrupt patch at line .... SWE-bench Pro then evaluates the unchanged base repository. This was reproduced during the real Qwen3-8B native-harness validation for MB-1694.Write the diff to a private, unique file outside the task repository and download it through the sandbox file API. Preserve line endings, no-newline markers, committed/untracked changes, pristine-image filtering, and existing sandbox/session cleanup. Remove both temporary files on success and failure. A nested
finallyensures cancellation during remote patch-file removal still stops the sandbox and clears its extraction state, while preservingCancelledErrorpropagation. The shared sandbox provider is unchanged.This focused follow-up is based on #3554 and applies to all SWE-bench Pro harnesses. It is separate from OpenCode #3639, OpenClaw #3637, and Codex #3640. History review of #2498 confirms that extraction must retain committed, modified, and untracked changes relative to the task base; its review discussions did not require exec-log transport. Those semantics remain covered.
Validation:
resources_servers/swebench_pro/tests: 63 passed locally and 63 passed on Linux/Python 3.13.14. A private launcher binds the existing validation interpreter to the isolated checkout. New tests create actual Git repositories, reproduce a malformed patch after terminal-newline loss, then apply the downloaded patch and verify exact LF, CRLF, and no-final-newline contents. Git/download failure cleanup and existing native HTTP lifecycle tests pass.asyncio.Task.cancel()calls during git execution, download, and remotermverify cancellation propagation, sandbox stop, session-state removal, and local temporary-directory cleanup. Before the nested-finally fix, thermcase failed with zero sandbox-stop calls; all three cases now pass.pre-commit run --all-filesandgit diff --check: passed. Commit has DCO sign-off.c4691cfa6): Qwen3-8B through OpenCode 1.17.11, native HTTP EnvironmentServer, OpenSandbox, and the real pinned SWE-bench Pro Ansible task. The harness completed after six model calls. The extracted candidate reached the fresh verifier with its terminal newline intact; all 16 required outcomes were reported. The model did not solve the task (reward 0). Full process and session-file cleanup was inspected before verification, and all owned sandboxes were stopped. The subsequent cancellation-only fix was validated with the regression tests above; model inference was not rerun for that change.This changes patch transport correctness, so affected historical zero scores may need regrading. It does not change grading rules, model behavior, or test requirements. Keep draft pending upstream integration and repository runtime CI; this is functional evidence, not an accuracy baseline.