Skip to content

fix(swebench-pro): preserve patch bytes during extraction - #3643

Merged
yaoyu-33 merged 2 commits into
NVIDIA-NeMo:codex/hermes-swe-pro-validationfrom
yaoyu-33:codex/swepro-preserve-patch-newline
Sep 23, 2026
Merged

yaoyu-33 merged 2 commits into
NVIDIA-NeMo:codex/hermes-swe-pro-validationfrom
yaoyu-33:codex/swepro-preserve-patch-newline

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Integrated into #3554 and mirrored to NVIDIA-NeMo/Gym:codex/hermes-swe-pro-validation at 00db9de0a4159dcbba0e77f14cc6c8a658de9c96. This merged PR retains the focused review and validation record.

OpenSandbox command logs can omit the final newline from git diff, turning a valid model patch into corrupt patch at line .... SWE-bench Pro then evaluates the unchanged base repository. This was reproduced during the real Qwen3-8B native-harness validation for MB-1694.

Write the diff to a private, unique file outside the task repository and download it through the sandbox file API. Preserve line endings, no-newline markers, committed/untracked changes, pristine-image filtering, and existing sandbox/session cleanup. Remove both temporary files on success and failure. A nested finally ensures cancellation during remote patch-file removal still stops the sandbox and clears its extraction state, while preserving CancelledError propagation. The shared sandbox provider is unchanged.

This focused follow-up is based on #3554 and applies to all SWE-bench Pro harnesses. It is separate from OpenCode #3639, OpenClaw #3637, and Codex #3640. History review of #2498 confirms that extraction must retain committed, modified, and untracked changes relative to the task base; its review discussions did not require exec-log transport. Those semantics remain covered.

Validation:

  • resources_servers/swebench_pro/tests: 63 passed locally and 63 passed on Linux/Python 3.13.14. A private launcher binds the existing validation interpreter to the isolated checkout. New tests create actual Git repositories, reproduce a malformed patch after terminal-newline loss, then apply the downloaded patch and verify exact LF, CRLF, and no-final-newline contents. Git/download failure cleanup and existing native HTTP lifecycle tests pass.
  • Cancellation regression: real asyncio.Task.cancel() calls during git execution, download, and remote rm verify cancellation propagation, sandbox stop, session-state removal, and local temporary-directory cleanup. Before the nested-finally fix, the rm case failed with zero sandbox-stop calls; all three cases now pass.
  • Independent review: all 10 patch-extraction tests passed; no remaining findings.
  • pre-commit run --all-files and git diff --check: passed. Commit has DCO sign-off.
  • Real model smoke on the original file-transfer implementation (c4691cfa6): Qwen3-8B through OpenCode 1.17.11, native HTTP EnvironmentServer, OpenSandbox, and the real pinned SWE-bench Pro Ansible task. The harness completed after six model calls. The extracted candidate reached the fresh verifier with its terminal newline intact; all 16 required outcomes were reported. The model did not solve the task (reward 0). Full process and session-file cleanup was inspected before verification, and all owned sandboxes were stopped. The subsequent cancellation-only fix was validated with the regression tests above; model inference was not rerun for that change.
  • A separate verifier-only replay restored only the dropped terminal newline of the original saved OpenCode candidate. Patch application changed from false to true; the model candidate still failed one of 16 required tests. No new inference or golden patch was used in that replay. Original and replay artifacts are retained privately.

This changes patch transport correctness, so affected historical zero scores may need regrading. It does not change grading rules, model behavior, or test requirements. Keep draft pending upstream integration and repository runtime CI; this is functional evidence, not an accuracy baseline.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 added bug Something isn't working area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers labels Sep 23, 2026
@copy-pr-bot

copy-pr-bot Bot commented Sep 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
yaoyu-33 merged commit 00db9de into NVIDIA-NeMo:codex/hermes-swe-pro-validation Sep 23, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant