You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(terminal-bench): distinguish verifier infrastructure failures from zero scores #4571
Terminal-Bench 2.1 verifier infrastructure failures can become cached, unmasked zero rewards. This pre-existing behavior was raised during #3902 review and is separate from the sandbox lifecycle repair.
At e98eac14ac5708020472d88387d1be62634b9327, _verify catches verifier upload/setup/exec errors (the repair candidate retains this behavior in the extracted _verify_sandbox), then returns evaluation_completed=false, reward=0.0 without setting the common failure/masking fields. The Resources session caches this response. A repeated verify does not execute the failed operation again, and the resulting zero can be confused with a valid incorrect task result.
Reproduction and evidence
Independent local reproduction on the #3902 repair candidate uses the real verify → _verify → _verify_sandbox path and the canonical session replay, with only the sandbox provider fixture mocked:
Seed one Resources session.
Make _upload_folder raise RuntimeError for the verifier upload.
Call verify twice with the same request.
Observe both responses: evaluation_completed=false, reward=0.0, mask_sample=false, failure_kind=null, failure_reason=null.
The upload executes once, no reward is downloaded, and owner close still stops the sandbox once.
Source: verifier error path. This is a deterministic fixture reproduction, not an observed model-scoring failure or benchmark prevalence estimate.
Preserve legitimate zero scores as unmasked task evidence.
Make replay consistent with the chosen terminal/retryable outcome; do not rerun a potentially mutating verifier implicitly.
Add regressions for upload/exec/reward-download failures versus valid zero reward, including session replay and reporting coverage.
Verify aggregate reporting does not silently include unscored infrastructure attempts in the measured-score denominator.
No benchmark scoring change is included in the focused lifecycle repair. Owner: unassigned; benchmark/core outcome maintainers should select the contract before implementing it.
Activity
feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw (NVIDI…