Skip to content

fix(terminal-bench): distinguish verifier infrastructure failures from zero scores #4571

Open

Description

@yaoyu-33

Problem

Terminal-Bench 2.1 verifier infrastructure failures can become cached, unmasked zero rewards. This pre-existing behavior was raised during #3902 review and is separate from the sandbox lifecycle repair.

At e98eac14ac5708020472d88387d1be62634b9327, _verify catches verifier upload/setup/exec errors (the repair candidate retains this behavior in the extracted _verify_sandbox), then returns evaluation_completed=false, reward=0.0 without setting the common failure/masking fields. The Resources session caches this response. A repeated verify does not execute the failed operation again, and the resulting zero can be confused with a valid incorrect task result.

Reproduction and evidence

Independent local reproduction on the #3902 repair candidate uses the real verify → _verify → _verify_sandbox path and the canonical session replay, with only the sandbox provider fixture mocked:

  1. Seed one Resources session.
  2. Make _upload_folder raise RuntimeError for the verifier upload.
  3. Call verify twice with the same request.
  4. Observe both responses: evaluation_completed=false, reward=0.0, mask_sample=false, failure_kind=null, failure_reason=null.
  5. The upload executes once, no reward is downloaded, and owner close still stops the sandbox once.

Source: verifier error path. This is a deterministic fixture reproduction, not an observed model-scoring failure or benchmark prevalence estimate.

Requested follow-up

  • Select the benchmark contract for infrastructure failures: a typed no-result outcome (feat(core): define typed rollout outcomes and recovery #2135), or a completed masked result using the fields introduced by feat(core): define completed verifier-result failure semantics #2608.
  • Preserve legitimate zero scores as unmasked task evidence.
  • Make replay consistent with the chosen terminal/retryable outcome; do not rerun a potentially mutating verifier implicitly.
  • Add regressions for upload/exec/reward-download failures versus valid zero reward, including session replay and reporting coverage.
  • Verify aggregate reporting does not silently include unscored infrastructure attempts in the measured-score denominator.

No benchmark scoring change is included in the focused lifecycle repair. Owner: unassigned; benchmark/core outcome maintainers should select the contract before implementing it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions