Repository navigation
feat(terminal-bench-science): run three more harnesses in the task sandbox - #5459
Open
gwarmstrong wants to merge 26 commits into
Open
gwarmstrong wants to merge 26 commits into
gwarmstrong wants to merge 26 commits into
Conversation
Benchmarks that grade a harness started inside the task container (Harbor's agents) need Pi to run with its own tools and prompt, call the benchmark's model endpoint, and leave logs where the verifier side collects them. Pi's sandbox sessions could only call a Gym model server with Gym's runtime guards, and they lost Pi's context-limit outcomes with Pi 0.80.10. All new settings are off or empty by default, so existing configs behave as before: - model_base_url and model_api_key: Pi calls an OpenAI-compatible endpoint directly from the sandbox instead of a Gym model server (set exactly one). Gym does not capture those calls. - thinking_level_map and provider_compat for the generated provider, so that `--thinking max` can send chat reasoning_effort "max"; settings merged recursively over Gym's settings.json (for example retry counts). - sandbox_env, and sandbox_runtime_guards: false to run Pi's own tools and system prompt without Gym's bash cap and prompt note. - session_prompt_cache_key: one prompt_cache_key per session on every call, so a gateway that routes on it keeps the session on one replica. - sandbox_logs_dir and log_requests: Pi's JSON events (pi.txt), its stderr and a per-request log without the conversation in a sandbox directory such as Harbor's /logs/agent. - raise_oom_score_adj: Pi and its tools become the memory-cgroup OOM killer's preferred victims, not the exec daemon or the supervisor. - verify_after_agent_error: a Pi run that failed while its sandbox still runs and answers is returned as a gradable failed response, as Harbor grades after any agent exit, unless its terminal error is a transient model-endpoint failure (after Pi's own retries) or cleanup is unconfirmed. The runner drops Pi's streaming message_update and tool_execution_update events, which repeat the whole partial message on every delta, so long turns no longer grow the event log quadratically. Pi 0.80.10 routes extension console output to stderr in JSON mode, so outcome.mjs writes its records to a session file that the runner appends to the event log; before, context-limit outcomes never reached the adapter in sandbox sessions. Local runs keep printing them. The task's HOME, the agent timeout from the task metadata and the runner's cleanup on capture errors are upstream behaviour and unchanged. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
opencode_agent sandbox sessions could only call a Gym model server. A benchmark that grades a harness started in the task container, as Harbor runs OpenCode, has the harness call the model endpoint itself, and routing those calls through a Gym model server changes the requests the endpoint sees. model_base_url and model_api_key (both unset by default) point the session's OpenAI-compatible provider at an endpoint the sandbox reaches, sending `model` as the model ID; set exactly one of model_server and model_base_url. Gym does not capture those calls, so the model-call correlation plugin is not loaded for them. Sessions with a model server and local calls are unchanged. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
A sandbox session's provider sent no reasoning effort, replayed reasoning only as reasoning_content, capped every call at max_output_tokens and told OpenCode the model's size, so it compacted before the endpoint's real limit. A harness started in a task container with its own model config does none of that, and some endpoints read the replayed reasoning from another field. Four settings, whose defaults keep today's requests: - reasoning_effort: the model's reasoningEffort option, sent as chat reasoning_effort on every request. - interleaved_reasoning_field: the assistant-message field that replays each turn's reasoning (default reasoning_content). - output_token_policy: remaining_context loads a plugin that drops max_tokens from the session provider's requests, so the endpoint picks the output budget, as opencode_sandboxed_agent's policy of that name does. - declare_model_limits: false leaves the model's size unknown to OpenCode, which then compacts only when the endpoint reports an overflow. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Two gaps kept a sandbox session from running OpenCode the way a harness started in the task container does: - Task images without a WORKDIR start in /, and sessions rejected that workdir because the adapter's runtime and session files under /tmp sit inside it. allow_root_workdir accepts exactly / (the files stay under /tmp, as a harness's own files do there); /tmp and the adapter's own directories are still rejected, and other workdirs keep the full overlap check. - A launcher that passes the instruction as one argument, as Harbor 0.24's opencode agent does, makes `opencode run` wrap it in double quotes (inner quotes escaped) before the model sees it. Sessions send the instruction on stdin, unquoted. argv_prompt_quoting sends the text OpenCode builds from that argument instead. Both are off by default. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Harbor runs the verifier after any agent exit that is not a timeout, so a trial whose agent died while its sandbox still answers is graded, not rerun. opencode_agent sessions raised an execution error for every failed OpenCode run (a nonzero exit, an error on an assistant turn, no terminal result), so the Environment Server skipped verification and retried the episode from scratch: an OpenCode the kernel's OOM kill hit, one whose own pkill hit it, or a request the endpoint rejected. verify_after_agent_error (off by default) returns such a run as a `failed` response (`incomplete` when it also hit its time limit), with the error in metadata.graded_agent_error, which Resources grades as a nonzero agent exit. It applies only when the supervisor confirmed cleanup, the run's session export was read, the last assistant turn's error is not a transient model-endpoint failure (OpenCode's retryable APIError: 429, 5xx, reset or lost connections, header and stream timeouts) and the sandbox still reports running and answers a probe command. Anything else stays an execution error and is retried. raise_oom_score_adj (off by default) makes OpenCode the OOM killer's preferred victim in its sandbox, as codex_agent's setting of that name does: the runner raises its own oom_score_adj, which OpenCode and its tools inherit, to 1000, or to 0 when the sandbox starts it below zero. The exec daemon and the supervisor keep the sandbox's own score. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Sandbox sessions pass the instruction to `claude -p` as an argument after `--`. Harbor 0.24's claude-code agent pipes it on stdin instead, and on a command line the task's own words are a target: an agent that runs `pkill -f <word from its task>` stops its own Claude Code. instruction_on_stdin (off by default) leaves the instruction out of argv and has the runner feed it to the CLI on stdin from a session file. Without it, the runner keeps stdin empty, as before. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Running Claude Code against a self-hosted model the way its provider's Claude Code guide configures it needs three client settings the agent could not express: the reasoning effort (`--effort`, which Harbor passes from its reasoning_effort), the per-call output cap (CLAUDE_CODE_MAX_OUTPUT_TOKENS) and further client environment such as API_TIMEOUT_MS or CLAUDE_CODE_AUTO_COMPACT_WINDOW. A settings file on the agent host could carry environment, but not from a run config, and not the CLI flag. reasoning_effort, max_output_tokens and extra_env (all unset by default) apply to local and sandbox launches alike. extra_env rejects the variables the agent sets itself (endpoint, credentials, model, CLAUDE_CONFIG_DIR, the output cap), and YAML integers become strings. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
A task image without a WORKDIR lends "/" as its working directory, and sandbox sessions refuse it because the runtime and session files under /tmp sit inside it, so every such trial fails at session start while a harness started in the container runs there. allow_root_workdir (off by default) accepts exactly "/". The runtime and session files stay under /tmp; /tmp itself, the adapter's own directories and relative or `..` paths are still refused. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…essions Before Harbor 0.24 installs Claude Code in a task container, its claude-code agent runs ensure_system_dependencies for curl, bash, nodejs, npm and procps: each one whose commands are missing is installed as root with the first package manager found. Agents graded under Harbor therefore have node, npm, ps and pkill even where the task image lacks them, and a sandbox session that skips the step runs Claude Code in a different environment. harbor_system_dependencies (off by default) runs the same decisions as one POSIX sh script (system_dependencies.py: Harbor's checks, package names, manager order and install command lines, logged line by line) before the session installs Claude Code, bounded by system_dependencies_timeout_seconds (default 1800). Where Harbor errors the trial on a failed install, the session logs the failure and runs Claude Code anyway. The script's tests run it under real shells with stand-in package managers. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
… extension writes them Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…dbox-opt-ins Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> # Conflicts: # nemo_gym/agent_utils/sandbox_session.py # responses_api_agents/opencode_agent/app.py # responses_api_agents/pi_agent/app.py # responses_api_agents/pi_agent/tests/test_sandbox.py
…ndbox Claude Code, OpenCode and Pi profiles for Terminal-Bench-Science v0.1, composed like the Codex profile: the TB4 resources server lends the task sandbox, each harness runs in it with the PR 8 opt-ins that match Harbor 0.24's agents, and single_agent_turn_legacy grades the episode. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Contributor
Author
|
/ok to test 84ac5a7 |
With verify_after_agent_error, Pi and OpenCode graded any failed run while the sandbox still ran. A run the harness runner could not start (exit 125, 126 or 127), a Pi run whose Gym MCP tools did not load (exit 78), or an OpenCode run whose required Gym MCP tools did not connect or were never confirmed is a setup failure: it now stays an execution error and is retried, never graded as the agent's result. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…bench-science-more-harnesses-v2 Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Contributor
Author
|
/ok to test 830d6a6 |
github-actions
Bot
added
the
sla:triage-overdue
…dbox-opt-ins Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> # Conflicts: # responses_api_agents/opencode_agent/app.py # responses_api_agents/pi_agent/app.py # responses_api_agents/pi_agent/tests/test_sandbox.py # tests/unit_tests/test_harness_runner.py
…bench-science-more-harnesses-v2 Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Contributor
Author
|
/ok to test 0a8dcf3 |
…dbox-opt-ins Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> # Conflicts: # nemo_gym/agents/__init__.py # responses_api_agents/pi_agent/tests/test_sandbox.py
…-ins' into georgea/terminal-bench-science-more-harnesses-v2 Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Contributor
Author
|
/ok to test bcf17eb |
…dbox-opt-ins Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> # Conflicts: # responses_api_agents/opencode_agent/app.py
…-ins' into georgea/terminal-bench-science-more-harnesses-v2 Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Contributor
Author
|
/ok to test 6780292 |
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…-ins' into georgea/terminal-bench-science-more-harnesses-v2 Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Contributor
Author
|
/ok to test 815e56e |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #5458 (Pi, OpenCode and Claude Code settings), which is not merged yet: until it is, this PR's diff against main also shows #5458's commits. This PR's own change is one commit, 9 files, plus merges of #5458: gwarmstrong/Gym@georgea/harness-sandbox-opt-ins...georgea/terminal-bench-science-more-harnesses-v2
#5219 (TB4 sandbox sessions and the Codex profile) is on main. It does not need #5457: each harness calls its endpoint directly.
What does this PR do?
Adds Terminal-Bench-Science profiles for Pi (0.80.10), OpenCode (1.17.8) and Claude Code (2.1.207). Each runs inside the task sandbox through #5219's TB4 sandbox sessions and calls its endpoint directly from the sandbox: chat completions for Pi and OpenCode, Anthropic Messages for Claude Code. Pi and OpenCode default to DeepSeek V4.1 Flash at effort max; Claude Code takes its model and client settings from the run config, and the README gives DeepSeek's. A model error the harness reports itself goes to the environment's outcome policy (#4834).
The profiles share
resources.yamlwith the Codex profile, so they get its sandbox settings, including #5334's disk floor and the opt-out from the cell's S3 mount. Like the Codex and mini-SWE profiles, each runs one trial per task;++num_repeats=3runs the leaderboard's three.Until #5227 merges, Claude Code's token usage on these runs is counted as #5158 counts it. A run that ends on its own reports only the uncached part of its input. Against a server like vLLM, a run cut off at the time limit reports each call's whole prompt as input, with no cache split and no output. Grading is unaffected.
No issue: profiles for the benchmark added in #4879.
Validation
terminal_bench_4350 passed;pi_agent368,opencode_agent676 andclaude_code_agent223 passed.tests/unit_tests/environment/test_environment_validation.py, which also checks the three harness manifests against their runtime composition: 49 passed. Every server block of the three profiles validates against its config class with no unknown keys.pre-commitclean.main, before feat(agent): report native causes for shared grading policy #4834's outcome policy, OpenSandbox, 70 tasks × 3 trials, 8 h, self-hosted DeepSeek V4.1 Flash: Pi 20/210 and OpenCode 20/210 (OpenCode then through the legacyopencode_sandboxed_agent).Rollouts
At 84ac5a7 (before #5458's setup-failure fix, which changes only failed runs), on OpenSandbox against self-hosted DeepSeek V4.1 Flash (one replica behind the multiplexer, with a recording proxy as each harness's endpoint), one task per harness with the official instruction and a 30-minute limit. All three ran in the task sandbox, stopped at the limit and were graded. Every model call returned 200.
reasoning_effort: "max",max_tokens393,216, a per-sessionprompt_cache_keyreasoning_effort: "max", nomax_tokensdeepseek-flash-gap(fromdeepseek-flash-gap[1m]),max_tokens128,000, adaptive thinking, effortmaxcached_tokens0, no outputpi.txt, 1.0 MB after 30 minutes) holds no update events, andpi-requests.jsonlhas one line per call.oom_score_adj1000, while the supervisor and the exec daemon kept the sandbox's 999.Compatibility and benchmark impact
New profiles only.