Repository navigation
feat(hle_diamond): run the text tool presets as opencode_agent and pi_agent sessions - #5702
Draft
weijiac0619 wants to merge 58 commits into
Draft
weijiac0619 wants to merge 58 commits into
weijiac0619 wants to merge 58 commits into
Conversation
Benchmarks that grade a harness started inside the task container (Harbor's agents) need Pi to run with its own tools and prompt, call the benchmark's model endpoint, and leave logs where the verifier side collects them. Pi's sandbox sessions could only call a Gym model server with Gym's runtime guards, and they lost Pi's context-limit outcomes with Pi 0.80.10. All new settings are off or empty by default, so existing configs behave as before: - model_base_url and model_api_key: Pi calls an OpenAI-compatible endpoint directly from the sandbox instead of a Gym model server (set exactly one). Gym does not capture those calls. - thinking_level_map and provider_compat for the generated provider, so that `--thinking max` can send chat reasoning_effort "max"; settings merged recursively over Gym's settings.json (for example retry counts). - sandbox_env, and sandbox_runtime_guards: false to run Pi's own tools and system prompt without Gym's bash cap and prompt note. - session_prompt_cache_key: one prompt_cache_key per session on every call, so a gateway that routes on it keeps the session on one replica. - sandbox_logs_dir and log_requests: Pi's JSON events (pi.txt), its stderr and a per-request log without the conversation in a sandbox directory such as Harbor's /logs/agent. - raise_oom_score_adj: Pi and its tools become the memory-cgroup OOM killer's preferred victims, not the exec daemon or the supervisor. - verify_after_agent_error: a Pi run that failed while its sandbox still runs and answers is returned as a gradable failed response, as Harbor grades after any agent exit, unless its terminal error is a transient model-endpoint failure (after Pi's own retries) or cleanup is unconfirmed. The runner drops Pi's streaming message_update and tool_execution_update events, which repeat the whole partial message on every delta, so long turns no longer grow the event log quadratically. Pi 0.80.10 routes extension console output to stderr in JSON mode, so outcome.mjs writes its records to a session file that the runner appends to the event log; before, context-limit outcomes never reached the adapter in sandbox sessions. Local runs keep printing them. The task's HOME, the agent timeout from the task metadata and the runner's cleanup on capture errors are upstream behaviour and unchanged. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
opencode_agent sandbox sessions could only call a Gym model server. A benchmark that grades a harness started in the task container, as Harbor runs OpenCode, has the harness call the model endpoint itself, and routing those calls through a Gym model server changes the requests the endpoint sees. model_base_url and model_api_key (both unset by default) point the session's OpenAI-compatible provider at an endpoint the sandbox reaches, sending `model` as the model ID; set exactly one of model_server and model_base_url. Gym does not capture those calls, so the model-call correlation plugin is not loaded for them. Sessions with a model server and local calls are unchanged. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
A sandbox session's provider sent no reasoning effort, replayed reasoning only as reasoning_content, capped every call at max_output_tokens and told OpenCode the model's size, so it compacted before the endpoint's real limit. A harness started in a task container with its own model config does none of that, and some endpoints read the replayed reasoning from another field. Four settings, whose defaults keep today's requests: - reasoning_effort: the model's reasoningEffort option, sent as chat reasoning_effort on every request. - interleaved_reasoning_field: the assistant-message field that replays each turn's reasoning (default reasoning_content). - output_token_policy: remaining_context loads a plugin that drops max_tokens from the session provider's requests, so the endpoint picks the output budget, as opencode_sandboxed_agent's policy of that name does. - declare_model_limits: false leaves the model's size unknown to OpenCode, which then compacts only when the endpoint reports an overflow. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Two gaps kept a sandbox session from running OpenCode the way a harness started in the task container does: - Task images without a WORKDIR start in /, and sessions rejected that workdir because the adapter's runtime and session files under /tmp sit inside it. allow_root_workdir accepts exactly / (the files stay under /tmp, as a harness's own files do there); /tmp and the adapter's own directories are still rejected, and other workdirs keep the full overlap check. - A launcher that passes the instruction as one argument, as Harbor 0.24's opencode agent does, makes `opencode run` wrap it in double quotes (inner quotes escaped) before the model sees it. Sessions send the instruction on stdin, unquoted. argv_prompt_quoting sends the text OpenCode builds from that argument instead. Both are off by default. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Harbor runs the verifier after any agent exit that is not a timeout, so a trial whose agent died while its sandbox still answers is graded, not rerun. opencode_agent sessions raised an execution error for every failed OpenCode run (a nonzero exit, an error on an assistant turn, no terminal result), so the Environment Server skipped verification and retried the episode from scratch: an OpenCode the kernel's OOM kill hit, one whose own pkill hit it, or a request the endpoint rejected. verify_after_agent_error (off by default) returns such a run as a `failed` response (`incomplete` when it also hit its time limit), with the error in metadata.graded_agent_error, which Resources grades as a nonzero agent exit. It applies only when the supervisor confirmed cleanup, the run's session export was read, the last assistant turn's error is not a transient model-endpoint failure (OpenCode's retryable APIError: 429, 5xx, reset or lost connections, header and stream timeouts) and the sandbox still reports running and answers a probe command. Anything else stays an execution error and is retried. raise_oom_score_adj (off by default) makes OpenCode the OOM killer's preferred victim in its sandbox, as codex_agent's setting of that name does: the runner raises its own oom_score_adj, which OpenCode and its tools inherit, to 1000, or to 0 when the sandbox starts it below zero. The exec daemon and the supervisor keep the sandbox's own score. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Sandbox sessions pass the instruction to `claude -p` as an argument after `--`. Harbor 0.24's claude-code agent pipes it on stdin instead, and on a command line the task's own words are a target: an agent that runs `pkill -f <word from its task>` stops its own Claude Code. instruction_on_stdin (off by default) leaves the instruction out of argv and has the runner feed it to the CLI on stdin from a session file. Without it, the runner keeps stdin empty, as before. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Running Claude Code against a self-hosted model the way its provider's Claude Code guide configures it needs three client settings the agent could not express: the reasoning effort (`--effort`, which Harbor passes from its reasoning_effort), the per-call output cap (CLAUDE_CODE_MAX_OUTPUT_TOKENS) and further client environment such as API_TIMEOUT_MS or CLAUDE_CODE_AUTO_COMPACT_WINDOW. A settings file on the agent host could carry environment, but not from a run config, and not the CLI flag. reasoning_effort, max_output_tokens and extra_env (all unset by default) apply to local and sandbox launches alike. extra_env rejects the variables the agent sets itself (endpoint, credentials, model, CLAUDE_CONFIG_DIR, the output cap), and YAML integers become strings. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
A task image without a WORKDIR lends "/" as its working directory, and sandbox sessions refuse it because the runtime and session files under /tmp sit inside it, so every such trial fails at session start while a harness started in the container runs there. allow_root_workdir (off by default) accepts exactly "/". The runtime and session files stay under /tmp; /tmp itself, the adapter's own directories and relative or `..` paths are still refused. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…essions Before Harbor 0.24 installs Claude Code in a task container, its claude-code agent runs ensure_system_dependencies for curl, bash, nodejs, npm and procps: each one whose commands are missing is installed as root with the first package manager found. Agents graded under Harbor therefore have node, npm, ps and pkill even where the task image lacks them, and a sandbox session that skips the step runs Claude Code in a different environment. harbor_system_dependencies (off by default) runs the same decisions as one POSIX sh script (system_dependencies.py: Harbor's checks, package names, manager order and install command lines, logged line by line) before the session installs Claude Code, bounded by system_dependencies_timeout_seconds (default 1800). Where Harbor errors the trial on a failed install, the session logs the failure and runs Claude Code anyway. The script's tests run it under real shells with stand-in package managers. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
… extension writes them Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…dbox-opt-ins Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> # Conflicts: # nemo_gym/agent_utils/sandbox_session.py # responses_api_agents/opencode_agent/app.py # responses_api_agents/pi_agent/app.py # responses_api_agents/pi_agent/tests/test_sandbox.py
With verify_after_agent_error, Pi and OpenCode graded any failed run while the sandbox still ran. A run the harness runner could not start (exit 125, 126 or 127), a Pi run whose Gym MCP tools did not load (exit 78), or an OpenCode run whose required Gym MCP tools did not connect or were never confirmed is a setup failure: it now stays an execution error and is retried, never graded as the agent's result. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…dbox-opt-ins Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> # Conflicts: # responses_api_agents/opencode_agent/app.py # responses_api_agents/pi_agent/app.py # responses_api_agents/pi_agent/tests/test_sandbox.py # tests/unit_tests/test_harness_runner.py
The deprecated pi_sandboxed_agent and opencode_sandboxed_agent each ship a configs/benchmark.yaml profile. A caller that composes a benchmark itself, such as an evaluation framework that runs HLE, loads that profile instead of naming a Resources server. The session harnesses that replace them had no such profile, so moving those callers off the deprecated agents meant copying the `resources_server.name: ???` binding into the caller, where some config systems reject a mandatory-missing value. Add the same profile to pi_agent, opencode_agent, codex_agent, openclaw_agent, hermes_agent and claude_code_agent, at configs/profiles/benchmark.yaml. Each leaves the Resources server unset, so Gym composes the agent onto the benchmark's own agent instance, and each session creates its own sandbox through the `sandbox` provider at an absolute address. The Claude Code profile also points the CLI at the `policy_model` Gym model server. Image, limits and prompts stay with the caller. Agent discovery reads only the top level of configs/, so, like opencode_agent's permission profiles, these are not listed as standalone agent variants. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
A session agent got tool grants only from its episode's own Resources server, so the HLE search presets, which add a Tavily server through tool_servers, had no session path and stayed on the deprecated sandboxed agents. single_agent_turn now takes tool_servers: each episode seeds its own session on every listed Resources server, with the episode's task data, grants that server's MCP tools as a required MCPToolAccess, and closes the session at cleanup. Unset, it falls back to the bound agent's tool_servers, declared on the base agent config, so a pairing moved off the legacy_agent relay keeps the tool servers its /run seeded, and a harness overlay can add search without naming generated instances. As with the Resources server's own MCP grant, an episode that grants one restarts from its input after a crash. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Benchmarks whose Resources server supplies no sandbox, such as HLE, have pi_agent create one per session. pi_sandboxed_agent ran those sandboxes with model-only egress; pi_agent could not, so a reasoning benchmark moved to agent sessions would gain open internet. - network_access (inherit, model_only, model_and_tools) sets an OpenSandbox egress policy on a sandbox the agent creates. A borrowed task sandbox keeps its owner's network, so a session that borrows one rejects any other value. The policy helper, restrict_sandbox_network, lives in nemo_gym.agents.utils.sandbox_network and is exported from nemo_gym.agents so other session agents can share it. - sandbox_runtime_dir points setup at a runtime baked into the image and checks its version, since a model-only sandbox cannot download Node or Pi. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Match pi_agent so reasoning benchmarks without task sandboxes can move off opencode_sandboxed_agent with the same model-only network: network_access (inherit, model_only, model_and_tools) sets the OpenSandbox egress policy on a sandbox OpenCode creates, and a session that borrows a task sandbox rejects any value other than inherit. It cannot be combined with sandbox_config.provider_options.network_policy. The policy comes from the shared restrict_sandbox_network helper, imported from nemo_gym.agents. The preinstalled runtime path already exists (sandbox_executable_path). Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…dbox-opt-ins Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> # Conflicts: # nemo_gym/agents/__init__.py # responses_api_agents/pi_agent/tests/test_sandbox.py
Startup now also rejects a tool server that is not a Resources server, the episode's own Resources server, and a Resources server without expose_tools_over_mcp, each of which otherwise failed every episode. The partner_if_dispatcher and nooa_agent tests build this server from a bare client, so they now pass tool_servers=[] instead of reading their agent's. The README separates the Resources transports from tool-server grants and says every seeded session is closed. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Composed onto HLE, every session harness rejected the first seed with a 422. HLE's Environment Server grants the agent its Resources server over direct_http, which no session harness can use; HLE Diamond, GPQA and APEX Shortlist do the same. The judge offers the agent no tools, so each adapter README now gives the pairing setting that clears the grant, `resources_tool_transports: []` under the benchmark's Environment Server. The profiles also fill the gaps an episode reached after that: - Claude Code pins `claude_code_version: "2.1.108"`, which sandbox sessions require. - Each profile declares `sandbox_config` with a mandatory image, so a missing image fails at resolve and the image can be set under the composed agent's name. All but Hermes set `ttl_s: 14400`, without which OpenSandbox ends the sandbox after 10 minutes; Hermes derives its TTL from its own timeouts. - `head_server.host` stays on 127.0.0.1. `use_absolute_ip` otherwise puts the head server, whose config endpoint is unauthenticated, on the address sandboxes reach. A new test composes each profile onto HLE, seeds the composed agent with the request HLE's Environment Server builds, and checks that the seed reaches sandbox creation with the setting and is refused without it. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
equivalence_llm_judge, math_with_judge and mcqa register no tool routes, but their Environment Servers granted the agent their Resources server over direct_http, a leftover of #4953's switch from the legacy_agent relay. Every config of these servers binds simple_agent, which accepts a seed with no grant, so dropping the transport changes nothing for it. A session harness, which rejects a required direct_http grant, now runs HLE, HLE Diamond, GPQA and APEX Shortlist without a per-benchmark override. The adapter READMEs and profile comments drop that override. The profile test now seeds each composed agent through HLE as shipped, and keeps a case that grants direct_http to show the 422 still guards pairings whose Resources server serves tools. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
The core test that seeded each composed agent imported all six adapters, so core coverage measured their modules, about 6,700 statements that only the adapters' own suites exercise, and fell to 87.57%, under its 95% floor. The seed check now lives in tests/session_harness_profiles.py and runs from a test in each adapter's suite, in that adapter's environment, which also runs the Hermes case that the core venv had to skip. The core test keeps the config-only checks. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
#5550 moved the HLE seed check out of the core test into each adapter's suite. This applies this branch's change there: the seed test runs through HLE as shipped, with a case that grants direct_http to keep the 422, and the core test reads the unset transports as empty. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…dbox-opt-ins Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com> # Conflicts: # responses_api_agents/opencode_agent/app.py
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
use_absolute_ip gave the agent server the address sandboxes reach. An unseeded request to its /v1/responses runs the harness on the Gym host with its built-in tools, so a sandbox could start one there. Each profile now binds its agent to 127.0.0.1, like the head server; only the model server, which the sandboxes need, takes the reachable address. No adapter's sandbox calls its own agent server. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
restrict_sandbox_network raised ValueError when a restricted network_access could not be enforced: an explicit provider network policy as well, a provider without network policies, or an endpoint the sandbox cannot reach. The seed route answered it with a 500, which the Environment Server retries, though no attempt can fix the configuration. The helper now raises AgentSeedRejectedError, still a ValueError, so every adapter that calls it refuses the seed once with HTTP 422, before any sandbox is created. Pi also built its tool egress from every MCP grant, including an optional one it leaves out at activation, so such a grant at a loopback URL failed the seed. Egress now covers only the grants Pi keeps. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…hange Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
#5108 made the Environment Server's resources_server required again and dropped the fallback to the agent's. The startup check for tool servers now compares against the configured resources_server; the agent fallback for tool_servers stays. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
… with a 422 With #5554's restrict_sandbox_network change, a restricted network_access the provider cannot enforce (here an explicit network_policy as well) refuses the seed with HTTP 422 before any sandbox is created, instead of a retried 500. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Both sides added public exports to nemo_gym.agents; both are kept. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
#5550 landed squashed, so its files conflicted as added on both sides. This branch's versions are kept: main had not changed them since #5550, except the Claude Code and Hermes READMEs, which take main's text with this change's paragraph applied. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…code egress change Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
The Add or Update a Harness page and the add-harness skill now say that an adapter that creates its own sandbox exposes network_access and passes the spec through restrict_sandbox_network, that settings it cannot enforce are refused with AgentSeedRejectedError, and that a restricted sandbox gets its runtime by upload or from a preinstalled sandbox_runtime_dir. They also list the tests, the model_only pairing run, and an egress exit check, so a new harness cannot leave a sandbox with open egress on a benchmark graded on the model's own knowledge. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Addresses the review of the Pi egress change: - A borrowed session with a restricted network_access, and a preinstalled runtime that is missing or runs another version, raise AgentSeedRejectedError: a final, logged 422 instead of a retried 500. A runtime check that hits a transport error or timeout stays retryable. - A test covers the MCP-only filter on tool egress: an optional direct_http grant opens no host. - The Pi README describes the runtime directory as copied whole (with libstdcpp/ on older Alpine), says a restricted sandbox needs it or a prebaked default path, recommends absolute_ip_server_types so only the model server leaves loopback, and notes that full isolation needs the OpenSandbox dns+nft egress mode. - The harness guide lists agents/utils/sandbox_network.py and the egress notes, and the OpenSandbox page points at restrict_sandbox_network. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Addresses the second review of the Pi egress change: - Pi's benchmark profile sets network_access: model_only, like pi_sandboxed_agent's profile, so an HLE or GPQA run that uses it as shipped no longer gets open internet. A search preset overrides it with model_and_tools. sandbox_runtime_dir is required (???), because a restricted sandbox cannot download Pi; null keeps an image prebaked at the default path. - restrict_sandbox_network takes the provider instead of its name, so the move to SandboxSpec.egress_rules and provider capabilities (#5363) only changes the helper's body, not its callers. - The README says a borrowed sandbox relies on its owner's policy. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
The profile tests derive which profiles restrict egress and require a runtime directory from the profiles themselves, so a harness that restricts its profile changes only its own files and the stacked PRs do not conflict. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…viewed Pi change restrict_sandbox_network now takes the provider, and a borrowed session with a restricted network_access is refused with AgentSeedRejectedError, so the 422 is logged like the other adapters'. OpenCode writes every MCP grant into its config, so its tool egress already matches the grants it keeps. The README notes that a borrowed sandbox relies on its owner's network policy. The benchmark profile keeps inherit: OpenCode installs its runtime from GitHub unless the image supplies sandbox_executable_path and sandbox_executable_sha256. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Reading the restricted profiles from the files keeps the stacked PRs apart but cannot catch a profile losing model_only. Pi's own seed-test file now expects model_only and a required runtime directory, so removing either fails it, and the shared helper supplies any harness setting a profile requires (a runtime directory, or a baked binary and its checksum) and checks that each one fails at resolve when unset. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
… server The profile sets network_access: model_only, as the reviewed Pi change does, and requires sandbox_executable_path and sandbox_executable_sha256, because a restricted sandbox cannot download OpenCode. Its own test expects model_only and the required binary, so removing either fails it. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Addresses Yu Yao's review of the Pi egress change: - restrict_sandbox_network takes the required tool grants and refuses them under model_only with AgentSeedRejectedError. The policy blocks their hosts, so a session would otherwise fail only after the sandbox and the runtime exist, as a retried 5xx. Pi passes its required MCP grants. - A host written with a trailing dot no longer yields a rule the egress sidecar never matches, and provider_options: null is restricted like an empty mapping instead of raising a 500. - Pi's tests cover both restricted values for a borrowed sandbox, a model URL that fails to resolve (the provider is closed), and a preinstalled runtime that exits nonzero or reports a version prefix. - The profile comment says that with sandbox_runtime_dir: null the image must hold the runtime at the default path with its ready marker. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Under model_and_tools, a grant at an unreachable address refuses the seed even when optional, and model_only refuses a required grant. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Addresses Yu Yao's review: - OpenCode passes its required MCP grants to restrict_sandbox_network, so under model_only a required grant is refused at seed with a 422 instead of failing after the sandbox exists. - The README and the network_access description say that a restricted sandbox needs sandbox_executable_path (or the runtime at the cache path) and ripgrep in the image, which grants a restricted seed refuses, and that full isolation needs OpenSandbox's dns+nft egress mode. - The egress test uses a required and an optional MCP grant on different hosts plus a direct_http grant that must stay out of the policy, the borrowed refusal covers both restricted values, and a route test covers the required-grant refusal. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…ion agents) Signed-off-by: weijia chen <weijiac@nvidia.com>
…dboxes) Conflicts resolved: opencode_agent/app.py keeps both imports (restrict_sandbox_network from #5555, verification_failure_fields from main); pi_agent/tests/test_sandbox.py keeps main's comment wording. Signed-off-by: weijia chen <weijiac@nvidia.com>
…arbor-style sandbox runs) Conflicts with main resolved: - opencode_agent/app.py: _sandbox_prepare_command takes both allow_root (#5458) and export_dir (main); the seed keeps main's sandbox_id check and passes both; Gym's remaining-context plugin loads before owner-staged plugins; request temperature is declared on the session's model_id. - opencode_agent and claude_code_agent READMEs keep both sections. Signed-off-by: weijia chen <weijiac@nvidia.com>
…_agent sessions The opencode, opencode_search, opencode_reference, pi and pi_search presets leave the legacy_agent relay and the deprecated sandboxed agents. They run as opencode_agent / pi_agent sessions through each agent's benchmark profile, driven by the benchmark's own single_agent_turn_legacy Environment Server. The judge binding and routing name are unchanged. Each session starts its own sandbox with network_access model_only or model_and_tools. Such a sandbox cannot download a runtime, so OpenCode runs the science image's preinstalled binary at the digest its Dockerfile checks, and Pi runs the Pi image's /opt runtime. Search presets list Tavily in the agent's tool_servers. Reasoning stays high: OpenCode sets reasoning_effort, Pi sets thinking high plus provider_compat.supportsReasoningEffort, so Pi requests now carry reasoning_effort. The environment note moves to system_prompt, because a session's opencode_config takes only permission and tools. opencode_full and opencode_search_full stay on opencode_sandboxed_agent behind the relay, since sessions take text input only. Their agent is renamed hle_diamond_opencode_full_agent so sandbox_docker_local.yaml can override both kinds of agent. README and tests are updated; the relay switch test drops the four hle_diamond rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
…Code limits The relay presets set execution_failure_reward_zero, so a harness crash, nonzero exit or deadline became a reward-0 row in the mean. The session agents default verify_after_agent_error to false, so such a run became a failure row outside the denominator. opencode, opencode_reference and pi (and the search presets that compose them) now set it to true, and the session preset tests check it. The README now notes that OpenCode stalls about 70 s per start offline while it tries to install @opencode-ai/plugin (neither science image ships the plugin dependency bundle), that session presets use opencode_agent's default permissions instead of '*': allow, and that the harness transcript under the artifacts directory exists only for the full presets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
…ecrets detector Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: weijia chen <weijiac@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The HLE-Diamond text presets with tools ran through the
legacy_agent/runrelay on the deprecatedopencode_sandboxed_agentandpi_sandboxed_agent. They now run asopencode_agentandpi_agentagent sessions, through each agent's benchmark profile (configs/profiles/benchmark.yaml), driven by the benchmark's own Environment Server,hle_diamond_equivalence_llm_judge_environment_server(single_agent_turn_legacy). The judge binding, routing name, dataset, prompts and grading are unchanged.opencode.yaml,opencode_search.yaml,opencode_reference.yaml(OpenCode) andpi.yaml,pi_search.yaml(Pi).network_access: model_onlyfor the code-only presets andmodel_and_toolsfor search (feat(pi): restrict the egress of agent-created sandboxes #5554, feat(opencode): restrict the egress of agent-created sandboxes #5555). Such a sandbox cannot download a runtime. OpenCode therefore runs the science image's/usr/local/bin/opencodeat the digest thatoffline_science_image/Dockerfilechecks (sandbox_executable_path,sandbox_executable_sha256); the reference image installs the same binary. Pi runs the Pi science image's/opt/nodeand/opt/piruntime (sandbox_runtime_dir: /opt,pi_version: 0.85.1).tool_servers. The Environment Server seeds a Tavily session for each episode and grants its MCP tools (feat(env): grant MCP tools of extra tool servers to session agents #5551). The judge serves no tools and grants none (fix(configs): grant no tools from judge servers that serve none #5590).reasoning_effort: high(feat(agents): Pi, OpenCode and Claude Code opt-ins for Harbor-style sandbox runs #5458). Pi keepsthinking: highand addsprovider_compat.supportsReasoningEffort: true(feat(agents): Pi, OpenCode and Claude Code opt-ins for Harbor-style sandbox runs #5458).verify_after_agent_error: true(feat(agents): Pi, OpenCode and Claude Code opt-ins for Harbor-style sandbox runs #5458) on both agents, so a run that fails after the harness started stays in the denominator (see Compatibility).system_prompt, because a session'sopencode_configaccepts onlypermissionandtools. OpenCode reads it as its instructions file; Pi appends it to its system prompt.sandbox_docker_local.yaml: blocks for the session agents, with the note assystem_prompt. The reference preset's acknowledgement gate is unchanged.Left alone:
opencode_full.yamlandopencode_search_full.yamlstay onopencode_sandboxed_agentbehind the relay, because agent sessions take text input only. Their agent is renamedhle_diamond_opencode_full_agentso thatsandbox_docker_local.yamlcan override both kinds of agent, and each preset now carries its own config instead of inheritingopencode.yaml.benchmarks/hle/is not changed.No issue is linked: the work is tracked internally.
Compatibility
execution_failure_reward_zero: true, so a harness crash, nonzero exit or deadline became a reward-0 row in the mean. The session agents defaultverify_after_agent_errorto false, which would make such a run a failure row outside the denominator and inflate accuracy. The presets set it to true. The failed run is then judged on what it produced (normally 0) and stays in the denominator. Unlike the relay, a transient model-endpoint failure, unconfirmed cleanup or a sandbox that no longer answers stays an execution error and the episode is retried.opencode_agent'sconfigs/permissions/default.yaml: everything is allowed except listed bash commands (process killing,rm -rfof system paths, raw disk writes, git network commands and others). The sandboxed presets allowed every command ('*': allow) and denied only web fetch, web search and subagents. Subagents stay off (tools.task: false).compaction.auto: false, because a session'sopencode_configaccepts onlypermissionandtools. The presets setdeclare_model_limits: false, so OpenCode compacts only after a context-overflow error, plusoutput_token_policy: remaining_contextand a 14,400 stimeout.reasoning_effort. The deprecated Pi presets never sent it, because the generated provider declaredsupportsReasoningEffort: false.slices/partition/*andjudgement_parsing_issue_rate, because the Environment Server forwards aggregation to the judge and the relay did not. Rows carry session fields (response.status,ng_agent_observations,failure_kind) instead ofopencode_*andpi_*. The harness transcript under the artifacts directory is written only by the full presets; the README says so.pi_agentruns one worker. Withnum_workersabove 1 its server does not start, because uvicorn finds no module-levelapp.opencode_agentkeeps 4 workers.hle_diamondrows are dropped, because those presets no longer use the relay.benchmarks/hle_diamond/opencode_hle_tools.yaml, which composesopencode.yamland overrideshle_diamond_opencode_agent.responses_api_agents.opencode_sandboxed_agent. After this PR that instance isopencode_agent, so Rshahbazyan/hle web tools #5176 needs rework to settool_servers,network_accessand the note (assystem_prompt) underopencode_agent. There is no git conflict, so nothing flags it automatically.Validation
Unit tests
At
6c05864be, Linux, the repository venv,pytest -q -n 16:resources_servers/equivalence_llm_judge/tests/test_hle_diamond.pytests/unit_tests/tooling/test_add_legacy_agent_environment_servers.pytests/unit_tests/environmenttests/unit_tests/test_public_api_facades.pytests/unit_tests/test_public_api.pytests/unit_tests/agentstests/unit_tests/toolingtests/unit_tests/configresponses_api_agents/opencode_agent/testsresponses_api_agents/pi_agent/testsresponses_api_agents/claude_code_agent/testsenvironment_servers/single_agent_turn/testsenvironment_servers/single_agent_turn_legacy/testsresources_servers/equivalence_llm_judge/testsThe one Pi failure,
test_app.py::test_mcp_setup_exit_is_request_failure_and_cleans_workspace[False], depends on test order. It fails only in the parallel run of the whole directory with the default options. It passes alone (three runs), in its own file, and in the parallel run without them. Neitherpi_agent/app.pynortest_app.pydiffers frommainon this branch.The session preset tests (
TestAgentSessions) check the following:single_agent_turn_legacy, with no relay and no leftover harness instance.OpenCodeAgentConfigandPiAgentConfig, withverify_after_agent_errortrue.package.json.Real runs
vllm_modelserver.python:3.11-slim-bookwormwithnetwork_access: inherit, because the science images were not available to that sandbox deployment. Egress was not restricted on these runs.6c05864be.reasoning_effort: highhle_diamond/opencodeslices/partition/{reasoning,knowledge},judgement_parsing_issue_ratehle_diamond/pislices/partition/{reasoning,knowledge},judgement_parsing_issue_rateBoth runs: no failures, no masked rows, coverage complete, and
verify_after_agent_error: truein the resolved config.Egress probe, on the earlier build of this branch, before the rebuild on current
main.opencode.yamlwith itsmodel_onlysetting ran on the same image without a preinstalled runtime. The seed was refused because the runtime installer could not reach the Debian mirror (apt-get: "Unable to locate package curl"), while theinheritrun had installed it. So the network policy is applied to agent-created sandboxes. The row failed asenvironment_server_failed, and no rollout ran.Checks
pre-commit runon the changed files: passed (ruff, ruff-format, whitespace, end of file, guardwords).git merge-treeagainst currentmain: no conflicts.Known limits
cais/hle-diamonddataset is gated and was not accessible for the smoke, so the rows were synthetic. No benchmark score is claimed.network_access: inherit. Only the seed refusal above shows the policy in effect.@opencode-ai/plugininto each session's fresh config directory at every start, unless the plugin dependency bundle is staged (sandbox_plugin_dependencies_path). Theopencode_agentREADME documents a stall of about 70 s offline before that install fails. Neither science image ships the bundle yet, so expect that delay on every row ofopencode,opencode_searchandopencode_reference. The README records this as a known limitation. It was not reproduced here: that needs an image with OpenCode preinstalled under restricted egress.opencode_search,pi_searchandopencode_referencewere not run, because no Tavily key was available./optruntime were checked against their Dockerfiles andpackage.jsonin tests only, not in a sandbox. The local Docker override was not run.opencode_fullandopencode_search_fulltoopencode_agentonce session image input lands (feat(opencode): attach prompt images in agent sessions #4922). feat(pi): attach prompt images in agent sessions #4923 would allow a Pi full preset.benchmarks/hle/.🤖 Generated with Claude Code