feat(browsecomp_agent): run the session harness inside a sandbox

benchmarks/browsecomp/README.md

@@ -1,5 +1,5 @@
 # Run instructions
-1. Update your env.yaml file with the necessary keys and configs. You can leave `exclude_domains_file_path` as a path to an empty json.
+1. Update your env.yaml file with the necessary keys and configs. Set `exclude_domains_file_path` to a JSON file containing `{"notices": []}` when no additional domain exclusions are needed; an empty list or `{}` does not match the parser schema.
 ```yaml
 browsecomp_benchmark_resources_server:
   resources_servers:
@@ -35,3 +35,24 @@ ng_e2e_collect_rollouts \
     ++policy_api_key=??? \
     ++policy_model_name=???
 ```
+
+## Run through the Environment Server
+
+`config.yaml` now runs the benchmark through `single_agent_turn_legacy` with sandboxed agent and resources tool execution. It requires a Docker daemon by default. Replace the included Docker provider config and both sandbox images when selecting another provider; do not merge two provider blocks under `sandbox`. The sandboxes reach the model, resources and agent servers at the host's address (`use_absolute_ip: true`); `absolute_ip_server_types` keeps the head server, whose config endpoint returns the merged config, on loopback. BrowseComp sessions are not exported to checkpoints, so after a crash an episode starts over from its input.
+
+Tasks, prompts and correct-answer grading are unchanged. The sandbox path adds a 3600-second activation cap by default (`sandbox_runner_timeout_seconds`); when `rollout_timeout_s` is unset, that cap also bounds the loop. Account for this limit when comparing with previous step-only runs. Compared with the previous host `/run` recipe, failure handling changes the scored denominator:
+
+| Case | Previous host `/run` recipe | Current sandbox session recipe |
+| --- | --- | --- |
+| Rollout reaches `rollout_timeout_s` | Reward 0 without a judge call, `timed_out: 1` | The same |
+| Model call returns 400, 413 or 422 | Reward 0, with `agent_error` | Verified with an empty answer, which scores 0 |
+| Model call returns 5xx after the client's retries, or the sandbox cannot connect to the model or tools, at the activation's start or for about 90 seconds later | Set aside unscored (`kill_shaped`) for resume | A non-terminal failure row (`_ng_failure_*`), unscored, which `gym eval run --resume` retries |
+| Model call returns 408, 425 or 429 after the client's retries | Reward 0, with `agent_error` | A non-terminal failure row, which a resume retries |
+| Model call returns any other 4xx | Reward 0, with `agent_error` | A terminal failure row, not scored and not retried |
+| Any other error in the agent turn | Reward 0, with `agent_error` | A non-terminal failure row, which a resume retries |
+
+The sandbox checks that it can connect to the model, tool and token-count endpoints before the loop starts. A connection lost later is resent for about 90 seconds; if the endpoint does not come back, the activation fails with a non-terminal failure row instead of scoring as a timeout.
+
+So an error that the previous recipe scored 0 now leaves the scored set instead of counting as a wrong answer. Check the failure rows and the coverage file alongside the accuracy.
+
+The [BrowseComp adapter README](../../responses_api_agents/browsecomp_agent/README.md) documents Evaluation capability, explicit evidence gaps, sandbox ownership and unsupported controls. For a hosted model without `/tokenize`, set `save_model_call_using_vllm_tokenize_endpoint: false` explicitly. The research agent and read-only shell each run in an independently owned sandbox; the Resources Server keeps search credentials and verification on the host.

benchmarks/browsecomp/config.yaml

@@ -1,6 +1,10 @@
 config_paths:
 - nemo_gym/sandbox/providers/docker/configs/docker.yaml
 
+use_absolute_ip: true
+# Only the servers the sandboxes call get the host address; the head server's config endpoint stays on loopback.
+absolute_ip_server_types: [responses_api_models, resources_servers, responses_api_agents]
+
 # Judge: a hosted reasoning model, independent of the policy under test. Keeping
 # the judge off the policy endpoint stops a weak policy from grading itself.
 judge_model:
@@ -67,9 +71,12 @@ browsecomp_benchmark_agent:
       # Estimate prompt tokens via the vLLM /tokenize endpoint before each model
       # call to decide context reset (skips the otherwise-wasted generation).
       save_model_call_using_vllm_tokenize_endpoint: true
-      resources_server:
-        type: resources_servers
-        name: browsecomp_benchmark_resources_server
+      resources_server: null
+      sandbox_provider: sandbox
+      sandbox_config:
+        image: python:3.13-slim
+        workdir: /sandbox
+        ttl_s: 7200
       model_server:
         type: responses_api_models
         name: policy_model
@@ -83,8 +90,14 @@ browsecomp_benchmark_agent:
 
 browsecomp_benchmark_environment_server:
   environment_servers:
-    legacy_agent:
+    single_agent_turn_legacy:
       entrypoint: app.py
+      resources_server:
+        type: resources_servers
+        name: browsecomp_benchmark_resources_server
       agent_server:
         type: responses_api_agents
         name: browsecomp_benchmark_agent
+      resources_tool_transports: [direct_http]
+      default_episode_timeout_seconds: 6000
+      cleanup_timeout_seconds: 300

fern/versions/latest/pages/agent-server/add-or-update-a-harness.mdx

@@ -262,6 +262,7 @@ The current adapters support the following tool sources:
 
 | Adapter | Built-in tools | Resources Server tool grants | Model API |
 | --- | --- | --- | --- |
+| BrowseComp | `update_progress` when configured. | At most one `direct_http` grant; required MCP grants are rejected. The research tools remain on the Resources Server, which isolates `bash_command` in its own sandbox. | Responses |
 | LangGraph | Reflection, orchestration, parallel thinking and ReWOO run in the Agent Server and call only the Model Server. | No environment tools or sandbox access; required tool grants and sandbox grants are rejected. Keep `resources_tool_transports: []`. Evaluation sessions report missing native invocation and turn evidence. | Responses |
 | Hermes | Hermes toolsets, selected with `enabled_toolsets` and `disabled_toolsets`, for example `[terminal]`. | `mcp` only. Each granted MCP server becomes a Hermes toolset with the same name. Required `direct_http` grants are rejected. | Chat Completions |
 | Pi | Pi's built-in sandbox tools. `sandbox_bash_timeout_seconds` limits each bash call. | None in agent sessions. Required grants are rejected; keep `resources_tool_transports: []`. | Chat Completions |

fern/versions/latest/pages/agent-server/integrate-existing-agents.mdx

@@ -14,10 +14,11 @@ To use an agent that Gym does not integrate yet, see [Add an agent that is not l
 
 ## Sandboxed harnesses
 
-These adapters run a coding harness inside a task sandbox that a Resources server provides, and are driven by the `single_agent_turn` Environment Server. Each row summarizes the adapter's README; the README is the reference for supported features, runtime requirements, and limits.
+These adapters run a harness inside a sandbox and are driven by the `single_agent_turn` Environment Server. They borrow a task sandbox when the Resources server supplies one; otherwise a configured provider creates an agent-owned sandbox. Each row summarizes the adapter's README; the README is the reference for supported features, runtime requirements, and limits.
 
 | Harness | Model API | Resources tool grants | Token IDs and logprobs | SWE-bench Pro | Terminal Bench 2.1 |
 | --- | --- | --- | --- | --- | --- |
+| BrowseComp (`browsecomp_agent`) | Responses | Direct HTTP | Preserved from Model Server responses; sandbox path qualified for evaluation only | Not declared | Not declared |
 | Hermes (`hermes_agent`) | Chat Completions | MCP only | Returned in a result file that the model can also write; evaluation only | Declared | Not declared |
 | Pi (`pi_agent`) | Chat Completions | None | Not wired up; evaluation only | Declared | Declared |
 | Codex (`codex_agent`) | Responses (streaming) | MCP only | Not wired up; evaluation only | Declared | Declared |

nemo_gym/config/global_config.py

@@ -219,6 +219,7 @@ def _installed_version(distribution: str) -> Optional[str]:
 # so this list is the source of truth; a unit test keeps it in step with the agent apps.
 AGENT_SESSION_AGENT_TYPES = frozenset(
     {
+        "browsecomp_agent",
         "claude_code_agent",
         "codex_agent",
         "harbor_oracle_agent",

resources_servers/browsecomp_advanced_harness/README.md

@@ -51,7 +51,9 @@ Enroot must provide a PID namespace separate from the resources-server host, wit
 namespace. Setup checks the running sandbox's namespace IDs before permitting model commands; a version string
 or `ENROOT_UNSHARE_PID` setting alone is insufficient because older Enroot builds may ignore that setting. If the
 check fails, upgrade or configure Enroot with working PID isolation, or select another isolating provider. The
-failed session follows the usual sandbox cleanup and close-retry path. Only Enroot is probed. Choose any other
+failed session follows the usual sandbox cleanup and close-retry path. A probe that shows the host's namespace or a
+mismatched `/proc` mount refuses the seed with 422 once the sandbox is stopped, since no retry can pass it; a probe
+that does not finish fails the seed with a retryable error. Only Enroot is probed. Choose any other
 provider for sandboxes that cannot see the host filesystem: Apptainer, for example, bind-mounts the host's home
 directory and `/tmp` unless configured otherwise.
 
@@ -66,11 +68,14 @@ version pinned into its isolated Python runtime. The first seed builds that runt
 reuse its host-side artifact cache. Setup failures surface to the caller and retain ownership when cleanup fails,
 so a later close can retry. Commands are supervised inside the sandbox; their guard, supported commands, output
 truncation, and exit-code envelope are unchanged. A command that outlives the supervisor deadline answers
-`[exit_code=-1]` and a sandbox failure answers `[exit_code=-2]`, as on the host. There is no fallback to host
-execution after a sandbox failure. A sandbox that cannot start or cannot run a command marks its session, and
-`/verify` keeps the judged reward but masks the sample (`mask_sample: true`, `failure_kind:
-provider_unavailable`), since the model lost its tool to the infrastructure. A blocked command answers without
-starting a sandbox. A missing provider block or an unknown `sandbox_config` key refuses the seed with 422.
+`[exit_code=-1]` and a sandbox failure answers `[exit_code=-2]`, as on the host. Only the supervisor's deadline
+reads as the command's timeout; a provider timeout (an upload, the exec itself, or reading the output) is a sandbox
+failure. There is no fallback to host execution after a sandbox failure. A sandbox that cannot start or cannot run a
+command marks its session, and `/verify` keeps the judged reward but masks the sample (`mask_sample: true`,
+`failure_kind: provider_unavailable`), since the model lost its tool to the infrastructure. A masked sample is
+returned even when the judge cannot be reached. After a failed start, later commands answer `[exit_code=-2]` without
+starting another sandbox. A blocked command answers without starting a sandbox. A missing provider block, an
+unknown provider, or an unknown `sandbox_config` key refuses the seed with 422.
 
 `allow_host_execution: true` explicitly enables the former host interpreter for development when no provider is
 configured, or permits the non-isolating `local` provider. It defaults to `false`; production and benchmark configs
@@ -83,21 +88,40 @@ Each rollout has its own session: the provider-call metrics that `/verify` repor
 - **Environment Server.** `/seed_session` takes a `ResourcesSeedSessionRequest` and keys the session by its `resources_session_id`. The seed is refused with 422 when `task_data` lacks the `question` and `ground_truth` that `/verify` needs, or, with a workspace, when the ID is not a plain directory name. `/close_session` removes the workspace and the metrics, and is idempotent. A close that cannot remove the workspace fails and keeps the session, so the caller can retry it. After a close, a seed for the same ID is refused with 409, and a tool call for it gets 404 (`session_lost`) instead of recreating the workspace. `/verify` of a closed session returns a masked sample (`reward: 0`, `mask_sample: true`, `failure_kind: session_lost`). The reward depends only on the response and the task, so the mask is kept on purpose: a closed session reads the same on every server under the resources session contract. `single_agent_turn` verifies before it closes, so only an out-of-order verify gets the mask.
 - **An agent's `/run`.** `/seed_session` takes the flat row and keys the session by the cookie. `/verify` releases the sandbox and removes the workspace, since this path sends no close. The sandbox starts with the first `bash_command`, not at the seed. A run whose agent never sends `/verify`, as `browsecomp_agent` does after a timeout or an aborted run, keeps its sandbox until the provider's `ttl_s` reclaims it or the server shuts down; Enroot ignores `ttl_s`. Sandboxed runs should use the Environment Server path, which always closes.
 
-`/verify` and `/close_session` stop the owned sandbox before deleting its workspace and metrics. A failed stop leaves
-the session available for a close retry. Shutdown also stops any remaining tool sandboxes. Tool calls, verification,
-and close are serialized per session, so cleanup cannot delete files while a tool is running. `/verify` drops the
-session's metrics after reporting them, unless `dump_session_id_to_metrics_on_exit` keeps them for the dump at shutdown.
+`/close_session`, and `/verify` on the `/run` path, stop the owned sandbox before deleting its workspace and metrics.
+A failed stop leaves the session available for a retry. `/verify` of an Environment Server session releases nothing,
+so a re-sent verify still finds the session's metrics and its sandbox failure; its close releases them. Tool calls,
+verification, and close are serialized per session, so cleanup cannot delete files while a tool is running. Releasing
+a session drops its metrics, unless `dump_session_id_to_metrics_on_exit` keeps them for the dump at shutdown.
+
+Shutdown tries to stop any remaining tool sandboxes, but only on a best-effort basis. `gym env` kills the server one
+second after its interrupt unless `server_shutdown_timeout_seconds` is raised to cover the closes, and Enroot ignores
+`ttl_s`, so an Enroot sandbox left behind stays until `enroot remove` deletes it.
 
 ## Running
 
+The example pairing uses `single_agent_turn_legacy`: existing flat dataset rows still work, while the environment
+server owns resources/agent session setup, verification, and close. The agent loop runs in its own sandbox and
+receives the resources session through the `direct_http` transport; its `resources_server` config is `null`.
+The config includes the Docker provider and `use_absolute_ip: true` so the sandbox can reach the Gym servers;
+`absolute_ip_server_types` limits that to the model, resources and agent servers, so the head server, whose config
+endpoint returns the merged config, stays on loopback.
+Replace the included Docker provider config to run on a different backend, and set both sandbox image fields for that provider. Do not merge two provider blocks under `sandbox`.
+
+The example dataset exposes only `search` and `browse`, so the example config keeps `workspace: none` and inline
+page content. To enable the resources-owned tool sandbox, use `workspace: per_session` with dataset rows exposing
+`bash_command`, as the BrowseComp benchmark does. This preserves the example prompts and tool schemas.
+
 ```bash
-# If you want to run with browsecomp benchmark instead of the example samples, need to change the datasets part to the one like `benchmarks/browsecomp/config.yaml` in `resources_servers/browsecomp_advanced_harness/configs/browsecomp_advanced_harness.yaml`
 config_paths="resources_servers/browsecomp_advanced_harness/configs/browsecomp_advanced_harness.yaml,\
 responses_api_models/vllm_model/configs/vllm_model.yaml"
 
 ng_run "+config_paths=[${config_paths}]"
 ```
 
+To run the benchmark instead, use `benchmarks/browsecomp/config.yaml` as the first config path. Its dataset,
+workspace, and tool schemas are configured together.
+
 
 # Licensing information
 Code: ?