[yaoyu-33]

Stack: #5484 (resources sessions and sandboxed workspace tools) is merged into main, so this PR now targets main. It also carries the fixes from #5484's last review pass, which reached that server after #5484 merged (see "Review fixes" below).

BrowseComp's agent sessions previously ran the research loop on the Agent Server host. The session path now launches the same repository loop inside a supervised sandbox, and both shipped pairings use single_agent_turn_legacy.

Execution and lifecycle

• The adapter borrows a granted direct sandbox or creates an owned sandbox from configuration. Missing isolation, the Local provider, required MCP grants, multiple direct HTTP grants and unsupported task user/environment settings are rejected. Host /run and unseeded /v1/responses require explicit allow_host_execution: true.
• Enroot startup verifies the actual PID namespace and its /proc mount before staging rollout input. A shared host namespace or failed probe assertion refuses the seed with 422; a probe that times out or hits a provider error fails it as retryable. Upstream Enroot 4.2+ is required.
• Seed installs a private pinned Python runtime and uploads the exact Gym/harness sources. The pinned ANTLR dependency has no wheel, so only its hash-checked source archive uses the shared installer's host-build path. No benchmark command executes on the host.
• Model calls use the rollout-qualified Gym Model Server URL; tools use the supplied direct HTTP URL, headers and cookie jar. Upstream credentials are not serialized into the runner payload. Optional vLLM token counting stays behind a session-authenticated adapter endpoint.
• One activation is bound to the seeded rollout and request. Identical HTTP retries replay its result; conflicting requests return 409. SandboxSession supervises execution, collection and release. Failed startup/teardown retains state for cleanup retry, including failed file delivery followed by failed provider stop.
• The environment closes the agent before verification, then closes resources. BrowseComp workspace tools run in the separate sandbox introduced by feat(browsecomp_advanced_harness): sandbox workspace tools and serve sessions #5484. The example keeps inline search/browse results because its unchanged tool schema has no shell tool.

Compatibility and capability

The adapter declares Evaluation capability. Close always returns an observation bundle with native_turn_evidence_unavailable or observation_capture_failed; native invocation/turn records remain unimplemented. Model-call capture and Responses output remain available. Function-calling shell conformance is applicable but optional at this level and is not registered or qualified. No Observability, Training, SWE-bench or Terminal-Bench qualification is claimed.

Sampling, progress, compaction, resets and empty-answer retry behavior stay in the existing loop. Hosted endpoints without /tokenize must explicitly disable save_model_call_using_vllm_tokenize_endpoint; sandbox sessions reject host snap_dir.

The default sandbox activation cap is 3600 seconds and also bounds the loop when rollout_timeout_s is unset. Session outcome policy can exclude/retry errors that the former host /run counted as zero-reward answers. Compare coverage and failure rows alongside score. Timed-out responses still score zero without invoking the judge. Tasks, prompts and answer grading were not changed.

No separate issue is needed for this focused migration of the existing BrowseComp integration.

Validation

• responses_api_agents/browsecomp_agent/.venv/bin/pytest responses_api_agents/browsecomp_agent/tests -q: 116 passed, including a real subprocess/HTTP boundary test and owned-start failure/cleanup retry.
• resources_servers/browsecomp_advanced_harness/.venv/bin/pytest resources_servers/browsecomp_advanced_harness/tests -q: 226 passed.
• Core session-contract, environment/configuration and public-import checks: 305 passed; one catalog test failed on the workstation's unresolvable hostname. That exact catalog test passed on Linux.
• Linux pocketshell suite: 78 passed.
• pre-commit run --all-files: passed; focused checks also passed after the bootstrap fix.
• npm run check in fern/: 0 errors, 1 warning.
• Both final published heads passed automatic Copyright, Component locks, Secrets detector and Code linting CI. Full maintainer-triggered core/component CI remains pending.
• Independent review rechecked the startup cleanup fix and bootstrap dependency change; no remaining confirmed source defect in its scoped review.

After merging the updated #5484 and two review passes, at 1adaa74def1d (macOS, no Docker, so no sandbox smoke for these changes):

• pytest responses_api_agents/browsecomp_agent/tests -q: 126 passed. The tests added in these passes fail on the previous code:
  • An unreachable model URL ends the runner with a ConnectionError receipt within seconds (before: it spun until rollout_timeout_s).
  • A tool-free session answers an ungranted tool call.
  • Retries share one deadline, and that deadline counts from the runner's start, so the endpoint checks spend the rollout's budget, not the supervisor's 30-second grace.
  • The seed is refused with 422 for any of these, since no retry can fix them: a loopback model, tool or token-count URL (under Docker the shared check only warns, because a host-network container can reach loopback; a bridge-network container then fails the activation's reachability check, which is retryable); a missing Enroot PID namespace; an unknown provider; an unknown sandbox_config key.
  • An isolation probe that times out fails the seed as retryable instead of 422.
  • A new test also covers the host activation path: the task's deadline clamp, the runner payload and an error receipt.
• pytest resources_servers/browsecomp_advanced_harness/tests -q: 234 passed.
• pytest tests/unit_tests/tooling/test_component_dependencies.py tests/unit_tests/test_public_api_facades.py tests/unit_tests/config/test_global_config.py: 1041 passed. scripts/component_locks.py: 269 projects, 0 failed (main's opencode_search lock fix is merged in).
• pre-commit run --files on the changed files: passed.

Review fixes on 74ea2592bc00 (third review pass)

Merged main (now with #5484 as a squash commit, plus #5528's ChatCompletionsConverter rename) and fixed what a third review found.

browsecomp_advanced_harness (code from #5484):

• Only the supervisor's deadline answers a shell command with [exit_code=-1]. A provider timeout (docker cp, the provider's exec, reading the output) is a sandbox failure: [exit_code=-2] and a masked sample (provider_unavailable). Before, any TimeoutError read as the model's own timeout and was scored.
• /verify of an Environment Server session no longer releases it; close_session does. A re-sent verify (the client resends after a dropped connection) or a retry after a judge failure therefore still finds the shell failure and the metrics; before, the retry came back unmasked with empty metrics. A judge failure on a masked sample returns the masked sample instead of raising. An agent's /run session is still released by its verify.
• A typed seed whose provider cannot be created, or whose Enroot sandbox shows the host's PID namespace or a mismatched /proc mount, answers 422 once the sandbox is stopped (was a retryable 500 for every episode). A probe that does not finish stays retryable. The agent applies the same rule.
• After a failed start, later commands answer [exit_code=-2] without starting another sandbox.
• The shell runner writes UTF-8 with errors="replace", so a lone surrogate in the output no longer loses the whole result and its exit-code envelope.
• The sandbox timeouts must be finite; README says shutdown cleanup is best-effort (gym env stops the server one second after its interrupt unless server_shutdown_timeout_seconds is raised, and Enroot ignores ttl_s).

browsecomp_agent and configs (this PR):

• use_absolute_ip is scoped with absolute_ip_server_types: [responses_api_models, resources_servers, responses_api_agents] in both shipped configs, so the head server, whose unauthenticated config endpoint returns the merged config including API keys, stays on loopback.
• The sandboxed loop's model, tool and token-count calls resend a refused connection for about 90 seconds, then fail the activation with an error receipt (a retryable failure row). Before, a mid-episode outage resent until the rollout deadline and scored as a timed-out zero.
• update_progress arguments that are valid JSON but not an object get a tool error back, like invalid JSON; before, they crashed the activation into a retried, unscored row.
• The isolation probe now follows the resources server's rule: only a finished probe that shows a shared or mismatched namespace is final (422); a non-zero exit or an unparseable answer is retryable.
• Upstream error text and the runner log in error messages are capped at 4000 characters.
• Removed the unreachable snap_dir branch of the session loop and config_single_agent_turn.yaml, an alias that main never had (config.yaml already runs this path).
• The agent's timeout fields must be finite. Docstrings for responses and run.

Validation at 74ea2592bc00 (macOS, no Docker):

• pytest responses_api_agents/browsecomp_agent/tests: 147 passed.
• pytest resources_servers/browsecomp_advanced_harness/tests: 246 passed. The new tests fail on the previous code (provider timeouts, re-sent verify, 422 refusals, commands after a failed start, the lone-surrogate runner, endpoint loss mid-episode, non-object update_progress arguments).
• tests/unit_tests/config/test_global_config.py, test_public_api_facades.py, test_session_contract_repo.py, benchmarks/browsecomp/tests: 1052 passed.
• tests/unit_tests/environment/test_benchmarks.py::test_every_repo_benchmark_appears_in_listing fails on this workstation only because its hostname does not resolve for use_absolute_ip; it fails the same way on main.
• pre-commit run --files on the changed files: passed.

Not run: no sandbox smoke (Docker or Enroot) at this head; the last real pairing runs below predate the second and third review passes. The earlier CI failure responses_api_models/vllm_model/tests/test_framework_context.py::test_cancelled_chat_keepalive_retains_uncertain_capture[False] is a timing flake that reproduces on main (about 1 in 10 runs locally); these PRs do not touch that server.

Real pairing evidence

Functional cluster run smoke-04 (job 6446261), folders smoke-04/{browsecomp,reasoning}, at be3925b94589eca0685dcd746fda50492476d2dd: five unchanged BrowseComp tasks completed scoring (0/5 correct), plus two unchanged reasoning_gym examples completed through configuration alone (2/2 correct). Policy: nvidia/nemotron-3.5-bolt-e4b; independent judge: z-ai/glm-5.3-flash; real Tavily search/browse. Policy settings: 24 steps, 360-second loop, 480-second runner, 2048 output tokens per call, temperature 0, concurrency 1. The unchanged verifier sets the judge to 10240 output tokens and temperature 1.0; captured requests, not the overridden YAML values, are the effective settings.

The run exercised 22 searches, 3 browse calls, 20 shell calls and 26 progress updates. All seven usage totals matched captured policy calls and all agents closed before verification. No timeout, infrastructure failure or masked sample occurred. Health reported 0 unhealthy and 7 unobserved rows because native turn/invocation and model-call ownership records are absent. Strict evidence exited 1: TE-3 (steps), TE-6 (absent evaluation_completed and sandbox observation records), TE-8/9 (joins), and additionally TE-5 (tools) for BrowseComp. These are explicit Evaluation limits; raw captured calls and separate runtime audits supply the smoke evidence. Supervisor and owned containers cleaned up; task credential copies were removed and the artifact scan found no exact credential values.

Isolation correction: system Enroot 3.4.1 ignores the PID-isolation setting, so smoke-04 is functional evidence only. Actual final-source diagnostics on the cluster confirmed a shared host PID namespace and successful signal-0 visibility of a nonsecret host sentinel (environment reading denied). Both BrowseComp guards correctly rejected it. Complete task-local Enroot 4.2.1 (ead3a25ed974948235d28f99fc387ac82435ce62) provided distinct PID/user/mount namespaces, matching container self/PID1, and an absent/unreadable sentinel with signal 0 returning ProcessLookupError; both guards accepted it.

Final-source run smoke-05 (job 6446620), at aebd6b9184909d1f84390205a59e07dd5d4d471b, completed the first unchanged row from each pairing on Enroot 4.2.1: BrowseComp 0/1 correct and reasoning_gym 1/1 correct, with no infrastructure failure. The smaller post-fix scope fits a 15-minute allocation; it retains the earlier 5+2 results without combining repeated tasks into an accuracy estimate. Natural BrowseComp tools: 4 searches, 1 browse, 5 shell calls and 6 progress updates. Both rows passed stock health checking with 0 unhealthy and 2 unobserved rows; strict evidence exited 1 for the same declared gaps. Captured usage matched all 18 policy calls (17 BrowseComp, 1 reasoning); the separate judge made 1 call. Agent close preceded verification, supervisor exited 0, and owned container inventories were empty. Final exact-value scan found 0 credential occurrences across 6,428 task text artifacts; temporary cluster keys were deleted. Live agent/PID1 audits and source hashes matched; this targeted run did not catch the short-lived shell runner itself. All 5 shell outputs were retained (4 exit 0, one grep exit 1, whose exact semantic cause is unclassified).

The first isolation diagnostic stopped before model calls because its diagnostic script did not handle a denied /proc/1 read; the corrected retry retained that failure and kept the production source and strict acceptance conditions unchanged. This PR is ready for maintainer CI/review; it has not been merged.

Earlier failed configuration and fresh-runtime ANTLR attempts are retained in validation records. Smoke used Tavily with an explicit empty additional-domain filter ({"notices": []}); Exa was not exercised. No accuracy baseline or score-parity claim is made.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yaoyu-33]

/ok to test 65c7c1c

[yaoyu-33]

/ok to test 615a506

[yaoyu-33]

/ok to test 5dd4dd1

[yaoyu-33]

/ok to test 1adaa74

[yaoyu-33]

/ok to test 74ea259

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42077542830