fix(agent): build Hermes and Stirrup runtimes on the host

fern/docs.yml

@@ -385,10 +385,14 @@ redirects:
     destination: "/nemo/gym/nemo-gym/nemo_gym/agents/utils/task_timeout"
   - source: "/nemo/gym/:version/nemo-gym/nemo_gym/agent_utils/task_timeout"
     destination: "/nemo/gym/:version/nemo-gym/nemo_gym/agents/utils/task_timeout"
+  - source: "/nemo/gym/nemo-gym/nemo_gym/agents/utils/uv_bootstrap"
+    destination: "/nemo/gym/nemo-gym/nemo_gym/agents/utils/sandbox_python"
+  - source: "/nemo/gym/:version/nemo-gym/nemo_gym/agents/utils/uv_bootstrap"
+    destination: "/nemo/gym/:version/nemo-gym/nemo_gym/agents/utils/sandbox_python"
   - source: "/nemo/gym/nemo-gym/nemo_gym/agent_utils/uv_bootstrap"
-    destination: "/nemo/gym/nemo-gym/nemo_gym/agents/utils/uv_bootstrap"
+    destination: "/nemo/gym/nemo-gym/nemo_gym/agents/utils/sandbox_python"
   - source: "/nemo/gym/:version/nemo-gym/nemo_gym/agent_utils/uv_bootstrap"
-    destination: "/nemo/gym/:version/nemo-gym/nemo_gym/agents/utils/uv_bootstrap"
+    destination: "/nemo/gym/:version/nemo-gym/nemo_gym/agents/utils/sandbox_python"
   - source: "/nemo/gym/nemo-gym/nemo_gym/agent_outcome"
     destination: "/nemo/gym/nemo-gym/nemo_gym/agents/agent_outcome"
   - source: "/nemo/gym/:version/nemo-gym/nemo_gym/agent_outcome"

fern/versions/latest/pages/agent-server/add-or-update-a-harness.mdx

@@ -350,7 +350,7 @@ The [`nemo_gym/agents/utils`](https://github.com/NVIDIA-NeMo/Gym/tree/main/nemo_
 
 ### Code layout
 
-Organize a new adapter under `responses_api_agents/<harness>/`. The adapter's `sandbox.py` runs on the Agent Server. Inside the sandbox, the shared `harness_runner.sh` starts the harness CLI. A harness that is itself a Python program runs an adapter-specific `sandbox_runner.py` shim with a Python runtime that does not come from the task image: a pinned standalone Python that the Agent Server verifies and uploads, as NOOA does, or a runtime prepared outside the task container and mounted read-only, as `hermes_sandboxed_agent` does. Three adapters still depend on Python or the network inside the sandbox, and these paths are being removed rather than copied: Codex runs its shim on the task image's Python 3.8 or newer, and Hermes and Stirrup build a `uv` virtual environment in the sandbox from the network.
+Organize a new adapter under `responses_api_agents/<harness>/`. The adapter's `sandbox.py` runs on the Agent Server. Inside the sandbox, the shared `harness_runner.sh` starts the harness CLI. A harness that is itself a Python program runs an adapter-specific `sandbox_runner.py` shim with a Python runtime that the Agent Server builds and uploads with `SandboxPythonRuntime`, as Hermes and Stirrup do. `hermes_sandboxed_agent` is being deprecated; use `hermes_agent` for new development and as the integration reference. One adapter still depends on Python inside the sandbox, and that path is being removed rather than copied: Codex runs its shim on the task image's Python 3.8 or newer.
 
 | Adapter file | Responsibility |
 | --- | --- |
@@ -361,15 +361,15 @@ Organize a new adapter under `responses_api_agents/<harness>/`. The adapter's `s
 | `requirements.txt` | The adapter's isolated environment: `-e nemo-gym[dev,sandbox] @ ../../` plus packages that run on the Agent Server, such as a framework or SDK. Without the `sandbox` extra, an SDK-backed provider such as OpenSandbox cannot import its SDK, and the first sandbox session fails. |
 | `tests/` | Host dispatch, session setup, request semantics, output parsing, installation, and lifecycle regressions. |
 
-The shared modules have separate responsibilities. All of them run on the Agent Server, except `process_supervisor.sh` and `harness_runner.sh`, which are uploaded into the sandbox. Provider APIs and general file transfer stay in `nemo_gym/sandbox/`.
+The shared modules have separate responsibilities. All of them run on the Agent Server, except `process_supervisor.sh`, `harness_runner.sh`, and `install_python_runtime.sh`, which are uploaded into the sandbox. Provider APIs and general file transfer stay in `nemo_gym/sandbox/`.
 
 | Shared module | Responsibility |
 | --- | --- |
 | `agents/utils/sandbox_session.py` | `SandboxSession` owns the sandbox connection, the ownership flag, `workdir`, `session_dir`, the execution and finalization tasks, the termination receipt, artifacts, and retryable close state. It uploads the supervisor before launch. `SandboxCommand` carries the harness argv. `shell_assignments()` renders runner input as bash source. |
 | `agents/utils/supervisor_client.py` | Builds the launch and stop shell commands, defines control-file names and timeout budgets, validates termination receipts, and retires session directories safely. |
 | `agents/utils/process_supervisor.sh` | Runs inside the Linux sandbox and needs only bash, a readable `/proc`, a `sleep` that accepts fractions of a second, and `mv`. It leaves the provider's process group, enforces the harness deadline, stops the harness process group and, after a forced stop, the processes described under [Harness termination](#harness-termination), confirms that none still runs, and writes the termination receipt. The sandbox needs neither Gym nor Python. |
 | `agents/utils/harness_runner.sh` | Runs inside the sandbox with bash (built with programmable completion, for `compgen`), `dirname`, `sleep`, and, on bash older than 4.2, `date`. It sources the input that `shell_assignments()` wrote, applies the harness environment, runs the CLI with file-backed stdio, and optionally records each JSON line of stdout with the time it was seen. It exits with the CLI's status, with 125 when it fails before starting the CLI (including an `exit` in its setup code), or with 126 or 127 when the CLI could not be started. |
-| `agents/utils/uv_bootstrap.py` | `uv_for_sandbox()` chooses the `uv` binary to upload for sandboxed Python runtime installation. It reuses a compatible host binary or downloads the static musl build for the detected platform. If the platform probe fails, it warns and falls back to the host binary without confirming compatibility. Hermes and Stirrup use it to build their runtime inside the sandbox, a path that is being removed; do not use it in a new adapter. |
+| `agents/utils/sandbox_python.py` | `SandboxPythonRuntime` gives a Python harness a private runtime that needs no network, `uv`, or Python in the sandbox. On the Agent Server, it downloads a pinned python-build-standalone CPython for the sandbox's architecture and C library, checks its SHA-256, and installs the harness's requirements from wheels for that platform with `uv`, pinned to the versions in the agent environment. Only requirements given as direct URLs, such as a harness pinned to a commit, are built from source. Both archives are cached on the host under `sandbox_python` in Gym's `cache_dir`. In the sandbox, `install_python_runtime.sh` unpacks them under `/tmp`, checks the harness import, and publishes the runtime with an atomic rename, so concurrent seeds never see a partial runtime. Run an absolute harness script with the runtime's `python -E -s`, because it is not a virtual environment: `-E` ignores the task's `PYTHON*` settings and `-s` its user site-packages. `is_installed()` reuses a runtime built from the same inputs. Hermes and Stirrup use it. |
 | `agents/utils/task_timeout.py` | `resolve_agent_timeout(metadata, ceiling=...)` returns the harness time budget: a task's positive `metadata.agent_timeout_sec`, which Harbor task rows carry, capped by the configured timeout. A missing or invalid value keeps the configured timeout. Hermes, Pi, Codex, OpenCode, and OpenClaw use it. |
 | `sandbox/utils.py` | `upload_text()` and `read_text()` transfer UTF-8 text through `AsyncSandbox.upload()` and `download()`. Use those underlying APIs directly for binary files. |
 
@@ -439,7 +439,7 @@ Keep harness-specific response parsing and model and tool behavior in the adapte
 **1. Define the session state.** Subclass `AgentSessionState` and compose a `SandboxSession`. Implement two hooks:
 
 - `stage_activation` uploads the activation input and returns `SandboxCommand(argv=...)`. Install the runtime earlier, during seed. A CLI harness runs through the shared bash runner, whose name never mentions python. `stdlib_runner=True` exists only for Codex, whose standard-library runner still runs on the task image's Python: the runner then runs as `bin/gym-runner`, so a `pkill -f python3` in the task cannot stop it. Do not use it in a new adapter; it goes away once Codex no longer needs the task image's Python.
-- `collect` copies artifacts to the Agent Server after harness termination is confirmed, including partial output from interrupted runs. It may run a bounded snapshot command, but it must not restart the harness. Bound the size of what it downloads: trim large logs inside the sandbox first and keep the tail with a truncation marker. For example, the older `hermes_sandboxed_agent`, which calls the process supervisor directly, keeps the last 1 MiB of its runner log; apply the same bound in `collect`.
+- `collect` copies artifacts to the Agent Server after harness termination is confirmed, including partial output from interrupted runs. It may run a bounded snapshot command, but it must not restart the harness. Bound the size of what it downloads: trim large logs inside the sandbox first and keep the tail with a truncation marker, for example the last 1 MiB of a runner log.
 
 ```python
 @dataclass
@@ -546,7 +546,7 @@ All control files live under `session_dir`, and their names are defined in `supe
 | `cleanup.json` | The supervisor records here whether the harness process group terminated. If a stop wins the launch claim, the stop shell writes the same receipt shape with `return_code: null` and `timed_out: false`. |
 | `output.log` | The launch shell redirects supervisor and harness diagnostics here. Task output is a separate, adapter-owned artifact. |
 
-To migrate an adapter that launches the process supervisor itself, such as `hermes_sandboxed_agent`, move launch, termination, collection, and release into `SandboxSession`. Receipt types and timeout helpers that used to live in `process_supervisor.py` are now in `nemo_gym.agents.utils.supervisor_client`. Update the imports, hook arguments, control-file assumptions, test doubles, and tests together. The adapter imports these modules at startup, so a stale import also breaks host calls.
+To migrate an adapter that launches the process supervisor itself, move launch, termination, collection, and release into `SandboxSession`. For Hermes, use the existing `hermes_agent` integration; `hermes_sandboxed_agent` is being deprecated and should not receive new integration work. Receipt types and timeout helpers that used to live in `process_supervisor.py` are now in `nemo_gym.agents.utils.supervisor_client`. Update the imports, hook arguments, control-file assumptions, test doubles, and tests together. The adapter imports these modules at startup, so a stale import also breaks host calls.
 
 Out-of-tree adapters written for the Python supervisor also need these changes:
 
@@ -568,7 +568,7 @@ Implement the [agent-session API](https://github.com/NVIDIA-NeMo/Gym/blob/main/n
 
 Run the installation check at every seed, and reuse a compatible runtime if one is already present. Keep reusable runtime files separate from per-session writable state: input, output, logs, private harness configuration, and caches. Session close must not remove a shared runtime.
 
-1. Detect the target sandbox's OS, CPU architecture, libc, runtime prerequisites, and execution permissions. Select artifacts for that target, because an executable found on the Agent Server host may not run there. For example, `uv_for_sandbox()` probes the sandbox before it chooses a `uv` build.
+1. Detect the target sandbox's OS, CPU architecture, libc, runtime prerequisites, and execution permissions. Select artifacts for that target, because an executable found on the Agent Server host may not run there. For example, `SandboxPythonRuntime.install()` internally probes the architecture, C library, and glibc version, which select the Python build and the wheels that `SandboxPythonRuntime` uploads.
 2. Check any preinstalled, mounted, or previously installed runtime against the pinned version and a health check that runs it. The existence of a directory does not prove the runtime is ready.
 3. Verify the checksums of pinned artifacts on the Agent Server host, upload them through the sandbox file API, and run the harness's installer in a private directory outside the task repository. Preserve the task's interpreters and libraries. If the installer resolves packages, constrain the harness's dependencies to the versions the adapter is tested with, for example with a constraints file generated from the agent environment; an exact pin on the harness alone lets its dependencies change between episodes.
 4. Mark the runtime as ready only after installation and its health check succeed. Handle partial installations and concurrent setup without exposing an incomplete runtime. On failure, keep the commands, exit codes, and logs.
@@ -580,9 +580,11 @@ The supervisor and the shared runner need only bash in the task image. Do not re
 
 A harness that is itself a Python program needs a Python runtime that the adapter brings, never the task image's. The launch command checks that bash can parse the supervisor before it claims the launch.
 
+Launch the harness with an absolute interpreter and an absolute script path under its private runtime, without activating its environment. Hermes and Stirrup use `python -E -s`: the harness ignores task `PYTHON*` settings and user site-packages, while its tool subprocesses retain the task's `PATH`, `VIRTUAL_ENV`, `PYTHONPATH`, and `PYTHONHOME`. The trusted script directory stays importable for sibling modules; the task's current directory does not become the harness's import root. Set harness-specific state before the first framework import: Hermes sets `HERMES_HOME` to the session's private directory before importing Hermes, and its seed import checks use the runtime's private home. Keep the task's `HOME` unchanged for tools.
+
 The proposed shared installer must not depend on system package managers or root access. Not all existing adapter installers meet this goal yet. Until they migrate, document the supported images and any remaining prerequisites.
 
-[Hermes seed](https://github.com/NVIDIA-NeMo/Gym/blob/main/responses_api_agents/hermes_agent/sandbox.py) and Stirrup seed are exceptions that are being removed, not examples to follow. A cold setup uploads the `uv` binary that `uv_for_sandbox()` selects for the sandbox's platform, creates a private Python 3.13 environment in the sandbox, and installs the pinned package from the network. For a new Python harness, verify a pinned standalone Python on the Agent Server and upload it, as [NOOA](https://github.com/NVIDIA-NeMo/Gym/blob/main/responses_api_agents/nooa_agent/sandbox_runtime.py) does, or prepare the runtime outside the task container and mount it read-only, as [`hermes_sandboxed_agent/prepare_runtime.sh`](https://github.com/NVIDIA-NeMo/Gym/blob/main/responses_api_agents/hermes_sandboxed_agent/prepare_runtime.sh) does.
+For a Python harness, build the runtime on the Agent Server with [`SandboxPythonRuntime`](https://github.com/NVIDIA-NeMo/Gym/blob/main/nemo_gym/agents/utils/sandbox_python.py), as the [Hermes seed](https://github.com/NVIDIA-NeMo/Gym/blob/main/responses_api_agents/hermes_agent/sandbox.py) does. The sandbox needs x86_64 or aarch64 Linux with glibc 2.17 or newer or musl, `bash`, `tar` with gzip, and space under `/tmp`, but no network access. The Agent Server needs `uv` and access to the package index and the python-build-standalone releases on GitHub; `HTTPS_PROXY` applies, and `UV_PYTHON_INSTALL_MIRROR` selects an HTTP(S) mirror (`file://` mirrors are not supported). An offline host can prepopulate `cache_dir/sandbox_python/` with the pinned Python archive; its checksum is still verified. Package dependencies must also be available to host uv through a local cache or index. Every dependency except a direct-URL requirement must publish a wheel for the sandbox's platform, so a harness's dependencies can narrow these platforms: Stirrup's need glibc 2.28 or newer and do not support musl.
 
 <Note>
 The [shared installation proposal](https://github.com/NVIDIA-NeMo/Gym/pull/3961#issuecomment-6003983236) would add an Agent Server helper, `harness_install.py`, with `ensure_harness_installed(...)`. It would detect the platform, reuse or stage verified artifacts, run the adapter's installer at seed, and return the runtime location. Neither the helper nor its package location exists yet.

fern/versions/latest/pages/reference/configuration.mdx

@@ -20,7 +20,7 @@ Two **opt-in** top-level config keys that servers can consult for state that out
 | Key | Default | Read today by |
 |-----|---------|---------------|
 | `results_dir` | `<working dir>/results` | the `swe_agents` server (per-run results) and the W&B run directory |
-| `cache_dir` | `<working dir>/cache` | the `swe_agents` server (setup trees) and the `uv_cache_dir` default |
+| `cache_dir` | `<working dir>/cache` | the `swe_agents` server (setup trees), the `uv_cache_dir` default, and `sandbox_python/` (host-built harness runtime archives) |
 
 These are opt-in settings, not a global redirect: components that don't consult them (profiling output, rollout capture, logs, Hugging Face caches, other agents) keep their own locations, and the module-level `RESULTS_DIR`/`CACHE_DIR` constants remain unchanged defaults for such consumers. New artifact producers are encouraged to read these keys.
 

nemo_gym/agents/__init__.py

@@ -60,6 +60,7 @@
     from nemo_gym.agents.testing.agent_session_conformance import check_agent_seed_rejected
     from nemo_gym.agents.testing.failure_assertions import raises_agent_failure
     from nemo_gym.agents.utils.sandbox_observation import lifecycle_observation
+    from nemo_gym.agents.utils.sandbox_python import SandboxPythonRuntime
     from nemo_gym.agents.utils.sandbox_session import (
         HARNESS_COMMAND_NOT_STARTED,
         HARNESS_RUNNER_FAILED,
@@ -84,7 +85,6 @@
         parse_cleanup_receipt,
     )
     from nemo_gym.agents.utils.task_timeout import AGENT_TIMEOUT_METADATA_KEY, resolve_agent_timeout
-    from nemo_gym.agents.utils.uv_bootstrap import DEFAULT_UV_CACHE_DIR, uv_for_sandbox
 
 
 # What code outside the library, such as servers, imports from this package. Each name's submodule loads
@@ -111,7 +111,6 @@
     "CLEANUP_PHASE_COUNT": "utils.supervisor_client",
     "CleanupReceipt": "utils.supervisor_client",
     "DEFAULT_CLEANUP_TIMEOUT": "utils.supervisor_client",
-    "DEFAULT_UV_CACHE_DIR": "utils.uv_bootstrap",
     "HARNESS_COMMAND_NOT_STARTED": "utils.sandbox_session",
     "HARNESS_RUNNER_FAILED": "utils.sandbox_session",
     "HARNESS_RUNNER_FILE": "utils.sandbox_session",
@@ -126,6 +125,7 @@
     "SUPERVISOR_FILE": "utils.supervisor_client",
     "SUPERVISOR_SCRIPT": "utils.supervisor_client",
     "SandboxCommand": "utils.sandbox_session",
+    "SandboxPythonRuntime": "utils.sandbox_python",
     "SandboxSession": "utils.sandbox_session",
     "SimpleResponsesAPIAgent": "base_responses_api_agent",
     "agent_outcome_policy": "agent_outcome",
@@ -145,7 +145,6 @@
     "seed_mcp_servers": "agent_tools",
     "shell_assignments": "utils.sandbox_session",
     "stage_skills": "skills",
-    "uv_for_sandbox": "utils.uv_bootstrap",
     "verify_agent_response": "agent_tools",
 }
 __getattr__, __dir__ = lazy_exports(__name__, _PUBLIC_API)