[d-molinari]

What does this PR do?

Lets Gym's sandboxed harnesses work on OfficeEnv tasks. With sandbox_provider set, the OfficeEnv server from #5009 lends each Environment Server episode its own sandbox (Docker or OpenSandbox). The harness works in that sandbox, and OfficeEnv grades the simulated apps' final state as before. Without sandbox_provider, nothing changes.

#5009 is merged, so this PR now sits on main. It also carries #5373 and #5374, folded in, and OfficeEnv's Environment Server, which ends an episode when its task ends (below; it replaces #5624).

How an episode runs

• Start and stop. The seed starts the episode's sandbox and writes a short guide (README.md) into the workspace. /verify stops the sandbox before grading, so nothing the harness left running can change the reward.
• Seeds take turns. Each seed builds its world on the server's single event loop. At about 1,000 episodes in flight, a wave of seeds held the loop for minutes, and Pi's MCP setup timed out after 30 s. Seeds now run one at a time, and before each the loop serves waiting requests for 100 ms, so a burst of harness setups is answered between builds. Each seed logs its place in the queue, its wait and its build time.
• Reaching the apps. The row picks one of three ways:
  • MCP tools, for Pi and Codex;
  • a command per app in the shell (<app> <command> --parameter=value), for any harness with a shell;
  • HTTP endpoints the model looks up over MCP. The documentation tool goes by the name the row's exposure gives it: api_fetch, or api_docs with OfficeEnv 1.11.0's docs_tool option.
• Files go through the workspace, not the conversation.
  • Downloads from Drive, Box, SharePoint and OneDrive, and Gmail attachments, are saved as files and answered with their path.
  • Uploads send a workspace file through a per-episode file API, and an email attaches a workspace file by its path.
  • Pasted base64 is refused, with the command to use instead.
• The sandbox sees an ordinary company API (COMPANY_API_URL, company_* tools). Nothing this server shows the sandbox names the environment or Gym, and a test checks all of it.
• Guarded. Only Gym's node (and trusted_networks) may seed, verify or call the tool routes, and sessions are signed with a per-process secret. block_internet is opt-in and works on OpenSandbox only; it limits a sandbox's network to this server and the model servers.
• Provider failures. If the provider fails while saving or reading a file, the call fails (502) and the episode's reward is masked (mask_sample), as fix(harness_agent): mask rollouts the sandbox provider reports as failed #5354 does for a lost sandbox.
• Shutdown. When the server shuts down (a job's end), it stops every lent sandbox at once, 64 at a time, within Gym's server_shutdown_timeout_seconds less 2 s. It logs any it could not stop at ERROR, for a reaper: the lent ones apart from those still being created, each with their sessions.

OfficeEnv's Environment Server (environment_servers/officeenv, the top two commits), following the review on #5624:

• It runs the single-agent-turn protocol unchanged (it subclasses single_agent_turn_legacy).
• With max_steps set, it drives the agent one step per activation (the simple agent with max_steps: 1), passing the conversation so far as input.
• Between steps it reads the OfficeEnv server's new GET /task_state. The episode ends before the next model call once complete_task has succeeded, when the model answers without a tool call, when a step doesn't complete, or at max_steps.
• The resources server only answers the read, with the state each session's last tool call left, published when that whole call is over. With close_tools_on_task_end it refuses tool calls once the task has ended, so the calls left in that step change nothing.
• configs/officeenv.yaml now puts it in front of the simple agent (50 steps) in place of legacy_agent. The sandboxed configs keep single_agent_turn_legacy, so their harnesses still end their own runs.

Gym core: three opt-in hooks, with today's behavior as their defaults:

1. SimpleServer.get_session_secret(): the key that signs session cookies and MCP tokens.
2. SimpleServer.get_session_cookie_name(): the session cookie's name.
3. mcp_server_name and mcp_bearer_token on BaseResourcesServerConfig: an MCP endpoint listed under its own name, with a standard Authorization: Bearer token.

Shutdown needs no new hook. Gym interrupts each server through its process group (#5141), so this shutdown runs at a job's end. Under NeMo RL, whose Ray workers block that interrupt, servers also need #5623. The sandboxed configs set Gym's server_shutdown_timeout_seconds (also #5141) to 30 s, because its 1 s default is too short to stop a collection's sandboxes. The setting applies to the whole run that composes those configs; other runs keep the default.

Configs, data and images

• Configs for Pi (Docker and OpenSandbox), Codex CLI, mini-SWE, OpenCode, OpenClaw and Terminus 2.
  • Pi, Codex, OpenCode and OpenClaw each run a limited number of episodes at once. Their configs cap the Environment Server's admission to match, so no episode waits for its harness while its sandbox's TTL runs.
• Data: nine sandbox tasks in data/sandbox_pilot.jsonl.
• Images:
  • an optional documents image (Word, PDF and LibreOffice tools), for rows that ask for it;
  • one image per harness with its runtime preinstalled, for sandboxes without the internet.
• Docs: the server README's "Sandboxed harnesses" section covers the setup.

Validation

• gym env test --resources-server officeenv +should_validate_data=true: 201 passed with OfficeEnv 1.10.0 installed. The tests use a fake sandbox provider. Three api_docs tests skip on 1.10.0 and pass on OfficeEnv's main branch.
• gym env test +entrypoint=environment_servers/officeenv: 14 passed. single_agent_turn and single_agent_turn_legacy pass as on main.
• End to end, with the real simple agent, Environment Server and OfficeEnv server and a stub policy: a task the model completes takes one model call, and the call left in that step is refused; a task without one takes two (tool call, then answer). Both are graded.
• Gym's core suite (pytest -m "not sandbox") fails only where main does (the same 33 tests).
• pre-commit and detect-secrets pass on the changed files, and every commit is signed off.
• Rebased onto main 3a19fead73, which has refactor(sessions): one name per session lifecycle concept before release #5083's session names. The component lock check passes for all 271 projects.

Rollouts

Smoke runs with gpt-5.5 on Docker sandboxes, on earlier heads of this branch:

• Pi, Codex CLI, mini-SWE, OpenCode, OpenClaw and Terminus 2 across the sandbox tasks: 103 rollouts, all at reward 1.0. mini-SWE needed a one-line fix of its own on macOS, which belongs in the agent.
• No file content passed through the conversation, and no rollout shows this server's environment names.
• Not yet run: episodes on an OpenSandbox cell, and block_internet with a harness image.

Compatibility and benchmark impact

• Without sandbox_provider: the routes are feat(officeenv): add the OfficeEnv resources server #5009's, plus GET /task_state. Episodes run through OfficeEnv's Environment Server instead of legacy_agent, and end at the step that completes the task. A run that raised the simple agent's max_steps now raises the Environment Server's. Gym core's defaults are unchanged.
• Dependency: the server's pyproject.toml takes nemo-gym[dev,sandbox], locked as main's other sandbox servers lock it. OfficeEnv installs apart as in feat(officeenv): add the OfficeEnv resources server #5009, and the sandbox tests skip without it.
• Known limits: the harness adapters still show some Gym names (their folders, variables and system prompts). The README lists them.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally.
• Documentation added or updated (resources_servers/officeenv/README.md, "Sandboxed harnesses"; the MCP tutorial).
• Pre-commit checks pass locally.
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s).

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

Review of 20676db. Each item has a failing test I can share.

1. A provider failure reading an attachment reaches the model as a missing file (app.py L1453-L1458)
_read_attachments catches every exception from episode.sandbox.download, then copied_file reports the path as missing. A timeout or a stopped sandbox becomes "there is no file at /workspace/notes.csv in the workspace", so an infra failure is scored as a model mistake. Suggest catching only the provider's not-found case and letting the rest fail the call.

2. _save_downloads can race /verify releasing the sandbox (app.py L1608-L1616)
After the mkdir await, _stop_before_grading can run _release, which sets episode.sandbox = None. The next episode.sandbox.upload raises AttributeError, answered as 502 "Could not save ... in the workspace" after the world change already happened. Holding the episode lock for the save, or re-checking the sandbox after each await, would close it.

3. (nit) shutdown can miss a sandbox a concurrent seed is still starting (app.py L1333-L1338)
shutdown walks the episodes without the per-episode lock, and the seed inserts the episode before it assigns episode.sandbox. That sandbox is left to the provider TTL.

4. (nit) _locks entries are never removed (app.py L1133)
setdefault at L1247, L1371, L1400 and L1409 with no pop, including close calls for never-seeded ids, so the dict grows for the life of the process.

[d-molinari]

Thanks for the review. All four are fixed on the new head (199654c34, rebased onto main 154d77914), each in the commit that introduced the code, with tests that fail without the fix. #5373 and #5374 are now folded into this PR (commits 13–19), so their routes are covered here too.

1. Provider failure scored as a missing attachment (feat(officeenv): keep Gmail attachments out of a sandboxed episode's conversation)

• Only a file that is gone counts as missing now. After a failed download, the path is checked again: if it still holds a regular file, the provider failed, and the call fails (502). The provider's words go to the server's log and the verify's failure_reason, never to the model. If the file is gone since the check, it is missing, as before.
• A check that doesn't run fails the call the same way. That covers stat's exec failing, and fewer output lines than paths. Before, such a check read as "every path missing".
• Either failure also masks the episode's verification (mask_sample, failure_kind: provider_unavailable), the convention fix(harness_agent): mask rollouts the sandbox provider reports as failed #5354 uses on the harness side. So the reward isn't scored against the policy.
• The same applies to a save the provider fails in _save_downloads. A save the workspace blocks (a folder at the file's path, or a file where its folder goes) stays the model's doing: 502, graded as usual.

2. _save_downloads racing /verify (feat(officeenv): keep Drive file content out of a sandboxed episode's conversation)

• A call now holds the episode's lock through the whole call: the attachment read, the service call and the saves (_episode_call).
• _stop_before_grading releases the sandbox under the same lock, so verification waits for a call already under way. A call that comes after verification has begun is refused.
• The cli and endpoint routes (commit 13, from feat(officeenv): reach a sandboxed episode's apps as commands or endpoints, for any harness with a shell #5373) go through the same helper. The endpoint route re-raises the helper's HTTPExceptions, so a 409 or 502 doesn't turn into its 500.
• The test drives your scenario: /verify starts at the save's mkdir. Without the lock, it gets the 502 from the AttributeError.

3. shutdown missing a sandbox a seed is still starting (feat(officeenv): lend each Environment Server episode a sandbox)

• shutdown takes each episode's lock, so it waits for a seed that is still starting a sandbox. That includes a seed still waiting for its lock, which has no episode yet.
• It sets a flag first, so a seed that comes later is refused (503) before it starts a sandbox.
• retire_session_state had the same shape: it took the lock only if the episode already had a sandbox. It now takes the lock whenever there's an episode, so it waits for a starting sandbox and stops it too.

4. _locks never shrinking (same commit)

• _locks is a weakref.WeakValueDictionary: a lock stays only while a request holds or awaits it.
• So closed sessions leave nothing behind, and neither does a close for an ID that was never seeded.

Since your review: episodes waiting with a sandbox lent (fix(officeenv): admit no more sandboxed episodes than the harness runs at once, commit 21)

A training run with Pi on an OpenSandbox cell found another lifetime gap:

• The seed lends an episode its sandbox, and the sandbox's TTL runs from its create. Pi runs 8 episodes at once by default, and the Environment Server admitted every episode a collection sent.
• So the rest waited for Pi with their sandboxes lent: of 128 episodes seeded at once with a 1 h TTL, 30 lost their sandbox ("Sandbox not found").
• The harness configs now cap the Environment Server's admission at the harness's own concurrency, by interpolation, so raising one raises both. That covers Pi, Codex, OpenCode and OpenClaw. The other episodes wait there, before their seed, holding no sandbox, for up to a day.
• mini-SWE and Terminus 2 have no cap of their own and run every episode they are given, so their configs are unchanged.
• tests/test_sandbox_configs.py composes each config as gym env start does. Its admission checks fail on the previous configs.

If your failing tests differ from the ones added here, I'd be glad to run them against the new head.

[d-molinari]

Two commits since my reply above, both on the shutdown path from your item 3:

• fix(officeenv): stop every lent sandbox at once when the server shuts down. Shutdown now stops the lent sandboxes concurrently, 64 at a time, within Gym's server_shutdown_timeout_seconds less 2 s; the sandboxed configs set it to 30 s. Before, it stopped them one by one with no deadline. It still waits for a seed that is starting a sandbox, but only until that deadline. Past it, it logs the sandboxes left running or starting at ERROR, with their sessions, for a reaper.
• fix(officeenv): seeds take turns on the event loop, so a harness's MCP setup is not queued behind them. World builds now run one at a time, and before each the loop serves waiting requests for up to 10 ms. A seed still waiting for its turn when shutdown begins is refused (503) in its turn, before anything is built.