Clarify Agent, Environment, Task, and Sandbox boundaries

Problem

NeMo Gym is trying to support a familiar RL-style setup in a modern, LLM-agent setting:

Experiment with LLM-based Agents on Tasks sampled from different Environments, collect rollouts, evaluate performance, and optionally train the Agent or Policy Model from those rollouts.

The concepts are not new: an Agent interacts with an Environment, produces a trajectory, and receives reward. In today's LLM setting, the Environment is often interfaced through language, tools, APIs, filesystems, browsers, sandboxes, or MCP servers. The Task may be partially observable, long-context, and agentic. The reward may be terminal-only, as in many RLVR workflows, or it may eventually be emitted per turn, tool call, or environment step.

The confusion is in how we map those concepts onto NeMo Gym's async-first, server-first architecture.

The docs currently use environment in a few overlapping ways. The concepts page says an environment contains dataset, agent harness, verifier, and state. The terminology page then says a Gym environment is "a resources server together with an agent and dataset(s)" and that every environment has exactly one resources server.

That wording is hard to map onto the architecture. It also showed up in SWE-bench work such as #1738, where task setup, sandbox lifecycle, and grading could drift into agent-server code instead of being modeled as Environment or World responsibilities.

The goal of this issue is to clarify the mental model so we can extend the system without mixing up Agent, Environment, Task, Resources Server, Model Server, and Sandbox.

This is an architecture clarification and design direction, not a claim that every contract is final.

Proposed mental model

Agent

An Agent is the decision-making system:

Policy Model + Agent Harness

The Policy Model is the LLM whose weights we may evaluate or train.

The Agent Harness is the orchestration around the model: tool calls, model calls, planning loops, context management, stopping behavior, CLI invocation, memory, critics/value functions, or other adaptive pieces. Today, the most common learning target is the Policy Model weights, but the broader Agent design space can include adaptive memory, critics, value functions, or harness-level learning.

An Agent Server is how Gym deploys this orchestration, usually under responses_api_agents/.

Black-box CLI agents such as Claude Code or Codex still fit this model. We wrap the CLI in an Agent Server, but conceptually it is still an Agent Harness around model-backed behavior.

Environment

An Environment is the world or problem family the Agent interacts with.

It defines the state, the actions/tools available to the Agent, the observations returned to the Agent, the transition semantics of the world, and the reward authority. In a language-agent setting, those pieces may be exposed through text, tool calls, filesystems, shells, browsers, MCP servers, APIs, databases, or other runtime systems.

For SWE-bench, the Environment is the code-repair world. Its state includes the repository filesystem: the repo at the base commit, dependencies/build artifacts in the image, files changed during the attempt, command outputs, logs, and test execution state. The Sandbox hosts that filesystem and process runtime, but the Environment defines what that state means, how it is initialized from a Task, and how changes to it are scored.

A browser is not part of the standard SWE-bench Environment because the SWE-bench world is repo + shell/test execution + patch verification. But in a browser-based debugging Environment, browser DOM/page state, screenshots, clicks, and app runtime state could be part of the Environment state and action/observation interface.

Current RLVR-style Environments often produce sparse terminal reward through verify() after the full episode is collected. That remains valid. The design should also allow richer Environments to emit reward per step, per turn, or per tool call when the interaction is mediated by Gym.

Notice that nothing prevents the Agent to create a Model of the parts of the MDP (Environment) like for example a Reward Model to help with its own learning.

Task

A Task is one problem instance sampled from an Environment.

For SWE-bench, a task is one issue instance: repo, base state, problem statement, instance id, and privileged grading metadata.

This is different from the Environment. SWE-bench is the environment/world family; django__django-13741 is a task inside it. A dataset is a collection of tasks.

Resources Server

A Resources Server is an implementation mechanism for Environment capabilities.

It may expose task seeding, state, tools, MCP endpoints, verify(), optional step(), or a facade over other services. Many Environments today are implemented as one Resources Server, but we should not define Environment as exactly one Resources Server forever.

Model Server

A Model Server is separate from the Agent Server so inference and training can scale independently. This separation matters for NeMo Gym and NeMo-RL: rollout collection, model inference, and training should be able to scale without forcing the Agent Harness and model backend into the same process.

Sandbox

A Sandbox is runtime and isolation.

It can host Environment state, Agent execution, both together, or neither. For SWE-bench, Docker or Apptainer provides the repo filesystem and execution boundary.

The sandbox is not the Environment by itself. It is the runtime substrate where Environment state and/or Agent execution happen.

Benchmark

A Benchmark is a fixed evaluation product built on top of an Environment:

fixed task split
fixed metric
fixed protocol
baselines or leaderboard claims

For example, SWE-bench is an Environment/world family. SWE-bench Verified is a benchmark configuration built from a fixed task set and scoring protocol.

SWE-bench as the concrete example

Using SWE-bench as the running example:

Task: one SWE-bench instance, with problem statement, repo/image info, instance id, and grading metadata.
Environment: the SWE-bench world: repo state, patch-producing interaction, and test-based reward semantics.
Resources Server(s): implementation of SWE-bench setup and verification, such as seed_session and verify.
Agent: claude_code_agent, anyswe_agent, OpenClaw, or another harness wrapped as an Agent Server.
Policy Model: the LLM backing the Agent.
Model Server: optional serving layer for inference, decoupled from the Agent Server.
Sandbox: Docker, Apptainer, OpenSandbox, or another runtime.
Benchmark: SWE-bench Verified as a frozen task split and metric.

One naming caveat: SWE-bench's own "evaluation harness" is its grading/test execution machinery. That is not the same thing as a Gym Agent Harness.

Desired direction

We should update docs and architecture language so they say:

Agent = Policy Model + Agent Harness.
Agent Server deploys Agent orchestration.
Model Server serves inference separately for scalability and training integration.
Environment defines the world, state, actions/tools, observations, transitions, and reward semantics.
Resources Servers implement Environment capabilities, but are not always the whole Environment concept.
Task is a first-class problem instance inside an Environment.
Sandbox is runtime/isolation for the Agent, the Environment, or both.
Benchmark is a fixed evaluation protocol built on top of an Environment.

This should preserve what works today while leaving room for external Environments, black-box Agent internals, remote sandboxes, per-step rewards, and richer learning setups.

Why this matters

A natural requirement is:

Swap any Agent with any compatible Environment and collect rollouts.

The target composition should look like:

agent server x environment x sandbox backend x task set x model server

not one wrapper per agent/benchmark pair.

For SWE-bench, the long-term direction is that SWE-specific setup and verification live with the SWE-bench Environment, while black-box agents remain normal Agent Servers that consume the Environment contract.

Decouple SWE environment infrastructure from agent harnesses #1249: parent issue for decoupling SWE environment infrastructure from agent harnesses
Converge #1677 swe_env decoupling into anyswe + verify on SWE-bench Verified (#1249) #1738: SWE-bench convergence PR that exposed the boundary question
feat: Add initial prototype for Agent / Environment responsibility split. #1801: initial prototype for the Agent / Environment responsibility split
Docs: fern/versions/latest/pages/about/concepts/environments.mdx
Docs: fern/versions/latest/pages/about/concepts/key-terminology.mdx

This comment is the design note companion to the issue body. It is exploratory: a design direction for clearer boundaries, grounded in the prototype in #1801, not a claim that every part is final.

The useful way to read the prototype is step by step: a Task is sampled, an Environment accepts it and returns episode context, the Agent Harness orchestrates the agent side of the interaction, the Sandbox hosts runtime, and the Environment produces reward.

Bigger picture

At the end of the day, Gym is experimenting with LLM-based Agents on Tasks created in different Environments or Worlds.

Those Tasks may look like partially observable MDPs, contextual MDPs, tool-use problems, code-repair tasks, browser workflows, or long-context agentic tasks. The interface is often language plus tools rather than a small numeric action space, but the high-level loop is still familiar:

sample Task -> Agent interacts with Environment -> collect rollout -> score reward

The rollout can be used for:

Evaluation: measure Agent or Policy Model performance on tasks.
Behavior cloning / SFT: train from successful trajectories or demonstrations.
RL: train from rewards, usually updating Policy Model weights today.

The design should not assume the Policy Model is the only thing that can ever adapt. Future Agents may include adaptive harnesses, memory modules, critics, value functions, reward models, or other pieces. But today, the main learning path is still through the LLM / Policy Model.

NeMo Gym and NeMo-RL are trying to make this work in an async-first, server-first architecture: rollout collection, inference, environment interaction, verification, and training should scale independently.

What we are trying to fix

The current wording makes it too easy to treat an Environment, a Resources Server, an Agent Server, a Sandbox, and a Task as interchangeable pieces.

That confusion showed up in #1738. The PR does useful work by extracting SWE-bench setup, grading, and sandbox-provider logic, but the boundary is still blurry: SWE-specific world setup and verification can live under an agent-server path, and the agent wrapper becomes the place where environment behavior is assembled.

The simpler target is:

Keep agents as agents, keep environments as worlds, pass tasks between them, and use sandboxes as runtime.

Core concepts

Agent

The decision-making system: Policy Model plus Agent Harness. The harness orchestrates the agent side of the interaction: model calls, tool calls, CLI invocation, planning loops, context management, stopping behavior, memory, and similar logic.

In Gym, this orchestration is deployed as an Agent Server under responses_api_agents/.

Black-box CLIs such as Claude Code or Codex still fit this model. Their internals may not be native to Gym, but the wrapper Agent Server lets Gym collect rollouts from them.

Policy Model / Model Server

The Policy Model is the LLM. The Model Server is separate from the Agent Server so inference and training can scale independently. This is important for NeMo-RL integration: the rollout system should not require the model backend and the agent orchestration to live in one process.

Environment

The world/problem family. It defines state, actions or tools, observations, transition semantics, and reward authority.

Current RLVR-style environments often produce sparse terminal reward through verify() after the full episode is collected. That remains valid. But the design should also allow richer Environments to emit reward per step, per turn, or per tool call when the interaction is mediated by Gym.

The invariant is:

Reward belongs to the Environment, whether it is terminal, step-level, or both.

Task

One problem instance sampled from an Environment. For SWE-bench, this is one instance_id with problem statement, repo/base state, image or world-init data, and privileged grading metadata.

Resources Server

An implementation mechanism for Environment capabilities: task seeding, state, tools, verify(), optional step(), MCP endpoints, or a facade over other services.

A simple Environment may be implemented as one Resources Server. More complex Environments may be composed from multiple Resources Servers or external systems. The concept should not force exactly one process forever.

Sandbox

Runtime and isolation. It can host the Environment state, the Agent execution, both, or neither. Docker and Apptainer are sandbox backends for SWE-bench; they are not themselves the Environment.

Concrete Prototype #1801

PR #1801 makes the discussion concrete with a SWE-bench prototype:

resources_servers/swe_bench/ becomes the SWE-bench Environment resources server.
resources_servers/swe_bench/task.py introduces SweTask, an Environment-internal Task value built from dataset row metadata.
resources_servers/swe_bench/session.py introduces SessionDescriptor, the Environment response after accepting a task.
resources_servers/swe_bench/app.py exposes seed_session and verify.
responses_api_agents/claude_code_agent/app.py stays the Claude Code Agent Server, but learns how to consume the descriptor for topology agent_in_env.
nemo_gym/sandbox/providers/docker/ adds a Docker sandbox backend used by both the agent working sandbox and the verifier sandbox.

The important point is that this does not make Claude Code a SWE-specific environment. It lets the Claude Code Agent Harness interact with a SWE-bench Environment through a small contract.

Prototype flow, step by step

1. The rollout starts from a dataset row

The rollout input is still a JSONL row with responses_create_params and verifier_metadata.

For SWE-bench, the row carries public prompt data plus privileged fields such as instance_id, dataset_name, instance_dict, test metadata, and optionally a container formatter.

At this point, the data is just storage format. It becomes a Task when the Environment parses it.

2. The Agent Server asks the Environment to seed the task

claude_code_agent.run() sends the run body to:

POST swe_bench /seed_session

The Environment parses the row into a SweTask using parse_task_from_request().

Conceptually:

dataset row -> SweTask

SweTask includes:

task identity: instance_id / task_id
initial observation: problem_statement
world-init data: image, repo workdir, base commit
privileged verifier metadata: full SWE instance record, tests, benchmark family

This is the Task, not the Environment and not the Agent.

3. The Environment returns episode context

swe_bench.seed_session() selects the SWE harness family, builds a SandboxSpec, and returns a SessionDescriptor.

The descriptor is not the Task. It is the Environment saying:

I accepted this task; here is the context needed for the Agent Harness to interact with this environment instance.

In #1801 it includes:

public task identity
placement topology, currently agent_in_env
sandbox spec: image, workdir, env, resources
egress hints for model/API access
verifier metadata that must round-trip back to /verify

This is where the Environment owns the SWE-bench world setup instead of asking the Agent Server to know SWE-bench details.

4. The episode is the Agent-Environment interaction

The Agent Server does not “run the episode” by itself. The episode is the interaction between the Agent and the Environment.

The Agent Harness orchestrates the agent side of that interaction: model calls, CLI invocation, tool use, stopping behavior, and artifact capture.

For topology agent_in_env, the prototype interaction is:

Claude Code Agent Harness
  -> starts sandbox from descriptor
  -> runs claude -p inside /testbed
  -> Claude Code edits the repo using its normal tools
  -> Agent Harness captures the resulting patch

SWE-bench is terminal-verification style in this prototype: the agent acts in the repo world, then the Environment scores the final patch.

5. The sandbox hosts the runtime

For SWE-bench, the sandbox is the runtime for the repo world and the black-box CLI execution.

In #1801, claude_code_agent reads the descriptor's sandbox.spec, starts an AsyncSandbox with the configured provider, injects model egress environment variables, stages Claude settings, and runs claude -p inside the task workdir.

After Claude Code stops, the Agent Harness harvests the artifact with:

git add -A && git diff --cached

That patch becomes verifier_metadata.model_patch.

6. The Agent returns the artifact for verification

After the interaction, claude_code_agent sends:

POST swe_bench /verify

with:

the original task/run body
the NeMoGymResponse trajectory built from Claude Code stream-json
merged verifier metadata
the captured model_patch

The Agent is not the reward authority. It only returns what happened and the artifact it produced.

7. The Environment verifies in a fresh sandbox

swe_bench.verify() parses the same task again, applies the submitted patch to the task value, and calls verify_task().

verify_task() then:

selects the benchmark-family grading plugin
builds a fresh eval sandbox spec
acquires a fresh sandbox
resets the repo to the base commit
materializes the model patch and test patch
runs the eval command
parses the results
returns reward and diagnostics

This is the hermetic twin: verification does not reuse the agent's working sandbox, so agent-side artifacts do not accidentally affect scoring.

8. The rollout receives reward

The Environment returns a response with:

reward
resolved
patch_exists
mask_sample
error_kind
task/environment identity

That result flows back through the Agent Server to the rollout driver.

Reward timing: terminal today, step-level later

The current SWE-bench prototype uses terminal verify() because raw Claude Code shell use is not mediated by a Gym step API. Gym sees the final patch and grades it.

The design should also support less sparse rewards when the Environment mediates the interaction:

Terminal reward: collect the whole trajectory, then call verify().
Per-tool reward: an MCP tool or Environment-owned tool call returns an observation plus reward.
Per-step reward: a Gymnasium-like step(action) returns observation, reward, done, and metadata.
Hybrid reward: shaping rewards during the episode plus a final verifier reward.

For black-box agents, per-step reward requires the relevant action boundary to be visible to Gym: MCP tools, a shell broker, a browser broker, or another Environment API. If the agent directly controls a raw shell inside a sandbox, Gym may only observe the final artifact unless we instrument that boundary.

What this clarifies

The prototype separates responsibilities like this:

The Task is the SWE-bench problem instance.
The Environment owns SWE-bench setup, verifier metadata, grading semantics, and reward.
The Resources Server is the HTTP implementation of the SWE-bench Environment contract.
The Agent Harness owns Claude Code orchestration and patch capture.
The Agent Server deploys that harness and connects it to the Environment contract.
The Model Server serves inference separately so rollout collection and training can scale.
The Sandbox is the runtime where the repo world and/or agent process execute.
The Benchmark is SWE-bench Verified: a fixed task split and scoring protocol on top of the SWE-bench Environment.

Why this is better than an agent-specific SWE wrapper

The target composition should be:

agent server x environment x sandbox backend x task set x model server

not a new wrapper for every agent and benchmark pair.

In this model:

claude_code_agent remains a general Agent Server for Claude Code.
SWE-bench setup and verification live with swe_bench.
Other black-box agents can consume the same Environment contract.
Other environments can reuse the same Agent Server if they return compatible episode context.
External Environments can be integrated without forcing them into Gym-native internals.
The verified: marker belongs on the Environment configuration once the protocol is baselined.

How to read #1738 from this perspective

#1738 is useful as a stepping stone. It validates several pieces we want:

shared sandbox-provider logic
SWE-bench task/setup code extracted from older agent-specific paths
patch capture and hermetic grading behavior
support for different sandbox backends

But it should not become the long-term abstraction if it means SWE-specific Environment behavior is owned by an agent wrapper. The follow-up design should move toward a SWE-bench Environment contract that black-box Agents can consume without becoming SWE-specific Agents themselves.

Open design questions

A few details still need design work:

What fields should every Task carry across environments, and which fields should remain environment-specific?
Should seed_session return a formal session descriptor with sandbox, MCP, egress, and placement fields?
Which Environments need step-style rewards versus terminal verify() only?
What is the right interface for per-tool or per-step rewards in language/tool-mediated environments?
When both the Agent and Environment need isolation, who owns the sandbox lifecycle?
Do we need a sandbox broker Resources Server, or is the local provider API enough for now?
Which parts of in-box execution are generic sandbox helpers, and which parts are specific to each Agent Harness?
How should external Environments plug into rollout collection without adopting all Gym internals?

The immediate next step is to align the docs and terminology so people can discuss these questions without first untangling overloaded words.

Clarify Agent, Environment, Task, and Sandbox boundaries #1792

Description