docs(harbor): add a SWE-Interact run recipe on the Harbor task shape by ffrujeri · Pull Request #4728 · NVIDIA-NeMo/Gym

ffrujeri · 2026-10-19T05:24:49Z

Summary

This PR adds a run recipe for SWE-Interact. The recipe has two files in environments/harbor/swe_interact/:

swe_interact.yaml: a deployment overlay for the harbor environment server.
README.md: the run command and an explanation of each setting.

The PR adds no engine code. It only adds configuration and documentation.

Important

This PR is built on main. The recipe needs #4819 (from the Harbor stack) and #5064 (gym eval prepare). Do not merge this PR before both merge. The PR stays a draft until a smoke rollout exists.

Dependencies on the Harbor environment server PRs

flowchart LR
    MAIN["main (M20, M24, M27, M28, M22, M33 merged)"] --> M34["#4819 M34: steps as turns, MCP grants, compose_variables, builds.force"]
    M34 --> REST["#4827 ... #4951"]
    REST --> TS["#4998 turn_sessions"]
    MAIN --> PREP["#5064 gym eval prepare"]
    M34 -. "required" .-> THIS["#4728 SWE-Interact recipe"]
    PREP -. "required by the README" .-> THIS
    MAIN --> THIS

PR	Use in this recipe
#4819 (M34)	Required. It adds `[[steps]]` as turns, `task_mcp_grants`, `compose_variables` with `model_server`, and `builds.force`.
#5064	Required by the README. `gym eval prepare` builds the 150 image contexts once before the runs.
#4998 (`turn_sessions`)	Not used. The published runs do not resume the agent across steps. The recipe keeps the default: a new agent session for each step.
#4951 (per-step workdir, `setup.sh`, artifacts, separate verification)	Not used. SWE-Interact steps have only `instruction.md` and `tests/`.
#4863 (Compose overlay, variable precedence)	Compatible. Operator `compose_variables` override the task values, as the recipe needs.
#4827, #4860, #4861, #4862, #4864, #4865, #4872	Not used by SWE-Interact.

Why

SWE-Interact measures multi-turn, user-driven coding. Models score about 50% on the single-turn tasks and about 25% on the interactive version.

It is the first Harbor dataset that needs all of these features together:

Multi-step tasks ([[steps]]), with each step run as one turn.
An MCP sidecar that simulates the user.
Compose values that the deployment supplies, not the task.

Related: #4707 (Harbor tasks with a simulated user), #3083 (user simulation for multi-turn episodes).

What changed since the first version

The first version of this PR also added engine code. That code is now in the Harbor stack, with the same configuration names. This PR now contains only the recipe.

First version of this PR	Where it is now
`[[steps]]` as turns, `multi_step_reward_strategy`	#4819
`task_mcp_grants: off \| optional \| required`	#4819
`compose_variables` with a `model_server` source	#4819
Sidecar `build:` contexts, `builds.force`	M24 (merged) and #4819
SWE-Interact recipe	This PR

Other changes to the recipe:

The recipe moved from benchmarks/harbor/swe_interact/ to environments/harbor/swe_interact/. Other Harbor dataset recipes on main use this location.
The overlay no longer sets default_episode_timeout_seconds. The server now applies the task's own time budget (enforce_task_timeouts, on by default).
The README no longer says that Codex must accept MCP tool grants. feat(codex): use MCP tool grants in sandbox sessions #4590 added this support.
The PR targets main, not adasif/harbor-m20-dockerfile-builds. feat(harbor): Dockerfile builds, the third image mode (RFC M20) #4566 merged.
The README runs gym eval prepare once before gym eval run (feat(harbor): gym eval prepare for Harbor targets, with resumable, rate-limit-safe image records #5064).
SIM_USER_MODEL is now openai/gpt-5.5, as in the published run configs. The sidecar calls LiteLLM at ${LITELLM_BASE_URL}/v1. The openai/ prefix tells LiteLLM to use the OpenAI-compatible route for any model name. The user_model server selects the actual model.

How a run works

Components

flowchart LR
    subgraph Host["Gym host"]
        CLI["gym eval run"]
        ENV["harbor environment server"]
        AGENT["codex_agent"]
        POLICY["policy model server"]
        USER["user_model server"]
    end
    subgraph Sandbox["OpenSandbox Compose group (one per task)"]
        MAIN["main: repository and agent work"]
        SIDE["user-server: simulated user (MCP)"]
    end
    LLM["User model endpoint (USER_MODEL_BASE_URL)"]

    CLI --> ENV
    ENV -- "seed, verify" --> Sandbox
    ENV -- "one session per turn" --> AGENT
    AGENT -- "model calls" --> POLICY
    AGENT -- "shell commands" --> MAIN
    AGENT -- "ask_user (MCP grant)" --> SIDE
    SIDE -- "inspects commits" --> MAIN
    SIDE -- "LITELLM_BASE_URL" --> USER
    USER --> LLM

One episode

Each task has five steps. The server runs each step as one turn on the same sandbox. Each turn starts a new agent session.

sequenceDiagram
    participant S as harbor server
    participant B as Compose group
    participant A as codex_agent
    participant U as user-server sidecar
    participant M as user_model server

    S->>B: Start main and user-server (fills compose_variables)
    loop For each step: plan, implement, handoff, write_tests, test_handoff
        S->>A: Start session (step instruction, MCP grant for user-server)
        A->>U: ask_user(question)
        U->>B: Read the agent's commits
        U->>M: Generate the user reply
        M-->>U: Reply
        U-->>A: Answer
        A->>B: Edit, commit
        A-->>S: Turn ends
        S->>B: Run the step verifier
        B-->>S: Reward for the step
    end
    S->>S: Use the final step's reward (strategy "final")
    S->>B: Stop the Compose group

If a turn has no reward file, or the server masks a turn, the episode stops.

The agent does not keep its conversation between steps. It gets context from the repository, its commits and /tmp/plan.md. The user-server sidecar lives for the full episode and keeps its own conversation.

Data flow of the overlay settings

flowchart TD
    Y["swe_interact.yaml"] --> BF["builds.force: true"]
    Y --> MG["task_mcp_grants: required"]
    Y --> CV["compose_variables"]
    Y --> UM["user_model server block"]

    BF --> PREP["Prepare: build every task Dockerfile and every user-server sidecar (150 contexts)"]
    PREP --> IMG["Recorded images, used at seed"]

    MG --> GRANT["Each turn: the agent gets the user-server MCP server. A harness that cannot use it fails the seed."]

    CV --> URL["LITELLM_BASE_URL = sandbox-reachable user_model URL with the rollout prefix"]
    CV --> KEY["LITELLM_API_KEY = unused, SIM_USER_MODEL = openai/gpt-5.5"]
    URL --> COMPOSE["Compose document for the episode"]
    KEY --> COMPOSE
    UM --> URL
    ENVV["USER_MODEL_BASE_URL, USER_MODEL_API_KEY on the Gym host"] --> UM

Results of this design:

The provider keys for the user model stay on the Gym host. The sandbox gets only a Gym URL.
The user model calls go through Gym with the rollout prefix. Gym records them with the rollout. Gym does not use them as training tokens.

Settings in the overlay

Setting	Value	Reason
`builds.force`	`true`	The declared task images do not contain `repo_exec_server.py`. The simulated user needs it to read the repository. The published Harbor runs use `--force-build` for the same reason.
`task_mcp_grants`	`required`	The agent must reach `ask_user`. Without the grant, the seed fails. The episode does not run without the user.
`compose_variables`	`LITELLM_BASE_URL`, `LITELLM_API_KEY`, `SIM_USER_MODEL`	These variables connect the sidecar to the `user_model` Gym server.
`user_model`	`openai_model`	The simulated user's model. Set `USER_MODEL_BASE_URL` and `USER_MODEL_API_KEY`.

Compatibility

There is no change to existing behavior. The PR adds two new files only.

Scores are comparable only when you use the same user model. The published runs use GPT-5.5 (high).

Validation

I validated the overlay against the configuration classes at the top of the Harbor stack (feat(harbor): turn_sessions, resume the agent across a task's steps (M34 follow-up) #4998 head, 494526942):
- HarborEnvironmentServerConfig accepts all server keys, including task_mcp_grants and compose_variables.
- BuildOptions.from_config reads force=True.
I checked the recipe against the published dataset (scaleapi/SWE-Interact, 75 tasks):
- The sidecar reads LITELLM_BASE_URL, LITELLM_API_KEY and SIM_USER_MODEL, and calls ${LITELLM_BASE_URL}/v1. A Gym model server serves /v1/chat/completions under the rollout prefix.
- LiteLLM (>=1.80.8, as in the sidecar image) resolves openai/gpt-5.5 to the OpenAI-compatible route.
- The published runs use --force-build and do not use --resume-trajectory. The recipe matches both.
pre-commit run on the two files passes.
DCO sign-off is on the commit.
Unit tests: N/A. The PR has no Python code.

Rollout evidence

N/A at this time. The PR stays a draft until a smoke rollout exists. A rollout needs:

feat(harbor): multi-turn tasks, Harbor [[steps]] as turns (M34) #4819 and feat(harbor): gym eval prepare for Harbor targets, with resumable, rate-limit-safe image records #5064 merged into main.
Image builds with buildah or podman, and a builds.repository for OpenSandbox.
OpenSandbox with Compose networking (networking.enabled: true).

The first smoke run will use the three tasks in SWE-Interact's run_configs/multiturn/test_run.sh.

copy-pr-bot · 2026-10-19T05:24:53Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

yuhezhang-ai

Thanks for putting this together. Sharing two early correctness findings against 5c337713 while this is still a draft: grading files remain visible between steps, and explicit step timeouts bypass the dataset multiplier. Both have focused local reproductions; full SWE-Interact rollout validation remains outstanding.

yuhezhang-ai · 2026-10-19T21:38:02Z

+        results: list[HarborTurnResult] = []
+        stopped_after: str | None = None
+        for turn, turn_input in zip(task.turns, inputs):
+            response, observations = await self._agent_turn(


[P1] Clear the grading files before the next agent turn

_run_shared_verifier clears /tests and /logs/verifier only before verification, then leaves their contents in the reused sandbox. This loop starts the next agent before that cleanup runs again. In SWE-Interact, even the first planning-step verifier uploads the root tests/test.patch, so the implementation step can read the hidden grading tests. A local filesystem reproduction confirms that a patch absent at the first agent activation is readable at the second. Please clear the verifier-owned files after collecting diagnostics and before seeding/activating the next agent, and stop the episode if cleanup fails. A regression test should check what the second agent can actually read.

yuhezhang-ai · 2026-10-19T21:38:02Z

+        return bool(self.steps)
+
+    def step_agent_timeout_sec(self, step: HarborStep) -> float:
+        return step.agent.timeout_sec if step.agent.timeout_sec is not None else self.agent.timeout_sec


[P2] Apply the dataset timeout multiplier to explicit step budgets

apply_dataset_config scales only the task-level agent and verifier timeouts, while these new accessors prefer an explicit step timeout unchanged. With defaults.timeout_multiplier = 2, a step declaring 60 seconds for its agent and 30 seconds for verification still receives 60/30; a step inheriting task defaults correctly receives doubled budgets. SWE-Interact declares explicit per-step timeouts, so increasing the dataset multiplier does not extend those limits and runs can still time out at the original values. Please scale explicit step timeouts during dataset configuration resolution too, while ensuring inherited values are multiplied only once, and cover both cases.

Harbor multi-step tasks now load and run on the harbor environment server instead of being rejected. Each [[steps]] entry is one Gym turn in the same sandbox, as Harbor runs them without resume_trajectory: a fresh agent session per turn, the step's own instruction and agent budget, then the shared verifier with the task's tests/ and steps/<name>/tests/ on top. A turn that is masked, writes no valid reward file, or misses its min_reward ends the episode, and multi_step_reward_strategy (mean or final) rolls the turns up. Multi-step rows never carry the top-level instruction.md, which SWE-Interact uses for the hidden spec its simulated user holds. task_mcp_grants (off by default) grants the task's streamable-http [[environment.mcp_servers]] entries to the agent as MCP tool accesses, so a harness can reach a Compose sidecar such as SWE-Interact's ask_user server. Per-step workdir/, setup.sh, healthchecks, artifacts and separate verification are refused by name for now. Port note (round 13 layout, PR #4728 commit 59545d8): VerifierSettings and _turn_verifier_settings live in environment_servers/harbor/verifier.py beside _run_shared_verifier (HarborVerificationMixin); VerifierSettings carries the task's [[verifier.collect]] hooks, which the mixin runs for every turn; a step's user overrides the verifier user; the single-turn path still uploads the task's tests/ unconditionally (only step and multi-step folders may be absent). The conftest episode() helper builds a HarborEpisodeRequest. Port note (round 14, rebased onto the round-14 M24): the base now forwards the harness's own trajectory from the session close as ng_trajectory on the result (M7 round-14 fix), so _agent_turn returns it with the response and observations, HarborTurnResult carries it per turn as `trajectory` (left out when None, as ng_trajectory is) and a multi-turn episode forwards the last turn's, beside the last turn's observations; test_multi_turn covers it. The base's rewritten README rows (artifacts and hooks, image ENTRYPOINT), its verifier_image_config docstring and its _refuse_verifier_group seed check are kept as they are. Port note (round 20, rebased onto M33 1dd755f): the base's run() now bounds the episode with the task deadline (enforce_task_timeouts), seeds the harness with agent_timeout_sec, tolerates a timed-out session close and hands the verifier an ATIF export of the turn, so _agent_turn carries all four per turn (the step's own agent budget as agent_timeout_sec) and run() wraps both the single-turn path and _run_turns in one deadline, _post_seed_budget_s summing every turn's setup allowance, agent budget, hooks and verifier time for a multi-step task; a _TurnProgress holder gives the deadline failure the running turn's stage, response and observations. _verified_outcome exports each turn's transcript from that turn's own request (_export_trajectory takes it as a third argument) before _run_shared_verifier, which keeps its trajectory argument. VerifierSettings carries network_mode, which _score reports as verifier_network, and HarborTurnResult carries verifier_network. A multi-step task is refused at seed when separate_verifier_for_artifacts would force it into separate mode. materialize_task keeps the base's dataset_digest keyword on multi-turn rows. test_multi_turn covers the per-turn budget and transcript, the summed deadline and the seed refusal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-authored-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>

A Harbor task's Compose file may leave a variable to the deployment, as SWE-Interact's simulated user does with ${LITELLM_BASE_URL}, ${LITELLM_API_KEY} and ${SIM_USER_MODEL}. Prepare used to fail on any unresolved name. It now leaves a name used only in a service's environment to the deployment, since an environment value cannot change which images run; elsewhere it still fails. The harbor server's compose_variables fill those names at seed from a literal value, a process environment variable, or a Gym model server. A model_server value is that server's sandbox-reachable URL with the episode's rollout prefix, so a sidecar's model calls are recorded with the rollout and never become training tokens. A name still unset at seed fails the episode terminally before provisioning. A document holding these values goes to a private folder removed after the group starts, never under artifacts_dir. Port note (round 13 layout, PR #4728 commit 95804b3): the stack's loader already follows docker compose for every unset name at prepare (an empty string with one warning, M19), which covers the environment-only deferral, so _blank_deployment_values is not carried; with operator_variables the loader substitutes strictly, keeping the seed-time refusal. _compose_variable_values and the episode-keyed _compose_document live in sandboxes.py (HarborSandboxMixin); the per-digest cached document stays for servers without compose_variables, and the private per-episode folder is used when they are set. The seed pre-check reads the document through load_compose_document, so it does not touch the provider block. Port note (round 20, rebased onto M33 1dd755f): _create_compose on this base routes sidecars per provider block, plans volumes and starts the group through _start_with_pull_retries (M13/M22/M28), so the private per-episode document is removed in a finally around that call, after the last pull attempt has read it; _compose_document keeps the base's field check and volume planning around the operator_variables load. The test's recording Compose stub takes the adapter's extra keywords. Everything else applied unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-authored-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>

SWE-Interact's tasks cannot prepare without two things Harbor does at run time. Every task's simulated user is a Compose sidecar with a build: section. Prepare now builds each sidecar context once, from that context's own Dockerfile, keyed by Harbor's content hash like main's build, and records it; the recorded image then stands in for the build: section everywhere the Compose file is read. A sidecar build takes only a context inside environment/, and its section passes the same host-safety checks as main's. With docker, the compose build of main then names main alone. The tasks' declared docker_image lacks the helper their Dockerfiles copy in, which is why their run configs pass Harbor's --force-build. builds.force builds the Dockerfile of a task that declares an image, and the loader then prefers the recorded build of the task's own Dockerfile, as Harbor does. Validating the main compose build also leaves environment values that only the deployment can fill blank, since they cannot change what is built. Port note (round 13 layout, PR #4728 commit df0a130): on this stack M24 (Compose service builds) already builds every service with a build: section, records it by service name under the context's digest and substitutes it at load (sidecar_build_services, ImageBuilder.build_many, with_built_images), and the M19 loader already leaves every unset Compose name blank at prepare, so the sidecar-build and blank_unresolved_environment parts are not carried. What is carried is builds.force: BuildOptions.force and BUILD_SETTINGS (build_options.py), plan_builds/build_images/_build_dockerfiles force (build_plan.py, image_build.py, prepare_builds.py) and the loader's preference (task.py). To keep the stack's rule that a recorded main is never substituted for a task pinned to a prebuilt image, the record names the tasks whose declared image a forced prepare built over (forced_tasks, written on build and on reuse) and the loader prefers the build for those tasks only. Port note (round 14, rebased onto the round-14 M24): the base's _build_dockerfiles now plans a second time without the tasks refused_before_build names (20 Oct review fix); that replan passes builds.force too, or a forced main would drop out of the plan whenever another task of the dataset is refused (test_harbor_compose_prepare covers it). Everything else applied unchanged. Port note (round 20, rebased onto M33 1dd755f): the loader on this base reads built-images.json from the dataset root (nested task folders, M27), so the forced_tasks record is read from the same root and _build_required keeps its root argument. The base's pre-build screen now reads keys only (a value such as featurebench's `count: all` is refused after the build), so the forced-replan test refuses its other task through a main.build.args key instead. Everything else applied unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-authored-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>

A config overlay and README for running the published SWE-Interact dataset through the harbor environment server with Codex: force-built images and sidecars, the user-server MCP grant on every turn, the simulated user's model bound to a Gym model server, and an episode budget that fits the Atlas tasks. The overlay validates against the server's config model; native Codex sessions still need to accept MCP tool grants before a run can start. Port note (round 13 layout, PR #4728 commit 5c33771): unchanged apart from one README sentence, since on this stack the user-server sidecar is built as a second service of each task's context (M24 Compose service builds), not as a context of its own. Port note (round 14, rebased onto the round-14 M24): the recipe lives under environments/harbor/swe_interact/ (the stack moved benchmarks/harbor/ to environments/harbor/ beside terminal_bench_4), and the --config paths in the README and the overlay's header follow; the files are otherwise unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-authored-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>