Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions benchmarks/nemotron_3.5_super/eval_container_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,10 @@ config_paths:
- benchmarks/swebench/verified/opencode.yaml
- benchmarks/swebench/multilingual/opencode.yaml
- benchmarks/deepswe/opencode.yaml
- benchmarks/terminal_bench_2_1/opencode.yaml
- benchmarks/terminal_bench_2_1/terminus_2.yaml
- benchmarks/swebench/pro/opencode.yaml
- responses_api_models/vllm_model/configs/vllm_model.yaml
# - nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml
# SWE Bench Pro
- benchmarks/nemotron_3.5_super/sandbox_utils.yaml

########################################
# START Secrets
Expand Down Expand Up @@ -109,9 +108,9 @@ opencode_sandboxed_agent_deepswe:
responses_api_agents:
opencode_sandboxed_agent:
datasets: []
opencode_sandboxed_agent_terminal_bench_2_1:
terminus_2_sandboxed_agent_terminal_bench_2_1:
responses_api_agents:
opencode_sandboxed_agent:
terminus_2_sandboxed_agent:
datasets: []
opencode_sandboxed_agent_swebench_pro:
responses_api_agents:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,9 @@ opencode_sandboxed_agent_deepswe:
prepare_script: benchmarks/deepswe/prepare.py
num_repeats: 5

opencode_sandboxed_agent_terminal_bench_2_1:
terminus_2_sandboxed_agent_terminal_bench_2_1:
responses_api_agents:
opencode_sandboxed_agent:
terminus_2_sandboxed_agent:
datasets:
- name: terminal_bench_2_1
type: benchmark
Expand Down
21 changes: 21 additions & 0 deletions benchmarks/terminal_bench_2_1/terminus_2.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
config_paths:
- responses_api_agents/terminus_2_sandboxed_agent/configs/terminus_2_sandboxed_agent.yaml
- resources_servers/terminal_bench_2_1/configs/terminal_bench_2_1.yaml


terminus_2_sandboxed_terminal_bench_2_1_resources_server:
_inherit_from: terminal_bench_2_1_resources_server

terminus_2_sandboxed_agent_terminal_bench_2_1:
_inherit_from: terminus_2_sandboxed_agent
responses_api_agents:
terminus_2_sandboxed_agent:
resources_server:
type: resources_servers
name: terminus_2_sandboxed_terminal_bench_2_1_resources_server
datasets:
- name: terminal_bench_2_1
type: benchmark
jsonl_fpath: benchmarks/terminal_bench_2_1/data/benchmark.jsonl
prepare_script: benchmarks/terminal_bench_2_1/prepare.py
num_repeats: 8
39 changes: 39 additions & 0 deletions responses_api_agents/terminus_2_sandboxed_agent/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Harbor Terminus 2 Agent

This agent runs Harbor's `Terminus2` control loop in the task sandbox supplied by
the NeMo Gym resources server. It adapts the small Harbor environment interface
that Terminus uses (`exec` and `is_dir`) to `AsyncSandbox`; task state therefore
remains owned by the resources server.

```bash
gym env start \
--config responses_api_models/vllm_model/configs/vllm_model.yaml \
--config nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml \
--config responses_api_agents/terminus_2_sandboxed_agent/configs/terminus_2_sandboxed_agent.yaml \
--config resources_servers/swebench/configs/swebench.yaml
```

To run one row from a benchmark JSONL after starting the servers:

```bash
python responses_api_agents/terminus_2_sandboxed_agent/client.py \
+benchmark_jsonl=benchmarks/swebench/data/swebench_verified_benchmark.jsonl
```

The agent calls the configured model server exclusively through the Responses
API. Its returned response contains every model request and response from the
Terminus trajectory. Set `dump_trajectory: true` to also have Harbor write its
per-turn JSON trajectory files; it is `false` by default.

## Download and mount Tmux binary
```bash
curl -fL \
-o tmux-3.7c-linux-x86_64.tar.gz \
https://github.com/tmux/tmux-builds/releases/download/v3.7c/tmux-3.7c-linux-x86_64.tar.gz
tar -xzf tmux-3.7c-linux-x86_64.tar.gz
mv tmux tmux-3.7c-linux-x86_64

# e.g. copy to mounted S3 bucket
aws s3 cp tmux-3.7c-linux-x86_64 \
s3://tmux/tmux-3.7c-linux-x86_64
```
2 changes: 2 additions & 0 deletions responses_api_agents/terminus_2_sandboxed_agent/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
Loading
Loading