Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
ae45be9
refactor(agent_utils): run the harness process supervisor in bash
hemildesai Oct 26, 2026
30681cd
feat(agent_utils): add a shared bash harness runner
hemildesai Oct 26, 2026
7aae745
docs(agent_utils): describe the bash supervisor and runner; update he…
hemildesai Oct 26, 2026
79d9041
test(hermes_agent): expect stop requests as the stop file, not a signal
hemildesai Oct 26, 2026
40547ce
refactor(kilocode): run the harness without Python in the sandbox
hemildesai Oct 26, 2026
226d2a7
refactor(claude_code_agent): run the harness without Python in the sa…
hemildesai Oct 26, 2026
3fe0155
refactor(openclaw_agent): run the harness without Python in the sandbox
hemildesai Oct 26, 2026
b1b6823
refactor(opencode): run the harness without Python in the sandbox
hemildesai Oct 26, 2026
1dbb639
refactor(pi_agent): run the harness without Python in the sandbox
hemildesai Oct 26, 2026
83197dd
fix(opencode): keep pre-port outcomes when the session database is un…
hemildesai Oct 26, 2026
edeb1e6
style: format bash supervisor tests
hemildesai Oct 26, 2026
55f59ef
refactor(agent): adapt GitHub-only harness paths to the bash supervisor
yaoyu-33 Oct 26, 2026
a298d67
fix(agent_utils): keep runner events on old bash and ignore inherited…
yaoyu-33 Oct 26, 2026
626cab1
fix(agent_utils): stop tools that left the harness group and harden t…
yaoyu-33 Oct 26, 2026
6ca344c
fix(agent_utils): give the events-mode CLI default interrupts and no …
yaoyu-33 Oct 26, 2026
fa254f1
fix(agent): make installer locks without flock crash-safe and portabl…
yaoyu-33 Oct 26, 2026
28448d1
fix(opencode,kilocode): close host-side SQLite connections
yaoyu-33 Oct 26, 2026
844e9c6
docs(agent_utils): describe the stop-time cleanup of tools in their o…
yaoyu-33 Oct 26, 2026
7552c46
fix(agent_utils): reject shell words bash cannot hold; tighten Hermes…
yaoyu-33 Oct 26, 2026
01ac0f1
style: apply ruff formatting to the ported tests
yaoyu-33 Oct 26, 2026
ec0eadf
test(pi_agent): stop in-flight detached tools and keep services from …
yaoyu-33 Oct 26, 2026
a1f9e5c
docs(agent_utils): say which harnesses keep a Python shim in the sandbox
yaoyu-33 Oct 26, 2026
27a2877
Merge branch 'main' of github.com:NVIDIA-NeMo/Gym into refactor/bash-…
yaoyu-33 Oct 26, 2026
f6471c3
Merge branch 'main' into refactor/bash-sandbox-runtime
yaoyu-33 Oct 26, 2026
a61af13
fix(agent): crash-safe installer queue, checked version probe, JSON-s…
yaoyu-33 Oct 27, 2026
c5fd070
test(opencode,openclaw): pass the bash runner's payload to the neutra…
yaoyu-33 Oct 27, 2026
6198d4d
test(agent_utils): find the stdlib runner as the tool's parent
yaoyu-33 Oct 27, 2026
bb1661b
fix(agent_utils): reach processes that leave the harness during a stop
yaoyu-33 Oct 27, 2026
d303a76
fix(agent_utils): linear CR strip, errexit guard and distinct runner …
yaoyu-33 Oct 27, 2026
c69adb8
fix(agent): clean up staged OpenCode copies; bootstrap bash for Claud…
yaoyu-33 Oct 27, 2026
7908559
docs(agent-server): describe sandbox Python and uv paths as exceptions
yaoyu-33 Oct 27, 2026
fb642f9
test(agent_utils): find the supervisor through the runner after it le…
yaoyu-33 Oct 27, 2026
05bac05
perf(agent_utils): make the supervisor's forced stop cheaper on busy …
yaoyu-33 Oct 27, 2026
b1c63b8
Merge main (7520cf42d) into #5094's head as the base for #5137
yaoyu-33 Oct 27, 2026
656e173
fix(agent): close gaps in the flock-less installer queue and root-wor…
yaoyu-33 Oct 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion benchmarks/swe_together/opencode.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,6 @@ swe_together_candidate:
model: ${policy_model_name}
opencode_version: 1.15.13
sandbox_model_id: ${policy_model_name}
sandbox_python_executable: /opt/gym-runtime/python/bin/python3
sandbox_path_prefix: [/opt/gym-runtime]
sandbox_executable_path: /opt/gym-runtime/opencode
sandbox_executable_sha256: 8e97f83175c84daeb51e96b34baf9ee9839ae54e5e848a3cce1e0e32dcf41a6c # pragma: allowlist secret (public runtime digest)
Expand Down
27 changes: 13 additions & 14 deletions benchmarks/terminal_bench_2_1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,24 +156,23 @@ a successful smoke, and one task does not establish a benchmark baseline.

## Tasks that start services

The supervisor stops live processes in the harness group and reaps its children before
capture or verification. On timeout or close, it first records its already adopted
children. It then stops the harness and repeatedly drains newly adopted children,
including in-flight tools in separate sessions, within the cleanup deadline. Previously
adopted children outside the harness group remain for Resources to stop after verification
or episode failure. This includes unfinished background jobs such as `make &` left by an
The supervisor stops live processes in the harness group before capture or verification.
On timeout or close, it stops the harness, then kills within the cleanup deadline the group,
the harness's descendants in other process groups (such as in-flight tools in separate
sessions) and processes that inherited the harness's `NEMO_GYM_SUPERVISOR` environment
marker. Processes that left the harness before the stop remain for Resources to stop after verification or episode
failure. This includes unfinished background jobs such as `make &` left by an
earlier tool call, as well as services; either can continue writing during verification.

To survive this forced-stop cleanup, a service must be outside the harness group and,
before the stop, adopted by the supervisor or descended from a process that was. In the
usual tool-launch path, let the process that launched it exit before the harness deadline
or stop so that adoption can occur. A service still parented by an in-flight tool is
adopted when that tool exits or is killed during the stop, and is drained with the
in-flight tools.
To survive this forced-stop cleanup, a service must be outside the harness group and no
longer descend from the harness when the stop begins. In the usual tool-launch path, let the
process that launched it exit before the harness deadline or stop. A service still parented
by an in-flight tool descends from the harness at the stop and is killed with the in-flight
tools.

Natural CLI exit has no such cutoff: unfinished tools in separate sessions cannot be
distinguished from services and may still change task files. Adoption races, PID reuse,
and a preserved service orphaning more children during cleanup also limit this rule.
distinguished from services and may still change task files. PID reuse and a tool
that starts a new session after the stop is recorded also limit this rule.
A confirmed receipt is not a guarantee that the sandbox is idle or a security boundary.
Unconfirmed termination blocks verification; a final failure receipt still lets agent
close release its connection for owner teardown.
Expand Down
110 changes: 58 additions & 52 deletions fern/versions/latest/pages/agent-server/add-or-update-a-harness.mdx

Large diffs are not rendered by default.

10 changes: 6 additions & 4 deletions nemo_gym/agent_utils/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@
implements the shell protocol over session control files. All three run on the
agent server.

Only process_supervisor.py is copied from this package into the task sandbox. It
uses the standard library to enforce deadlines, stop and reap the harness process
group, and write its termination receipt. Explicitly detached task services remain
for the sandbox owner to stop. Core sandbox APIs and providers live in
Only process_supervisor.sh and harness_runner.sh are copied from this package into
the task sandbox; they need bash, not Python. The supervisor enforces deadlines, stops
the harness process group and writes its termination receipt. A forced stop also kills the
harness's descendants in other groups and processes that inherited its marker; processes
that had left the harness before the stop, and detached processes after a natural exit,
remain for the sandbox owner to stop. Core sandbox APIs and providers live in
:mod:`nemo_gym.sandbox`.
"""
77 changes: 77 additions & 0 deletions nemo_gym/agent_utils/harness_runner.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Harness runner uploaded into task sandboxes; runs under process_supervisor.sh with only bash.
#
# usage: harness_runner.sh PARAMS
#
# PARAMS is bash source written by shell_assignments() on the agent server. It sets:
# cwd, command (array), env (array of KEY=VALUE), stdin, stdout, stderr
# and optionally:
# setup bash code run first, before env is applied
# unset_env array of extglob patterns naming inherited variables to drop
# events file receiving each JSON object line of stdout as [seen_at, object]
# Writes runtime.json beside PARAMS. Exits with the command's status, 125 when the runner fails before
# starting it (including an exit from setup), or 126 or 127 when the command could not be started.

# An inherited BASH_ENV or SHELLOPTS must not stop the runner at an expected failure.
set +o errexit +o nounset +o pipefail
IFS=$' \t\n'
shopt -s extglob
# Inherited variables with these names must not configure the runner or leak into the CLI.
unset -v cwd command env stdin stdout stderr setup unset_env events
trap 'exit 125' EXIT
source "$1" || exit
printf '{"hostname": "%s", "pid": %d}\n' "$HOSTNAME" "$$" >"$(dirname "$1")/runtime.json"
eval "${setup-}" || exit
for pattern in "${unset_env[@]}"; do
for name in $(compgen -e); do
[[ $name == $pattern ]] && unset "$name"
done
done
((${#env[@]})) && export "${env[@]}"
cd "$cwd" || exit
# A failed redirection does not end bash at exec; the CLI did not start.
if [[ -z ${events-} ]]; then
trap - EXIT
exec "${command[@]}" <"$stdin" >"$stdout" 2>"$stderr"
exit 126
fi

# Stamp stdout lines as they appear. A file, unlike a pipe, cannot be held open by a detached tool.
: >"$stdout" && exec 3<"$stdout" 4>"$events" || exit
trap - EXIT
# The runner's own descriptors stay out of the CLI.
{ exec "${command[@]}" <"$stdin" >>"$stdout" 2>"$stderr" 3<&- 4>&-; exit 126; } &
cli=$!
trap 'kill -TERM "$cli" 2>/dev/null' TERM
pending=""
emit() {
# ${1%$'\r'} would try the suffix at every position of a long line.
local line=$1
[[ ${line: -1} == $'\r' ]] && line=${line%?}
[[ $line == {* ]] && printf '[%s, %s]\n' "$seen" "$line" >&4
}
# Stamp new complete lines, keeping a partial one for the next pass. Bash < 5 has whole seconds;
# bash < 4.2 has no strftime in printf and forks date.
stamp() {
local line
seen=${EPOCHREALTIME/,/.}
[[ -n $seen ]] || printf -v seen '%(%s)T' -1 2>/dev/null
[[ $seen =~ ^[0-9.]+$ ]] || seen=$(date +%s)
while IFS= read -r line <&3; do
emit "$pending$line"
pending=""
done
pending+=$line
}
while kill -0 "$cli" 2>/dev/null; do
stamp
sleep 0.05
done
wait "$cli"
status=$?
stamp
emit "$pending"
exit "$status"
227 changes: 0 additions & 227 deletions nemo_gym/agent_utils/process_supervisor.py

This file was deleted.

Loading
Loading