[adil-a]

Stacked on #4872 (M38 second slice). Follow-up to M34 #4819: the per-step Harbor features the loader refused, from Felipe's gap analysis (gaps 3 and 6).

feat(harbor): per-step workdir/setup.sh, healthcheck, artifacts and separate verification for [[steps]] tasks (M34 follow-up)

Summary

M34 runs a Harbor [[steps]] task as turns on one sandbox, but its loader refused every per-step feature of Harbor 0.22.0's MultiStepTrial by name: steps/<name>/workdir/ and setup.sh, [steps.healthcheck], per-step artifacts, and per-step (or task-level) separate verification. The hub's gabeorlanski/slopcodebench (36 tasks) could not load at all because every checkpoint declares artifacts. This stack implements the five follow-ups, one commit each, with Harbor 0.22.0's semantics (trial/multi_step.py, trial/trial.py, models/task/verifier_mode.py) cited in code and docs. The one Harbor file Gym still cannot honour, the seed trajectory.json, is handled as Harbor's own oracle handles it: the oracle runs the task, every other agent is refused at seed naming the file.

What changes

1. steps/<name>/workdir/ and workdir/setup.sh (environment_servers/harbor/steps.py, new HarborStepsMixin): before each turn's agent session, after the step's user switch, the folder is merged into the working directory (an upload over what previous turns left, as root with the folder's modes, the digest's ignored names left out) and bash <workdir>/setup.sh runs from the working directory as the turn's agent user, within the new step_setup_timeout_s (unset by default: the step's own agent budget, [steps.agent].timeout_sec else [agent].timeout_sec; Harbor bounds the script only by the trial timeout). The bound is counted in the episode budget for every step that ships the script. A non-zero exit (or a run past the bound) is the step's failure, as Harbor's step exception_info: the turn is recorded with failure_kind: harbor:step_setup_failed, no response and no verifier result, no agent session or verifier runs for it, and the episode stops there; under mean the turns before it are averaged (Harbor's _aggregate_step_rewards), under final the episode is masked with that kind, since Harbor's trial then has no reward at all. A first turn that fails there is a terminal agent-stage failure with that kind. An entry of the folder the sandbox refuses to unpack over what the previous turns left (a non-empty directory at a shipped file's name, which the previous turn's agent can plant in the working directory it owns) is the step's failure as well, harbor:step_workdir_failed, as Harbor's docker cp refuses the same mismatch and its trial errors: never a retryable outage, so an agent cannot turn every later attempt into an unscored retry. A symbolic link the previous turn's agent planted at the name of an entry the folder ships is removed, as root, in the same shell line right before the extraction (upload_dir(replace_links=True)): GNU tar and docker cp replace such a link with the shipped entry, BusyBox tar (Alpine images) keeps it and writes the shipped files through it, as root, wherever it points; a real-BusyBox test shows the diversion without the option and the merge at its own names with it, on both tars. Nothing stops what an earlier turn left running before the script runs, so a step that is root after a non-root step can have its setup.sh replaced in that window (Harbor shares it): documented, and the loader warns about such a task. HarborTurnResult.response is optional for such a turn. A setup.sh beside instruction.md (not a Harbor file) only warns.
2. [steps.healthcheck]: the seed's healthcheck loop is extracted into HarborEntrypointMixin._poll_healthcheck (Docker's start-period and retries semantics) and reused per turn after the setup (not after a failed one), from the working directory as the turn's agent user. Only a check that ran and never passed (HealthcheckFailed) is the step's failure (harbor:step_healthcheck_failed); a sandbox stopped under the poll, or a provider that could not run the command, is a retryable outage, as for setup.sh and the collect hooks. An attempt that overruns its timeout_sec counts as one failed attempt (Docker's HEALTHCHECK semantics, and what the seed's task-level loop does); Harbor 0.22.0 differs there (its Docker exec raises on the timeout and no caller of run_healthcheck catches it, so the first such attempt errors the whole trial), which the code, the tests and the docs state instead of claiming parity. The task's [environment.healthcheck] ran at seed and is not run again per step, as Harbor runs it once at environment start. The episode budget counts the check's worst case per step (start period, the attempt that may start just inside it and its start_interval_sec sleep, then the counted attempts with an interval_sec sleep between them, each timeout_sec plus the exec grace), checked against a simulated poll at its worst alignment.
3. Per-step artifacts, collected in shared mode too: Harbor's per-step collection pass after the step's agent and before its verifier (_collect_step_artifacts): the task's [[verifier.collect]] hooks and then the step's own [[steps.verifier.collect]] hooks (Harbor's _collect_hooks_for; the step's table now loads, HarborStepVerifier.collect, and is budgeted), /logs/artifacts, the task's artifacts and the step's own, staged under numbered roots in that turn's verifier logs folder (artifacts/NN/<destination>, the separate path's layout); the hooks run once per turn. Every pass, the separate path's included, writes artifacts/manifest.json (index, source, service, origin, destination, type, status) as Harbor's artifact handler writes its manifest. A collection the sandbox cannot serve masks the turn as an outage. Fixture: the real task.toml and Dockerfile of gabeorlanski/code_search from the pulled slopcodebench dataset.
4. Per-step separate verification, modes mixed by step: HarborTurn.verifier_mode resolves as Harbor's resolve_step_verifier_mode (the step's explicit mode, else separate with a [steps.verifier.environment], else the task's); the turn's verifier image follows the single-step rules on the step's effective environment ([steps.verifier.environment], else [verifier.environment], else a copy of [environment]). _run_separate_verifier is split into the collection-and-stop phase and _verify_separately; a separate turn runs the latter on a per-turn view of the task (_turn_verifier_view), fed the turn's collection pass, in a verifier sandbox created and stopped around that turn under the episode's verifier key. The agent's sandbox keeps running for the next turn, except on the last turn of a task whose last step verifies separately: there, as Harbor's _run_step does on the last step of a separate-mode task, main is stopped before the sidecar pass and the whole agent side before the verifier grades. A shared turn's collected archives are removed once its pass ends (it restores nothing). separate_verifier_for_artifacts applies per turn; the seed checks every separate turn's image, env templates and user group; the budget counts a verifier sandbox start (and overlay) per separate turn; each turn reports verifier_mode, the episode's is the shared one or mixed. A step's verifier runs within its own [steps.verifier].timeout_sec, else Harbor's step default of 600 s, never the task's [verifier].timeout_sec (Harbor's StepConfig.verifier is a full VerifierConfig with that default, on 0.22.0 and main alike), in both modes and in the budget; the loader warns once per task, naming the steps the task's figure does not reach. A step that says environment_mode = "shared" and declares a [steps.verifier.environment] is refused at load, as Harbor's validator refuses it. The pre-turn reset of /tests, /logs/verifier and the server's transcript runs only after a turn that verified in the agent's sandbox; after a separate turn the image's /tests stays, as under Harbor 0.22.0's separate mode and Harbor main's gate on a preceding shared verifier. A step whose verifier Harbor would build from its own steps/<name>/tests/Dockerfile is refused naming the image to declare (prepare builds tests/Dockerfile only; that build is now planned whatever the task's own mode, verifier_build_environment). A [verifier] network restriction needs every step separate. The tests patch _start_with_pull_retries, so the real _create_verifier_sandbox runs on the turn's view and the spec it hands the provider is what they assert.
5. Seed trajectory.json, run for the oracle alone: the loader records steps/<first step>/trajectory.json (a multi-step task's top-level file and a later step's are ignored, as Harbor's _task_trajectory_path ignores them) or a single-step task's trajectory.json as HarborTask.seed_trajectory, with a warning. Harbor 0.22.0's Trial._resolve_load_trajectory loads that file into the agent's session for every agent but the oracle and nop; no harness here can be seeded with it yet, so the environment server runs the task for the oracle (gym dataset validate works as Harbor's does) and refuses it at seed, terminally, for any agent whose implementation is known and is not the oracle, naming the file, before any sandbox is paid for; an agent block the server cannot read (several implementations under the instance) warns and runs, as the solution placement treats that shape. The file stays in the folder digest and package hash.

Left untouched, as decided: the dataset multiplier, resume across steps. (The unset step verifier timeout no longer falls back to the task's: Harbor never does, see 4.)

Docs: environment_servers/harbor/README.md (Multi-step tasks section, feature table), environment_servers/harbor/configs/harbor.yaml (step_setup_timeout_s) and fern/versions/latest/pages/data/multi-step-tasks.mdx describe every behaviour, with what Harbor 0.22.0 does beside it.

Test plan

• environment_servers/harbor/tests responses_api_agents/harbor_oracle_agent/tests -k "not partial_eval and not buildah and not validate_ and not real_image": 535 passed on the base, 564 passed at the head (+29: workdir/setup.sh and its bound, the refused workdir entry, the failed-setup roll-up under mean and final, healthcheck, its outage cases and its budget against a simulated poll, artifacts and manifest, a step's own collect hooks, archive removal, slopcodebench checkpoints, separate turns with the last-turn stop and the real sandbox spec, mixed modes, forced separate per turn, budget per separate turn, the seed trajectory for the oracle, a model agent and an unreadable agent block). Every commit's suites green from a git-archive copy (see the report).
• tests/unit_tests/test_harbor_tasks.py (+ test_harbor_build_plan.py, test_harbor_image_build.py, test_harbor_run_summary.py, test_harbor_validation.py, test_sandbox_transfer.py): 472 passed at the head (the transfer file's BusyBox tests run where busybox is installed, as on the test node).
• ruff check, ruff format, pre-commit run --files <changed> at every commit.
• Live (OpenSandbox, oracle, python:3.12-slim): gym dataset validate of four tasks: stepfeat (four steps: workdir/ + setup.sh, [steps.healthcheck], per-step artifacts in both modes, a step-level [[steps.verifier.collect]] hook whose output the separate verifier reads, and a separate last step graded after the agent side is stopped), setupfail (a second step's setup.sh past step_setup_timeout_s = 5 under final: the episode is masked with harbor:step_setup_failed), seedtraj (a first step's trajectory.json and an ignored top-level one, run by the oracle) and plantdir (the first step's oracle solution leaves /app/setup.sh/x/ where the second step ships workdir/setup.sh: the second turn is harbor:step_workdir_failed, unmasked, under mean the episode scores the first turn, nothing retried). Result at the head: stepfeat, seedtraj and plantdir oracle solved 1.0, setupfail masked as intended. See the report's runC section, including the provider observation (OpenSandbox reports a killed command as exit -1 signal: killed and a failed tar as exit status 2 without the command's stderr).
• Live (OpenSandbox, oracle, BusyBox tar): gym dataset validate of the same four tasks plus plantlink on bash:5.2 (Alpine plus bash, BusyBox tar 1.37.0, ENTRYPOINT []), whose first step's oracle plants ln -s /etc /app/fixtures and ln -s /etc /app/sub/deep where the second step ships workdir/fixtures/input.txt and workdir/sub/deep/x.txt: the second turn's test.sh asserts both paths are directories holding the shipped files, nothing reached /etc and setup.sh ran; result plantlink oracle solved 1.0 (the other four as before). See the report's runD section.
• Live, the turn cleanup against Harbor (OpenSandbox, oracle, python:3.12-slim, no Compose): a two-step task whose task-level tests/ ships a hidden marker, whose second step's solution records what /tests, /logs/verifier and /logs/agent hold when its agent starts, and whose second step's test.sh rewards 1 only when none of the first turn's grading files were there (multi_step_reward_strategy = "final"). Harbor 0.22.0's oracle: the second step saw /tests/hidden_marker.txt, /tests/test.sh and three files under /logs/verifier (its reset runs only right before each step's shared verifier), reward 0. Harbor main (b4732c06, 0.24.0, a throwaway venv): /tests and /logs/verifier empty, reward 1 (its _prepare_step resets them when the previous step ran a shared verifier). Gym at this head through gym dataset validate: /tests and /logs/verifier empty, no trajectory.json, reward 1 (the pre-turn reset of M34's 3c1887440, gated as in 4 above). All three sides agree on /logs/agent: no transcript reaches the second agent. Harbor's OpenSandbox environment reached this deployment only with its self-signed certificate trusted through SSL_CERT_FILE and the proxy API-key header added by a launcher shim (both things Gym's provider does by default; nothing in Harbor's step logic changed). See the round-4 report.
• gym dataset pull harbor:gabeorlanski/slopcodebench (36 tasks): the base loader refused all 36 (per-step healthcheck and artifacts are not supported yet); the real code_search task.toml loads and runs its five checkpoints in the unit tests (its Dockerfile sets SHELL, so it is a prepare-built image: not run live here).

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally.
• Documentation added or updated.
• Pre-commit checks pass locally on the changed files.
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off.

Test fixture provenance

The test fixture tests/unit_tests/fixtures/harbor_slopcodebench/code_search/ (task.toml, environment/Dockerfile,
step instructions and stub test.sh scripts) is synthetic: it reproduces only the shape of the Harbor hub dataset
gabeorlanski/slopcodebench (five steps with per-step /app artifacts, min_reward on two reward keys, a build-mode
Dockerfile with a SHELL line). Its step names (scaffold, lexer, index, query, report), budgets,
resources, exclude list and reward gates are the tests' own; the only lines it shares with the dataset's task.toml are
TOML table headers and schema keys (schema_version, artifacts = [], workdir = "/app", difficulty, category).
No text of the dataset's task.toml, Dockerfile, instructions or tests is copied, so no external source URL, revision
or third-party license applies to any file of this PR.

Migration note for datasets whose sidecars write verifier inputs

The shared verifier's reset now deletes every undeclared file under /logs/verifier before test.sh runs, so an
already-pulled dataset whose sidecar writes an input there (kumo: the verifier service appends
/logs/verifier/actions.jsonl, which main's tests/score.py reads) scores 0, unmasked, until its dataset.toml
declares the file:

[gym.tasks."<task id>".verifier]
inputs = ["actions.jsonl"]

Declared names are literal top-level file names (no path separators, no *, ? or [: the server keeps them with
find -path, which matches a glob) and never a grader output name. The declaration is part of the task digest. No
dataset.toml in this repository declares actions.jsonl, so the kumo dataset's owners must add it out of band.

Behaviour notes

• Later step preparation failures (workdir/setup.sh/healthcheck) count the step as 0 and stop the episode.
• A step without [steps.verifier].timeout_sec is graded within Harbor's 600 s step default, not the task's
  [verifier].timeout_sec, which changes existing grading budgets.
• The shared verifier's reset now empties /logs/verifier (except the task's declared [verifier].inputs) before
  test.sh, in single-step shared tasks too.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[adil-a]

/ok to test 78a6161

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4951-33234040328a.docs.buildwithfern.com/nemo/gym

[adil-a]

/ok to test 2f5bab5

[adil-a]

/ok to test 55b208c

[adil-a]

/ok to test 1b41124

[adil-a]

/ok to test 3657d15

[ffrujeri]

If a non-last turn's verifier stop fails, _start_with_pull_retries overwrites self._sandboxes[_verifier_key(...)] on the next separate turn and the old sandbox leaks until ttl_s. Could you retry the stop, or mask the turn as provider_unavailable, before reusing the key?

[adil-a]

/ok to test 2e24547

[adil-a]

Fixed in 2e24547 (folded into the per-step separate verification commit): before a separate turn's verifier is created under the episode's verifier key, a previous turn's verifier still owned under that key (its stop raised in the finally) is stopped again once; if that retry fails too, the turn is masked provider_unavailable and the key is left alone, so the old sandbox stays registered for the episode cleanup's close_sandbox, which stops it. Tests in test_multi_turn.py: a stop that fails once is retried and both turns grade (stop calls [2, 1], table empty at the end); a stop that keeps failing masks the second turn, starts no second verifier, and the same sandbox object is the one stopped at cleanup. Both failed before the change. Also widened the test_a_seed_slower_than_the_ready_timeout_leaves_the_turns_their_budget margins (0.3 s step budgets to 0.5 s, 0.8 s to 1.6 s headroom) in this commit, since the test file is owned here; the flake did not reproduce locally in 20 runs, so the next shard-0 run is the check.

[adil-a]

/ok to test 924ccb0

[adil-a]

/ok to test 7b2dcb6

[ffrujeri]

Thanks for the fix, LGTM.

[adil-a]

/ok to test 11757aa

[adil-a]

/ok to test f7e0d51

[yuhezhang-ai]

Thanks for extending Harbor's multi-step support. I reviewed f7e0d51a1054bd219028a7ea043957aa555a1dbc against its immediate stack parent #4872 at 816e65bcb2cfc0312cc79dee2e4c1fa0e78c7051.

Please address the five grading/isolation findings before merge: retained test reports can produce a false passing score, a step's environment network restriction is dropped, separate steps miss agent-entrypoint failures, shared opt-out steps can use the agent's user, and artifact-forced separate steps can fail image resolution. The four remaining inline comments are marked non-blocking.

Please also align the PR description with the current behavior: later step preparation failures count as zero, the 600-second step verifier default changes existing grading budgets, and the partial log cleanup affects single-step shared tasks.

Validation used existing tests and focused local reproductions; no live sandbox provider or model rollout was run for this review. Codex independently checked the findings. Claude delivered a second opinion, but its execution audit remained incomplete after a malformed tool call and its filesystem checks encountered sandbox restrictions.

[adil-a]

/ok to test 3e51c0b

[adil-a]

/ok to test 3f2c52c

[ffrujeri]

LGTM. The five blocking findings from @yuhezhang-ai look fixed at this head, each with a test; leaving it to them to clear their request.

[yuhezhang-ai]

Thanks for addressing the review. Rechecked 3f2c52ccaf7df9356f63de0c065ae53677bbbff8 against its immediate parent #4872: the five blocking findings are fixed, with regression coverage, and the four non-blocking comments have also been addressed.

The required checks and focused verification pass. Approving this change; end-to-end RL testing on main and any resulting patches can follow in separate PRs, as discussed.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40551394222

[adil-a]

/ok to test faf15e2

[yuhezhang-ai]

Rechecked faf15e267acaad86efa26475b4c257412bb6487d against the current stack parent #4872 (8034d4568ce33b53ebab58b9f49a2d9d9694d837). This is a rebase of the previously approved version: all six patches and all 35 affected files are unchanged. The five blocking fixes remain intact, and all review threads are resolved.

Required checks and the current unit/server suites pass. Renewing approval after the rebase; end-to-end RL validation on main and any resulting patches can follow in separate PRs, as discussed.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40612812940

[adil-a]

/ok to test 49e0dc5

[adil-a]

/ok to test 3323404

[yuhezhang-ai]

Rechecked 33234040328a3366b3e6f55f2bd1685058cfd091 against the current stack parent #4872 (fb511ae6d9211418beec4711c68becd115fb16be). All six PR changes remain equivalent to the previously approved version, and the earlier blocking fixes are intact. No remaining blocking findings.

Required checks and the current unit/server suites pass. Renewing approval after the rebase; end-to-end RL testing on main and any resulting patches can follow in separate PRs, as discussed.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40727571028