[yaoyu-33]

What does this PR do?

Gives cline_agent the agent-session lifecycle (seed, Responses activation behind the session cookie, close) that the merged sandboxed harnesses have, so Cline runs inside the benchmark's task sandbox under a single_agent_turn Environment Server.

• Runtime without Node. Cline 3.0.55 ships a self-contained glibc binary per platform in @cline/cli-linux-{x64,arm64}. At seed, install_cline_runtime.sh downloads that npm tarball inside the sandbox, checks it against a SHA-512 pinned in the adapter (CLINE_RUNTIME_SHA512 in sandbox.py, npm's own dist.integrity), and publishes only the binary in /tmp/nemo-gym-cline-runtime-<version> with one rename; later sessions in the same sandbox reuse it. Publication is serialized with a 60-second-bounded flock; without flock, cold installation/reuse work but stale-runtime replacement fails safely without deleting another session's runtime.
• Seed connects to a borrowed direct sandbox, or starts an agent-owned one when sandbox_provider is set. An agent-owned sandbox gets a default ttl_s (install timeout + run timeout + 10 minutes, unless sandbox_config sets one), the provider's default metadata, then sandbox_config metadata, then gym-agent, gym-session and gym-rollout labels. Seed refuses (HTTP 422, AgentSeedRejectedError) required tool grants, a missing or unpinned cline_version, a missing model_server, and host-only settings (command, env, extra_args, repo_dir, openai_*).
• Activation writes Cline's provider settings directly: the rollout-prefixed Model Server URL plus an x-session-id header that owns every model attempt. It also turns off telemetry and self-update, then runs Cline in SandboxAccess.workdir under the shared process supervisor (harness_runner.sh). --config, --data-dir and the CLINE_* state paths point into a private session directory, and tool shells keep the task's HOME/XDG_*. A developer message counts as the system message, and metadata: {} is accepted.
• Outcome, by Cline's own exit rule, read from Cline's final run_result only:
  • finishReason: completed gives completed.
  • finishReason: error gives failed, with classify_outcome reading Cline's terminal error. "Model returned empty response" is agent_invalid_output.
  • A context overflow, the output-token cap, an abort, or a supervisor deadline with recorded model output gives incomplete (still verified). A deadline without recorded model output gives failed/unknown with a timeout marker, so it is retried by default rather than scored as an empty model zero. No model transport cause is invented.
  • No run result, including a done event without one, fails the activation.
  • A rollout stop before launch, or before the runner opened Cline's output, is graded as a stop with an empty transcript instead of failing the activation.
• Close returns an AgentObservationBundle (invocation, per-tool timing from Cline's event timestamps, sandbox record with sandbox_id). Usage comes from Cline's final run_result; cache reads are already inside openai-compatible input counts, so they are reported as cached_tokens and not added again.
• Host path (/run, sessionless /v1/responses, harness_agent, anyswe_agent) is unchanged, except that Cline now installs on the first host call (shared, retried after a failure) instead of at server startup.
• configs/cline_agent.yaml becomes the session config (concurrency: 32); the host config moves to configs/cline_local_agent.yaml with the same cline_agent key (environments/cline_math updated). cline_agent joins AGENT_SESSION_AGENT_TYPES, and the adapter's pyproject.toml depends on nemo-gym[dev,sandbox] (like Kilo and Pi), with its uv.lock regenerated by uv 0.13.4.
• Docs: the adapter README (the Terminal-Bench 2.1 recipe is labeled as the unsupported development pairing the smokes used), and Cline rows in the Tools table of add-or-update-a-harness.mdx and the Sandboxed harnesses table of integrate-existing-agents.mdx.

Validation

Current review corrections at cb1d35f5e (November 6)

• Preserves the distinction between the supervisor deadline and an independently observed provider-wait failure. The latter survives in the canonical failure report, including original causal event identities, and is forwarded on close without changing the gradable partial response.
• All runtime publication is under the bounded lock when available; no-lock stale replacement refuses safely. Linux tests cover concurrent publication, ready reuse, stale refusal without root/package-manager tools, and lock wait failure.
• HTTP-looking error text remains unknown with no invented status and default retry. Output-cap endings remain incomplete without a typed failure. README and regressions state those limits.
• Agent-owned network restriction remains unsupported and is documented; no new network-policy feature is claimed.

Checks:

• python -m pytest responses_api_agents/cline_agent/tests -q -o addopts='': 148 passed on Linux; 140 passed, 8 Linux-only skips on macOS.
• Original-source regressions fail for empty deadlines, missing deadline markers and discarded provider diagnostics; fixed tests pass.
• Changed-file pre-commit, git diff --check and DCO sign-off passed.
• Current-source stock TB2.1 regex-log smoke, Resources-owned Docker sandbox, 300-second harness budget: the N35 attempt returned native Too Many Requests and remained an ungraded, retry-eligible failure. A second attempt with NIM minimaxai/minimax-m3.1-flash made a successful model response and two completed tool calls, then reached the deadline. Its partial response stayed incomplete, verification completed unmasked with reward 0.0, strict --require-sandbox evidence and profiling passed. Health was 0 flagged, 1 unobserved, because the missing final run result leaves aggregate usage unknown. Both attempts' sandboxes and Gym listeners were removed.
• The second run validates deadline/partial-work plumbing; it is not a successful task solve or full benchmark qualification.

Follow-up smoke at 9c2253186 (November 6)

• Corrected the README and source comment: a fixed 300-second per-request bound does not apply universally to Gym's streaming-compatible route. The configured agent timeout is the supervisor's wall-clock budget. This commit changes only documentation/comments; Python AST comparison confirms unchanged execution logic. Changed-file pre-commit, git diff --check, and DCO sign-off passed.
• Successful current-head smoke: stock TB2.1 regex-log, Resources-owned Docker sandbox, NIM nvidia/nemotron-3.5-super-vl-120b-a12b, 900-second agent budget, one repeat/concurrency 1. Response completed, verifier reward 1.0, unmasked, evaluation_completed: true; 267.82 seconds end to end. All 8 model calls succeeded. Seven tool calls include two missing-Python failures followed by package installation and recovery. Response usage exactly matches capture totals: 167,504 input, 17,898 output, 185,402 total tokens (138,752 cached input).
• Health 1 passed, 0 flagged, 0 unobserved; model responses 0/8 faulted. Strict --require-sandbox evidence and profiling passed. Native runtime exited 0; task containers and reserved Gym listeners were empty after cleanup.
• The separate MiniMax diagnostic at cb1d35f5e, with the same task/sampling and a 900-second agent budget, still timed out. Its first two calls succeeded in 107.66 and 223.85 seconds; a model call inside Cline's spawn_agent remained pending beyond 352 seconds and did not return before the supervisor deadline. Its reward was 0.0 because the answer file was absent. Deadline/partial-tool observations, verifier execution, strict sandbox evidence, profiling, and cleanup behaved as expected; this is not a successful solve. The observations locate an outstanding model call but do not distinguish upstream queueing, generation, or network waiting.
• Original N35 failure was confirmed from capture as structured upstream HTTP 429 after six successful calls. These runs did not reproduce an adapter execution defect. They are integration smokes, not full benchmark qualification; subagent-hierarchy and reasoning-usage gaps remain explicit.

At 0fb78fd32 (review fixes in f2e0ec180 and a docstring fix, on top of 98ed91450)

On macOS. Each fix has a regression test, and each of those tests fails with its fix reverted.

• responses_api_agents/cline_agent/tests, in the adapter's locked venv: 131 passed, 5 skipped (the installer tests run only on Linux).
• tests/unit_tests/test_session_contract_repo.py: 2 passed. tests/unit_tests/tooling/test_add_legacy_agent_environment_servers.py: 122 passed. tests/unit_tests/test_fern_docs_links.py: 29 passed.
• pre-commit run --files <the 8 changed files>: passed.
• The new staging test runs the real staged runner directly, because the process supervisor needs Linux /proc. Neither it nor the rollout-stop tests have run on Linux yet.

At 98ed91450 (after merging current main, which brings in the locked component layout from #4837)

macOS with uv 0.13.4 on PATH:

• gym env test +entrypoint=responses_api_agents/cline_agent (fresh locked venv): 114 passed, 5 skipped (the installer tests run only on Linux).
• tests/unit_tests/test_install_cline_runtime.py on Linux (a workstation, before the merge; the script is unchanged by it): 5 passed.
• uv run --no-project --python 3.13.14 python scripts/component_locks.py: 269 projects checked, 0 failed. uv lock --project responses_api_agents/cline_agent --check passes.
• pytest tests/unit_tests/test_session_contract_repo.py tests/unit_tests/config/test_global_config.py: 231 passed.
• pytest tests/unit_tests/tooling/test_add_legacy_agent_environment_servers.py: 122 passed.
• pytest tests/unit_tests/cli/test_locked_env.py tests/unit_tests/cli/test_component_compatibility.py tests/unit_tests/cli/test_validate_server_configs.py: 50 passed.
• pytest responses_api_agents/harness_agent/tests: 57 passed. pytest responses_api_agents/anyswe_agent/tests -k cline: 3 passed.
• pre-commit run --files <the 15 changed files>: passed. It was scoped to these files so auto-fix hooks do not touch unrelated files.
• Mutation spot-checks: dropping x-session-id, mapping the deadline to failed, accepting required grants, and ignoring failed tool entries each fail a test.
• gym env resolve for environments/cline_math/config.yaml with the renamed host config.
• Harness conformance probes were not run. The adapter claims the Evaluation level, where they are optional; the saved-rollout evidence check below was run.

Rollouts

Both smokes ran before the merges of main and before the review fixes in f2e0ec180. The merges changed only the adapter's dependency files under responses_api_agents/cline_agent (the pyproject.toml/uv.lock layout). The review fixes change rollout-stop grading, the owned sandbox's TTL and labels, and how the activation status is read. The current-source November 6 smoke is recorded above; the following results remain historical.

1. Borrowed task sandbox, Terminal-Bench 2.1, Docker on a workstation (smoke-docker-05, commit 74a747a59)

• Why not the Slurm cluster: its Enroot provider cannot lend a Resources-owned sandbox to an in-sandbox harness. A cluster job failed every TB2.1 seed with provider 'enroot' does not support serialize()/connect(), and OpenSandbox pods cannot reach the cluster nodes.
• Model: minimaxai/minimax-m3.1-flash on NVIDIA NIM through the Gym vLLM Model Server, with reasoning_field: reasoning_content and max_tokens: 16384. Earlier attempts on nvidia/nemotron-3-ultra-550b-a55b lost every row to NIM 429s (a shared key under heavy use), and z-ai/glm-5.3-flash stalled on NIM 504s.
• Settings: single_agent_turn, resources_tool_transports: [], agent timeout: 900, --concurrency 1, --allow-unsupported-pairing. TB2.1 tasks at 7131e437.

Task	Response	Tool calls	Reward
regex-log	incomplete (900 s deadline)	7	1
log-summary-date-ranges	completed	8	1
fix-git	completed	14	1
openssl-selfsigned-cert	completed	8	1
git-leak-recovery	incomplete (900 s deadline)	14	0

• Grading: all 5 rows have evaluation_completed: true and harness_execution: sandbox.
• Usage: on completed rows, response usage equals Gym's captured calls exactly.
• Health: 3 healthy, 0 unhealthy, 2 unobserved. The two deadline rows report usage as unknown, so the token-count check has nothing to read.
• Strict evidence: inspect_harness_conformance.py --require-sandbox fulfills TE-1 to TE-9.
• Cleanup: no task containers remained after the run.
• Earlier run on ad61a8c88: the same tasks gave rewards 0, 1, 1, 1, 0, but TE-5 failed on the tool call cut off by the deadline, and one deadline row showed a token-count mismatch from Cline's lagging running total. Both are fixed in 74a747a59.

2. Agent-owned Enroot sandbox, cline_math (math_with_judge), Slurm cluster (commit ad61a8c88)

• Setup: single_agent_turn_legacy, NIM nvidia/nemotron-3-ultra-550b-a55b.
• Result: 3 of 5 graded (rewards 1, 1, 0), all completed, 3 to 6 tool calls each. Two rows failed on NIM 429 after Cline's retries; they were reported as retryable unknown outcomes and not graded as zeros.
• Health: 3 healthy.
• Evidence: fails only TE-6 evaluation_completed, a field math_with_judge does not emit.
• Cleanup: the agent close stopped every owned sandbox.
• Source: this run predates the usage and cut-off-tool changes in 74a747a59, and the owned-sandbox TTL and labels in f2e0ec180.

Compatibility and benchmark impact

• Config path: configs/cline_agent.yaml is now the agent-session config (resources_server: null, num_workers: 1, concurrency: 32). Configs that imported it for the host /run path must import configs/cline_local_agent.yaml instead; it keeps the same cline_agent key and values.
• Host install timing: the host path installs Cline on its first call instead of at server startup.
• Composition: cline_agent is now in AGENT_SESSION_AGENT_TYPES, so an unbound cline_agent composes with a session Environment Server rather than a legacy relay.
• Benchmarks: none changed. Cline is not added to TB2.1's allowed_agents; that needs the full TB2.1 evaluation and the benchmark owner's review.

Known limits

• Capability: Evaluation level; no token IDs or logprobs.
• Sandbox requirements:
  • Linux x86_64/aarch64 with glibc (musl is refused).
  • HTTPS access to registry.npmjs.org.
  • Root, or a preinstalled curl/tar/gzip/sha512sum.
  • A new cline_version needs its tarball checksums added to CLINE_RUNTIME_SHA512.
• Left in the sandbox after close: the runtime directory, packages the seed installed, and Cline's checkpoint commits under refs/cline/checkpoints/. Cline's checkpoints leave the working tree and branches untouched.
• Cline 3.0.55 behavior:
  • The prompt is read only from argv or a pipe, so it travels as an argument: sessions refuse prompts over 128 KiB (after quoting), and the prompt is visible in the sandbox's process list.
  • --timeout does not bound a --json run; the process supervisor enforces the deadline.
  • Do not rely on a fixed 300 s per-request bound on Gym's streaming-compatible route: a live call remained pending longer while Gym sent keepalives. The process supervisor enforces the configured agent timeout for the entire run. If Cline emits The operation timed out., it is classified as model_transport_error. Setting timeout in Cline's provider settings breaks connections; use the agent timeout for the wall-clock budget.
• Tools: MCP and direct_http grants are not wired into Cline. Required grants are refused and optional ones skipped.
• Model Server setting: the vLLM Model Server's default reasoning_field: both makes some NIM endpoints (MiniMax) reject replayed reasoning; set reasoning_field: reasoning_content there.
• Host path: it still adds cacheReadTokens to inputTokens, which double-counts cached tokens with the openai-compatible provider. Left as is in this PR.
• Not done here:
  • Current-head agent-owned sandbox smoke (the current-head borrowed TB2.1 smoke is recorded above).
  • TB2.1 8-repeat full evaluation.
  • The GDPVal-AA v2.1 / TB4.0 pairings.
  • Conformance-probe registration.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Documentation added or updated, or N/A with justification. The adapter README and the harness catalog tables are updated.
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass). Run on the changed files, not --all-files.
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yaoyu-33]

/ok to test c7b36e2

[yaoyu-33]

/ok to test d533de1

[yaoyu-33]

/ok to test 98ed914

[yaoyu-33]

/ok to test 0fb78fd

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5546-01e0bbbeefd1.docs.buildwithfern.com/nemo/gym

[svcnvidia-nemo-ci]

SHIP WITH CARE

I found one RISK: concurrent installers can remove a runtime another session has just published. The inline finding describes the failure, repair invariant and regression checks.

I inspected all 17 changed files and traced the affected session, model-capture, outcome, configuration and host-call contracts. One delegated report wasn't available; I independently covered its configuration, host compatibility, dependency and documentation assignment.

This was a static review. I didn't execute tests or rollouts, and the supplied real-model smokes predate the final fixes.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[svcnvidia-nemo-ci]

RISK — Concurrent replacement can delete a newly published runtime

Failure: With an existing partial runtime, two sessions installing the same version into one sandbox can both fail the initial mv -T and both pass ! ready. Installer A then removes the partial directory, publishes its verified runtime and returns successfully. Installer B subsequently executes this rm -rf, deleting A's ready runtime. If A's activation starts before B's rename, Cline cannot launch and the activation fails with no run result. The publication branch therefore doesn't provide the concurrent publication guarantee its comment describes. The blast radius is sessions sharing that runtime path.

Correction: Could you serialize the final readiness check, stale removal and publication using a per-runtime lock outside the replaced directory, and recheck readiness after acquiring it? Downloads and checksum verification can remain concurrent. The invariant is that an installer must never remove a verified runtime another installer published. The lock also needs bounded waiting and release on failure. The available sandbox tools don't establish a portable locking implementation, so I'm not suggesting an unchecked shell patch.

Preservation and acceptance: Add a proposed deterministic regression with a partial runtime and two installers. Use barriers so both reach stale removal, let A publish, then release B. After A publishes, an executable probe must never find the runtime missing; both installers must succeed and leave only cline and ready. Retain the existing installer assertions: ready-runtime reuse performs no download, sequential partial-runtime replacement succeeds, and checksum/version failures publish no replacement. These are proposed checks, not executed results.

[yaoyu-33]

Fixed in 01e0bbb: the readiness re-check and the replacement now run under flock on <runtime>.lock when the image has flock (as Copilot's installer does), so a runtime another installer published meanwhile is kept. test_a_runtime_another_installer_published_is_not_replaced holds the lock as the other installer, checks that the waiting installer leaves the stale runtime alone, then checks that it keeps the runtime published meanwhile; it fails against the previous script (run on Linux). Without flock the installer keeps its previous best-effort publication.

[yaoyu-33]

/ok to test 01e0bbb

[AdamRajfer]

The session contract is complete: seed refusals are final 422s, one idempotent activation, close returns a bundle, no host fallback for an unknown session. A run with no run_result raises, a rollout stop before launch is graded as a stop, and the tests cover an infra fault (test_run_without_a_result_fails_the_activation) and model-side endings (test_activation_status_follows_clines_exit_rule, test_classify_outcome_reads_clines_terminal_error). The gaps are in typing, both inline.

Mergeable once an unanswered model call at the deadline is typed as a transport failure and the model request status is typed, or documented as untyped. Needs changes.

[AdamRajfer]

Blocking: elif timed_out: status = "incomplete" applies to every deadline, even when stdout holds no model output. #5654 turns "deadline before any model call returned" into failed with model_transport_error. Here a hung model server is scored as the model's incomplete zero. Add the same split and a test.

[AdamRajfer]

Blocking: classify_outcome types only the context overflow, transport errors and the empty-response case. Every Cline error finish with an HTTP status (400, 429, 5xx) stays unknown, so model_request_error is never produced. Parse the status from the event, or state in the README that these endings are untyped and retried, and test that.

[AdamRajfer]

Non-blocking: output-token-cap endings are incomplete through _reached_model_limit with no typed kind; consistent with #5654, but say so in the README.

[AdamRajfer]

Non-blocking: owned sandboxes get a TTL but no network_access / restrict_sandbox_network.

[giuliolovisotto]

Reviewed failure reporting at this head. One non-blocking evidence-retention follow-up below. Harness conformance gaps are not a merge requirement for this review. Source inspection only; no new tests or native rollout were run.

— @codex

[giuliolovisotto]

Non-blocking: retain the provider wait failure separately from the confirmed harness timeout. This branch clears a real TimeoutError, while the saved SandboxObservation only contains the supervisor timeout and exit status. The warning does not retain the original exception details in the rollout. Keep the current incomplete/gradable result, but also preserve the distinct provider error in saved diagnostics (and its original failure event when available). Extend test_provider_timeout_after_a_confirmed_deadline_is_still_a_timeout to assert both facts survive, rather than only the response status.