Skip to content

docs(agent): add harness onboarding and upgrade guide - #3917

Closed
yaoyu-33 wants to merge 35 commits into
NVIDIA-NeMo:mainfrom
yaoyu-33:codex/docs-harness-onboarding
Closed

yaoyu-33 wants to merge 35 commits into
NVIDIA-NeMo:mainfrom
yaoyu-33:codex/docs-harness-onboarding

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Add a contributor guide for integrating or updating harnesses, linked from Integrate Existing Agents. It covers independent benchmark/agent composition, explicit host and sandbox execution modes, installation, request and error semantics, cleanup, and observability. Both Mermaid diagrams identify the protocols: HTTP/JSON between FastAPI Gym services, provider-specific execution and file transfer, subprocess/signals, and local JSON files. The architecture diagram includes model requests from both host and sandbox harnesses.

The experiment walkthrough provides four composable YAML files for model/provider, benchmark, agent, and EnvironmentServer. New experiments use single_agent_turn with benchmark-owned taskset declarations and environment_server_routes; gym eval run collates native task_id/task_input rows, starts servers, collects one task, and shuts down. Stable instance names support a Hermes → Pi swap and a separate experimental Terminal Bench 2.1 benchmark swap. The old flat-row adapter is retained only as a migration note. The examples target current-main native adapters and require no pair-specific presets or conversion script.

The lifecycle comparison and sequence diagram make Resources Server responsible for sandbox selection, reuse, and destruction. The TB2.1 session implementation at #3902 (d4f9f1aa2) verifies in the original sandbox and leaves destruction to final Resources close. SWE-Pro exports the patch, releases the original sandbox, and creates/stops a fresh verifier per attempt. The scope is the normal borrowed-session path; the TB2.1 claim is pinned to its implementation revision rather than assumed available in every checkout.

The sandbox guidance follows the Hermes #3961 revision a309dfb9a and its review comments:

  • Shared execution helpers live in nemo_gym/agent_utils/; general file transport stays in nemo_gym/sandbox/utils.py.
  • SandboxSession uploads the supervisor, accepts stage_activation and collect hooks, and manages cleanup, capture, and release. SandboxCommand contains only argv and the installed supervisor interpreter.
  • The guide documents session control files, shared timeout calculation, adapter-local optional runtime metadata, and crash-cleanup requirements. Hermes now defaults an owned sandbox's TTL to installation timeout + runner timeout + 600 seconds, preserving explicit values (including null) and borrowed-sandbox ownership. Every supervised harness requires Python 3 in the sandbox, including standalone CLI binaries.
  • Process control uses provider exec, shell commands, signals, and files. Model requests originate from the harness. The current Hermes path uses Chat Completions while the Gym Agent exposes a Responses API contract; the supervisor does not forward API requests.

Pi #3626 (d91469b81) and Codex #3640 (5151a3a74) now use the same agent_utils session API, updated hooks and fields, and adapter-local runtime metadata. The guide removes the stale migration warning while retaining host-only startup and per-pairing runtime validation requirements. Shared installation and provider mounts remain follow-ups; the page distinguishes them from the actual Hermes installation prerequisites.

This consolidates guidance from #3961 (comment) and the subsequent reviews. Tracks GYM-1011. The observability section links the conformance tooling from #3766 and distinguishes local evidence probes from remote sandbox lifecycle validation.

Updates after 0c72cec9d

  • feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw #3902 second review (d8a306135): follows the second #3902 review as resolved in ab9f3cc27. On timeout or stop the supervisor records already adopted children, stops the harness group, then drains newly adopted children across sessions; natural exit only drains the group, and the remaining limits are stated. Adds rules to keep every harness process in the supervised group (OPENCLAW_NO_RESPAWN=1), make harness hooks fail closed with a readiness receipt (OpenCode task-environment.ready), and bound downloaded logs; notes the Terminal Bench 2.1 close path without automatic retry; adds the AGENTS.md runtime-install rule; and expands termination and test-hygiene validation. feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw #3902 links now pin ab9f3cc27 (line ranges re-checked).
  • Integration path and capabilities (14c537bab): adds a starting point that compares Remote Agent, an Agent Server adapter, and a custom Environment Server; a capability table (evaluation, observability, environment tools and sandboxes, training) with the adapter work and validation each requires; and a split between what an adapter writes and what Gym provides. Adds guidance on routing every policy-generating model call (including summarization, compaction, and subagent calls) through the rollout-prefixed Model Server, echoing Model Server cookies, explicit handling of configuration and request fields, runtime profiles, lossless output conversion, tool source selection, and per-session isolation. Facts were checked against main: Remote Agent has no model_server; Pi and Codex reject unsupported request fields; SimpleAgent echoes Model Server cookies; the vLLM Model Server routes by rollout ID or session cookie; the DeepAgents adapter sets the model profile used for summarization.
  • feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw #3902 termination contract (6ae477488): aligns the guide with the #3902 review as resolved in c2a6dec84: process-group termination with detached task services preserved, retryable missing evidence versus a final negative receipt, task HOME/XDG_* preserved for tool shells, Terminal Bench 2.1 verification caching and close fencing, and the per-callback cleanup_timeout_seconds. A Note states that main keeps the earlier policy until feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw #3902 lands, so this PR should merge after feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw #3902. Also documents agent_utils/uv_bootstrap.py, which Hermes uses on main.
  • Contracts and structure (f094bf957): adds a terminology table, explicit Agent Server contracts with the Environment Server, the task sandbox, and tool grants, an adapter configuration table, and a step-by-step SandboxSession wiring guide. Links Hermes, Pi, and Codex to main now that agent_utils is merged there.
  • Review follow-ups from @lafrancef: verification-policy selection criterion and GYM_HOST discovery (ce4d98e2b).

Validation

Final documentation commit: 0c72cec9d9ce6e0966693b353ee18c431b9b503a.

At d8a306135:

  • python -m pytest tests/unit_tests/test_fern_docs_links.py — 23 passed.
  • pre-commit run --all-files — all hooks passed.
  • fern check — 0 errors; redirect validation skipped without Fern authentication. Full-page Fern preview remains unverified.
  • git diff --check passed; MDX scan found balanced <Note>/<Warning> blocks and code fences, and no unescaped </{ in prose.

Earlier validation at 0c72cec9d:

  • python tests/unit_tests/test_fern_docs_links.py — 18 passed.
  • python -m pre_commit run --all-files — all seven applicable hooks passed.
  • FERN_NO_VERSION_REDIRECTION=true FERN_DISABLE_TELEMETRY=true node <cached-fern-cli>/cli.cjs check --warnings — 0 errors; redirect validation skipped without local Fern authentication. Full-page Fern preview remains unverified.
  • git diff --check and git diff --cached --check — passed; commit has DCO sign-off.
  • Extracted the six final YAML blocks and validated all four SWE-Pro/Terminal Bench × Hermes/Pi compositions against isolated main 30f63c2bca0ffcc79dc0a110b2ee23b5bd2b4c82. Used offline GlobalConfigDictParser, dataset discovery, actual TrainDataProcessor.run with strict task-data validation and local synthetic fixtures, collector preprocessing, taskset routing, and SingleAgentTurnRequest validation. Confirmed source JSONL preservation, stable references, native input conversion, rejection of flat input/missing routes, and Terminal Bench pairing rejection/explicit opt-in. No model or sandbox was launched.
  • Parsed all three displayed Gym commands through the real CLI dispatcher, normally and with the experimental override: six checks passed. Verified gym eval run selects the end-to-end entry point without an agent selector or flat --input.
  • Lifecycle source review: TB2.1 feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw #3902 at d4f9f1aa209b5c20f946a781c391d493a7d821b4; SWE-Pro at 7f16ec541534b95dde258c449be09a461b33e8d4, with the extraction/verifier cleanup block unchanged at current main above. Checked EnvironmentServer close-before-verify and final Resources close ordering.
  • The sequence diagram is unchanged from 90c0a2386, where it rendered with the installed Mermaid CLI and existing Chrome 147 and both branches were visually inspected. Standalone diagram validation does not establish full-page rendering.

Earlier implementation evidence: targeted Hermes lifecycle/ownership tests passed at the separately recorded a309dfb9a and 2093960be revisions (6 and 258 tests respectively; 13 Linux-only skips in the latter). This historical evidence does not qualify the new example pairings.

Rollouts

N/A: documentation only; no runtime behavior changed. Real model/task rollouts and Linux process cleanup remain part of each harness implementation's validation.

Compatibility and benchmark impact

No runtime or benchmark-result changes. The guide preserves sessionless host execution, rejects implicit host fallback for episode sessions, documents the three adapters' shared API, and identifies remaining installation and provider-validation work.

Checklist

  • Read the contributing guidelines.
  • Focused documentation change.
  • Documentation checks and full pre-commit pass.
  • Documentation added and linked.
  • New source-file headers and model rollouts are N/A for this docs-only change.
  • Commits include DCO sign-off.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 added docs Documentation-only updates or documentation debt area:agent Agent harnesses and Responses API agent behavior labels Oct 2, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…boarding

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
yaoyu-33 marked this pull request as ready for review October 5, 2026 18:37
@yaoyu-33 yaoyu-33 added complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer labels Oct 5, 2026
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 6, 2026
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

@lafrancef lafrancef left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for observability/conformance section, two optional suggestions for other sections.

@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 13, 2026
yaoyu-33 and others added 14 commits October 13, 2026 13:02
…o codex/docs-harness-onboarding

Keep the verification-policy guidance and GYM_HOST discovery from
ce4d98e, placed in the restructured sections.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…boarding

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…ntract

Address the lifecycle findings from the NVIDIA-NeMo#3902 review
(NVIDIA-NeMo#3902 (review))
as resolved in c2a6dec:

- The supervisor stops and reaps the harness process group; explicitly
  detached task services survive until the sandbox owner tears down.
- Separate retryable missing evidence from a final negative receipt,
  which releases a borrowed connection without making the run gradable.
- Keep the task HOME and XDG directories for tool shells.
- Describe Terminal Bench 2.1 verification caching, close fencing, and
  the per-callback cleanup timeout without keep_sandbox_for_close.
- Add termination, service, and task-environment validation checks.

Also document uv_bootstrap.py, which Hermes now uses to pick a uv build
for the sandbox platform.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Help users choose between Remote Agent, an Agent Server adapter, and a
custom Environment Server before writing code, and show which Gym
capability each level of adapter work enables. Split the work into what
an adapter writes and what Gym already provides.

Document routing every policy-generating model call, including
auxiliary middleware calls, through the rollout-prefixed Model Server;
Model Server cookies; explicit handling of configuration and request
fields; runtime profiles; lossless output conversion; tool source
selection; and per-session isolation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Follow the second NVIDIA-NeMo#3902 review as resolved in ab9f3cc:

- On timeout or stop, the supervisor records already adopted children,
  stops the harness group, then drains newly adopted children across
  sessions. Natural exit only drains the group. State the remaining
  limits and that harnesses start each tool command in its own session.
- Keep every harness process in the supervised group; disable
  self-respawn, as with OPENCLAW_NO_RESPAWN=1.
- Make harness hooks fail closed with a readiness receipt, and bound
  downloaded logs.
- Note the Terminal Bench 2.1 close path that has no automatic retry.
- Add the AGENTS.md runtime-install rule for sandbox and host paths.
- Expand termination validation and test hygiene.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Cross-checking the Stirrup sandbox adapter in NVIDIA-NeMo#4178 against this guide
found gaps the guide did not state:

- Return 4xx for deterministic seed and request errors, since the
  Environment Server retries 5xx and re-seeds the sandbox.
- Size the episode timeout to cover seed, install, run, close and
  verification, and use admission to queue before the deadline starts.
- Handle SIGTERM in the sandbox runner and write partial output.
- Serialize non-string grant tool results instead of validating them as
  arguments.
- Constrain transitive dependencies of in-sandbox installs and document
  package-index egress.
- Keep direct_http session cookies, which reach non-tool routes, out of
  files the model can read.

Also align the termination section with NVIDIA-NeMo#4614: preserved children include
unfinished background jobs, services must be adopted before the stop, and
descendants of a preserved process are not drained.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
The execution-mode table listed borrowed, agent-owned, and host paths as
equal options. State that a borrowed sandbox is the default, limit
agent-owned sandboxes to benchmarks without a task sandbox or setups
without a shared sandbox service (starting with Docker), and mark the
Local provider and host path as unisolated, for development only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
"Need only evaluation scores?" read as if other paths did not produce
scores. Name the missing capabilities in the decision node and lead the
Remote Agent limits with "Scores only" followed by each "No ...".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
… current adapters

- Repin the Terminal Bench 2.1 and process_supervisor links to the NVIDIA-NeMo#3902
  merge commit 8bd25f3 and drop the pre-merge hedges.
- Name OpenCode and OpenClaw alongside Hermes, Pi, and Codex as agent_utils
  adapters, and add their tool-source rows.
- Note where the reference adapters and Resources Servers do not follow the
  guide's rules yet: Hermes seed errors surface as HTTP 500, Hermes pins its
  version in requirements.txt, and Terminal Bench 2.1 and SWE-bench Pro
  return a cached reward of 0 on verification infrastructure failures.
- Describe agent close as about five per-step timeouts, and size
  cleanup_timeout_seconds accordingly.
- Set default_episode_timeout_seconds in the example so it follows the
  episode-timeout rule, and say to raise the admission settings with
  --concurrency.
- Say that Terminal Bench 2.1 rows supply each task's image, add
  --require-sandbox to the pairing check, and clarify the session cookie,
  model_endpoint(), checkpoint opt-in, and the log-trimming example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test 3619379

@github-actions

Copy link
Copy Markdown
Contributor

Describe the Environment Server, the seed/activate/close/verify/cleanup
episode flow, and Resources Server sandbox ownership on the Architecture
page, and link the harness guide to it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
yaoyu-33 marked this pull request as draft October 17, 2026 15:42
oyilmaz-nvidia pushed a commit to anushaknvidia/Indic_Gym that referenced this pull request Oct 20, 2026
…NVIDIA-NeMo#4711)

* docs(architecture): describe Environment Server episode orchestration

Add the Environment Server to the architecture page: a table row, an
episode lifecycle (seed, agent loop, close and verify, clean up) with a
sequence diagram, a server-type section, a sandbox-ownership note and a
sandbox-transport note. Split from NVIDIA-NeMo#3917 so the concept change can be
reviewed on its own; the harness guide and agent-server pages follow.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* docs(architecture): scope episode orchestration to native Environment Servers

Most shipped configurations still run through the legacy_agent relay, where
the agent seeds and verifies itself. Scope the ordered steps, the component
table row, the Environment Server section, and the sandbox-ownership note to
native Environment Servers, describe the legacy_agent path, and stop saying
that agent harnesses orchestrate rollouts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

---------

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
oyilmaz-nvidia pushed a commit to anushaknvidia/Indic_Gym that referenced this pull request Oct 20, 2026
* docs(agent): add harness onboarding and upgrade guide

Add a contributor guide for integrating or updating an agent harness that
runs in a task sandbox: independent benchmark and agent composition, the
episode lifecycle and sandbox lifetime policies, the Agent Server contracts,
configuration, a worked experiment with a Hermes to Pi and a SWE-bench Pro
to Terminal Bench 2.1 swap, the agent_utils session helpers, input and error
mapping, retry and close safety, and per-pairing validation.

The page content is the guide from NVIDIA-NeMo#3917 at 3955a97, unchanged. It is
split out so the architecture change and the agent-server page structure
can be reviewed separately.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

* docs(agent): bring the harness guide up to date with main

- Step 3 resolves the time budget with resolve_agent_timeout(), runs under
  the adapter's concurrency semaphore, and names the session lookup
  helpers; list agent_utils/task_timeout.py, and note that Pi accepts
  metadata.agent_timeout_sec (NVIDIA-NeMo#4653).
- Pass --allow-unsupported-pairing only to gym eval run, which checks the
  pairing at dispatch; prepare and resolve do not accept it.
- Set cleanup_timeout_seconds: 360 in the run.yaml example so Pi's close
  fits, and note that the benchmark sandbox ttl_s is a separate limit.
- Close the provider when AsyncSandbox.connect() fails in the borrower
  snippet.
- Correct the OpenCode version pin, the Hermes 500 cases, the migration
  note, and the SWE-bench Pro empty-patch path; add the requirements.txt
  row with the sandbox extra.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

---------

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
@yaoyu-33 yaoyu-33 closed this Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior complexity:low Localized change in one scope with a small, straightforward review surface docs Documentation-only updates or documentation debt needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants