Repository navigation
Conversation
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…boarding Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
yaoyu-33
marked this pull request as ready for review
October 5, 2026 18:37
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
7 tasks done
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Draft
7 tasks done
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
lafrancef
reviewed
Oct 13, 2026
lafrancef
left a comment
Contributor
There was a problem hiding this comment.
LGTM for observability/conformance section, two optional suggestions for other sections.
…o codex/docs-harness-onboarding Keep the verification-policy guidance and GYM_HOST discovery from ce4d98e, placed in the restructured sections. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…boarding Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…ntract Address the lifecycle findings from the NVIDIA-NeMo#3902 review (NVIDIA-NeMo#3902 (review)) as resolved in c2a6dec: - The supervisor stops and reaps the harness process group; explicitly detached task services survive until the sandbox owner tears down. - Separate retryable missing evidence from a final negative receipt, which releases a borrowed connection without making the run gradable. - Keep the task HOME and XDG directories for tool shells. - Describe Terminal Bench 2.1 verification caching, close fencing, and the per-callback cleanup timeout without keep_sandbox_for_close. - Add termination, service, and task-environment validation checks. Also document uv_bootstrap.py, which Hermes now uses to pick a uv build for the sandbox platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Help users choose between Remote Agent, an Agent Server adapter, and a custom Environment Server before writing code, and show which Gym capability each level of adapter work enables. Split the work into what an adapter writes and what Gym already provides. Document routing every policy-generating model call, including auxiliary middleware calls, through the rollout-prefixed Model Server; Model Server cookies; explicit handling of configuration and request fields; runtime profiles; lossless output conversion; tool source selection; and per-session isolation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Follow the second NVIDIA-NeMo#3902 review as resolved in ab9f3cc: - On timeout or stop, the supervisor records already adopted children, stops the harness group, then drains newly adopted children across sessions. Natural exit only drains the group. State the remaining limits and that harnesses start each tool command in its own session. - Keep every harness process in the supervised group; disable self-respawn, as with OPENCLAW_NO_RESPAWN=1. - Make harness hooks fail closed with a readiness receipt, and bound downloaded logs. - Note the Terminal Bench 2.1 close path that has no automatic retry. - Add the AGENTS.md runtime-install rule for sandbox and host paths. - Expand termination validation and test hygiene. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Cross-checking the Stirrup sandbox adapter in NVIDIA-NeMo#4178 against this guide found gaps the guide did not state: - Return 4xx for deterministic seed and request errors, since the Environment Server retries 5xx and re-seeds the sandbox. - Size the episode timeout to cover seed, install, run, close and verification, and use admission to queue before the deadline starts. - Handle SIGTERM in the sandbox runner and write partial output. - Serialize non-string grant tool results instead of validating them as arguments. - Constrain transitive dependencies of in-sandbox installs and document package-index egress. - Keep direct_http session cookies, which reach non-tool routes, out of files the model can read. Also align the termination section with NVIDIA-NeMo#4614: preserved children include unfinished background jobs, services must be adopted before the stop, and descendants of a preserved process are not drained. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
The execution-mode table listed borrowed, agent-owned, and host paths as equal options. State that a borrowed sandbox is the default, limit agent-owned sandboxes to benchmarks without a task sandbox or setups without a shared sandbox service (starting with Docker), and mark the Local provider and host path as unisolated, for development only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
"Need only evaluation scores?" read as if other paths did not produce scores. Name the missing capabilities in the decision node and lead the Remote Agent limits with "Scores only" followed by each "No ...". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
… current adapters - Repin the Terminal Bench 2.1 and process_supervisor links to the NVIDIA-NeMo#3902 merge commit 8bd25f3 and drop the pre-merge hedges. - Name OpenCode and OpenClaw alongside Hermes, Pi, and Codex as agent_utils adapters, and add their tool-source rows. - Note where the reference adapters and Resources Servers do not follow the guide's rules yet: Hermes seed errors surface as HTTP 500, Hermes pins its version in requirements.txt, and Terminal Bench 2.1 and SWE-bench Pro return a cached reward of 0 on verification infrastructure failures. - Describe agent close as about five per-step timeouts, and size cleanup_timeout_seconds accordingly. - Set default_episode_timeout_seconds in the example so it follows the episode-timeout rule, and say to raise the admission settings with --concurrency. - Say that Terminal Bench 2.1 rows supply each task's image, add --require-sandbox to the pairing check, and clarify the session cookie, model_endpoint(), checkpoint opt-in, and the log-trimming example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test 3619379 |
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-3917-3619379d53a3.docs.buildwithfern.com/nemo/gym |
Describe the Environment Server, the seed/activate/close/verify/cleanup episode flow, and Resources Server sandbox ownership on the Architecture page, and link the harness guide to it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
yaoyu-33
marked this pull request as draft
October 17, 2026 15:42
This was referenced Oct 17, 2026
oyilmaz-nvidia
pushed a commit
to anushaknvidia/Indic_Gym
that referenced
this pull request
Oct 20, 2026
…NVIDIA-NeMo#4711) * docs(architecture): describe Environment Server episode orchestration Add the Environment Server to the architecture page: a table row, an episode lifecycle (seed, agent loop, close and verify, clean up) with a sequence diagram, a server-type section, a sandbox-ownership note and a sandbox-transport note. Split from NVIDIA-NeMo#3917 so the concept change can be reviewed on its own; the harness guide and agent-server pages follow. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * docs(architecture): scope episode orchestration to native Environment Servers Most shipped configurations still run through the legacy_agent relay, where the agent seeds and verifies itself. Scope the ordered steps, the component table row, the Environment Server section, and the sandbox-ownership note to native Environment Servers, describe the legacy_agent path, and stop saying that agent harnesses orchestrate rollouts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
oyilmaz-nvidia
pushed a commit
to anushaknvidia/Indic_Gym
that referenced
this pull request
Oct 20, 2026
* docs(agent): add harness onboarding and upgrade guide Add a contributor guide for integrating or updating an agent harness that runs in a task sandbox: independent benchmark and agent composition, the episode lifecycle and sandbox lifetime policies, the Agent Server contracts, configuration, a worked experiment with a Hermes to Pi and a SWE-bench Pro to Terminal Bench 2.1 swap, the agent_utils session helpers, input and error mapping, retry and close safety, and per-pairing validation. The page content is the guide from NVIDIA-NeMo#3917 at 3955a97, unchanged. It is split out so the architecture change and the agent-server page structure can be reviewed separately. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * docs(agent): bring the harness guide up to date with main - Step 3 resolves the time budget with resolve_agent_timeout(), runs under the adapter's concurrency semaphore, and names the session lookup helpers; list agent_utils/task_timeout.py, and note that Pi accepts metadata.agent_timeout_sec (NVIDIA-NeMo#4653). - Pass --allow-unsupported-pairing only to gym eval run, which checks the pairing at dispatch; prepare and resolve do not accept it. - Set cleanup_timeout_seconds: 360 in the run.yaml example so Pi's close fits, and note that the benchmark sandbox ttl_s is a separate limit. - Close the provider when AsyncSandbox.connect() fails in the borrower snippet. - Correct the OpenCode version pin, the Hermes 500 cases, the migration note, and the SWE-bench Pro empty-patch path; add the requirements.txt row with the sandbox extra. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Add a contributor guide for integrating or updating harnesses, linked from Integrate Existing Agents. It covers independent benchmark/agent composition, explicit host and sandbox execution modes, installation, request and error semantics, cleanup, and observability. Both Mermaid diagrams identify the protocols: HTTP/JSON between FastAPI Gym services, provider-specific execution and file transfer, subprocess/signals, and local JSON files. The architecture diagram includes model requests from both host and sandbox harnesses.
The experiment walkthrough provides four composable YAML files for model/provider, benchmark, agent, and EnvironmentServer. New experiments use
single_agent_turnwith benchmark-ownedtasksetdeclarations andenvironment_server_routes;gym eval runcollates nativetask_id/task_inputrows, starts servers, collects one task, and shuts down. Stable instance names support a Hermes → Pi swap and a separate experimental Terminal Bench 2.1 benchmark swap. The old flat-row adapter is retained only as a migration note. The examples target current-main native adapters and require no pair-specific presets or conversion script.The lifecycle comparison and sequence diagram make Resources Server responsible for sandbox selection, reuse, and destruction. The TB2.1 session implementation at #3902 (
d4f9f1aa2) verifies in the original sandbox and leaves destruction to final Resources close. SWE-Pro exports the patch, releases the original sandbox, and creates/stops a fresh verifier per attempt. The scope is the normal borrowed-session path; the TB2.1 claim is pinned to its implementation revision rather than assumed available in every checkout.The sandbox guidance follows the Hermes #3961 revision
a309dfb9aand its review comments:nemo_gym/agent_utils/; general file transport stays innemo_gym/sandbox/utils.py.SandboxSessionuploads the supervisor, acceptsstage_activationandcollecthooks, and manages cleanup, capture, and release.SandboxCommandcontains only argv and the installed supervisor interpreter.Pi #3626 (
d91469b81) and Codex #3640 (5151a3a74) now use the sameagent_utilssession API, updated hooks and fields, and adapter-local runtime metadata. The guide removes the stale migration warning while retaining host-only startup and per-pairing runtime validation requirements. Shared installation and provider mounts remain follow-ups; the page distinguishes them from the actual Hermes installation prerequisites.This consolidates guidance from #3961 (comment) and the subsequent reviews. Tracks GYM-1011. The observability section links the conformance tooling from #3766 and distinguishes local evidence probes from remote sandbox lifecycle validation.
Updates after
0c72cec9dd8a306135): follows the second #3902 review as resolved inab9f3cc27. On timeout or stop the supervisor records already adopted children, stops the harness group, then drains newly adopted children across sessions; natural exit only drains the group, and the remaining limits are stated. Adds rules to keep every harness process in the supervised group (OPENCLAW_NO_RESPAWN=1), make harness hooks fail closed with a readiness receipt (OpenCodetask-environment.ready), and bound downloaded logs; notes the Terminal Bench 2.1 close path without automatic retry; adds theAGENTS.mdruntime-install rule; and expands termination and test-hygiene validation. feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw #3902 links now pinab9f3cc27(line ranges re-checked).14c537bab): adds a starting point that compares Remote Agent, an Agent Server adapter, and a custom Environment Server; a capability table (evaluation, observability, environment tools and sandboxes, training) with the adapter work and validation each requires; and a split between what an adapter writes and what Gym provides. Adds guidance on routing every policy-generating model call (including summarization, compaction, and subagent calls) through the rollout-prefixed Model Server, echoing Model Server cookies, explicit handling of configuration and request fields, runtime profiles, lossless output conversion, tool source selection, and per-session isolation. Facts were checked againstmain: Remote Agent has nomodel_server; Pi and Codex reject unsupported request fields;SimpleAgentechoes Model Server cookies; the vLLM Model Server routes by rollout ID or session cookie; the DeepAgents adapter sets the model profile used for summarization.6ae477488): aligns the guide with the #3902 review as resolved inc2a6dec84: process-group termination with detached task services preserved, retryable missing evidence versus a final negative receipt, taskHOME/XDG_*preserved for tool shells, Terminal Bench 2.1 verification caching and close fencing, and the per-callbackcleanup_timeout_seconds. A Note states thatmainkeeps the earlier policy until feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw #3902 lands, so this PR should merge after feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw #3902. Also documentsagent_utils/uv_bootstrap.py, which Hermes uses onmain.f094bf957): adds a terminology table, explicit Agent Server contracts with the Environment Server, the task sandbox, and tool grants, an adapter configuration table, and a step-by-stepSandboxSessionwiring guide. Links Hermes, Pi, and Codex tomainnow thatagent_utilsis merged there.GYM_HOSTdiscovery (ce4d98e2b).Validation
Final documentation commit:
0c72cec9d9ce6e0966693b353ee18c431b9b503a.At
d8a306135:python -m pytest tests/unit_tests/test_fern_docs_links.py— 23 passed.pre-commit run --all-files— all hooks passed.fern check— 0 errors; redirect validation skipped without Fern authentication. Full-page Fern preview remains unverified.git diff --checkpassed; MDX scan found balanced<Note>/<Warning>blocks and code fences, and no unescaped</{in prose.Earlier validation at
0c72cec9d:python tests/unit_tests/test_fern_docs_links.py— 18 passed.python -m pre_commit run --all-files— all seven applicable hooks passed.FERN_NO_VERSION_REDIRECTION=true FERN_DISABLE_TELEMETRY=true node <cached-fern-cli>/cli.cjs check --warnings— 0 errors; redirect validation skipped without local Fern authentication. Full-page Fern preview remains unverified.git diff --checkandgit diff --cached --check— passed; commit has DCO sign-off.30f63c2bca0ffcc79dc0a110b2ee23b5bd2b4c82. Used offlineGlobalConfigDictParser, dataset discovery, actualTrainDataProcessor.runwith strict task-data validation and local synthetic fixtures, collector preprocessing, taskset routing, andSingleAgentTurnRequestvalidation. Confirmed source JSONL preservation, stable references, native input conversion, rejection of flat input/missing routes, and Terminal Bench pairing rejection/explicit opt-in. No model or sandbox was launched.gym eval runselects the end-to-end entry point without an agent selector or flat--input.d4f9f1aa209b5c20f946a781c391d493a7d821b4; SWE-Pro at7f16ec541534b95dde258c449be09a461b33e8d4, with the extraction/verifier cleanup block unchanged at current main above. Checked EnvironmentServer close-before-verify and final Resources close ordering.90c0a2386, where it rendered with the installed Mermaid CLI and existing Chrome 147 and both branches were visually inspected. Standalone diagram validation does not establish full-page rendering.Earlier implementation evidence: targeted Hermes lifecycle/ownership tests passed at the separately recorded
a309dfb9aand2093960berevisions (6 and 258 tests respectively; 13 Linux-only skips in the latter). This historical evidence does not qualify the new example pairings.Rollouts
N/A: documentation only; no runtime behavior changed. Real model/task rollouts and Linux process cleanup remain part of each harness implementation's validation.
Compatibility and benchmark impact
No runtime or benchmark-result changes. The guide preserves sessionless host execution, rejects implicit host fallback for episode sessions, documents the three adapters' shared API, and identifies remaining installation and provider-validation work.
Checklist