docs(agent): add the add-harness skill and document onboarding pitfalls by yaoyu-33 · Pull Request #5074 · NVIDIA-NeMo/Gym

yaoyu-33 · 2026-10-26T02:53:13Z

What changed and why

Third-party coding agents that onboard a harness need the same guidance add-benchmark gives for benchmarks. This PR adds a public add-harness skill and fills the doc gaps that two onboarding test drives (GitHub Copilot CLI and LangChain DeepAgents) hit.

Skill (.agents/skills/add-harness/, with the usual .claude/skills and .codex/skills links and Codex metadata). The skill links Add or Update a Harness and the other docs for every contract and command. It adds only:

the order of work
the decisions to make before writing code: whether an adapter already exists, the execution model, the capability level, and the first pairing with its allowed_agents
short code checklists
an exit-check table: tests, conformance or why it does not apply, real pairing, full TB 2.1 evaluation for harnesses claiming support, health, evidence, composition, the pairing declaration and the catalog tables

Docs:

add-or-update-a-harness.mdx:
- what an Evaluation-level adapter's health and evidence look like (unobserved, failed evidence check)
- when the conformance probes apply: a harness that offers a function-calling shell tool
- that gym eval run --input needs --no-serve, and that a config that resolves and validates can still fail when its servers start
- a full Terminal-Bench 2.1 evaluation before merging a new harness or upgrade that claims TB 2.1 support, at every capability level. Requires all tasks and declared repeats at a pinned revision with a real model, retained artifacts, per-task and aggregate results, explicit coverage and failures, and matched before/after runs for upgrades.
- MCP grant integration: an example typed grant, URL/header propagation, required-server readiness before inference, runtime/plugin failure enforcement, timeout semantics, sandbox ownership, and authenticated tool-call/cleanup validation. The skill links this guidance from its implementation and test checklists.
integrate-existing-agents.mdx: a single_agent_turn_legacy block names both resources_server and agent_server.
contribute/agent-skills.mdx: lists the new skill. add-benchmark now points its harness section at the new skill.

Copilot adapter review lessons. The Copilot CLI adapter (#5082) was built by following this skill. Its review found problems the skill did not warn about, now covered:

add-or-update-a-harness.mdx: set the response status by the harness's own rule for a failed run, so a tolerated subagent error or a recovered model call does not fail it. Report aggregate usage only when every model call has a usage record; otherwise leave it unknown with a gap.
Skill checklists:
- stdlib_runner and RUNNER_FILE for a sandbox runner, so a task's pkill -f python3 cannot stop it
- tests for a background process that holds stdout open, for tolerated errors and for a missing usage record
- a later 422 from the Model Server means the harness replays a response field; normalize only empty values in nemo_gym/openai_utils.py
- catalog entries match the adapter's limits, rejected fields and token-ID capture
- test_add_legacy_agent_environment_servers.py checks every config, so a session-only or unbound adapter needs its type in the migration script
- merge current main before the final checks

Related: #4994 adds the Copilot test drive's lessons to the harness guide, and #5008 makes gym env validate check Environment Server configs. This PR does not repeat either. The DeepAgents lessons come from #5003.

Validation

Every relative link and anchor in add-harness/SKILL.md and add-benchmark/SKILL.md resolves; checked with a script.
pre-commit run --all-files with the new files staged: clean, including the private-guardwords check.
nemo_gym.skills.load_skill_directory(".agents/skills") loads all 12 skills, including add-harness. pytest tests/unit_tests/test_skills.py tests/unit_tests/test_session_contract_repo.py: 26 passed.
fern check: 0 errors. The one warning is the redirect check, which needs Fern login; no pages moved.
TB 2.1 requirement update: reran staged pre-commit run --all-files, checked all 31 relative links and anchors in add-harness, and loaded all 12 skills through Gym from the edited checkout. npm run check -- --warnings reports 0 errors and the same authentication warning. The standalone skill-creator validator rejects the existing <CLI> text in the unchanged description on both the original and updated skill; Gym's loader accepts it.
Copilot review lessons (5611a9f7d): staged pre-commit run --all-files passes, including the guardwords check. All 34 relative links and anchors in add-harness resolve, and Gym loads all 12 skills from the edited checkout. pytest tests/unit_tests/test_skills.py tests/unit_tests/test_session_contract_repo.py: 26 passed. npm run check -- --warnings: 0 errors, the same authentication warning. A trial merge with current main (48a58f5a2) is clean. Each claim was checked against main: RUNNER_FILE and stdlib_runner in nemo_gym/agent_utils/sandbox_session.py, the two exemption checks in scripts/add_legacy_agent_environment_servers.py, and the Pi composition test.
Blind test drive. A fresh agent with no other context followed the skill in a read-only dry run: add the Aider CLI paired with Terminal-Bench 2.1. It reported gaps, which were checked against the code before being fixed:
- when to use --allow-unsupported-pairing, and that allowed_agents is checked at dispatch
- ng_test is deprecated; use gym env test +entrypoint=...
- the conformance probes require a function-calling shell tool
- the capability matrix is published by CI, so a local table is a preview only
- test_session_contract_repo.py covers every adapter but isn't found by searching for the adapter's name
- the catalog tables to update
- the 409 wording for activations
One reported gap was wrong: langchain_deepagents_agent does serve sessions, inherited from SimpleAgent. The skill keeps it as the framework example and adds simple_agent.
MCP grant guidance (5b88ac6bf): pre-commit run --all-files passed; PYTHONPATH="$PWD" python -m pytest tests/unit_tests/test_skills.py tests/unit_tests/test_session_contract_repo.py tests/unit_tests/test_fern_docs_links.py -q -o addopts='' -o log_cli=false -p no:cacheprovider passed (55 tests, 24 subtests). cd fern && npm run check: 0 errors, 1 warning. Gym's skill loader accepts the edited directory; the new relative link and anchor resolve; the JSON example validates against MCPToolAccess. The standalone skill validator's pre-existing <CLI> frontmatter rejection was reproduced on the unchanged baseline.

Rollout evidence

N/A: documentation and agent-skill changes only; no runtime code changes.

🤖 Generated with Claude Code

copy-pr-bot · 2026-10-26T02:53:17Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

yaoyu-33 · 2026-10-26T02:53:26Z

/ok to test e7d2904

github-actions · 2026-10-26T02:56:49Z

🌿 Preview your docs: https://nvidia-preview-pr-5074-47e2a7337464.docs.buildwithfern.com/nemo/gym

yaoyu-33

Team review of the add-harness skill and the harness-guide edits. Every command, flag, path, symbol and link the skill names was checked against the code at e7d29040: the CLI flags, the conformance runner and inspector flags, the session-contract symbols, the single_agent_turn_legacy and legacy_agent config shapes, and all 32 links and anchors resolve. No blockers.

Inline: one RISK and four NOTEs. The RISK is the pairing override. In the skill's gym env start + gym eval run --no-serve flow, the pairing check reads the head server's config, so the flag has to go on gym env start.

Lower-confidence points from the same pass, not raised inline:

The conformance registry key must equal the adapter directory without _agent.
The probes recognize only certain shell-tool names and argument shapes, and they give the adapter its own local sandbox.
A server that declares no allowed_agents accepts any harness.
The "Sandboxed harnesses" table covers only sandboxed adapters.

PR hygiene: the body cites #4994, #5008 and #5003 but links no issue and does not say why none is needed (AGENTS.md, "Pull Requests").

yaoyu-33 · 2026-10-26T14:56:26Z

/ok to test 827b1ce

yaoyu-33 · 2026-10-26T16:18:26Z

/ok to test 93d1a55

yaoyu-33 · 2026-10-26T21:32:20Z

/ok to test dc09885

yaoyu-33 · 2026-10-27T20:44:46Z

/ok to test 5611a9f

github-actions · 2026-10-28T20:11:34Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40761036239

github-actions · 2026-10-28T20:13:08Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40761259556

github-actions · 2026-10-28T20:46:10Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40766194414

Add a public skill that walks a coding agent through adding or updating a harness adapter: scoping, the session contract, pairing configuration, harness conformance, real pairing runs, and the exit checks before merge. The skill links the harness docs for every contract and command instead of restating them. Fill the doc gaps that two onboarding test drives hit: - when the conformance probes apply (a harness with a shell tool) - what an Evaluation-level adapter's health and evidence look like - `gym eval run --input` needs `--no-serve`, and a config that resolves and validates can still fail when its servers start - a `single_agent_turn_legacy` block names both `resources_server` and `agent_server` Point add-benchmark's harness section at the new skill and list it on the Agent Skills page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Review fixes for the add-harness skill and the harness guide: - With `gym eval run --no-serve`, the pairing check reads the running servers' config, so pass `--allow-unsupported-pairing` to `gym env start`. Treat `allowed_agents` as optional on a Resources Server. - Add the required `--output` to the guide's `--no-serve` smoke command. - Apply the Conformance and Evidence exit checks from the Observability level up; an Evaluation-level adapter records its expected failure. - Explain `unobserved` by the missing input, not only by the level. - Stop citing langchain_deepagents_agent as a model until its `responses()` checks the session; take the close fallback from hermes_agent. - Name the conformance registry-key, shell-tool, and sandbox constraints, and make the catalog tables conditional. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

- Define a pairing (one harness on one benchmark through a single_agent_turn Environment Server) and rename Step 3 to say what is paired. - In Step 4, sort conformance failures by source: harness or adapter failures are fixed in the adapter; a probe that cannot drive the harness's tool gets a provider extension in the same PR; setup failures are fixed in the setup; a wrong check is fixed in a separate PR with a regression test, never to suit one harness. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

- Set the TB 2.1 full evaluation to 8 repeats per task, as the benchmark's OpenCode, Terminus 2, and Inkling configs declare; the README's run config declares none. - Note that exporting NEMO_GYM_ALLOW_UNSUPPORTED_PAIRING=1 also works with gym eval run --no-serve. - Link the deepagents session caveat to #5003, which fixes it. - Say in Step 4 that the probes are optional at the Evaluation level. - Replace "very few unhealthy rollouts" with a stated cause for each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

The review of the Copilot CLI adapter (#5082), built by following this skill, found problems the skill and harness guide did not warn about: - Harness guide: set the response status by the harness's own rule for a failed run, and report aggregate usage only when every model call has a usage record, with a gap instead of a subtotal. - Code checklist: stdlib_runner and RUNNER_FILE for sandbox runners, plus the two output rules above. - Unit tests: a background process holding stdout must not block the activation; tolerated errors and missing usage records are covered. - Real runs: a later 422 means replayed response fields; normalize only empty values in nemo_gym/openai_utils.py and keep the rest strict. - Catalog: entries must match the adapter's limits, rejected fields, and token-ID capture. - Pre-commit: the legacy relay migration test checks every config, so session-only or unbound adapters need an exemption in the script; merge current main before the final checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Review of the two most recent add-harness commits against current main: - stdlib runner: `stdlib_runner=True` is set on the `SandboxCommand`, and it protects against `pkill python3` (neutral process name) and `pkill -f '\.py'` (no suffix on RUNNER_FILE), not `pkill -f python3`. - 422 on replay: validation is strict for Chat and Responses requests, and `annotations: null` comes from the harness's SDK, not a Model Server (see the Copilot adapter, #5082). - Legacy relay script: name the two checks in `needs_environment_server` instead of "the matching check". - MCP grants: replace slash-joined terms and "readiness receipt" with what to do, explain the shared MCP timeout, and use "plugin or hook" consistently in the guide and the skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

ffrujeri · 2026-10-29T00:24:47Z

/ok to test 47e2a73

github-actions · 2026-10-29T01:01:54Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40797864725

github-actions · 2026-10-29T01:53:14Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40803081619

yaoyu-33 added docs Documentation-only updates or documentation debt area:agent Agent harnesses and Responses API agent behavior labels Oct 26, 2026

copy-pr-bot Bot temporarily deployed to public October 26, 2026 02:53 Inactive

copy-pr-bot Bot temporarily deployed to public October 26, 2026 02:56 Inactive

yaoyu-33 mentioned this pull request Oct 26, 2026

feat(copilot): add a sandboxed Copilot CLI agent server #5082

Open

yaoyu-33 commented Oct 26, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public October 26, 2026 14:57 Inactive

copy-pr-bot Bot temporarily deployed to public October 26, 2026 14:59 Inactive

yaoyu-33 commented Oct 26, 2026

View reviewed changes

Comment thread .agents/skills/add-harness/SKILL.md Resolved

yaoyu-33 commented Oct 26, 2026

View reviewed changes

Comment thread .agents/skills/add-harness/SKILL.md Resolved

copy-pr-bot Bot temporarily deployed to public October 26, 2026 16:19 Inactive

copy-pr-bot Bot temporarily deployed to public October 26, 2026 16:21 Inactive

copy-pr-bot Bot temporarily deployed to public October 26, 2026 16:22 Inactive

copy-pr-bot Bot temporarily deployed to public October 26, 2026 21:42 Inactive

copy-pr-bot Bot temporarily deployed to public October 26, 2026 21:54 Inactive

copy-pr-bot Bot temporarily deployed to public October 26, 2026 21:55 Inactive

copy-pr-bot Bot temporarily deployed to public October 26, 2026 21:56 Inactive

copy-pr-bot Bot temporarily deployed to public October 27, 2026 20:45 Inactive

copy-pr-bot Bot temporarily deployed to public October 27, 2026 22:54 Inactive

ffrujeri approved these changes Oct 28, 2026

View reviewed changes

athitten mentioned this pull request Oct 28, 2026

docs(skills): add score triage skill #5207

Merged

7 tasks done

yaoyu-33 added this pull request to the merge queue Oct 28, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 28, 2026

yaoyu-33 and others added 8 commits October 28, 2026 17:24

docs(agent): require full TB 2.1 evaluation for supported harnesses

7d5b6b6

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

docs(agent): clarify MCP grant integration requirements

3da742d

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

ffrujeri force-pushed the yuya/add-harness-skill branch from 204335d to 47e2a73 Compare October 29, 2026 00:24

copy-pr-bot Bot temporarily deployed to public October 29, 2026 00:25 Inactive

copy-pr-bot Bot temporarily deployed to public October 29, 2026 00:27 Inactive

copy-pr-bot Bot temporarily deployed to public October 29, 2026 00:28 Inactive

ffrujeri added this pull request to the merge queue Oct 29, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 29, 2026

ffrujeri added this pull request to the merge queue Oct 29, 2026

Merged via the queue into main with commit 6142f47 Oct 29, 2026
55 of 58 checks passed

ffrujeri deleted the yuya/add-harness-skill branch October 29, 2026 02:28

Conversation

yaoyu-33 commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed and why

Validation

Rollout evidence

Uh oh!

copy-pr-bot Bot commented Oct 26, 2026

Uh oh!

yaoyu-33 commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaoyu-33 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yaoyu-33 commented Oct 26, 2026

Uh oh!

Uh oh!

Uh oh!

yaoyu-33 commented Oct 26, 2026

Uh oh!

yaoyu-33 commented Oct 26, 2026

Uh oh!

yaoyu-33 commented Oct 27, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

Uh oh!

ffrujeri commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yaoyu-33 commented Oct 26, 2026 •

edited

Loading

github-actions Bot commented Oct 26, 2026 •

edited

Loading