tracking: close P0 gaps and contract drift in NeMo Gym agent skills #3122

Summary

Track the work required to make the repository agent skills accurately encode the current NeMo Gym authoring and runtime contracts.

The current skill set is strong for debugging, reward profiling, pivot datasets, and rollout analysis, but it does not yet cover several high-risk component-authoring workflows. Two existing skills also contain instructions that have drifted from main.

The priority in this issue is based on failure risk: P0 items can generate apparently successful runs while silently losing training data, trajectory information, session state, verifier fidelity, or repository correctness.

P0: repair current contract drift

• Rewrite or replace .agents/skills/add-benchmark for the current manifest-backed onboarding flow.
  • Separate creating a new Gym environment from integrating an existing third-party benchmark.
  • Cover manifest.yaml, integration profiles, config-authoritative composition mirrors, task-data schema validation, and verifier fixtures.
  • Remove outdated universal assumptions such as binary-only reward and 95% coverage.
• Bring .agents/skills/nemo-gym-docs in sync with fern/README.md.
  • Remove the nonexistent fern/versions/latest.yml alias workflow.
  • Treat /latest as a redirect to /main and derive release information from fern/docs.yml rather than pinning a release number in the skill.
• Reconcile contract ambiguities between AGENTS.md and current documentation.
  • Decide whether real rollouts are mandatory or optional for environment and agent changes.
  • Clarify the HTTP decision tree: self.server_client for configured Gym server calls, nemo_gym.server_utils.request() for lower-level aiohttp transport, and the Gym OpenAI client for model protocol semantics.
  • Replace the one-line cookie guidance with explicit session/cookie ownership rules for agent, model, and resources servers.
• Make canonical skill discovery mechanically consistent.
  • Ensure .claude/skills and .codex/skills compatibility links cover every canonical skill that requires a native discovery path.
  • Add a test that checks canonical directories, native links, relative references, and documented skill inventory.

P0: add missing contract skills

Agent Server authoring and review

• Add add-agent-server or nemo-gym-agent-server as a standalone skill.
• Cover capture-aware model URL forwarding, replay of prior function calls and outputs, complete output traces, cookie ownership, tool ownership, response IDs, usage, trajectory/token capture, request-local state, and concurrency isolation.
• Require contract tests that exercise a real tool turn and assert the exact response observed by verification and rollout collection.
• Extract and rebase the useful skill work from Hwolff/deepagents agent server #2929 independently of the DeepAgents implementation.

Environment creation

• Add nemo-gym-create-environment based on the current manifest-backed onboarding contract.
• Route among benchmark, environment, and component-only resources-server cases.
• Cover integration profiles, dataset/task schemas, config and manifest authority, verifier fixtures, state isolation, validation, testing, and rollout inspection.
• Reuse ideas from feat: Add foundational NeMo Gym agent skills. #2132 only after rebasing them onto the current onboarding APIs.

Verification and scoring

• Add nemo-gym-verification-and-scoring.
• Cover deterministic, execution/state, LLM-judge, and multi-reward designs.
• Define malformed-output versus infrastructure-failure behavior, reward range/direction, determinism, response preservation, re-verification, and reward-hacking checks.
• Align with Tutorial: How to build and design verifiers with a new agent skill in Gym #1702 and the response-preservation contract proposed in docs(verifier): require verify to return the scored response #2703.

Existing benchmark integration

• Add nemo-gym-integrate-benchmark as a separate workflow from native environment creation.
• Require preservation of the upstream version, split, prompts, harness semantics, scorer behavior, dependencies, task identity, and published baseline.
• Require before/after parity evidence and task-level investigation of unexplained deltas.

P1 follow-ups

• Add or refresh evaluation guidance without duplicating nemo-gym-reward-profiling and nemo-gym-blade-analysis.
• Add RL training guidance for supported frameworks.
• Add async/performance guidance with the maintained HTTP client decision tree.
• Evaluate a custom model-server authoring skill, especially for streaming and token-capture contracts.
• Evaluate migration and sandbox-provider skills based on observed contribution failures.

Generic lint, formatting, and basic testing guidance should remain in AGENTS.md, repository configuration, and automated checks unless a Gym-specific workflow demonstrates that a dedicated skill changes agent decisions.

Skill design contract

Each P0 skill should be concise at entry and use progressive references for detailed contracts. It should include:

1. Precise positive and negative invocation routing.
2. Authoritative code and documentation references.
3. Component and state ownership boundaries.
4. Must-preserve data and protocol invariants.
5. Failure semantics and explicit stop conditions.
6. Minimum behavior and contract tests.
7. Real rollout evidence where required by repository policy.

Do not copy large sections of AGENTS.md or Fern documentation into every skill. Keep global policy in AGENTS.md, schemas and evolving details in their authoritative docs/code, and task-specific decisions in skills.

Acceptance criteria

• No canonical skill contradicts AGENTS.md, current CLI behavior, current schemas, or fern/README.md.
• Every P0 authoring path has one unambiguous automatically discoverable skill.
• New skills pass structural validation and repository discovery/link tests.
• Each P0 skill is forward-tested on at least one realistic isolated implementation or review task.
• Forward tests assert observable behavior and protocol invariants rather than generated wording.
• The Agent Skills documentation lists the exact canonical set and links to all of them.

Related work

• epic: nemo gym agent skills #1235
• Tutorial: How to build and design verifiers with a new agent skill in Gym #1702
• docs: end-to-end guide for adding a benchmark or environment to NeMo Gym #2545
• feat: Add foundational NeMo Gym agent skills. #2132
• docs(fern): overlay-first guide for adding a benchmark or environment #2546
• docs(verifier): require verify to return the scored response #2703
• Hwolff/deepagents agent server #2929