Harness evidence checks conformance by lafrancef · Pull Request #3766 · NVIDIA-NeMo/Gym

lafrancef · 2026-09-29T12:43:54Z

Motivation: for downstream analyses we require harnesses to output certain kinds of information, which we refer to as "evidence". Before attempting to launch such analyses we want to make sure that the harness actually supports outputting the required evidence.

What: This change introduces (1) a library that defines what evidence we are interested in, as well as allows checking existing rollouts to validate whether they contain the evidence (2) a tool that runs specific harnesses under fake scenarios designed to exercise paths that should generate the evidence. Together they can be used by first running the exercise tool to capture rollouts, then the library to output pass/fail verdicts for each harness x evidence.

Mental model: this change is like a behavioral/integration test for harness observability. It is complementary to observability itself (which is like adding logs) and to health checks (which is like reading the logs).

Future direction: add this to Gym CI informationally, and/or regularly regenerate this page: https://docs.nvidia.com/nemo/gym/reference/trajectory-capabilities/

copy-pr-bot · 2026-09-29T12:43:58Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-09-29T12:45:36Z

🌿 Preview your docs: https://nvidia-preview-pr-3766-adc58180847a.docs.buildwithfern.com/nemo/gym

Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>

lafrancef · 2026-10-02T09:23:39Z

@yaoyu-33 I addressed your comment on TE-6 in 213b911, the checker now aligns with the case you mentioned

lafrancef · 2026-10-02T09:23:51Z

/ok to test 213b911

copy-pr-bot · 2026-10-02T09:23:55Z

/ok to test 213b911

@lafrancef, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

lafrancef · 2026-10-02T09:26:55Z

/ok to test 213b911

github-actions · 2026-10-02T19:30:22Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37054531818

Stacked on #4041. Validate each piece of evidence once at its designated saved location, with JSON Schema for fields and explicit semantic checks for relationships. Missing collections block dependent checks without invalidating unrelated evidence. - Keep call, step, tool, grading and sandbox requirements; resolve invocation/call references against canonical targets. All supplied reference identifiers must agree, and ambiguous targets fail. - Replace blanket Python-model validation and cross-copy reconciliation. Do not repeat Gym-owned uniqueness, clock-order or provider-usage normalization checks. - Retain independent comparisons with scripted exchanges, response metadata, token counts and tool outcomes. A behavioral failure does not erase artifact passes. - Emit one table per harness, with individual check rows grouped as Schema, Semantic and Behavioral, scenario columns, and compact status symbols. Process status stays separate. Missing normal rollouts leave artifact checks unassessed. The profile is `gym-p0/v3`. Check IDs, report format and authoritative locations change; regenerate existing reports. TE labels only group results. `ng_trajectory` supplies calls, steps, invocations and tool executions; sandbox/compaction observations retain their explicitly designated source. Capture sidecars are provenance only. Benchmark execution and scoring code are unchanged. Validation: - Focused checker/runner suite: **253 passed, 2 skipped** (`jj` unavailable). - `pre-commit run --all-files`: passed. - Fern check: zero errors, one warning. - Fresh scripted runs on main `b0e3fa6cf` plus this checker stack for simple_agent, SciCode, OpenCode, Terminus2, mini-SWE, Hermes, NOOA, Codex, Pi and Stirrup. OpenCode is also tested separately with observability stack tip `ccf9f997b` (#3872). Failure sidecars leave artifact checks unassessed. - Local additional-driver/checker suite: **272 passed, 2 skipped**. Combined current OpenCode stack, producer/capture and checker tests: **625 passed, 2 skipped**. Controlled-endpoint tests execute shell commands and inspect actual model/tool/verifier exchanges. These scripted runs do not qualify production benchmarks. Stirrup also ran all seven scenarios in Lima/Apptainer with a small shell-test image: six normal rollouts and one terminal-error failure record. The six normal rollouts retain model-call evidence but lack invocation, step and tool records. This uses a local native-tool driver; it does not qualify the full GDPval environment. Full server CI was not run locally. Follow-up to #3766 and #4041; no separate issue is needed for this focused integration. --------- Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

Pi loses ownership of model attempts that fail before returning a response ID, causing the retry and terminal-error scenarios from NVIDIA-NeMo#3766 to fail. Carry the invocation ID in provider headers through retries and keep HTTP error payloads as captured attempts without creating empty model turns. The collector correction is shared by the other harness conformance fixes. No separate issue is needed: these failures were reproduced with the existing conformance runner. All seven conformance scenarios pass with Pi 0.80.2 (previously 4/7). Reproduce with `python scripts/run_harness_conformance.py --harness pi --output /tmp/pi-conformance` using a fresh output directory. Error/retry ownership satisfies TE-8; TE-9 remains the alternative response-ID contract. `pre-commit run --all-files` is clean. A real gpt-4.1-mini rollout computed 17 × 23, wrote and read 391 through a shell tool, and returned CONFORMANCE_DONE; the independent verifier awarded 1.0 after checking the file and final answer (two provider calls). Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>

Stacked on #4041. Validate each piece of evidence once at its designated saved location, with JSON Schema for fields and explicit semantic checks for relationships. Missing collections block dependent checks without invalidating unrelated evidence. - Keep call, step, tool, grading and sandbox requirements; resolve invocation/call references against canonical targets. All supplied reference identifiers must agree, and ambiguous targets fail. - Replace blanket Python-model validation and cross-copy reconciliation. Do not repeat Gym-owned uniqueness, clock-order or provider-usage normalization checks. - Retain independent comparisons with scripted exchanges, response metadata, token counts and tool outcomes. A behavioral failure does not erase artifact passes. - Emit one table per harness, with individual check rows grouped as Schema, Semantic and Behavioral, scenario columns, and compact status symbols. Process status stays separate. Missing normal rollouts leave artifact checks unassessed. The profile is `gym-p0/v3`. Check IDs, report format and authoritative locations change; regenerate existing reports. TE labels only group results. `ng_trajectory` supplies calls, steps, invocations and tool executions; sandbox/compaction observations retain their explicitly designated source. Capture sidecars are provenance only. Benchmark execution and scoring code are unchanged. Validation: - Focused checker/runner suite: **253 passed, 2 skipped** (`jj` unavailable). - `pre-commit run --all-files`: passed. - Fern check: zero errors, one warning. - Fresh scripted runs on main `b0e3fa6cf` plus this checker stack for simple_agent, SciCode, OpenCode, Terminus2, mini-SWE, Hermes, NOOA, Codex, Pi and Stirrup. OpenCode is also tested separately with observability stack tip `ccf9f997b` (#3872). Failure sidecars leave artifact checks unassessed. - Local additional-driver/checker suite: **272 passed, 2 skipped**. Combined current OpenCode stack, producer/capture and checker tests: **625 passed, 2 skipped**. Controlled-endpoint tests execute shell commands and inspect actual model/tool/verifier exchanges. These scripted runs do not qualify production benchmarks. Stirrup also ran all seven scenarios in Lima/Apptainer with a small shell-test image: six normal rollouts and one terminal-error failure record. The six normal rollouts retain model-call evidence but lack invocation, step and tool records. This uses a local native-tool driver; it does not qualify the full GDPval environment. Full server CI was not run locally. Follow-up to #3766 and #4041; no separate issue is needed for this focused integration. --------- Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

Conformance reporting previously coupled check dispatch and pass/fail gates to TE groups, and behavioral failures could obscure valid artifact evidence. Define schema, semantic and behavioral checks as typed objects with explicit IDs, tiers, evidence labels, locations, failure reasons and evaluation logic. All check definitions live under `nemo_gym/harness_capabilities/`; scripts supply scenarios, observations and expectations. Each check reports pass, fail, or not applicable, with reasons and blocking prerequisites. Missing required input, blocked checks and absent results display a cross. The P0 gate requires every applicable individual P0 check to pass; all current checks are P0, including ownership and call-to-step checks. TE labels only group results. Artifact, behavioral and execution results remain separate. The permanent first-request HTTP 400 scenario explicitly excludes step evidence; retry-and-recovery scenarios still require it. Previous-response history is resolved within its model server, avoiding false failures when servers reuse response IDs. Removes unused wrappers, internal record copies and redundant result conversions; consolidates overlapping tests. Includes the focused evidence checks previously stacked in NVIDIA-NeMo#4042. Removes string-based dispatch, whole-object validation and cross-copy fallback/consistency checks. Retains pre-PR profile/schema version identifiers. Schema helpers and model-call reference grammar remain in checker.py; schemas.py is removed. Follow-up to NVIDIA-NeMo#3766; no separate issue is needed. Harness unit tests assert native observability and health behavior without importing the conformance checker. Removes the copy-only `hydrate_record()` wrapper and `reader.py`; strict JSONL reading and hashing live in the conformance CLI, shared by the scripts. Validation: - At rebased head `b4f30dc96` (on `main` at `e76a5c629`): 276 checker/runner tests passed, 2 skipped; mini-SWE harness suite: 124 passed, 1 skipped. Tests use the current checkout via `PYTHONPATH` with existing dependency environments. - Cross-server response-ID reuse, same-server ambiguity, strict JSONL parsing and immutable report publication remain covered. - `pre-commit run --all-files` passed. - Earlier Fern validation: 0 errors, 1 warning; this cleanup changes no Fern files. - Earlier validation reinspected 76 saved conformance scenarios across 11 harness variants; all 5,775 individual statuses match the preceding report. Restoring all checks to P0 makes ten scenario gates fail on their existing ownership/attribution failures. Regenerated and visually checked the HTML tables. Default pytest discovery is limited to `tests`. Conformance runner tests remain available through an explicit `pytest scripts/harness_conformance/tests` invocation; dedicated CI for that module is deferred. The core coverage threshold remains 95%. This avoids adding incidentally imported harness implementations to the core coverage measurement without their separate server suites. At `da612c420`, core behavioral-check tests live in `tests/unit_tests/harness_capabilities/test_behavior.py`. Fifteen cases exercise model exchanges, response IDs, token counts and tool witnesses without importing harness implementations. A shared synthetic helper supplies independent saved evidence and witness records; scenario integration and protocol-normalization tests remain with the runner. Validation for this test relocation: - Core conformance and runner suites: 277 passed, 2 skipped. - Moved behavioral tests alone: 15 passed; `behavior.py` coverage 96.74% (89 of 92 statements). Verified no `responses_api_agents` modules were imported. - `pre-commit run --all-files` passed. - Full [CI for this head](https://github.com/NVIDIA-NeMo/Gym/actions/runs/37628725215) passed: core coverage **95.26%** against the unchanged 95% threshold; core tests 6,671 passed / 2 skipped, sandbox tests 916 passed / 5 skipped, and all eight server-test shards passed. Production harness and environment behavior is unchanged. This validation reuses saved controlled-run artifacts; no new live-model benchmark evaluation was performed. Full server CI was not run locally. --------- Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>

Follow-up to NVIDIA-NeMo#3766: run its existing P0 harness conformance suite in Gym CI and publish **Harness conformance P0** directly in the PR checks list. Approved PR mirrors and merge-queue revisions probe affected registered harnesses; shared code and dependency changes select all four. Probe jobs install pinned runtimes and retain summaries, logs, and evidence. A separate reporting job publishes a failed P0 check for conformance violations or unavailable measurements, and success when all selected harnesses pass. The check includes failed scenario counts, the first failing assertion or execution issue per harness, copy-pastable local rerun commands, and direct links to the evidence artifacts. After publication, the reporting job also fails if P0 fails or a probe job fails or is cancelled, making the outcome visible in the workflow job list. Only that reporting job receives permission to write checks. The P0 check is optional until maintainers add its fixed name to the repository's required checks; the existing `Nemo_CICD_Test` gate remains independent. Scheduled and manual CI runs exercise all harnesses. Documentation-table refresh is deferred. The probes were exercised on GitHub-hosted Linux in [this CI run](https://github.com/NVIDIA-NeMo/Gym/actions/runs/37681002351). Replaying that evidence through the new reporter produces **1 passed, 2 failed, 1 unavailable**: Pi passes, Codex and OpenCode fail conformance, and Hermes's terminal-model-error scenario yields no rollout. The [updated CI run](https://github.com/NVIDIA-NeMo/Gym/actions/runs/37686345111) published the [P0 check](https://github.com/NVIDIA-NeMo/Gym/runs/113018108848) with that same breakdown. The required CI aggregate passed; Hermes's probe job correctly reports the execution error separately. --------- Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>

…Mo#4048) Hermes retries lose model-attempt ownership when no response ID exists, and local terminal provider failures previously aborted before collection and verification. Propagate invocation headers through SDK retries and retain local provider failures so the verifier and collector receive their evidence. Provider-failed rollouts are always marked `mask_sample: true` and `failure_kind: agent_request_failed`, even if the verifier awards a positive reward. They remain excluded from scores. This intentionally changes the local failure behavior established in NVIDIA-NeMo#3554; sandbox sessions retain their retryable failure path. Depends on NVIDIA-NeMo#4045 for the shared collector correction; this PR is based on that branch to keep its diff harness-specific. No separate issue is needed: the failures reproduce with the conformance suite from NVIDIA-NeMo#3766. All seven conformance scenarios pass with Hermes commit 26bb847a88493342ca1b194e0455b479073ae21d (previously 4/7). Reproduce with `python scripts/run_harness_conformance.py --harness hermes --output /tmp/hermes-conformance` using a fresh output directory. Error/retry ownership satisfies TE-8, with TE-9 as the alternative response-ID contract. `pre-commit run --all-files` is clean. A real gpt-4.1-mini rollout computed 17 × 23, wrote and read 391 through a shell tool, and returned CONFORMANCE_DONE; the independent verifier checked both the file and final answer and awarded 1.0 (two provider calls). The live smoke disabled `chat_template_kwargs_enabled` for OpenAI compatibility. Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>

## What does this PR do? Prefer the agent session interface in the harness conformance runner, with legacy agent `/run` fallback when session setup/close are unimplemented. Follow-up to NVIDIA-NeMo#3766. Detect implementations through the shared agent contract, including inherited handlers, and reuse `single_agent_turn_legacy` EnvironmentServer orchestration. Migrating an adapter to that contract automatically switches its probes. Agent session setup, activation, and cleanup failures remain visible and never trigger legacy fallback. CI preserves the pinned uv executable on PATH for Hermes session setup in later steps. ## Validation Reproduction commands for the Linux probes: ```bash python scripts/run_harness_conformance.py \ --harness codex --harness pi --harness opencode \ --timeout 300 --output results/conformance-sessions python scripts/run_harness_conformance.py \ --harness hermes --scenario tool_success --scenario tool_failure \ --timeout 300 --output results/conformance-hermes ``` ## Rollouts All four real harnesses produced agent session rollouts with verification and Resources cleanup. Pi and Hermes tool-success/failure probes passed P0. The measurements retain Codex trajectory/ownership gaps, Pi retry ownership gaps, OpenCode local-sandbox identity gaps, and terminal model-error failure sidecars. ## Compatibility and benchmark impact Probes using agent sessions require Linux and install pinned harness runtimes during session setup. Legacy probes retain their preinstalled runtime requirement. Runtime metadata records the selected interface and resolved version pins. Probe expectations and benchmark scoring remain unchanged. --------- Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>