Harness evidence checks conformance

fern/versions/latest/pages/observability/harness-conformance.mdx

@@ -0,0 +1,121 @@
+---
+title: "Harness Conformance"
+description: "Check P0 trajectory evidence and compare harness artifact coverage."
+position: 3
+---
+
+# Harness Conformance
+
+Check the evidence retained by a harness/benchmark pair after collecting an evaluation:
+
+```bash
+python scripts/inspect_harness_conformance.py \
+    --bundle results/my-harness/rollouts.jsonl \
+    --output results/my-harness/evidence
+```
+
+The `gym-p0/v1` profile checks the currently supported P0 trajectory evidence
+(TE-1–TE-9). Every applicable TE-1–TE-7 must pass, plus **TE-8 or TE-9** on
+every rollout. It replaces the earlier experimental C0–C11 profiles.
+
+| Evidence | Persisted contract |
+| --- | --- |
+| TE-1: model call status | Exact call/model identity, HTTP or transport outcome, response status and termination metadata preserved from the response. Optional call clocks are checked when present. |
+| TE-2: token counts | Supplied prompt, completion, reasoning, total and cached counts preserved on successful and failed attempts. Provider omission remains unknown. Availability is reported separately. |
+| TE-3: steps | Task, repeat, rollout and step identity; timestamp; model-visible prompt; answer/tool or reasoning fields; resolution or explicit unknown. No step is inferred from an HTTP call. |
+| TE-4: history | Ordered saved model inputs and outputs, including tool results and changed context. Missing media, opaque content and unavailable prior-response history fail. |
+| TE-5: tool record | Execution id/name/status, originating request with dispatched arguments, and output/error. Tool clocks are outside P0. A tool request alone does not prove execution. |
+| TE-6: verifier outcome | The current Gym verification surface: finite rollout reward, including for masked results, and correctly scoped terminal resolution when present. `mask_sample=true` excludes the reward from scores. Masked or explicitly incomplete verification requires nonblank `failure_kind` and `failure_reason`. Extended verifier scope/version and judge provenance are not certified by this profile. |
+| TE-7: payloads | Saved request and response/error bodies. Empty returned bodies and explicit no-response transport failures remain distinct. |
+| TE-8: run join | Exactly one explicit invocation owner per captured attempt, including attempts without response ids; consistent parent relationships and exact references. |
+| TE-9: step join | Every retained policy attempt belongs to exactly one identified step. Retries remain attempts on that step. Explicit compaction-helper refs are excluded from policy steps. |
+
+TE-2 distinguishes preservation from availability: an absent provider metric is
+not an available zero or a five-count availability pass. Derived totals follow
+Gym's documented prompt-plus-completion convention. Anthropic prompt totals
+include cache reads and cache creation. Missing usage remains visible to health.
+
+The profile covers retained artifacts. It does not certify live failure/retry
+scenario coverage, closed-run completeness, health, accuracy, or deployment
+readiness. Independent completeness required by some health consumers is separate
+from TE-9's retained reference check. Infra taxonomy (TE-10) and P1 efficiency
+requirements are outside this profile.
+
+## Applicability and delivery
+
+By default the pair has policy steps, tools, and a verifier. Use `--no-tools`,
+`--no-verifier`, or `--no-steps` only when that feature does not apply to the
+pair. Missing evidence never automatically becomes N/A. Contradictory retained
+evidence rejects an exclusion. A step-free pair still needs TE-8.
+
+`--bundle` accepts a JSONL file, or a directory with exactly one `rollouts.jsonl`
+or `evaluator_rollouts.jsonl`, at its root or under `artifacts/`.
+
+The delivery surface is the rollout JSONL, including `ng_trajectory` payloads.
+An optional `--capture-dir /absolute/path/to/model-calls` cross-checks exact
+identity sets, payloads, metadata and incomplete markers. It cannot repair a
+missing JSONL payload for a passing delivery verdict.
+
+Each inspection atomically publishes an immutable directory containing:
+
+- `evidence_summary.json`: gate, per-TE counts, token availability, applicability,
+  source/checker/schema hashes and qualification limits.
+- `evidence_results.jsonl`: per-rollout findings and field locations.
+- `evidence_report.md`: evidence table.
+
+Exit codes are **0** for a met gate, **1** for unmet evidence, and **2** for an
+input/checker error. An empty input is an error. Reports omit payload contents.
+Identical inputs and checker versions reuse the same report directory.
+
+## Compare harnesses
+
+Use the same task and capture settings when comparing harnesses:
+
+```bash
+python scripts/inspect_harness_conformance.py matrix \
+    --harness opencode=results/opencode/rollouts.jsonl \
+    --harness pi=results/pi/rollouts.jsonl \
+    --harness codex=results/codex/rollouts.jsonl \
+    --harness hermes=results/hermes/rollouts.jsonl \
+    --output results/harness-evidence
+```
+
+The command writes per-harness reports plus `harness_evidence.md` and
+`harness_evidence.json`. Applicability flags apply to every input pair in that
+comparison. A matrix FAIL identifies missing or inconsistent evidence; it is not
+a benchmark accuracy result.
+
+Checker unit tests use hand-authored synthetic evidence contracts, independent of
+harness implementations. Mutations verify missing, conflicting, and malformed
+evidence without prescribing which checks a real harness should pass.
+
+The [Trajectory Capability Matrix](/main/reference/trajectory-capabilities) is
+generated separately from fresh scripted harness runs. Its regeneration command
+requires passing runner, harness adapter, and checker unit tests at a specific
+Gym commit before replacing the table.
+
+## Onboard a producer
+
+Enable `observability_enabled: true` and set `model_call_capture_dir` to a shared
+absolute directory. Route requests through Gym's correlated model URL and retain
+all attempts. See [Model-call Capture](/main/model-server/model-call-capture).
+
+Return `ng_agent_observations` with `AgentInvocation` and exact `ModelCallRef`
+keys (`model_call_id`, or unique `model_ref` plus `response_id`). Forward the
+invocation id as `x-session-id` for failed-call joins through the collector.
+A timestamp or shared model name never establishes ownership.
+
+Supply explicit `TrajectoryTurn` records with the actual prompt and decision
+boundaries. Preserve tool request/result pairs and `ToolCallObservation` outcomes,
+including failures and terminal submission. Keep resolution unknown on
+intermediate steps. Retain the resources server's verification response.
+See [Rollout Evidence](/main/observability/rollout-evidence).
+
+Inspect a small collected evaluation, fix producer gaps, and collect again.
+Run [rollout health checks](/main/evaluation/rollout-health) separately. Evidence
+checks neither rewrite `quality_summary.json` nor treat a truthfully recorded
+failed model call as missing evidence.
+
+```bash
+python -m pytest tests/unit_tests/harness_capabilities -q
+```

fern/versions/latest/pages/reference/trajectory-capabilities.mdx

@@ -4,21 +4,109 @@ description: "Rollout-evidence coverage by agent and execution path."
 position: 5
 ---
 
+# Trajectory Capability Matrix
+
 Use this reference to check which evidence an agent path provides. Start with
 [Rollout Evidence](/main/observability/rollout-evidence) for collection and artifact relationships.
 
+## Generated trajectory evidence (TE)
+
+This table runs the local harnesses against scripted model replies, tool executions,
+retries, and failures, then checks the collected rollouts. The
+[TE-1–TE-9 contracts](/main/observability/harness-conformance) cover model outcomes,
+token preservation, steps, history, tools, verification, payloads, and call joins.
+TE-2 checks that supplied usage is preserved; it does not require every provider
+metric to be available. Remote sandboxes, multimodal inputs, compaction, parallel
+execution, TE-10, and P1 are outside this suite.
+
+{/* BEGIN GENERATED TE TABLE */}
+
+Gym commit: [`939cace5af1436789ead920d0f0b3e663784ad73`](https://github.com/NVIDIA-NeMo/Gym/commit/939cace5af1436789ead920d0f0b3e663784ad73).
+Suite: `harness-p0-probes/v1`. [Generation report](../../../../assets/trajectory-capabilities/425964eadebb395fea46f594177e2373b4195530934ed44be4ead0d755892c90.json).
+
+Cells show **passing / exercised / required** scenarios. PASS requires every required scenario;
+FAIL includes missing exercise or missing evidence. **Not run** makes no capability claim.
+
+| Harness | TE-1 | TE-2 | TE-3 | TE-4 | TE-5 | TE-6 | TE-7 | TE-8 | TE-9 | P0 |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| `opencode` | PASS 7/7/7 | PASS 7/7/7 | FAIL 0/6/6 | PASS 7/7/7 | PASS 6/6/6 | PASS 6/6/6 | PASS 7/7/7 | PASS 7/7/7 | FAIL 0/7/7 | FAIL |
+| `pi` | PASS 7/7/7 | PASS 7/7/7 | FAIL 0/6/6 | PASS 7/7/7 | PASS 6/6/6 | PASS 6/6/6 | PASS 7/7/7 | FAIL 4/7/7 | FAIL 0/7/7 | FAIL |
+| `codex` | FAIL 6/6/7 | FAIL 6/6/7 | FAIL 0/5/6 | FAIL 6/6/7 | FAIL 0/5/6 | FAIL 5/5/6 | FAIL 6/6/7 | FAIL 0/6/7 | FAIL 0/6/7 | FAIL |
+| `hermes` | Not run | Not run | Not run | Not run | Not run | Not run | Not run | Not run | Not run | Not run |
+
+P0 requires TE-1–TE-7 and either TE-8 or TE-9 **per scenario**.
+
+### Tested dependencies
+
+- `opencode`: `1.17.11`
+- `pi`: `0.80.2`
+- `codex`: `codex-cli 0.155.0-alpha.9.2`
+
+### Unit-test gates
+
+| Suite | Passed | Skipped |
+| --- | ---: | ---: |
+| `runner` | 49 | 0 |
+| `opencode` | 82 | 0 |
+| `pi` | 34 | 0 |
+| `codex` | 46 | 0 |
+| `checker` | 77 | 0 |
+
+{/* END GENERATED TE TABLE */}
+
+### Regenerate the TE table
+
+From a Gym checkout matching the full source commit SHA, install Gym's development
+dependencies and the selected agents' Python dependencies and runtimes. Run:
+
+```bash
+python -m scripts.harness_conformance.table \
+    --commit <40-character-gym-commit-sha> \
+    --output /absolute/path/to/new-conformance-results
+```
+
+The default selects OpenCode, Pi, Codex, and Hermes. Repeat `--harness NAME` to
+select fewer; all others are explicitly marked **Not run**. Every selected harness
+runs the full scenario suite. The probes execute local shell tools in fresh working
+directories and require no model service or model credentials. See
+[`scripts/harness_conformance/README.md`](https://github.com/NVIDIA-NeMo/Gym/blob/main/scripts/harness_conformance/README.md)
+for runtime setup, scenarios, and the execution scope.
+
+Regeneration requires passing runner unit tests, each selected harness's adapter
+unit tests, and checker unit tests at that commit. Failing, empty, or entirely
+skipped suites stop regeneration. Skipped tests in otherwise passing suites are
+reported. Source changes during testing or probing, missing runtimes, and
+execution/checker errors also stop publication, preserving the existing table.
+An unmet evidence requirement produces a FAIL cell and can be published.
+
+The command replaces only the generated block above and writes a content-addressed
+JSON generation report under `fern/assets/trajectory-capabilities/`. The report
+records the Gym commit, runtime versions/source hashes, test counts and log hashes,
+scenario results, and artifact hashes. Full logs, captures, and rollouts remain in
+`--output`; keep them when reviewing a generation. Save source changes first, then
+commit the generated page and report in a subsequent commit so their source pin
+remains stable. Git and jj checkouts are supported.
+
+## Deprecated coverage table (C-table)
+
+<Warning>
+The manually maintained C-table below is deprecated. Use the generated TE table
+above for tested evidence coverage at a specific commit. The historical claims
+remain here temporarily for comparison.
+</Warning>
+
 ✅ **Full** means complete for the documented path; 🟡 **Partial** means some fields or paths are missing;
 ❌ **Unavailable** means the path does not supply that evidence. These are manually maintained coverage
 claims, not guarantees for every configuration.
 
 <Note>
 The statuses below retain the coverage reported in [main at 88d9d453](https://github.com/NVIDIA-NeMo/Gym/blob/88d9d45385f15b1351cdd055fff396186000f84f/fern/versions/latest/pages/reference/trajectory-capabilities.mdx).
 That source does not record a full runtime-validation date or per-row test configuration.
-Contributors changing an agent should update its row with the config, execution path, and validation
-evidence in the same pull request.
+New validation results belong in the generated TE table above; these historical rows are retained
+for comparison.
 </Note>
 
-## Model Calls and History
+### Model Calls and History
 
 Capture requires `observability_enabled`, a shared `model_call_capture_dir`, and a rollout-prefixed
 Gym Model Server URL. Direct-provider routes bypass capture. Token counts depend on provider usage
@@ -63,7 +151,7 @@ fields. History may come from captured payloads or harness conversations.
 | `toolsandbox_agent` | ✅ | ✅ | ✅ |
 | `verifiers_agent` | ✅ | ✅ | ✅ |
 
-## Agent Activity
+### Agent Activity
 
 Turns require explicit semantic-turn records. Tool execution covers output, status, and timestamps;
 parallel timing requires independent tool intervals. Resource/sandbox coverage means that the
@@ -85,7 +173,7 @@ benchmark path emits standardized agent evidence, not that every resource metric
 The other agents listed above report these four dimensions as unavailable. Model HTTP capture
 alone does not establish tool execution or sandbox activity.
 
-## Path Qualifications
+### Path Qualifications
 
 | Path | Qualification |
 | --- | --- |

nemo_gym/harness_capabilities/README.md

@@ -0,0 +1,85 @@
+# Harness trajectory evidence
+
+Check P0 evidence from a collected evaluation:
+
+```bash
+python scripts/inspect_harness_conformance.py \
+    --bundle results/my-harness/rollouts.jsonl \
+    --output results/my-harness/evidence
+```
+
+Use `matrix --harness NAME=PATH` to compare multiple harnesses. The checker uses
+TE-1–TE-9 and the `gym-p0/v1` profile. Reports are `evidence_summary.json`,
+`evidence_results.jsonl`, and `evidence_report.md`.
+
+For example, this excerpt from `evidence_report.md` shows retained model-call
+evidence passing while one tool record is missing its name:
+
+```markdown
+Profile: `gym-p0/v1`. Gate: **not_fulfilled**. Records: 1.
+
+| Evidence | Artifact verdict | Passing / applicable records |
+|---|---|---|
+| TE-1: model_call_status | fulfilled | 1/1 |
+| TE-5: tool_record | not_fulfilled | 0/1 |
+```
+
+One corresponding entry in that rollout's `findings` array in
+`evidence_results.jsonl` identifies the missing field:
+
+```json
+{
+  "evidence": "TE-5",
+  "assertion": "identity.required",
+  "location": "rollouts.jsonl:1/ng_trajectory/tool_calls/0/tool_name",
+  "reason": "nonempty value is required"
+}
+```
+
+These verdicts describe evidence conformance, not task success. A rollout with
+zero reward can have fully conforming evidence; a successful task can have
+incomplete evidence.
+
+Known producer gap: `miniswe_sandboxed_agent` retains its final submit tool call
+as `incomplete`, without output or error evidence. This fails TE-5 with
+`tool.terminal` and `tool.outcome`, so the P0 gate is `not_fulfilled` for those
+rollouts. mini-SWE must save a terminal result correlated with the submit call
+before exiting, and Gym must retain it in the projected evidence. See the
+[mini-SWE observability limits](../../responses_api_agents/miniswe_sandboxed_agent/README.md#native-behavior-and-observability).
+
+TE-6 requires a finite numeric `reward` even when `mask_sample=true` excludes it
+from scoring. Masked results and explicitly incomplete verification require
+nonblank `failure_kind` and `failure_reason`; a masked `reward=0.0` is valid
+evidence. The Gym profile uses the default unmasked behavior when `mask_sample`
+is omitted and checks `evaluation_completed` when supplied. Both flags must be
+booleans when present.
+
+Evidence objects must validate Gym's shared models at these paths:
+
+| JSON path | Model |
+|---|---|
+| `ng_model_call_capture.calls[]` | `ModelCallRecord` |
+| `ng_trajectory.model_calls[]` | `TrajectoryModelCall` |
+| `ng_trajectory.turns[]` | `TrajectoryTurn` |
+| `ng_trajectory.invocations[]` | `AgentInvocation` |
+| `ng_trajectory.tool_calls[]` | `TrajectoryToolCall` |
+| `ng_agent_observations.records[]` | `AgentObservationRecord` (discriminated by `kind`) |
+
+Validation is strict and reports the object's JSON path. Invalid objects are
+record integrity failures; missing evidence is evaluated by the TE checks and
+explicit applicability. Both trajectory and observation representations are
+validated when present. Invocation and tool checks retain their supported
+fallback between these paths.
+
+TE requirements apply on top of model validity. For example, TE-3 requires a
+non-null `question`, a non-null `answer` or `reasoning_content`, and an explicitly
+present `resolved` value (including `null` for unknown). TE-5 requires a tool name
+and terminal status; TE-1 requires model/server identity and a supported dialect;
+TE-2 requires nonnegative counts consistent with retained provider usage. Reports
+hash the path/model registry and shared model sources as well as the checker.
+
+For live harness checks and regenerating the capability table, see the
+[runner guide](../../scripts/harness_conformance/README.md).
+
+See [Harness Conformance](../../fern/versions/latest/pages/observability/harness-conformance.mdx)
+for contracts, applicability, matrix usage, producer onboarding, and limitations.

nemo_gym/harness_capabilities/__init__.py

@@ -0,0 +1,14 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Portable artifact contracts for harness observability.
+
+Readers supply native Gym JSON;
+artifact conformance is deliberately separate from behavioral qualification.
+"""
+
+from .checker import inspect_record
+
+
+__all__ = ["inspect_record"]
+__version__ = "0.2.0"

nemo_gym/harness_capabilities/__main__.py

@@ -0,0 +1,7 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+from .cli import main
+
+
+raise SystemExit(main())