Skip to content

fix(conformance): probe through /run unless agent sessions are requested - #4635

Closed
dims wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
dims:fix-conformance
Closed

dims wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
dims:fix-conformance

Conversation

@dims

@dims dims commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Harness conformance P0 on main has been red for all four harnesses since #4489 made the agent-session interface the default for every adapter that implements session setup and close. The session path is not yet conformant for any registered adapter: terminal model errors become failure sidecars instead of rollouts (opencode, codex, pi), codex rollouts have no turns, hermes and pi fail the ownership checks, and opencode fails the sandbox identity check. The P0 check and the published table therefore measure nothing.

This makes the /run path the default again and keeps the session path as an explicit opt-in, --agent-interface sessions, on the runner and the episode. Requesting sessions for an adapter without session hooks is an execution error. The default can move to sessions once the registered adapters pass P0 through it.

Validation

pre-commit run --files scripts/harness_conformance/README.md scripts/harness_conformance/episode.py scripts/harness_conformance/runner.py scripts/harness_conformance/tests/test_runner.py
python -m pytest -o addopts= --import-mode=importlib -q scripts/harness_conformance/tests tests/unit_tests/harness_capabilities

On Linux x86_64 at this branch's base (ac51e85), the same tests plus the four adapter test directories (1580 passed, 9 environment skips), then the default probes:

python scripts/run_harness_conformance.py --harness <harness> --output <dir>

opencode, hermes, codex and pi each exit 0 with all seven scenarios fulfilled, the result main had before #4489. The opt-in path still runs:

python scripts/run_harness_conformance.py --harness pi --scenario tool_success --agent-interface sessions --timeout 300 --output <dir>

pi and hermes exit 0 with agent_interface: agent_sessions in runtime.json.

Rollouts

No environment or agent code changes. The Linux runs above produced one conformance report per harness with seven fulfilled scenarios each.

Compatibility and benchmark impact

CI probes return to the /run path. suite.json records the selected agent_interface. The rerun commands in the P0 check stay valid. No benchmark impact.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification.
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
  • New source files include the required Apache-2.0 SPDX header.
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

Preferring the agent-session interface for every adapter that implements
session setup and close removed the P0 signal for all four registered
harnesses: the session path is not yet conformant for any of them, so
main has been red on Harness conformance since that change.

Make the /run path the default again and keep the session path as an
explicit opt-in, --agent-interface sessions, on the runner and the
episode. Requesting sessions for an adapter without session hooks is an
execution error. The default can move to sessions once the registered
adapters pass P0 through it.

Signed-off-by: Davanum Srinivas <dsrinivas@nvidia.com>
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 15, 2026 14:38
@yaoyu-33 yaoyu-33 added area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer labels Oct 15, 2026
@ananthsub
ananthsub requested a review from lafrancef October 15, 2026 19:13
@dims dims closed this Oct 16, 2026

This branch was previously deployed

1 inactive deployment
public — ececfe75 Deployed Oct 15, 2026 by copy-pr-bot[bot] via release / finalize / notify #5500
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants