[marta-sd]

What does this PR do?

Runs Stirrup through the harness conformance checks, and fixes what that exposed. #4178 has merged, so this is based on main.

• Terminal-error episodes are inspected from the failure record. When rollouts.jsonl is empty and the scenario expects a terminal model error, the runner inspects rollouts_failures.jsonl and reports delivery="failure_record". For a harness that raises instead of reporting a failed response, the model_error run then shows two failing checks (model.terminal_error and the usage health check) where it showed 31, almost all "required input is unavailable". I measured this with the Stirrup agent as it was before the last fix below.
• The probe can end an episode with a finish tool. Stirrup ends episodes with a finish tool call. The probe calls an advertised finish or submit tool when its required arguments are strings or arrays, tool_checks leaves that call out of the shell-tool witness, and the verifier accepts the answer in the call.
• Tool output given as text parts compares equal to the same text. Stirrup's client sends every tool result as Chat Completions text parts, while the Responses-typed trajectory records a string, so tools.witness_output compared a list to a string. The exact request stays in the model-call capture.
• Stirrup is registered and installed in CI. stirrup joins registry.HARNESSES. ci.install_runtime passes --override overrides.txt when the adapter has one, as nemo_gym/cli/setup_command.py does, and returns after the requirements install because Stirrup installs its runtime in the task sandbox. The launch test imports an adapter module only when HARNESS names it, since a CI job installs only its own adapter's dependencies.
• A request the model server rejects is reported as a failed response, not raised. With test(conformance): check terminal errors reported to environments #4782, a harness must report a failed response carrying the model error to the environment server. Stirrup raised instead: on a model 400 the agent returned HTTP 500, and model_error failed seven checks. For 400, 413 and 422 the runner now ends the episode with the turns completed so far, and the agent returns status: failed, an error that contains the provider's message, and metadata.stop_reason: model_error, as the pi agent does. Other model server errors still fail the episode. An episode whose only request was rejected reports zero usage; a request killed in flight keeps unknown usage.
• The install_runtime test moves out of the unit job. In tests/unit_tests it imported each harness's setup module, which put about 230 statements (156 unexecuted there) into the measured set and took the unit job to 94.76% against its 95% gate. It now lives in scripts/harness_conformance/tests, next to the other conformance tooling tests, which the gate omits. This changes which modules the unit job imports. It adds no coverage: the setup modules have their own tests in the per-agent suites.

No separate issue: this follows the harness swappability work (#4489, #4562, #4178, #4782).

Validation

Run against the final diff:

• Main venv, which has no Stirrup and stands in for the other harness jobs: pytest scripts/harness_conformance/tests tests/unit_tests/harness_capabilities tests/unit_tests/test_harness_conformance_ci.py tests/unit_tests/test_harness_conformance_checks.py tests/unit_tests/test_cli_setup_command.py: 494 passed, 2 skipped.
• Venv built from responses_api_agents/stirrup_agent/requirements.txt with its overrides.txt, and HARNESS=stirrup (the CI validate step): pytest scripts/harness_conformance/tests tests/unit_tests/harness_capabilities responses_api_agents/stirrup_agent/tests: 509 passed, 2 skipped.
• python scripts/run_harness_conformance.py --harness stirrup in that venv: 7 of 7 scenarios fulfilled, and model_error delivers a rollout.
• ruff check and ruff format --check on the changed paths pass, and the pre-commit hooks ran on each commit.

Not run: pre-commit run --all-files, and GitHub Actions on this head. The previous head passed the Stirrup probe job but failed the unit job's coverage gate for the reason in the last item above. The pi, codex and opencode probe jobs fail the same way on main.

Rollouts

N/A for model rollouts. The conformance probes use a scripted model server. A rejected request in a real GDPVal episode was not run end to end. resources_servers/gdpval/app.py special-cases only stop_reason == "wall_time", so a model_error response is judged on whatever deliverables exist.

Compatibility and benchmark impact

• Stirrup: an episode whose model request is rejected (400, 413, 422) is returned as a failed response with its partial work, and the environment server verifies it under its own policy. Before, it failed with a 500 that was retried and then dropped from the score. GDPVal scores can change for runs that hit such rejections, for example a context overflow.
• CI: Stirrup adds a fifth probe job, selected whenever a file outside the docs changes. The conformance checks stay informational and non-blocking.
• The Stirrup agent README documents the model_error behavior.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Documentation added or updated, or N/A with justification.
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[marta-sd]

/ok to test 53cf250

[lafrancef]

Maybe check whether this already exists as a util somewhere

[marta-sd]

I found private _message_content_to_text:

Gym/nemo_gym/responses_converter.py

Line 83 in c2f1e3e

def _message_content_to_text(content: Any) -> str:

They handle corner cases differently but I think it's still worth to re-use. I made the function public and use it for this check in 130027c

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4687-130027c29fe4.docs.buildwithfern.com/nemo/gym

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39455595752