[rystewart-nvidia]

Summary

This PR adds support for --input to the all-in-one gym eval run flow. Currently, using an explicit input file requires starting servers separately with gym env start, then collecting rollouts with gym eval run --no-serve --input ....

With this change, users can pass an already-prepared JSONL file and let Gym handle server startup, rollout collection, and shutdown in one command. This simplifies running subsets or retrying selected tasks without adding a separate dataset configuration. Any required dataset preparation still happens beforehand.

Closes #3501

Design

• Accepts --input without requiring --split.
• Gives explicit input files precedence over configured splits.
• Uses the supplied prepared JSONL directly, bypassing preparation of configured dataset splits.
• Checks that the input file exists and is a file before starting servers.
• Passes the selected file through to the standard collector or a benchmark’s custom collection driver.
• Preserves existing split preparation and cache reuse when no input file is supplied.
• Updates the CLI help and reference documentation.

Validation

• 446 tests passed across CLI, rollout collection, and documentation checks.
• Regression coverage for explicit inputs, invalid paths, split preparation, cache reuse, and custom collection drivers.
• Live end-to-end smoke test using gym eval run --input, without --split or --no-serve. Ran one task from example_single_tool_call with gpt-5.6-sol, exercising actual server startup, model inference, tool execution, verification, and shutdown. The rollout completed with reward 1.0 and 232 tokens recorded.
• pre-commit run --all-files.
• Fern documentation validation completed with zero errors and one warning.

The orchestration unit tests mock server startup and collection. The separate live smoke test exercised the full flow without mocks.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.