Repository navigation
feat(eval): bidirectional trajectory conversion — ingress (ATIF → Gym-native) and egress (Gym-native → ATIF) #2468
Description
Summary
Gym should support ATIF trajectory conversion in both directions:
- Ingress: score an external ATIF v1.7 trajectory with an existing Gym verifier via
gym eval reverify, without rerunning the model or agent harness. - Egress: export a completed Gym-native trajectory (
ng_trajectory) as an ATIF v1.7 file for downstream consumers that don't read Gym-native output.
Why
Today gym eval reverify only reads Gym's own output.jsonl. Teams that ran an agent outside Gym can't apply a Gym verifier without rerunning the rollout, even when they have a standard ATIF trajectory. Conversely, teams that want a Gym-native trajectory visible in external tooling have no export path. Both are the same shape of problem — a format converter at the edge, without changing Gym's own trajectory model or verifier dispatch — and belong in the same piece of work.
Approach
- Conversion happens at the format boundary, not inside Gym's core trajectory model or verifier logic.
- Ingress: ATIF input joins to the original materialized Gym task (expected answer,
verifier_metadata, resources-server routing) via an explicit manifest, then flows through Gym's existing verifier dispatch. Initially restricted to stateless verifiers. - Egress: a completed Gym-native rollout exports as one ATIF file per rollout, with a manifest that pins provenance (source hash, trajectory/session ID).
- Initial format: ATIF v1.7 in both directions.
Flags below are illustrative, not a locked contract:
# Ingress: score an external ATIF trajectory with an existing Gym verifier
gym eval reverify \
--input-format atif \
--tasks materialized-tasks.jsonl \
--atif-manifest relay-trajectories.jsonl \
--output rewards.jsonl
# Egress: export a completed Gym run as ATIF
gym eval export \
--format atif \
--rollouts results/rollouts.jsonl \
--output-dir results/atifOut of Scope
- Live/streaming export during a running rollout.
- Any change to Gym's own verifier or trajectory schema.
- Stateful verifier support in the first pass.
- Broader reuse of an ingressed trajectory beyond reverify (e.g. aggregate metrics, cross-harness comparison) — worth revisiting once the core conversion is proven, not part of this pass.
Acceptance Criteria
- An external ATIF v1.7 trajectory, joined to its materialized Gym task via manifest, scores correctly through
gym eval reverifywithout rerunning the agent. - A completed Gym-native rollout exports as a valid ATIF v1.7 file via
gym eval export.
Activity
[-][feat] gym eval reverify for external rollouts — score any Relay-supported trajectory with a Gym verifier[/-][+]feat(eval): accept ATIF v1.7 trajectories in gym eval reverify[/+]afourniernv commentedon Aug 25, 2026
I spiked this end to end and opened #2737. Two corrections to the issue body based on the working implementation:
nvidia-nat-atif, Relay, or Harbor as a runtime dependency. The PR owns a narrow, version-gated parser for the tested Relay ATIF v1.7 subset.--force, resume caching, append, and judge-failure recovery are rejected rather than used to broaden that safety boundary.The implemented CLI is:
gym eval reverify \ --input-format atif \ --inputs materialized-inputs.jsonl \ --atif-manifest relay-trajectories.jsonl \ --output rewards.jsonlThe manifest explicitly maps each ATIF trajectory to the materialized Gym task and rollout and may pin its SHA-256. The implementation verifies canonical ATIF output against Relay's preserved raw provider response, requires complete tool-result correlation, persists safe provenance without source paths, and rejects unsupported/lossy shapes.
The live smoke used a real Codex -> Relay Responses trajectory and Gym's actual stateless MCQA verifier. It preserved the semantic
call-*ID fixed in NVIDIA/NeMo-Relay#887, returned reward1.0, and produced aggregate accuracy100%.I cannot edit the issue body because it was opened by another account, so recording the corrected contract here. The PR description contains the complete supported/unsupported boundary and validation receipt.
fix(observability): correlate Responses tool calls in ATIF (#887)
[-]feat(eval): accept ATIF v1.7 trajectories in gym eval reverify[/-][+]feat(eval): accept ATIF trajectories in gym eval reverify[/+][-]feat(eval): accept ATIF trajectories in gym eval reverify[/-][+]feat(eval): bidirectional trajectory conversion — ingress (ATIF → Gym-native) and egress (Gym-native → ATIF)[/+]feat(eval): reverify Relay ATIF trajectories (NVIDIA-NeMo#2737)
feat(eval): export Gym trajectories as ATIF (NVIDIA-NeMo#2767)