[qiaochuz-nv]

What does this PR do?

The gym eval run example in resources_servers/ifbench/README.md now runs. It passed
--prompt-config benchmarks/ifbench/prompts/default.yaml together with
--input resources_servers/ifbench/data/example.jsonl, but every example row already carries
responses_create_params.input. validate_prompt_compatibility (nemo_gym/prompt.py:76-86)
rejects that combination on purpose, so the documented command failed on all five rows before any
generation.

Fixes NVBug 6783016 (Jira NMGYM-57).

Cause

example.jsonl is pre-rendered, and the README asked Gym to render the prompt a second time. The
mismatch dates from #999, which added IFBench with pre-rendered rows and an ng_collect_rollouts +prompt_config=... example after the validation (#861) already existed. #1630 carried the flag
over into the gym eval run form.

Fix

Remove --prompt-config (and the trailing \ before it) from the README command. The data and the
validation stay as they are:

• example.jsonl must keep input: it is the type: example dataset of ifbench_simple_agent
  (resources_servers/ifbench/configs/ifbench.yaml), which has no prompt config.
• The model input does not change. For all five rows (ids 0, 79, 172, 260, 262) the shipped input
  is exactly [{"role": "user", "content": <prompt>}], which is what default.yaml
  (user: "{prompt}") renders.
• The --benchmark ifbench path is unaffected: benchmarks/ifbench/prepare.py writes rows without
  input, matching the prompt_config in benchmarks/ifbench/config.yaml.

Before the fix

Gym eed77e195 (main), the README command against the README's gym env start --benchmark ifbench --model-type openai_model:

gym eval run --no-serve \
    --agent ifbench_benchmark_simple_agent \
    --input resources_servers/ifbench/data/example.jsonl \
    --output /tmp/ifbench_rollouts.jsonl \
    --num-repeats 1 \
    --max-output-tokens 4096 \
    --temperature 0.0 \
    --prompt-config benchmarks/ifbench/prompts/default.yaml

Using prompt config: benchmarks/ifbench/prompts/default.yaml
Reading rows: 5it [00:00, 53773.13it/s]
Traceback (most recent call last):
  ...
  File ".../nemo_gym/rollout_collection.py", in _preprocess_rows_from_config
    validate_prompt_compatibility([row for _, _, row in raw_rows], prompt_cfg)
  File ".../nemo_gym/prompt.py", line 83, in validate_prompt_compatibility
    raise ValueError(
ValueError: Some rows have responses_create_params.input but prompt_config is also specified. These are mutually exclusive. Use one or the other. Violating rows: [0, 1, 2, 3, 4]
  exit 1 in 3.0s

After the fix

Gym 07b4ac277 (this branch), the README command as now documented, same environment:

gym eval run --no-serve \
    --agent ifbench_benchmark_simple_agent \
    --input resources_servers/ifbench/data/example.jsonl \
    --output /tmp/ifbench_rollouts.jsonl \
    --num-repeats 1 \
    --max-output-tokens 4096 \
    --temperature 0.0

  exit 0 in 36.3s
5 rollout rows with unique identities
every row carries responses_create_params/response/reward and a resp_* id
materialized and rollout identity sets match (5 pairs)
failures sidecar present (0 rows)
ifbench_benchmark_simple_agent: mean/reward 0.4 agrees with persisted rollouts
ifbench_benchmark_simple_agent: key_metrics is a subset of agent_metrics

Validation

• QA testcase testcases/environments/ifbench.py (nmfw_tests), which runs the README's own
  commands with a live openai_model backend on an H100 host:
  • before, Gym eed77e195: 0/1 passed (the ValueError above);
  • after, Gym 07b4ac277: 1/1 passed — five real generations, verify results recomputed
    independently, and twelve strict/loose/fractional/binary scoring controls.
• Locally (macOS, Python 3.13.14): the README command without --prompt-config reads all five rows
  and gets past preprocessing; with it, the same ValueError.
• pre-commit run --all-files: passed.
• pytest tests/unit_tests/test_fern_docs_links.py tests/unit_tests/test_cli_main.py: 243 passed.
• Rollout evidence: N/A beyond the QA run above — documentation-only change.

Compatibility

None. Documentation only; no code, config, or data changes.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[qiaochuz-nv]

/ok to test 07b4ac2

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4538-07b4ac277a8f.docs.buildwithfern.com/nemo/gym

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38296457509

[qiaochuz-nv]

Retrospective QA-fix tracking issue: #4700