Repository navigation
feat: add ifbench - #999
Merged
Merged
Conversation
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
fsiino-nvidia
marked this pull request as ready for review
April 7, 2026 18:27
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
bxyu-nvidia
approved these changes
Apr 7, 2026
cmunley1
pushed a commit
that referenced
this pull request
Apr 8, 2026
wandb: https://wandb.ai/nvidia/fsiino-gym-dev/runs/y00euryh ``` Key metrics for ifbench_benchmark_simple_agent: { "mean/reward": 0.573469387755102, "mean/follow_all_instructions": 0.5421768707482993, "mean/input_tokens": 100.78503401360544, "mean/output_tokens": 4974.808163265307, "mean/total_tokens": 5075.593197278911 } ``` --------- Signed-off-by: Frankie Siino <fsiino@nvidia.com> Signed-off-by: cmunley1 <cmunley@nvidia.com>
khazic
pushed a commit
to khazic/Gym-lao
that referenced
this pull request
Oct 13, 2026
…DIA-NeMo#4538) ## What does this PR do? The `gym eval run` example in `resources_servers/ifbench/README.md` now runs. It passed `--prompt-config benchmarks/ifbench/prompts/default.yaml` together with `--input resources_servers/ifbench/data/example.jsonl`, but every example row already carries `responses_create_params.input`. `validate_prompt_compatibility` (`nemo_gym/prompt.py:76-86`) rejects that combination on purpose, so the documented command failed on all five rows before any generation. Fixes NVBug [6783016](https://nvbugspro.nvidia.com/bug/6783016) (Jira NMGYM-57). ### Cause `example.jsonl` is pre-rendered, and the README asked Gym to render the prompt a second time. The mismatch dates from NVIDIA-NeMo#999, which added IFBench with pre-rendered rows and an `ng_collect_rollouts +prompt_config=...` example after the validation (NVIDIA-NeMo#861) already existed. NVIDIA-NeMo#1630 carried the flag over into the `gym eval run` form. ### Fix Remove `--prompt-config` (and the trailing `\` before it) from the README command. The data and the validation stay as they are: - `example.jsonl` must keep `input`: it is the `type: example` dataset of `ifbench_simple_agent` (`resources_servers/ifbench/configs/ifbench.yaml`), which has no prompt config. - The model input does not change. For all five rows (ids 0, 79, 172, 260, 262) the shipped `input` is exactly `[{"role": "user", "content": <prompt>}]`, which is what `default.yaml` (`user: "{prompt}"`) renders. - The `--benchmark ifbench` path is unaffected: `benchmarks/ifbench/prepare.py` writes rows without `input`, matching the `prompt_config` in `benchmarks/ifbench/config.yaml`. ### Before the fix Gym `eed77e195` (main), the README command against the README's `gym env start --benchmark ifbench --model-type openai_model`: ```bash gym eval run --no-serve \ --agent ifbench_benchmark_simple_agent \ --input resources_servers/ifbench/data/example.jsonl \ --output /tmp/ifbench_rollouts.jsonl \ --num-repeats 1 \ --max-output-tokens 4096 \ --temperature 0.0 \ --prompt-config benchmarks/ifbench/prompts/default.yaml ``` ```text Using prompt config: benchmarks/ifbench/prompts/default.yaml Reading rows: 5it [00:00, 53773.13it/s] Traceback (most recent call last): ... File ".../nemo_gym/rollout_collection.py", in _preprocess_rows_from_config validate_prompt_compatibility([row for _, _, row in raw_rows], prompt_cfg) File ".../nemo_gym/prompt.py", line 83, in validate_prompt_compatibility raise ValueError( ValueError: Some rows have responses_create_params.input but prompt_config is also specified. These are mutually exclusive. Use one or the other. Violating rows: [0, 1, 2, 3, 4] exit 1 in 3.0s ``` ### After the fix Gym `07b4ac277` (this branch), the README command as now documented, same environment: ```bash gym eval run --no-serve \ --agent ifbench_benchmark_simple_agent \ --input resources_servers/ifbench/data/example.jsonl \ --output /tmp/ifbench_rollouts.jsonl \ --num-repeats 1 \ --max-output-tokens 4096 \ --temperature 0.0 ``` ```text exit 0 in 36.3s 5 rollout rows with unique identities every row carries responses_create_params/response/reward and a resp_* id materialized and rollout identity sets match (5 pairs) failures sidecar present (0 rows) ifbench_benchmark_simple_agent: mean/reward 0.4 agrees with persisted rollouts ifbench_benchmark_simple_agent: key_metrics is a subset of agent_metrics ``` ### Validation - QA testcase `testcases/environments/ifbench.py` (nmfw_tests), which runs the README's own commands with a live `openai_model` backend on an H100 host: - before, Gym `eed77e195`: 0/1 passed (the `ValueError` above); - after, Gym `07b4ac277`: 1/1 passed — five real generations, verify results recomputed independently, and twelve strict/loose/fractional/binary scoring controls. - Locally (macOS, Python 3.13.14): the README command without `--prompt-config` reads all five rows and gets past preprocessing; with it, the same `ValueError`. - `pre-commit run --all-files`: passed. - `pytest tests/unit_tests/test_fern_docs_links.py tests/unit_tests/test_cli_main.py`: 243 passed. - Rollout evidence: N/A beyond the QA run above — documentation-only change. ### Compatibility None. Documentation only; no code, config, or data changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: qiaochuz <qiaochuz@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
wandb: https://wandb.ai/nvidia/fsiino-gym-dev/runs/y00euryh