[DanialTaheri]

Part of #3161 (22f94c44116ac263673ea3aafb5f80b558800e25) from super-v3.5-posttraining, ported to main. This PR now contains only the image_tools agent (GRPO) changes. The PivotRL terminal-answer changes from #3161 have been removed from this PR.

What does this PR do?

Caps the image_tools agent's final reward at 1.0, makes its auxiliary tool-use shaping smaller, and gives the agent config train/validation datasets.

Why. The agent adds auxiliary rewards (tool-success bonus, invalid-call and duplicate-call penalties) to the base reward from the delegated resources server. Before this change, the sum was uncapped, so a correct answer plus tool bonuses could score above 1.0, rewarding extra tool calls on rows the model already answers correctly. Capping at 1.0 keeps a correct answer at most 1.0, and the smaller shaping values keep the auxiliary terms small next to the base reward.

responses_api_agents/image_tools_agent/app.py

• Final reward is min(1.0, base_reward + aux_reward) instead of base_reward + aux_reward. Positive bonuses can still raise any score below 1.0; negative penalties still apply at every base score, including a correct answer's 1.0. base_reward and image_tools_aux_reward are still reported separately.

responses_api_agents/image_tools_agent/configs/image_tools_agent.yaml

• Adds train / validation datasets (grpo_train.jsonl / grpo_validation.jsonl) with the required license field. Without license, the server is silently demoted to an "almost-server" that never starts. The JSONLs are internal and not checked in.
• Smaller shaping: tool_success_reward 0.02 → 0.01, tool_success_reward_cap 0.20 → 0.05, invalid_tool_call_penalty -0.05 → -0.01. duplicate_tool_call_penalty is unchanged (-0.02).

Tests and docs

• responses_api_agents/image_tools_agent/tests/test_app.py: the tool-loop test is parametrized over base rewards 0.0 / 0.5 / 0.99 / 1.0 → expected 0.02 / 0.52 / 1.0 / 1.0, on top of main's entrypoint and capture-mode parametrization.
• README.md: environment table regenerated with scripts/update_env_list.py (adds the Image Tools Agent row now that the agent config has datasets).

Validation

• Branch history: cherry-pick of feat(image_tools): reward terminal answers in the pivot environment #3161, review fixes, a merge of main (2b910d2), then 1013cdf restores resources_servers/image_tools to main and removes the PivotRL test file, then fc6772c merges current main again (clean). The net diff against main is the 4 files above.
• Local tests at fc6772c (Python 3.13, nemo-gym[dev] + server requirements): resources_servers/image_tools/tests, responses_api_agents/image_tools_agent/tests and resources_servers/string_match/tests (the agent's grader): 70 passed.
• ruff check and ruff format --check (v0.9.9) pass; scripts/update_env_list.py leaves the README table unchanged.

Rollouts

Smoke rollout run on this branch at fc6772c (job on one GB300 node): 12 GRPO rows through image_tools_simple_agent → string_match (extraction_mode: boxed). Policy: Super-3.5 Omni served with vLLM 0.30 (TP4), temperature 0.6, top-p 0.95, max_output_tokens 32768. Rows are internal BabyVision-style VQA items, one per category plus two multiple-choice; answers are not reproduced here.

row	category	correct	base_reward	image_tools_aux_reward	reward	tool calls	how it ended
0	Paper Folding	no	0.00	+0.01	0.01	1	answered
1	3D Pattern Completion	no	0.00	+0.02	0.02	2	answered
2	3D Views	no	0.00	+0.05	0.05	13	answered
3	Rotation Patterns	no	0.00	-0.01	-0.01	3 (1 invalid)	duplicate warning, then another tool call (invalid, −0.01, ends)
4	Connect the lines	no	0.00	+0.01	0.01	5 (1 invalid)	duplicate warning, then another tool call (invalid, −0.01, ends)
5	Recognize numbers and letters	yes	1.00	+0.00	1.00	0	answered
6	Find the same	no	0.00	+0.05	0.05	20	20th call hit max_tool_calls; max_steps (21) used up before the forced final turn
7	Maze	yes	1.00	+0.03	1.00	3	answered
8	Count Same Patterns	no	0.00	+0.04	0.04	4	answered
9	Mirroring Patterns	no	0.00	+0.04	0.04	4	answered
10	2D Pattern Completion	no	0.00	+0.05	0.05	7	answered
11	Pattern and Color Completion	no	0.00	+0.02	0.02	7 (1 invalid)	duplicate warning, then another tool call (invalid, −0.01, ends)

• 12/12 rollouts completed; the failures sidecar is empty; no response hit the token limit (incomplete_details unset on every row).
• Cap: every row satisfies reward == min(1.0, base_reward + image_tools_aux_reward); no reward exceeds 1.0. Row 7 is the capped case: 1.00 + 0.03 → 1.00. Before this PR the sum was uncapped, and with the old shaping (0.02 per call) this row would have scored 1.06.
• Shaping values: every row's image_tools_aux_reward reconstructs exactly from its tool responses with this PR's config: +0.01 per successful call, −0.02 per duplicate / near-duplicate, −0.01 per invalid call, capped at 0.05 (rows 2, 6 and 10 reach the cap with 13, 20 and 7 calls; with the old values, 0.02 per call capped at 0.20, row 6 would have earned 0.20).
• Mean reward 0.190, mean base_reward 0.167, mean image_tools_aux_reward 0.026. Accuracy (2/12) isn't the point of a 12-row smoke test.
• Observation, not changed by this PR: with max_steps: 21 and max_tool_calls: 20, a rollout that uses all 20 calls gets the forced "final turn" prompt on its last step, so the model never answers it (row 6). max_steps ≥ max_tool_calls + 2 would leave room for that answer; happy to follow up separately if useful.

Compatibility and benchmark impact

• image_tools_agent rewards are capped at 1.0 and the auxiliary terms are smaller, so reward scales of runs using this agent will differ from earlier runs.
• No change to resources_servers/image_tools or the PivotRL path.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Documentation added or updated, or N/A with justification.
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yfw]

/ok to test fc6772c

[DanialTaheri]

@yfw do you have any other comments?

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40953490358