[qiaochuz-nv]

What does this PR do?

The nested conversational tool-use environment (resources_servers/conversational_tool_use_simulation with the
conversational_tool_use/simulation agent) now starts from a source checkout, and its rollouts are graded and
masked correctly. Before this PR its stack never came up: every model server bound to the agent crashed during
startup. Behind that crash were two more defects, which this PR also fixes. The trajectory judge often returned JSON
that failed validation, and the server's mask decision never reached the mask_sample field that Gym and trainers read.

Fixes NVBug 6847461. Closes #4497.

Cause

1. Startup crash. ServerClient.assistant_message_header (nemo_gym/server_utils.py:825, added by [1/5] feat(opencode): link assistant messages to captured model calls #3711)
   imports responses_api_agents.{harness} with the raw config key. Nested harnesses are keyed by directory path,
   so conversational_tool_use/simulation raises ModuleNotFoundError in policy_model setup
   (base_responses_api_model.py:286). The venv blocker in the same NVBug (a pin to nemo-gym==<version>) was
   already fixed by fix(cli): treat servers nested three levels deep as editable installs #4464.
2. Unparseable judge output. The judge prompt contains Evaluation.model_json_schema() as text, and the reply
   is parsed with model_validate_json, with no decoding constraint (since feat: add end-to-end synthetic conversational tool-use generation and rollouts #2111). With the QA model (temperature 1.0), the
   step-0 judgement failed all 3 attempts in 3 of 3 runs. Replies either wrapped the instance in
   {"properties": {...}} or left a string unquoted, so the row got verification_generation_error.
3. Mask dropped. feat: add end-to-end synthetic conversational tool-use generation and rollouts #2111 reported the mask decision only as instance_config.mask_sample. feat(core): report infrastructure failure from verify, not just a reward #2611 later made
   BaseVerifyResponse.mask_sample the contract. This environment was never migrated, so simulator, tool, and judge
   failures reached rollouts with mask_sample: false and reward 0.

Fix

1. Map / to . before importing the harness package.
2. When judge_responses_create_params.text is not configured, the judge requests Responses
   text.format (json_schema) for Evaluation, or for UserAgentEnvironmentEvaluation in combined mode. A
   configured text still wins. openai_model forwards it to the provider; the chat converter maps it to
   response_format.
3. The resources server's /verify and the agent's transient-failure response set top-level mask_sample.
   instance_config is removed, and the checked-in example_rollouts.jsonl is migrated. The README states the
   masking rule.

Rejected: catching ImportError in assistant_message_header. It would hide a declared header from any nested
harness, and the new test fails on it.

Before the fix

Gym 4a101eeb5 (main), QA case conversational_tool_use_simulation:

./run.sh test conversational_tool_use_simulation --ref 4a101eeb5 --no-ai-log

nemo_gym/base_responses_api_model.py", line 286, in setup_webserver
    assistant_message_header=self.server_client.assistant_message_header(self.config.name),
nemo_gym/server_utils.py", line 825, in assistant_message_header
ModuleNotFoundError: No module named 'responses_api_agents.conversational_tool_use/simulation'
RuntimeError: Process `policy_model` finished unexpectedly!
  ERROR (30.2s): StackStartupError: `gym env start` exited with code 1 during child startup
conversational_tool_use_simulation: 0/3 passed in 90.6s

With only the import fixed (intermediate commit), the stack started but the judge failed to parse:

FAIL (78.7s): original conversation has no generation or verification failures: expected [], got ['verification_generation_error']
generation_error = 2 validation errors for Evaluation
success
  Field required [type=missing, input_value={'properties': {'success'...tion', 'type': 'object'}, input_type=dict]

After the fix

Gym 29d57f66c (this branch before a rebase onto 977583d3d; the one new main commit touches no file in this diff), same command:

./run.sh test conversational_tool_use_simulation --ref 29d57f66c --no-ai-log

PASS   373.9s  the canonical one-example conversation uses live roles and verifies session disposal and grading gates
PASS    79.1s  enable_llm_judge=false scores a complete conversation 1.0 without calling the judge
PASS    49.8s  complete_trajectory_combined_evaluation judges the whole conversation once with per-role verdicts
conversational_tool_use_simulation: 3/3 passed in 502.8s

Validation

• QA testcase conversational_tool_use_simulation (3 cases: canonical message-level judge, enable_llm_judge=false,
  complete_trajectory_combined_evaluation): 0/3 on main (stack startup), 3/3 on this branch. The canonical case also replays the recorded conversation and checks the step limit (excessive_length, not masked), ###TRANSFER### with both scope labels, and rejection of malformed history (25 direct resources-server calls).
• Judge-format probe against the QA endpoint, 6 calls each: prompt-only schema gave 5/6 valid replies, text.format
  gave 6/6.
• After the rebase (head a3a6ab1f4), I reran test_server_utils.py (173), gym env test for the server (78), and the agent tests (36): all pass.
• tests/unit_tests/test_server_utils.py: 173 passed. The new test_model_header_imports_nested_harness_key fails on
  main, and also fails on a catch-ImportError fix.
• gym env test --resources-server conversational_tool_use_simulation: 78 passed. New judge structured-output
  tests cover both judge paths and a configured text. 8 mask assertions now check top-level mask_sample, and the
  example-rollout contract test checks the masking rule.
• responses_api_agents/conversational_tool_use/simulation/tests: 36 passed. 4 mask assertions moved to mask_sample.
• tests/unit_tests (Mac): same 43 pre-existing failures as main, plus the new passing test. In one full
  run, test_slurm_script.py::…multi_node_chain_survives_symmetric_run_entrypoint also failed (a bash subprocess
  with a 10 s timeout). It passes 3/3 alone on both trees and 226/226 as a file, and this diff does not touch it.
• pre-commit run --all-files: passed, no changes
• Rollout evidence: the QA case above runs live rollouts. The policy, simulator, and judge all use
  nvidia/nvidia/nemotron-3.5-super-vl-preview through openai_model.

Compatibility

• Consumers that read instance_config.mask_sample from this environment, such as older NeMo RL, no longer get a
  mask. They should read top-level mask_sample, like every other environment.
• The judge request now includes text.format. A provider that rejects json_schema structured output can turn it
  off by setting judge_responses_create_params.text.format.type=text.
• With a live judge, rows that it scores as user or tool failures are now masked. Before this PR they counted as
  reward-0 policy samples.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[qiaochuz-nv]

/ok to test a3a6ab1

[yaoyu-33]

LGTM. Reviewed the nested import, judge JSON schema, and top-level mask_sample changes; 250 focused tests passed locally. The Hermes model_error conformance check still needs to be cleared before merge.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38486441840

[qiaochuz-nv]

On the PR’s current head a3a6ab1, CI fails model_error because that base doesn’t have #4270. after rebase it work.