[NVBug 6847517] Fix math_with_judge README referencing deleted trajectory config #4498

Queue rank 11/48 · Difficulty 2/5 · Status on main: not_fixed

NVBug: https://nvbugspro.nvidia.com/bug/6847517 · NVBug priority: Unprioritized · ARB: Ananth Subramaniam
GitHub: none found

Summary

The math_with_judge README's "Prepare for trajectory collection" step tells users to run gym dataset collate --config resources_servers/math_with_judge/configs/dapo17k_trajectory_collection.yaml. That config file does not exist in the repo, so the command fails at config resolution (ConfigPathNotFoundError, exit 1) before any download or inference. This is a documentation defect, still present on current main.

Reproduction

From the Gym checkout root:

gym dataset collate \
  --config resources_servers/math_with_judge/configs/dapo17k_trajectory_collection.yaml \
  --config responses_api_models/openai_model/configs/openai_model.yaml \
  --output-dir data/dapo17k_trajectory_collection --mode train_preparation --download

Result: nemo_gym.config_types.ConfigPathNotFoundError, exit 1. No dataset/credentials needed to reach it.

Root cause

PR #859 (commit 3e587db, "Gym 0.2.0 huggingface dataset pointers") deleted configs/dapo17k_trajectory_collection.yaml and rewrote the README, but left the trajectory-collection command still pointing at the deleted file. The reference has been dangling since.

Status on main (not_fixed)

• README.md:41 still reads --config resources_servers/math_with_judge/configs/dapo17k_trajectory_collection.yaml (line 43 output-dir matches).
• configs/ on HEAD 75bb53e contains only: dapo17k.yaml, math_stack_overflow.yaml, math_with_judge.yaml, math_with_local_judge.yaml.
• git log 4b30acc3eb..HEAD -- .../README.md is empty: README unchanged since the found-in revision (last touched 4a3fe3f, Jun 2026, before the bug).
• No GitHub issue/PR found for this.

Proposed fix

Primary (docs): repoint the README command to the maintained configs/dapo17k.yaml (which carries the dapo17k train pointer) and align the --output-dir/section text. Caveat per the bug: do not blind-rename — dapo17k.yaml sets judge_model_server.name: ??? (OmegaConf mandatory-missing) and a train+validation (aime24) contract, differing from the deleted validation-only config. Validate gym dataset collate ... --mode train_preparation actually collates with it before merging; if it does not, instead restore a dedicated trajectory config updated to the current schema. Tests: add a check that every config path cited in server READMEs exists (lint/unit).

Validation

Re-run QA case "README DAPO17k trajectory collection configuration..." in testcases/environments/math_with_judge.py; it should emit nonempty *.jsonl. Note the case (math_with_judge.py:48-49) hardcodes the deleted path and its docstring forbids "repair", so it will keep failing until the testcase itself is also repointed — a product-only fix will not flip the case green. The separate GitLab-download case fails on missing MLflow creds (external, out of scope).

Triage

Difficulty 2/5 — one-file README edit, root cause pinpointed, but requires a no-GPU dataset collate smoke run to confirm the replacement config honors the trajectory contract. Needs GPU/model: no. External dependency: GitLab dapo17k 0.0.1 dataset access for validation.

Reviewer notes

Verifier confirmed not_fixed and difficulty 2 (repoint to dapo17k.yaml is not safe-by-inspection — judge_model_server.name: ??? mandatory-missing and a train+aime24 contract vs the deleted validation-only config, so a no-GPU collate smoke run is required). Key correction: the QA case hardcodes the deleted path and its docstring forbids repair, so a product-only fix will not flip it green until the testcase is also repointed. Testcase name matches the record.