Repository navigation
feat(indic_gsm8k): add Indic GSM8K benchmark - #3590
anushaknvidia wants to merge 3 commits into
Conversation
nemo-automation-bot
Bot
added
the
community-request
anushaknvidia
changed the title
yaoyu-33
added
area:environment
github-actions
Bot
added
the
sla:triage-overdue
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
yaoyu-33
removed
the
needs-review
svcnvidia-nemo-ci
added
the
benchmark-onboarding
Signed-off-by: Anusha Kamath <anushak@nvidia.com>
Add zero-shot chat preparation for the local wide Indic GSM8K corpus, retaining original English answer keys, source hashes and stable task IDs. Expose a portable source path and document the 12-language thinking run. Track the saved-campaign validator with LF-delimited JSONL parsing so Unicode line separators inside model text do not break completed runs. Check coverage, prompts, actual HTTP settings and recomputed scores before marking a shard complete. Preserve strict and flexible pass metrics without majority voting across incompatible answer extractors. Validation: - 27 focused preparation, scoring and validator tests passed. - pre-commit run --all-files passed; private guardword scan skipped because its catalog was unavailable. - 15,828 prepared tasks match the original 12-language campaign exactly. - All 12 updated Sarvam language reports and its two-question smoke report match saved results when verified with the rebased code. Signed-off-by: Anusha Kamath <anushak@nvidia.com>
Download and verify the pinned ai4bharat/indic-gsm8k test split by default. Consolidate the benchmark into one zero-shot thinking config with explicit Gym environment routing and the 12 requested languages. Remove intermediate preparation and validation variants, and simplify dataset documentation. Validated 32 preparation and verifier tests, Gym benchmark discovery and environment routing, full pre-commit checks, preparation of all 15,828 tasks, and unchanged scores on 1,321 saved model responses. Saved campaign source snapshots were preserved before cleanup. Signed-off-by: Anusha Kamath <anushak@nvidia.com>
f369a56 to
5abb2e5
Compare
| def build_rows(source: list[dict], *, languages: list[str] | None = None, source_sha256: str) -> list[dict]: | ||
| """Expand translated questions while retaining original numeric answer keys.""" | ||
| selected = list(DEFAULT_LANGUAGES) if languages is None else languages | ||
| if not isinstance(selected, list) or not selected or len(set(selected)) != len(selected): |
There was a problem hiding this comment.
Could we normalize Hydra’s language list before this check? The documented gym eval prepare --benchmark indic/gsm8k/config '+prepare_script_args.languages=[hi,ta,te]' passes an OmegaConf ListConfig, so this guard raises ValueError for a valid selection. I reproduced this with the pinned source file.
Please accept that list representation and add a regression through Gym’s configuration parsing. Moving the preparation tests under tests/unit_tests/ would also ensure normal CI collects them.
| @@ -0,0 +1,5 @@ | |||
| {"responses_create_params": {"input": [{"role": "user", "content": "Solve the following math problem. End your response with a line of the form \"#### <answer>\", using digits for the final numerical answer.\n\nIf one box has 2 pencils, how many pencils are in 3 boxes?"}], "max_output_tokens": 120000}, "expected_answer": "Three boxes have 3 * 2 = 6 pencils. #### 6", "language_code": "synthetic"} | |||
There was a problem hiding this comment.
Could we complete the new-benchmark onboarding and validate the applicable path before merging? The current changed-server validator fails because data/example_metrics.json is absent; data/example_rollouts.jsonl is also missing. The new-benchmark guide additionally requires manifest.yaml and a passing VERIFIER_FIXTURE, neither of which is included.
Please complete the manifest/fixture workflow and record how the server check is satisfied. If the legacy validator conflicts with that workflow, the contributor guide asks us to report the mismatch rather than generate duplicate legacy artifacts solely for the check.
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| GSM8KReferenceResourcesServer.run_webserver() |
There was a problem hiding this comment.
Could we add the standard worker-import branch here? With num_workers > 1, Gym imports resources_servers.gsm8k_reference.app:app, but this module only handles __main__. The import fails with Attribute "app" not found.
Please add the is_nemo_gym_fastapi_entrypoint(__file__) branch and a worker-import regression.
github-actions
Bot
removed
the
sla:triage-overdue
svcnvidia-nemo-ci
added
needs-review
What does this PR do?
Adds
indic/gsm8k/configfor zero-shot chat evaluation on ai4bharat/indic-gsm8k, using Gym's environment server,simple_agent, vLLM model adapter, and a deterministic GSM8K numeric verifier.bn,gu,hi,kn,mr,ml,ne,or,pa,ta,te, andur(15,828 tasks). Assamese and Sanskrit can also be selected explicitly.e21f3ff7e9fe703a7f4faeefdba27801be278eb5and verifies its SHA-256. An identical local Parquet file can be supplied through preparation arguments.#### <answer>line. Scores final assistant text separately from reasoning, reporting strict and flexible numeric exact match on Gym's 0–100 scale, both overall and by language. Strict match determines the rollout reward.gym eval prepareandgym eval runcommands. Generated benchmark data remains outside Git.No separate issue is required: this is a self-contained benchmark addition.
Validation
Checked the final implementation at commit
5abb2e54:python -m pytest benchmarks/indic/gsm8k/tests resources_servers/gsm8k_reference/tests -q --tb=short: 32 passed, with coverage collection enabled. Covers source verification, HF downloads, language selection, original answer keys, stable IDs, stale prepared data, Unicode separators, reasoning/final separation, and strict/flexible scoring.python -m pytest environment_servers/legacy_agent/tests -q --tb=short: 5 passed, covering request forwarding, cookies, error responses, and metric aggregation.python -m pytest tests/unit_tests/test_benchmarks.py::TestDiscoverBenchmarksInDir::test_every_repo_benchmark_appears_in_listing -q --tb=short: passed. Discovery and environment checks were run outside the sandbox to allow local sockets.pre-commit run --all-files: available checks passed. The private guardword scan was skipped because its catalog was unavailable.Preparation evidence
Ran
python -m benchmarks.indic.gsm8k.preparethrough the default HF download path, and exercisedgym eval prepare --benchmark indic/gsm8k/configwith a verified local copy supplied through+prepare_script_args.source.Both paths produced 15,828 tasks: 12 languages × 1,319 problems. The downloaded Parquet matches the published HF checksum:
Compared every prepared task with the saved evaluation inputs. Questions, original answer keys, UUIDs, language assignments, and generation limits match; the prepared rows additionally record the HF repository and revision. The evaluation prompt is unchanged.
Real model evidence
Replayed 1,321 saved Sarvam 105B responses from the updated checkpoint through the current verifier: the two-task smoke run and the complete Kannada split. Every reward and both extraction scores matched the saved results.
kn)The saved runs used thinking with a 120,000-token output limit, and all 1,321 responses contained reasoning output. This validates scoring against existing model generations; no fresh inference was launched for this cleanup, and the table is not an all-language benchmark average.
Compatibility and benchmark impact
Adds a benchmark and resources server without changing shared Gym inference or metric code. Serve the model with a compatible reasoning parser and enough context for the output budget; the evaluated setup used a 131,072-token context window. The default dataset selection contains the 12 languages listed above. Preserved task identities, prompts, and replayed scores allow comparison with the saved evaluations.