Skip to content

feat(indic mbpp): add Indic MBPP - #3644

Merged
yuhezhang-ai merged 4 commits into
NVIDIA-NeMo:mainfrom
anushaknvidia:indic/mbpp
Oct 29, 2026
Merged

feat(indic mbpp): add Indic MBPP #3644
yuhezhang-ai merged 4 commits into
NVIDIA-NeMo:mainfrom
anushaknvidia:indic/mbpp

Conversation

@anushaknvidia

@anushaknvidia anushaknvidia commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds ai4bharat/indic-mbpp as indic/mbpp: all 500 original MBPP test tasks in 14 Indic languages, plus a matching en baseline. No issue is linked because this is a standalone translated-dataset integration.

The existing English mbpp benchmark is MBPP+ (378 EvalPlus tasks). Only 224 IDs overlap with the translated original-MBPP dataset, and 115 of those have revised descriptions. This integration therefore preserves all 500 original tasks and their original tests, as requested, rather than silently discarding tasks or applying incompatible MBPP+ tests.

Reuses the existing English MBPP prompt template, simple agent, and SciCodePile assertion verifier, including extraction, subprocess execution, timeout handling, and pass@k metrics. Adds only benchmark preparation, configuration, manifest, documentation, and focused tests.

Compatibility and benchmark impact

This evaluates original MBPP, not MBPP+. Scores are not directly comparable to the existing 378-task English mbpp benchmark or its EvalPlus passing_plus_tests metric.

Default repeats: 1. Reward requires all three original tests to pass. Report pass@1/accuracy; repeated runs also expose pass@k and average single-attempt accuracy.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated edits are excluded.
  • Tests added and pass locally.
  • Documentation added.
  • Pre-commit checks pass locally (pre-commit run --all-files).
  • New source files include the required Apache-2.0 SPDX header.
  • All commits have DCO sign-off (git commit -s).

@copy-pr-bot

copy-pr-bot Bot commented Sep 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Sep 23, 2026
@yaoyu-33 yaoyu-33 added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Sep 23, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Sep 24, 2026
@github-actions github-actions Bot mentioned this pull request Sep 24, 2026
Open
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 25, 2026
@yaoyu-33 yaoyu-33 removed the needs-review PR is ready for code review and waiting on a reviewer label Sep 25, 2026
yuhezhang-ai
yuhezhang-ai reviewed Oct 6, 2026
View reviewed changes

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this and explaining the distinction from MBPP+. The local checks and the comparison against the public original MBPP reference solutions passed. Before merge, could you add a small real-model smoke run on the Indic data, ideally alongside the matching English baseline, with the configuration and a few inspected outputs/rewards? I couldn't access the private dataset to verify that part. I've also left two non-blocking suggestions to make result analysis and reproducibility easier.

Comment thread benchmarks/indic/mbpp/prepare.py Resolved
Comment thread benchmarks/indic/mbpp/prepare.py Resolved
@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-customer Waiting on the original author to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Oct 6, 2026
@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 6, 2026
anushaknvidia and others added 4 commits October 11, 2026 11:36
@anushaknvidia
feat(benchmarks): add Indic MBPP with original tests and English base…
3c2b8f8 
…line

Signed-off-by: Anusha Kamath <anushak@nvidia.com>
Anusha Kamath @anushaknvidia
remove dataset version
07346cf 
Signed-off-by: Anusha Kamath <anushak@cw-dfw-cs-001-vscode-01.cm.cluster>
@anushaknvidia
fix(benchmarks): address Indic MBPP reproducibility review
0765a45 
Preserve language-qualified task IDs in saved verifier results and record resolved dataset provenance without pinning upstream revisions.

Align the manifest split, collect preparation tests in CI, and document paired English/Hindi/Kannada real-model smoke outputs and settings.

Validate 114 tests, 500 reference solutions, six real-model examples, and all-files pre-commit checks.

Signed-off-by: Anusha Kamath <anushak@nvidia.com>
@anushaknvidia
fix(benchmarks): align Indic MBPP with main environment routing
7bb48b6 
Add the named legacy-agent relay and strict config-composition checks after rebasing onto upstream main at 4a9236e.

Record a fresh six-example English/Hindi/Kannada model smoke with inspected outputs and unchanged prompt and scoring contracts. Document the unobserved telemetry-health limitation separately from verifier results.

Validate 119 targeted tests, 500 reference solutions, the incorrect-solution control, all six real-model examples, and all-files pre-commit checks.

Signed-off-by: Anusha Kamath <anushak@nvidia.com>
@yuhezhang-ai

yuhezhang-ai commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

/ok to test 7bb48b6

yuhezhang-ai
yuhezhang-ai approved these changes Oct 15, 2026
View reviewed changes

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the feedback and adding the paired smoke results. The language IDs and dataset provenance look good, and I have no remaining review concerns. Approved, pending the normal CI checks.

@nemo-automation-bot
nemo-automation-bot Bot requested review from a team and anwithk October 15, 2026 14:22
@github-actions

github-actions Bot commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

🌿 Preview your docs: https://nvidia-preview-pr-3644-7bb48b6a2767.docs.buildwithfern.com/nemo/gym

@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Oct 16, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-customer Waiting on the original author to respond labels Oct 17, 2026
This was referenced Oct 22, 2026
Open
Open
@yuhezhang-ai
yuhezhang-ai added this pull request to the merge queue Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40907510581

@github-actions

github-actions Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40911076048

@github-actions

github-actions Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40914927845

@github-actions

github-actions Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40924872891

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Oct 29, 2026
@yuhezhang-ai
yuhezhang-ai added this pull request to the merge queue Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40958625104

@github-actions

github-actions Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40962959853

Merged via the queue into NVIDIA-NeMo:main with commit 0dbcaf3 Oct 29, 2026
49 of 51 checks passed
@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Oct 30, 2026

This branch was previously deployed

1 inactive deployment
public — 7bb48b6a Deployed Oct 15, 2026 by copy-pr-bot[bot] via release / finalize / notify #5497
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yuhezhang-ai yuhezhang-ai yuhezhang-ai approved these changes
@anwithk anwithk Awaiting requested review from anwithk

Assignees

No one assigned

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers community-request Issue reported or requested by someone from the community complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@anushaknvidia @yuhezhang-ai @yaoyu-33 @svcnvidia-nemo-ci