Skip to content

Add a Hindi and Tamil example suite with hand-written fixtures - #45

Open
tnalbant wants to merge 2 commits into
OpenWave-India:mainfrom
tnalbant:example/multilingual-suite
Open

tnalbant wants to merge 2 commits into
OpenWave-India:mainfrom
tnalbant:example/multilingual-suite

Conversation

@tnalbant

Copy link
Copy Markdown

Closes #28.

Adds examples/multilingual/ with six cases in Hindi and Tamil, following the structure of examples/extraction:

  • hindi-sentiment-positive, hindi-sentiment-neutral and tamil-sentiment-positive: sentiment classification with the labels written in the prompt's own language.
  • hindi-order-json: order extraction (id, vendor, amount, currency, city) with a JSON Schema that requires all five.
  • tamil-event-json: event extraction (event, venue, date) with a date format check.
  • hindi-exact-answer: one exact string answer.

Baseline and candidate outputs in recorded/ are hand-written replay fixtures, labelled as such in metadata.note and in the suite description. The candidate produces the intended three regressions and one passing change:

Case Candidate Result
hindi-sentiment-positive positive label flipped to negative regression
tamil-sentiment-positive positive label flipped to neutral regression
hindi-order-json currency dropped regression (schema)
tamil-event-json notes added passing output change

Verification (Python 3.14, repo venv):

  • modeldiff validate examples/multilingual/suite.yaml -> exit 0, 6 cases (3 classification, 1 exact, 2 json).
  • modeldiff compare out/multilingual-baseline.json out/multilingual-candidate.json -> exit 1, REGRESSIONS FOUND (3), one passing-with-changed-output entry.
  • pytest: 184 passed, including two new tests in tests/test_cli.py that pin the regression ids and assert both fixtures and the suite description label the outputs as hand-written.
  • ruff check clean, ruff format --check clean on the changed Python file.

One thing to review carefully: the Hindi and Tamil wording was drafted with an AI assistant and I am not a native speaker of either language. The sentences are short and everyday on purpose, and the suite description says the wording should be confirmed by a native speaker. Please route it to the Hindi and Tamil speakers you mentioned before merge; I will apply any wording corrections they suggest.

This change was prepared with an AI coding assistant.

Both existing examples are English only, so a model or quantization change that
hurts non-English output has no reproducer. This adds examples/multilingual with
six cases: two Hindi sentiment classifications, one Tamil classification, a
Hindi order extraction, a Tamil event extraction and one exact answer.

Baseline and candidate outputs are hand-written replay fixtures, labelled as
such in metadata.note and in the suite description. The candidate flips one
Hindi and one Tamil label and drops the currency field from the Hindi order,
which gives three regressions; it adds notes to the Tamil event, which surfaces
as a passing output change.

validate exits 0 and compare exits 1 with the three regressions. Two tests in
tests/test_cli.py pin the regression ids and the hand-written labelling.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your first pull request to ModelDiff! A maintainer will review it soon. CI for first-time contributors starts after a maintainer approves it.

While you wait, please check that:

  • The issue was assigned to you before you started (see One issue at a time).
  • The description links the issue (Closes #12) and says how you tested the change.
  • pytest, ruff check src tests examples and ruff format --check src tests examples pass locally.
  • The Files changed tab shows only changes for this issue.

@AdityaWagh06 AdityaWagh06 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tnalbant, this is really well done. The fixtures and the regression mix are exactly what #28 asked for. Two small README fixes and one question:

  1. The README says "two sentiment classifications", but the suite has three (two Hindi, one Tamil). Please change it to "three".
  2. Please remove the last sentence about Windows ("On Windows, redirecting this report..."). The UTF-8 fix (#32) is already merged, so it's no longer true.
  3. Were the Hindi and Tamil prompts checked by a native speaker? If not, that's okay. Just say so in the PR, and I'll ask for a native review before merging.

OpenWave-India#45)

Review on OpenWave-India#45: the README said the multilingual suite has "two sentiment
classifications" where it has three -- `hindi-sentiment-positive`,
`hindi-sentiment-neutral` and `tamil-sentiment-positive` -- and it kept a
sentence saying a redirected run only works once the UTF-8 output fix lands,
which OpenWave-India#32 already merged.

Co-authored-by: Hermes Agent (DeepSeek V4) <noreply@nousresearch.com>
@tnalbant

tnalbant commented Nov 3, 2026

Copy link
Copy Markdown
Author

Both README fixes are in, as d00cf80:

  1. It says three sentiment classifications now, and I listed them where it helps (hindi-sentiment-positive, hindi-sentiment-neutral, tamil-sentiment-positive).
  2. The Windows sentence is gone — you are right that fix: handle non-ASCII reports when stdout is redirected (cp1252 -> UTF-8) #32 made it untrue.

On the third question: no, they were not checked by a native speaker. Neither the Hindi nor the Tamil prompts. I do not have one to ask, so please do line up the native review you offered — I would much rather have someone read them than have the fixtures merged on my word alone.

Nothing else changed: the suite, the fixtures and the tests are untouched by this commit.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a multilingual example suite (Hindi and another Indian language)

2 participants