Repository navigation
Conversation
Both existing examples are English only, so a model or quantization change that hurts non-English output has no reproducer. This adds examples/multilingual with six cases: two Hindi sentiment classifications, one Tamil classification, a Hindi order extraction, a Tamil event extraction and one exact answer. Baseline and candidate outputs are hand-written replay fixtures, labelled as such in metadata.note and in the suite description. The candidate flips one Hindi and one Tamil label and drops the currency field from the Hindi order, which gives three regressions; it adds notes to the Tamil event, which surfaces as a passing output change. validate exits 0 and compare exits 1 with the three regressions. Two tests in tests/test_cli.py pin the regression ids and the hand-written labelling.
There was a problem hiding this comment.
Thanks for your first pull request to ModelDiff! A maintainer will review it soon. CI for first-time contributors starts after a maintainer approves it.
While you wait, please check that:
- The issue was assigned to you before you started (see One issue at a time).
- The description links the issue (
Closes #12) and says how you tested the change. pytest,ruff check src tests examplesandruff format --check src tests examplespass locally.- The Files changed tab shows only changes for this issue.
AdityaWagh06
left a comment
There was a problem hiding this comment.
Thanks @tnalbant, this is really well done. The fixtures and the regression mix are exactly what #28 asked for. Two small README fixes and one question:
- The README says "two sentiment classifications", but the suite has three (two Hindi, one Tamil). Please change it to "three".
- Please remove the last sentence about Windows ("On Windows, redirecting this report..."). The UTF-8 fix (#32) is already merged, so it's no longer true.
- Were the Hindi and Tamil prompts checked by a native speaker? If not, that's okay. Just say so in the PR, and I'll ask for a native review before merging.
OpenWave-India#45) Review on OpenWave-India#45: the README said the multilingual suite has "two sentiment classifications" where it has three -- `hindi-sentiment-positive`, `hindi-sentiment-neutral` and `tamil-sentiment-positive` -- and it kept a sentence saying a redirected run only works once the UTF-8 output fix lands, which OpenWave-India#32 already merged. Co-authored-by: Hermes Agent (DeepSeek V4) <noreply@nousresearch.com>
|
Both README fixes are in, as
On the third question: no, they were not checked by a native speaker. Neither the Hindi nor the Tamil prompts. I do not have one to ask, so please do line up the native review you offered — I would much rather have someone read them than have the fixtures merged on my word alone. Nothing else changed: the suite, the fixtures and the tests are untouched by this commit. |
Closes #28.
Adds
examples/multilingual/with six cases in Hindi and Tamil, following the structure ofexamples/extraction:hindi-sentiment-positive,hindi-sentiment-neutralandtamil-sentiment-positive: sentiment classification with the labels written in the prompt's own language.hindi-order-json: order extraction (id, vendor, amount, currency, city) with a JSON Schema that requires all five.tamil-event-json: event extraction (event, venue, date) with adateformat check.hindi-exact-answer: one exact string answer.Baseline and candidate outputs in
recorded/are hand-written replay fixtures, labelled as such inmetadata.noteand in the suite description. The candidate produces the intended three regressions and one passing change:hindi-sentiment-positivetamil-sentiment-positivehindi-order-jsoncurrencydroppedtamil-event-jsonnotesaddedVerification (Python 3.14, repo venv):
modeldiff validate examples/multilingual/suite.yaml-> exit 0, 6 cases (3 classification, 1 exact, 2 json).modeldiff compare out/multilingual-baseline.json out/multilingual-candidate.json-> exit 1,REGRESSIONS FOUND (3), one passing-with-changed-output entry.pytest: 184 passed, including two new tests intests/test_cli.pythat pin the regression ids and assert both fixtures and the suite description label the outputs as hand-written.ruff checkclean,ruff format --checkclean on the changed Python file.One thing to review carefully: the Hindi and Tamil wording was drafted with an AI assistant and I am not a native speaker of either language. The sentences are short and everyday on purpose, and the suite description says the wording should be confirmed by a native speaker. Please route it to the Hindi and Tamil speakers you mentioned before merge; I will apply any wording corrections they suggest.
This change was prepared with an AI coding assistant.