You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Follow-up to #1578: judge prompt placeholder injection ({gold} splice); unseeded bootstrap CI bounds; metric crash recorded as zero with SUCCESS status; allow_inclusion grades by substring #1697
Four defects at pin 3c52bd0 (extends #1578, which was fixed in #1588).
1. Judge prompt placeholder injection
Sequential str.replace over the whole prompt means a model answer containing the literal {gold} splices the reference answer into the predicted-answer section. A wrong answer scores 1.0.
2. Unseeded bootstrap confidence bounds
The arena bootstrap resamples with no seed; three fresh-process runs on identical battles produce different 97.5 percent bounds (0.6875, 0.7203, 0.7828).
3. Metric crash recorded as zero with SUCCESS status
The adapter's except-handler writes value=0 and keeps the default SUCCESS status, so a crashing metric plugin counts as a wrong answer in the mean denominator, contradicting the project's own ScoreStatus contract.
4. Accuracy with allow_inclusion grades by Python in
A list reference gives full credit for one-of-several required letters while the complete answer scores zero; alias lists become exact-match case-sensitive; string references become substring.
All four reproduce with deterministic offline stubs. Available as PRs with tests.
Activity