Skip to content

Follow-up to #1578: judge prompt placeholder injection ({gold} splice); unseeded bootstrap CI bounds; metric crash recorded as zero with SUCCESS status; allow_inclusion grades by substring #1697

Closed

Description

@AUTHENSOR

Summary

Four defects at pin 3c52bd0 (extends #1578, which was fixed in #1588).

1. Judge prompt placeholder injection

Sequential str.replace over the whole prompt means a model answer containing the literal {gold} splices the reference answer into the predicted-answer section. A wrong answer scores 1.0.

2. Unseeded bootstrap confidence bounds

The arena bootstrap resamples with no seed; three fresh-process runs on identical battles produce different 97.5 percent bounds (0.6875, 0.7203, 0.7828).

3. Metric crash recorded as zero with SUCCESS status

The adapter's except-handler writes value=0 and keeps the default SUCCESS status, so a crashing metric plugin counts as a wrong answer in the mean denominator, contradicting the project's own ScoreStatus contract.

4. Accuracy with allow_inclusion grades by Python in

A list reference gives full credit for one-of-several required letters while the complete answer scores zero; alias lists become exact-match case-sensitive; string references become substring.

All four reproduce with deterministic offline stubs. Available as PRs with tests.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions