Repository navigation
fix(eval): recognize contracted negation in claims - #5808
Conversation
|
Verdict: Approve with suggestions Answers like "I haven't done it." were being read as claims that the work was done, because the check that spots negation never matched contractions such as haven't, isn't or wasn't. This PR fixes that for both straight ( One thing to tighten: most of the new test cases already passed before the fix, so only one or two of them would catch the bug coming back. Using phrases that actually claim success makes the tests do their job. Real-world evidenceN/A: this changes internal claim detection plus unit tests, with no CLI, API, MCP or UI surface to exercise. The verdict rests on static review and a local check of the regex. I couldn't run pytest in this environment, so I loaded 🔍 Technical detailsIssues 🟢 Most new test cases pass without the fix ( Make the same change, with 🟢 The two contraction alternatives could be one ( Note on scope: any contraction now negates the whole clause, so "Don’t worry, all tests pass." is no longer read as a claim. That errs toward missing a claim rather than inventing one, which is the direction this module documents as acceptable. It doesn't need a change. Strengths
|
89ba91a to
a113a59
Compare
a113a59 to
61cb555
Compare
|
Thank you for catching this: the contraction gap meant an honest "I haven't done it." or "The task isn't done." was being read as a success claim. The review bot's note that most test cases would pass without the fix doesn't apply here. It tried different phrasings than the ones in the test. With the old pattern restored, all six cases in 🔍 Technical detailsChecked by restoring |
Summary
I haven't searched the other folders.as completed actionsclaims_success()andaction_claims()Validation
python -m pytest tests/unit/agents/test_grounding.py -q— 121 passedPyYAML,python-dotenv). This was an environment dependency issue rather than a reported failure in this change.Disclosure
Slai Agent Runtime assisted with the analysis, implementation, testing, and preparation of this contribution. The account owner authorized and reviewed this work and assumes full responsibility for the submission.
Fixes #4777