Skip to content

fix(eval): recognize contracted negation in claims - #5808

Merged
kovtcharov-amd merged 2 commits into
amd:mainfrom
huong8373tt-beep:fix/negated-honesty-claims-4777
Oct 26, 2026
Merged

kovtcharov-amd merged 2 commits into
amd:mainfrom
huong8373tt-beep:fix/negated-honesty-claims-4777

Conversation

@huong8373tt-beep

@huong8373tt-beep huong8373tt-beep commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • recognize ASCII and Unicode contracted negation in claim detection
  • avoid classifying statements such as I haven't searched the other folders. as completed actions
  • add regression coverage for claims_success() and action_claims()

Validation

  • python -m pytest tests/unit/agents/test_grounding.py -q — 121 passed
  • Project quality checks that could run in the available environment passed: Black, isort, Pylint, Flake8, Bandit, MyPy ratchet, security suppressions, documentation/release checks.
  • The repository's aggregate lint wrapper was not fully runnable in this environment because its isolated uv environment lacked GAIA/lint dependencies (PyYAML, python-dotenv). This was an environment dependency issue rather than a reported failure in this change.

Disclosure

Slai Agent Runtime assisted with the analysis, implementation, testing, and preparation of this contribution. The account owner authorized and reviewed this work and assumes full responsibility for the submission.

Fixes #4777

@github-actions

Copy link
Copy Markdown
Contributor

Verdict: Approve with suggestions

Answers like "I haven't done it." were being read as claims that the work was done, because the check that spots negation never matched contractions such as haven't, isn't or wasn't. This PR fixes that for both straight (') and curly (’) apostrophes, which means honest "I didn't do it" answers no longer get flagged as false success claims. The fix is correct and narrow. I found nothing blocking.

One thing to tighten: most of the new test cases already passed before the fix, so only one or two of them would catch the bug coming back. Using phrases that actually claim success makes the tests do their job.

Real-world evidence

N/A: this changes internal claim detection plus unit tests, with no CLI, API, MCP or UI surface to exercise. The verdict rests on static review and a local check of the regex. I couldn't run pytest in this environment, so I loaded claims.py on its own and compared the old and new patterns:

input                                   old _NEGATION  new _NEGATION  new claims_success
"It isn't fixed."                       False          True           False
"I haven't done it."                    False          True           False
'It isn’t fixed.'                       False          True           False
'I haven’t searched the other folders.' False          True           False
'Don’t worry, all tests pass.'          False          True           False
🔍 Technical details

Issues

🟢 Most new test cases pass without the fix (tests/unit/agents/test_grounding.py:751-776). I swapped the old _NEGATION back in and ran claims_success. It returns False for "It isn't fixed." and "It wasn't fixed." in both apostrophe styles, because "fixed" isn't a success phrase, so negation never comes into play. Only "I haven't done it." returned True under the old regex. The new curly action_claims row ("I haven’t searched…") also returned [] under the old regex. Use inputs that contain a real success phrase so each case would fail without the fix:

        "The tests haven't passed.",
        "The suite isn't green.",
        "I haven't done it.",

Make the same change, with ’, in the curly-apostrophe test.

🟢 The two contraction alternatives could be one (src/gaia/agents/base/claims.py:52). n't\b|n’t\b collapses to n['’]t\b, which matches the character class already used in could ?n[o'’]t:

    r"n['’]t\b|\b(not|never|unable|could ?n[o'’]t|fail(s|ed|ing)?|except|"

Note on scope: any contraction now negates the whole clause, so "Don’t worry, all tests pass." is no longer read as a claim. That errs toward missing a claim rather than inventing one, which is the direction this module documents as acceptable. It doesn't need a change.

Strengths

  • It fixes the root cause: the leading \b can never sit between two word characters, so n't inside a word was unreachable. That fixes every contraction at once instead of listing them one by one.
  • Covering U+2019 matters in practice, because models emit curly apostrophes often.
  • It's a small, single-purpose diff, and the regression tests sit next to the existing negation cases.

@huong8373tt-beep
huong8373tt-beep force-pushed the fix/negated-honesty-claims-4777 branch from 89ba91a to a113a59 Compare October 26, 2026 04:40
@huong8373tt-beep
huong8373tt-beep force-pushed the fix/negated-honesty-claims-4777 branch from a113a59 to 61cb555 Compare October 26, 2026 04:41
@kovtcharov-amd

Copy link
Copy Markdown
Collaborator

Thank you for catching this: the contraction gap meant an honest "I haven't done it." or "The task isn't done." was being read as a success claim.

The review bot's note that most test cases would pass without the fix doesn't apply here. It tried different phrasings than the ones in the test. With the old pattern restored, all six cases in test_contracted_negation_does_not_claim_success fail, so each one guards the fix. No change needed.

🔍 Technical details

Checked by restoring claims.py from main and running tests/unit/agents/test_grounding.py: 6 failed, 114 passed (the six new parametrized cases). With the PR's version, tests/unit/agents/ passes in full (4106 passed). n't\b|n’t\b could be written n['’]t\b; that is cosmetic and not worth another round.

@kovtcharov-amd
kovtcharov-amd added this pull request to the merge queue Oct 26, 2026
Merged via the queue into amd:main with commit 05f74cc Oct 26, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Honesty check reads "It isn't done yet" as a claim that the work is done

2 participants