fix(tools/skill-evals): require exact boolean for negate in assertions - #966
fix(tools/skill-evals): require exact boolean for negate in assertions#966surajthedev wants to merge 1 commit into
Conversation
Why: spec.get("negate") used Python truthiness, so a string like
"false" (non-empty, therefore truthy) would silently invert a
security assertion that the author intended to leave un-negated.
This is a silent correctness bug with no error surfaced.
The fix reads the negate field into a local variable and raises
TypeError immediately if the value is not an exact bool. Existing
callers that already pass true/false are unaffected.
Three new tests cover the three required cases:
- negate: true -> result is inverted
- negate: false -> result is unchanged
- non-boolean -> TypeError is raised with a clear message
|
Good catch on the bug. The bad-value loop is well chosen: Two notes, however. The rejection mechanism turns a one-assertion failure into a whole-run abort, which I'd change before this lands. And
|
Closes #942
tools/skill-evals/src/skill_evals/runner.py read the
negateassertion key using Python truthiness, so a value like "false" (a non-empty string, therefore truthy) would silently invert a security assertion the author intended to leave un-negated.This PR requires
negateto be an exact boolean. Any non-boolean value now raises a clear TypeError instead of silently producing the wrong result.Added three tests covering negate: true, negate: false, and non-boolean values (string, int, float, list, dict).