Skip to content

Commit 0793689

Browse files
committed
test(ai-slop): assert case 5 actually writes the findings file
Case 5's remaining persistence expectations were negative or procedural: no rubric finding in the file, and persistence treated as applicable. Neither requires a file to exist. context/persist-findings.md permits reporting without writing when the destination cannot be proven outside tracked space, so a run could call persistence applicable, write nothing, and satisfy both - green while asserting nothing, which is the shape this release removes. Adds the positive half: the findings file is written, and a run that reports the findings and persists nothing does not satisfy the case. The rubric boundary now constrains a file that must exist. Not a regression from dc5ca9a, which was also negative-only, but the assertion is gradeable under the specification design and so was worth having. Case 5 expectations only. Every prompt and expected_output in the file is byte-identical to b263df3, and cases 1-4 and 6-9 are untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g
1 parent b263df3 commit 0793689

2 files changed

Lines changed: 5 additions & 1 deletion

File tree

plugins/ai-slop/CHANGELOG.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,10 @@ name a committed fixture instead.
3737
that it treats persistence as applicable, fetches the producer contract first, and — case 4 —
3838
refuses to write when that fetch fails, rather than refusing because the target was out of tracked
3939
space. A case cannot verify real repository state, and pretending otherwise is what made case 4
40-
pass for the wrong reason.
40+
pass for the wrong reason. Case 5 also asserts the positive half — that the findings file is
41+
actually written — because `context/persist-findings.md` permits reporting without writing when the
42+
destination cannot be proven outside tracked space, so a negative-only case would be satisfied by a
43+
run that persists nothing at all.
4144
- **This reverses 0.1.0's no-fixtures decision, which was recorded in `detect.test.sh`'s header.**
4245
That decision holds for the *unit* suite, whose fixtures are still built inline in a tmpdir. It
4346
does not survive contact with the eval suite: an eval case is graded against a deterministic

plugins/ai-slop/skills/audit/evals/evals.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@
6565
"Reports the promotional-register tell as a rubric finding, quoting the offending text and naming the catalog entry it comes from",
6666
"Does NOT report the promotional wording as a script finding — the fixture's promotional words sit outside the mechanical vocabulary, so the detector's only finding is the em dash",
6767
"Treats persistence as applicable, since the audit examined a tracked file, rather than passing over the step as out of scope",
68+
"Writes the findings file, on the stated premise that the audited file is tracked — a run that reports the findings and persists nothing does not satisfy this case",
6869
"Puts NO rubric finding in the findings file — only the em-dash script finding belongs there",
6970
"States the V1 boundary explicitly rather than silently omitting the rubric verdict from the file"
7071
]

0 commit comments

Comments
 (0)