From dc5ca9ade367685423d853f4894f89081bc73556 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 01:43:35 +0000 Subject: [PATCH 1/8] test(ai-slop): back the audit evals with committed fixtures (0.3.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The audit eval cases described their input in prose, and nothing checked that the described input produced the finding the case graded. It drifted from the detector three times in one PR — each time a golden answer the scenario could not produce. Seven of the nine cases now name a committed fixture under skills/audit/evals/fixtures/ via files[], and every expected_output names the rules, lines and fired thresholds the detector actually emits, measured with detect.sh under an empty HOME and CLAUDE_PROJECT_DIR rather than asserted. Cases 3 and 9 keep narration: true — they grade repo-wide flow and consuming-repo config, not file content. This reverses 0.1.0's no-fixtures decision, which was recorded in detect.test.sh's header and is updated in place. The decision still holds for the unit suite, whose fixtures stay inline in a tmpdir; it does not survive the eval suite, where a case is graded against a deterministic detector run and its scenario has to satisfy an ERE the eval author cannot see. The fixtures carry real tells, so this repo's own audit declines them through an excluded_paths glob rather than an in-file marker: a file marker declines unconditionally, including under the config isolation the detector's tests use, which would leave detect.sh printing nothing and the eval author back to trusting prose. Closes #3041 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g --- .claude/ai-slop.json | 5 +- plugins/ai-slop/.claude-plugin/plugin.json | 2 +- plugins/ai-slop/CHANGELOG.md | 30 +++++++++ plugins/ai-slop/skills/audit/evals/evals.json | 62 +++++++++---------- .../evals/fixtures/em-dash-substitution.md | 8 +++ .../evals/fixtures/fix-guarded-rewrite.md | 8 +++ .../evals/fixtures/knowledge-cutoff-prose.md | 9 +++ .../audit/evals/fixtures/report-only.md | 9 +++ .../audit/evals/fixtures/rubric-boundary.md | 8 +++ .../skills/audit/evals/fixtures/triads.md | 14 +++++ .../skills/audit/scripts/detect.test.sh | 12 +++- 11 files changed, 128 insertions(+), 39 deletions(-) create mode 100644 plugins/ai-slop/skills/audit/evals/fixtures/em-dash-substitution.md create mode 100644 plugins/ai-slop/skills/audit/evals/fixtures/fix-guarded-rewrite.md create mode 100644 plugins/ai-slop/skills/audit/evals/fixtures/knowledge-cutoff-prose.md create mode 100644 plugins/ai-slop/skills/audit/evals/fixtures/report-only.md create mode 100644 plugins/ai-slop/skills/audit/evals/fixtures/rubric-boundary.md create mode 100644 plugins/ai-slop/skills/audit/evals/fixtures/triads.md diff --git a/.claude/ai-slop.json b/.claude/ai-slop.json index a016b3069..fd73462da 100644 --- a/.claude/ai-slop.json +++ b/.claude/ai-slop.json @@ -1,8 +1,9 @@ { - "_comment": "Three rules are disabled here, each for a measured, repo-specific reason; the shipped defaults stay neutral and this is the consuming repo's own decision, which is what the config layer is for. (1) rule-em-dash: this marketplace's house style uses em dashes deliberately, measured 34,999 matching lines across 1229 tracked markdown files (2026-08-19), which is a style choice, not a defect backlog. Disabling beats enumerating em_dash_allowed_paths, which would need near-total path coverage to be useful. (2) rule-curly-artifacts: punctuation glyphs were already ruled out of scope by the repo owner, twice in writing, and the rule's yield is almost entirely the material those rulings protect. Measured 2026-08-20 over 1214 tracked markdown files: 72 findings, of which 39 are quoted Pat Pattison book text, 32 are a verbatim pasted external Codex review, and exactly 1 was genuine paste residue (a stray curly apostrophe in plugins/disk-hygiene/README.md, fixed rather than suppressed in the same change as this entry). The rulings: plugins/songwriting/context/pat-pattison/research/book-references.md ('Do not sweep, measure, audit, or open work items on punctuation glyphs' - a session was spent on exactly that before the owner ruled it out of scope; it also notes the four books disagree with each other, so there is no fidelity axis to restore toward) and the songwriting CHANGELOG entry 'PUNCTUATION GLYPHS ARE NOT A FIDELITY AXIS - DO NOT SWEEP FOR THEM', which records a prior session normalizing 470 lines before the ruling landed. (3) rule-emoji-formatting: measured 2026-08-20 over the same corpus, all 21 findings are SEMANTIC markers and none is decoration - 7 wrong/right teaching pairs in docs-hygiene extract-ssot citation-form.md and planning interview loop.md, 4 coaching pairs in the pat-pattison coaching-protocol.md, 9 warning headings across kindle-dedrm README.md, the songwriting CHANGELOG and book-references.md, and 1 severity-legend marker in REVIEW.md that belongs to the same table as its two siblings. The rule is doing its job; this repo simply uses these glyphs as vocabulary rather than ornament. In-file block markers were the first choice and were rejected on cost: the marked files span four plugins, and the changelog-parity gate treats any edit under plugins// as version reuse, so silencing cosmetic audit noise would have published four plugin releases. A per-rule path exemption is the instrument that actually fits all three of these entries, and the config layer supports it only for rule-em-dash today; generalizing it is left to its own change. Revisit any of the three if a corpus of genuine LLM residue ever lands here.", + "_comment": "Three rules are disabled here, each for a measured, repo-specific reason; the shipped defaults stay neutral and this is the consuming repo's own decision, which is what the config layer is for. (1) rule-em-dash: this marketplace's house style uses em dashes deliberately, measured 34,999 matching lines across 1229 tracked markdown files (2026-08-19), which is a style choice, not a defect backlog. Disabling beats enumerating em_dash_allowed_paths, which would need near-total path coverage to be useful. (2) rule-curly-artifacts: punctuation glyphs were already ruled out of scope by the repo owner, twice in writing, and the rule's yield is almost entirely the material those rulings protect. Measured 2026-08-20 over 1214 tracked markdown files: 72 findings, of which 39 are quoted Pat Pattison book text, 32 are a verbatim pasted external Codex review, and exactly 1 was genuine paste residue (a stray curly apostrophe in plugins/disk-hygiene/README.md, fixed rather than suppressed in the same change as this entry). The rulings: plugins/songwriting/context/pat-pattison/research/book-references.md ('Do not sweep, measure, audit, or open work items on punctuation glyphs' - a session was spent on exactly that before the owner ruled it out of scope; it also notes the four books disagree with each other, so there is no fidelity axis to restore toward) and the songwriting CHANGELOG entry 'PUNCTUATION GLYPHS ARE NOT A FIDELITY AXIS - DO NOT SWEEP FOR THEM', which records a prior session normalizing 470 lines before the ruling landed. (3) rule-emoji-formatting: measured 2026-08-20 over the same corpus, all 21 findings are SEMANTIC markers and none is decoration - 7 wrong/right teaching pairs in docs-hygiene extract-ssot citation-form.md and planning interview loop.md, 4 coaching pairs in the pat-pattison coaching-protocol.md, 9 warning headings across kindle-dedrm README.md, the songwriting CHANGELOG and book-references.md, and 1 severity-legend marker in REVIEW.md that belongs to the same table as its two siblings. The rule is doing its job; this repo simply uses these glyphs as vocabulary rather than ornament. In-file block markers were the first choice and were rejected on cost: the marked files span four plugins, and the changelog-parity gate treats any edit under plugins// as version reuse, so silencing cosmetic audit noise would have published four plugin releases. A per-rule path exemption is the instrument that actually fits all three of these entries, and the config layer supports it only for rule-em-dash today; generalizing it is left to its own change. Revisit any of the three if a corpus of genuine LLM residue ever lands here. Separately, two excluded_paths entries name files that CONTAIN the tells on purpose and would otherwise be scanned as if they were defects: reference/catalog.md quotes the tells it catalogues, and skills/audit/evals/fixtures/** are the eval suite's committed slop samples (#3041 - a prose scenario cannot be checked against the detector and disagreed with it three times, so the input is a file now). Excluding the fixture path rather than marking each fixture with an in-file ai-slop-ignore-file marker is what keeps them measurable: a file marker declines the file unconditionally, including under the empty HOME + CLAUDE_PROJECT_DIR isolation the detector's own tests use, so `detect.sh ` would print nothing and the eval author would be back to trusting prose. An excluded_paths glob is a layer of THIS repo's config, so that same isolation lifts it and the fixture reports its real findings.", "excluded_paths": [ "plugins/*/skills/*/vendor/**", - "plugins/ai-slop/skills/audit/reference/catalog.md" + "plugins/ai-slop/skills/audit/reference/catalog.md", + "plugins/ai-slop/skills/audit/evals/fixtures/**" ], "disabled_rules": ["rule-em-dash", "rule-curly-artifacts", "rule-emoji-formatting"] } diff --git a/plugins/ai-slop/.claude-plugin/plugin.json b/plugins/ai-slop/.claude-plugin/plugin.json index a0510efd2..c0b4c5a68 100644 --- a/plugins/ai-slop/.claude-plugin/plugin.json +++ b/plugins/ai-slop/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "ai-slop", - "version": "0.2.2", + "version": "0.3.0", "description": "Detects and removes AI-writing tells (slop) in checked-in markdown prose: em dashes, emoji formatting, AI vocabulary, negative parallelisms, chatbot phrases, filler, stacked hedging, citation artifacts, and the rest of a catalog distilled from Wikipedia's Signs of AI writing. Read-only audit by default with a deterministic detector plus a judgment rubric; an explicit fix action rewrites findings behind a semantic-diff guard. Findings conform to the detector-findings convention so the review fanout fix relay can consume them.", "author": { "name": "Melodic Software", diff --git a/plugins/ai-slop/CHANGELOG.md b/plugins/ai-slop/CHANGELOG.md index 4985baa3d..e60ba1547 100644 --- a/plugins/ai-slop/CHANGELOG.md +++ b/plugins/ai-slop/CHANGELOG.md @@ -1,5 +1,35 @@ # Changelog +## [0.3.0] + +The audit eval cases described their input in prose. Nothing checked that the described input +produced the finding the case graded, and it drifted from the detector three times in one PR +(#3041) — each time a golden answer the scenario could not produce. Seven of the nine cases now +name a committed fixture instead. + +- **Six eval fixtures ship under `skills/audit/evals/fixtures/`**, referenced from each case's + `files` array: `report-only.md` (four rules on one file), `fix-guarded-rewrite.md` (two em dashes + and two filler phrases), `rubric-boundary.md` (one script finding, promotional register the + mechanical rules deliberately miss), `em-dash-substitution.md` (a single em dash), + `triads.md` (rule-of-three at 3 hits in 69 words) and `knowledge-cutoff-prose.md` (the recorded + false-positive class). Every case's `expected_output` now names the rules, lines and fired + thresholds the detector actually emits, measured rather than asserted. +- **This reverses 0.1.0's no-fixtures decision, which was recorded in `detect.test.sh`'s header.** + That decision holds for the *unit* suite, whose fixtures are still built inline in a tmpdir. It + does not survive contact with the eval suite: an eval case is graded against a deterministic + detector run, so its scenario has to satisfy an ERE the eval author does not have in front of + them. A committed fixture cannot disagree with the detector; prose describing one can, and did. +- Cases 3 and 9 keep `narration: true`. They grade repo-wide flow and consuming-repo config, not + file content, so there is nothing for a fixture to pin. +- The fixtures carry real tells, so a repo auditing its own tree has to decline them. Prefer an + `excluded_paths` glob over an in-file `ai-slop-ignore-file` marker: a file marker declines + unconditionally, including under the empty `HOME` + `CLAUDE_PROJECT_DIR` isolation, so + `detect.sh ` would print nothing and the eval author would be back to trusting prose. + An `excluded_paths` entry is a config layer, and that isolation lifts it. This repo's own + `.claude/ai-slop.json` carries the glob as the worked example. +- Consuming repos need no exclusion of their own: the audit scans `git ls-files '*.md'`, and an + installed plugin's files are not tracked in the repo that installs it. + ## [0.2.2] The in-file suppression the fix flow and the catalog both tell operators to reach for now works in diff --git a/plugins/ai-slop/skills/audit/evals/evals.json b/plugins/ai-slop/skills/audit/evals/evals.json index 62a9f74d6..0e5c22cfe 100644 --- a/plugins/ai-slop/skills/audit/evals/evals.json +++ b/plugins/ai-slop/skills/audit/evals/evals.json @@ -4,26 +4,26 @@ { "id": 1, "name": "bare-invocation-read-only", - "prompt": "/ai-slop:audit docs/example.md", - "expected_output": "A findings report for the target file — per-rule findings with file:line locations, fired thresholds, and a summary — with no file modified. The fix flow is offered but not run.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit evals/fixtures/report-only.md (path relative to the skill directory)", + "expected_output": "A findings report for the fixture — four findings across four rules (rule-filler-phrases on line 3, rule-significance-inflation on line 6, rule-em-dash and rule-chatbot-artifacts both on line 8), each with its file:line location and fired condition, plus the Summary rows — with no file modified. The fix flow is offered but not run.", + "files": ["evals/fixtures/report-only.md"], "expectations": [ "Runs detect.sh on the target and reports findings with file:line locations and the fired condition per finding", - "Does NOT edit the target file or any other file on bare invocation", + "Reports all four rules the fixture trips rather than a subset — filler phrase, significance inflation, em dash, and chat-turn residue", + "Does NOT edit the fixture or any other file on bare invocation", "Mentions the fix action as the explicit follow-up rather than applying rewrites" ] }, { "id": 2, "name": "fix-runs-semantic-guard", - "prompt": "/ai-slop:audit fix docs/example.md", - "expected_output": "The file's findings are rewritten per-file (em dashes reworded, stock phrases deflated), each file's before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs and the findings file is re-emitted so no stale findings survive.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit fix evals/fixtures/fix-guarded-rewrite.md (path relative to the skill directory)", + "expected_output": "The fixture's four findings — two em dashes (lines 3 and 7) and two filler phrases ('It is worth noting that' on line 3, 'Due to the fact that' on line 4) — are rewritten in one per-file pass, the before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs and the findings file is re-emitted so no stale findings survive.", + "files": ["evals/fixtures/fix-guarded-rewrite.md"], "expectations": [ "Applies rewrites only because fix was explicitly requested", - "Verifies every rewritten file with a fresh-context semantic-diff subagent and reverts flagged hunks", + "Verifies the rewritten file with a fresh-context semantic-diff subagent and reverts flagged hunks", + "Preserves every claim the fixture makes — that the cache invalidates on write, that the reader never sees a stale entry, that writes are rare, and that the policy is least-recently-used", "Re-runs the detector and re-emits the findings file after the last fixed file" ] }, @@ -43,10 +43,9 @@ { "id": 4, "name": "persist-refuses-without-contract", - "prompt": "/ai-slop:audit docs/example.md (with the detector-findings contract URL unreachable)", - "expected_output": "The audit report is delivered, but no findings file is written: the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit evals/fixtures/report-only.md (path relative to the skill directory), with the detector-findings contract URL unreachable", + "expected_output": "The audit report is delivered with the fixture's four findings intact, but no findings file is written: the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.", + "files": ["evals/fixtures/report-only.md"], "expectations": [ "Delivers the human-facing findings report regardless", "Does NOT write a findings file when the contract fetch fails", @@ -56,12 +55,12 @@ { "id": 5, "name": "rubric-findings-reach-the-report-not-the-findings-file", - "prompt": "/ai-slop:audit docs/example.md — the file contains promotional register (\"a breathtaking, must-visit dashboard sitting in the heart of the stack\") and one em dash. The promotional words are deliberately outside the mechanical vocabulary list, so the detector matches the em dash and nothing else.", - "expected_output": "Both layers report: the em dash appears as the single script finding, and the promotional tone as a rubric finding citing its catalog entry. The persisted findings file carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit evals/fixtures/rubric-boundary.md (path relative to the skill directory)", + "expected_output": "Both layers report: the em dash on line 7 appears as the fixture's single script finding, and the promotional register on lines 3-5 ('a breathtaking, must-visit surface sitting right in the heart of the platform') as a rubric finding citing its catalog entry. The persisted findings file carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.", + "files": ["evals/fixtures/rubric-boundary.md"], "expectations": [ "Reports the promotional-register tell as a rubric finding, quoting the offending text and naming the catalog entry it comes from", + "Does NOT report the promotional wording as a script finding — the fixture's promotional words sit outside the mechanical vocabulary, so the detector's only finding is the em dash", "Does NOT place any rubric finding in the persisted findings file — only the script finding appears there", "States the V1 boundary explicitly rather than silently omitting the rubric verdict from the file" ] @@ -69,10 +68,9 @@ { "id": 6, "name": "fix-never-swaps-one-tell-for-another", - "prompt": "/ai-slop:audit fix docs/example.md — the file's only finding is a single em dash in \"the parser accepts empty input — a change from 0.3.\"", - "expected_output": "The em dash is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit fix evals/fixtures/em-dash-substitution.md (path relative to the skill directory)", + "expected_output": "The fixture's single finding — the em dash on line 5, in 'The parser accepts empty input — a change from 0.3.' — is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.", + "files": ["evals/fixtures/em-dash-substitution.md"], "expectations": [ "Reads reference/rewrite-guide.md before applying the rewrite, per the fix flow's first step", "The rewritten line contains no em dash, en dash, parenthesis, or spaced hyphen standing in for the original punctuation", @@ -82,25 +80,23 @@ { "id": 7, "name": "triads-collapse-toward-the-strongest-item", - "prompt": "/ai-slop:audit fix docs/example.md — the file trips rule-of-three at 3 hits on two kinds of triad. Rhetorical: \"the tool is fast, simple, and reliable\" and \"it reads clean, tight, and portable\". Load-bearing: \"the cascade layers are user, team, and local\" — a complete set the reader needs, where dropping one names a resolution order that does not exist. All three match the density rule's ERE, so all three reach the fix flow as one finding.", - "expected_output": "The two rhetorical triads collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration is KEPT intact, with that judgment stated — enumerating three actual things is not a tell.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit fix evals/fixtures/triads.md (path relative to the skill directory)", + "expected_output": "The fixture trips rule-rule-of-three as one density finding (3 hits in 69 words, 43.5/1000 against a 3.0 threshold), and all three triads reach the fix flow together. The two rhetorical triads — 'fast, simple, and reliable' and 'clean, tight, and portable' — collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration 'user, team, and local' is KEPT intact, with that judgment stated: dropping one names a resolution order that does not exist.", + "files": ["evals/fixtures/triads.md"], "expectations": [ - "Collapses rhetorical triads toward one item rather than preserving all three in a reworded form", - "Keeps a triad when each element is load-bearing, and says why it was kept", + "Collapses the two rhetorical triads toward one item rather than preserving all three in a reworded form", + "Keeps the 'user, team, and local' triad because each element is load-bearing, and says why it was kept", "Does not treat every three-item enumeration as a tell — enumerating three actual things is not slop" ] }, { "id": 8, "name": "known-false-positive-routes-to-config-not-a-rewrite", - "prompt": "/ai-slop:audit docs/example.md — the file is documentation ABOUT model knowledge cutoffs, so rule-knowledge-cutoff-disclaimer fires on prose that is describing the concept rather than disclaiming on the document's own behalf.", - "expected_output": "The finding is reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit evals/fixtures/knowledge-cutoff-prose.md (path relative to the skill directory)", + "expected_output": "The three rule-knowledge-cutoff-disclaimer findings (lines 1, 3 and 5) are reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. The fixture is documentation ABOUT knowledge cutoffs, so the rule fires on prose describing the concept rather than disclaiming on the document's own behalf. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.", + "files": ["evals/fixtures/knowledge-cutoff-prose.md"], "expectations": [ - "Recognizes prose ABOUT knowledge cutoffs as the recorded false-positive class rather than deleting the sentence", + "Recognizes prose ABOUT knowledge cutoffs as the recorded false-positive class rather than deleting the sentences", "Recommends the in-file ignore marker or a config exclusion, citing the calibration record", "Does NOT propose weakening or disabling the shipped rule to make this corpus pass" ] diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/em-dash-substitution.md b/plugins/ai-slop/skills/audit/evals/fixtures/em-dash-substitution.md new file mode 100644 index 000000000..117bb1bab --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/em-dash-substitution.md @@ -0,0 +1,8 @@ +# Parser release notes + +## 0.4 + +The parser accepts empty input — a change from 0.3. Callers that relied on the +old rejection path have to check for the empty case themselves. + +The token table grew by two entries. Both of them are punctuation. diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/fix-guarded-rewrite.md b/plugins/ai-slop/skills/audit/evals/fixtures/fix-guarded-rewrite.md new file mode 100644 index 000000000..37b4ef297 --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/fix-guarded-rewrite.md @@ -0,0 +1,8 @@ +# Cache invalidation + +It is worth noting that the cache invalidates on write — the reader never sees +a stale entry. Due to the fact that writes are rare, the extra round trip costs +little. + +The eviction policy is least-recently-used — a choice made when the working set +was small, and one the team has not revisited since. diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/knowledge-cutoff-prose.md b/plugins/ai-slop/skills/audit/evals/fixtures/knowledge-cutoff-prose.md new file mode 100644 index 000000000..b76df12ed --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/knowledge-cutoff-prose.md @@ -0,0 +1,9 @@ +# What a knowledge cutoff is + +A model's knowledge cutoff is the date after which its training corpus contains +nothing. Documentation that explains the concept has to name it, which is why +this page writes "knowledge cutoff" as its subject rather than as a disclaimer +about this page's own currency. + +Readers sometimes confuse the cutoff with the release date. They are different +dates, and the gap between them is often months wide. diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/report-only.md b/plugins/ai-slop/skills/audit/evals/fixtures/report-only.md new file mode 100644 index 000000000..77164b491 --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/report-only.md @@ -0,0 +1,9 @@ +# Retry helper + +In order to retry a failed request, the helper doubles its wait between +attempts and gives up after five tries. + +The helper plays a crucial role in keeping the queue drained during an outage. + +I hope this helps — the defaults live in `config.toml` if you want to change +them. diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/rubric-boundary.md b/plugins/ai-slop/skills/audit/evals/fixtures/rubric-boundary.md new file mode 100644 index 000000000..738ef2509 --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/rubric-boundary.md @@ -0,0 +1,8 @@ +# Metrics dashboard + +The metrics dashboard is a breathtaking, must-visit surface sitting right in the +heart of the platform, and the team behind it has poured real craft into every +panel. Anyone who cares about latency will want to spend an afternoon here. + +Panels refresh every thirty seconds — the interval is fixed and not yet +configurable. diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/triads.md b/plugins/ai-slop/skills/audit/evals/fixtures/triads.md new file mode 100644 index 000000000..91cd25f7c --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/triads.md @@ -0,0 +1,14 @@ +# Configuration cascade + +## Why the tool reads well + +The tool is fast, simple, and reliable. The maintainers intend to keep it that +way. + +The generated output reads clean, tight, and portable, which is the whole point +of the generator. + +## The cascade + +The cascade layers are user, team, and local. Resolution runs in that order, +and dropping any one of them names an order that does not exist. diff --git a/plugins/ai-slop/skills/audit/scripts/detect.test.sh b/plugins/ai-slop/skills/audit/scripts/detect.test.sh index 219b2059e..c31168f48 100755 --- a/plugins/ai-slop/skills/audit/scripts/detect.test.sh +++ b/plugins/ai-slop/skills/audit/scripts/detect.test.sh @@ -1,7 +1,13 @@ #!/usr/bin/env bash -# Self-contained tests for detect.sh (no external test lib; fixtures are built -# inline in a tmpdir, so the plugin ships no slop samples for the audit to trip -# over). Per the shell-test-helpers convention, assertion helpers are local. +# Self-contained tests for detect.sh (no external test lib; THIS suite's +# fixtures are built inline in a tmpdir, so no unit-test slop corpus sits in +# the tree). The eval suite is the deliberate exception: evals/fixtures/ holds +# committed slop samples, because a prose scenario cannot be checked against +# the detector and drifted from it three times (#3041). This repo keeps its own +# audit off them with an excluded_paths glob in .claude/ai-slop.json, and they +# stay measurable through the same config isolation this suite uses — an empty +# HOME and CLAUDE_PROJECT_DIR give the shipped defaults back. +# Per the shell-test-helpers convention, assertion helpers are local. set -uo pipefail # Fixture git isolation: an inherited GIT_DIR/GIT_WORK_TREE/GIT_CONFIG would From 1c06107669e6558ca494f145c77daf6fd845e8aa Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 05:27:15 +0000 Subject: [PATCH 2/8] test(ai-slop): stage eval fixtures into a scratch repo before auditing them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fixture-backed cases pointed the invocation straight at the committed fixture, which reintroduced the failure class #3041 was meant to close. The fix cases (2, 6, 7) rewrite each flagged line in place, so the first run remediated the committed fixture and the second run graded already-fixed input, where the declared findings no longer fire. The persistence expectations (2, 4, 5) grade a branch conditioned on the audit having examined tracked files. An installed plugin's fixture is not tracked in the consuming repo, so the branch was unreachable, and case 4's contract-fetch refusal passed for the wrong reason: nothing was written because nothing was owed, not because the fetch failed. Every fixture-backed case now stages its target first — git init a temp directory, copy the fixture in as docs/.md, commit it, audit the copy. The rewrite lands on a throwaway file and the target is tracked, so the persistence branch is genuinely reached. Staged copies re-measured: identical rules, lines and densities at the new path. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g --- plugins/ai-slop/CHANGELOG.md | 11 ++++ plugins/ai-slop/skills/audit/evals/evals.json | 50 ++++++++++--------- 2 files changed, 38 insertions(+), 23 deletions(-) diff --git a/plugins/ai-slop/CHANGELOG.md b/plugins/ai-slop/CHANGELOG.md index e60ba1547..f5682157d 100644 --- a/plugins/ai-slop/CHANGELOG.md +++ b/plugins/ai-slop/CHANGELOG.md @@ -14,6 +14,17 @@ name a committed fixture instead. `triads.md` (rule-of-three at 3 hits in 69 words) and `knowledge-cutoff-prose.md` (the recorded false-positive class). Every case's `expected_output` now names the rules, lines and fired thresholds the detector actually emits, measured rather than asserted. +- **Every fixture-backed case stages its target before auditing it**: `git init` a temp directory, + copy the fixture in as `docs/.md`, commit it, and point the invocation at that copy. A case + that named the fixture directly was self-corrupting in two ways. The `fix` cases (2, 6, 7) rewrite + each flagged line in place, so the first run remediated the committed fixture and the second run + graded already-fixed input, where the declared findings no longer fire. And the persistence + expectations (cases 2, 4, 5) grade a branch conditioned on the audit having "examined tracked + files" — an installed plugin's fixture is not tracked in the consuming repo, so the branch was + unreachable and case 4's contract-fetch refusal passed for the wrong reason: nothing was written + because nothing was owed, not because the fetch failed. Staging fixes both at once: the rewrite + lands on a throwaway copy, and the copy is tracked, so persistence is genuinely owed. The staged + copies were re-measured — identical rules, lines and densities at the new path. - **This reverses 0.1.0's no-fixtures decision, which was recorded in `detect.test.sh`'s header.** That decision holds for the *unit* suite, whose fixtures are still built inline in a tmpdir. It does not survive contact with the eval suite: an eval case is graded against a deterministic diff --git a/plugins/ai-slop/skills/audit/evals/evals.json b/plugins/ai-slop/skills/audit/evals/evals.json index 0e5c22cfe..bbda75379 100644 --- a/plugins/ai-slop/skills/audit/evals/evals.json +++ b/plugins/ai-slop/skills/audit/evals/evals.json @@ -4,27 +4,27 @@ { "id": 1, "name": "bare-invocation-read-only", - "prompt": "/ai-slop:audit evals/fixtures/report-only.md (path relative to the skill directory)", - "expected_output": "A findings report for the fixture — four findings across four rules (rule-filler-phrases on line 3, rule-significance-inflation on line 6, rule-em-dash and rule-chatbot-artifacts both on line 8), each with its file:line location and fired condition, plus the Summary rows — with no file modified. The fix flow is offered but not run.", + "prompt": "/ai-slop:audit docs/report-only.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it, so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", + "expected_output": "A findings report for the staged copy `docs/report-only.md` — four findings across four rules (rule-filler-phrases on line 3, rule-significance-inflation on line 6, rule-em-dash and rule-chatbot-artifacts both on line 8), each with its file:line location and fired condition, plus the Summary rows — with no file modified. The fix flow is offered but not run.", "files": ["evals/fixtures/report-only.md"], "expectations": [ - "Runs detect.sh on the target and reports findings with file:line locations and the fired condition per finding", + "Runs detect.sh on the staged copy and reports findings with file:line locations and the fired condition per finding", "Reports all four rules the fixture trips rather than a subset — filler phrase, significance inflation, em dash, and chat-turn residue", - "Does NOT edit the fixture or any other file on bare invocation", + "Does NOT edit the staged copy, the committed fixture, or any other file on bare invocation", "Mentions the fix action as the explicit follow-up rather than applying rewrites" ] }, { "id": 2, "name": "fix-runs-semantic-guard", - "prompt": "/ai-slop:audit fix evals/fixtures/fix-guarded-rewrite.md (path relative to the skill directory)", - "expected_output": "The fixture's four findings — two em dashes (lines 3 and 7) and two filler phrases ('It is worth noting that' on line 3, 'Due to the fact that' on line 4) — are rewritten in one per-file pass, the before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs and the findings file is re-emitted so no stale findings survive.", + "prompt": "/ai-slop:audit fix docs/fix-guarded-rewrite.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/fix-guarded-rewrite.md` (path relative to the skill directory) in as `docs/fix-guarded-rewrite.md`, and commit it, so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", + "expected_output": "The staged copy's four findings — two em dashes (lines 3 and 7) and two filler phrases ('It is worth noting that' on line 3, 'Due to the fact that' on line 4) — are rewritten in one per-file pass, the before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs over the rewritten copy and the findings file is re-emitted (the target is tracked, so persistence is owed) so no stale findings survive.", "files": ["evals/fixtures/fix-guarded-rewrite.md"], "expectations": [ - "Applies rewrites only because fix was explicitly requested", + "Applies rewrites only because fix was explicitly requested, and only to the staged copy — `evals/fixtures/fix-guarded-rewrite.md` under the skill directory is byte-identical after the run", "Verifies the rewritten file with a fresh-context semantic-diff subagent and reverts flagged hunks", "Preserves every claim the fixture makes — that the cache invalidates on write, that the reader never sees a stale entry, that writes are rare, and that the policy is least-recently-used", - "Re-runs the detector and re-emits the findings file after the last fixed file" + "Re-runs the detector and re-emits the findings file after the last fixed file, because the audited target is tracked in the scratch repo" ] }, { @@ -43,62 +43,66 @@ { "id": 4, "name": "persist-refuses-without-contract", - "prompt": "/ai-slop:audit evals/fixtures/report-only.md (path relative to the skill directory), with the detector-findings contract URL unreachable", - "expected_output": "The audit report is delivered with the fixture's four findings intact, but no findings file is written: the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.", + "prompt": "/ai-slop:audit docs/report-only.md, with the detector-findings contract URL unreachable — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it, so the audit examines a tracked file and a findings file is owed.", + "expected_output": "The audit report is delivered with the staged copy's four findings intact, but no findings file is written: the target is tracked, so persistence is owed, and the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.", "files": ["evals/fixtures/report-only.md"], "expectations": [ "Delivers the human-facing findings report regardless", "Does NOT write a findings file when the contract fetch fails", - "States that persistence was refused because the contract was unreachable, rather than silently skipping it" + "States that persistence was refused because the contract was unreachable, rather than silently skipping it", + "Attributes the refusal to the failed contract fetch and not to the target sitting outside tracked space — the staged copy is committed, so the persistence branch is genuinely reached" ] }, { "id": 5, "name": "rubric-findings-reach-the-report-not-the-findings-file", - "prompt": "/ai-slop:audit evals/fixtures/rubric-boundary.md (path relative to the skill directory)", - "expected_output": "Both layers report: the em dash on line 7 appears as the fixture's single script finding, and the promotional register on lines 3-5 ('a breathtaking, must-visit surface sitting right in the heart of the platform') as a rubric finding citing its catalog entry. The persisted findings file carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.", + "prompt": "/ai-slop:audit docs/rubric-boundary.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/rubric-boundary.md` (path relative to the skill directory) in as `docs/rubric-boundary.md`, and commit it, so the audit target is tracked and a findings file is owed.", + "expected_output": "Both layers report on the staged copy: the em dash on line 7 appears as its single script finding, and the promotional register on lines 3-5 ('a breathtaking, must-visit surface sitting right in the heart of the platform') as a rubric finding citing its catalog entry. The target is tracked, so a findings file IS written, and it carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.", "files": ["evals/fixtures/rubric-boundary.md"], "expectations": [ "Reports the promotional-register tell as a rubric finding, quoting the offending text and naming the catalog entry it comes from", "Does NOT report the promotional wording as a script finding — the fixture's promotional words sit outside the mechanical vocabulary, so the detector's only finding is the em dash", - "Does NOT place any rubric finding in the persisted findings file — only the script finding appears there", + "Writes the findings file (the staged copy is tracked) and places NO rubric finding in it — only the em-dash script finding appears there", "States the V1 boundary explicitly rather than silently omitting the rubric verdict from the file" ] }, { "id": 6, "name": "fix-never-swaps-one-tell-for-another", - "prompt": "/ai-slop:audit fix evals/fixtures/em-dash-substitution.md (path relative to the skill directory)", - "expected_output": "The fixture's single finding — the em dash on line 5, in 'The parser accepts empty input — a change from 0.3.' — is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.", + "prompt": "/ai-slop:audit fix docs/em-dash-substitution.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/em-dash-substitution.md` (path relative to the skill directory) in as `docs/em-dash-substitution.md`, and commit it, so the rewrite lands on a tracked, disposable copy and the committed fixture stays byte-identical.", + "expected_output": "The staged copy's single finding — the em dash on line 5, in 'The parser accepts empty input — a change from 0.3.' — is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.", "files": ["evals/fixtures/em-dash-substitution.md"], "expectations": [ "Reads reference/rewrite-guide.md before applying the rewrite, per the fix flow's first step", "The rewritten line contains no em dash, en dash, parenthesis, or spaced hyphen standing in for the original punctuation", - "Preserves the sentence's claim — the version fact survives the rewrite" + "Preserves the sentence's claim — the version fact survives the rewrite", + "Rewrites only the staged copy — `evals/fixtures/em-dash-substitution.md` under the skill directory is byte-identical after the run" ] }, { "id": 7, "name": "triads-collapse-toward-the-strongest-item", - "prompt": "/ai-slop:audit fix evals/fixtures/triads.md (path relative to the skill directory)", - "expected_output": "The fixture trips rule-rule-of-three as one density finding (3 hits in 69 words, 43.5/1000 against a 3.0 threshold), and all three triads reach the fix flow together. The two rhetorical triads — 'fast, simple, and reliable' and 'clean, tight, and portable' — collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration 'user, team, and local' is KEPT intact, with that judgment stated: dropping one names a resolution order that does not exist.", + "prompt": "/ai-slop:audit fix docs/triads.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/triads.md` (path relative to the skill directory) in as `docs/triads.md`, and commit it, so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", + "expected_output": "The staged copy trips rule-rule-of-three as one density finding (3 hits in 69 words, 43.5/1000 against a 3.0 threshold), and all three triads reach the fix flow together. The two rhetorical triads — 'fast, simple, and reliable' and 'clean, tight, and portable' — collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration 'user, team, and local' is KEPT intact, with that judgment stated: dropping one names a resolution order that does not exist.", "files": ["evals/fixtures/triads.md"], "expectations": [ "Collapses the two rhetorical triads toward one item rather than preserving all three in a reworded form", "Keeps the 'user, team, and local' triad because each element is load-bearing, and says why it was kept", - "Does not treat every three-item enumeration as a tell — enumerating three actual things is not slop" + "Does not treat every three-item enumeration as a tell — enumerating three actual things is not slop", + "Rewrites only the staged copy — `evals/fixtures/triads.md` under the skill directory is byte-identical after the run" ] }, { "id": 8, "name": "known-false-positive-routes-to-config-not-a-rewrite", - "prompt": "/ai-slop:audit evals/fixtures/knowledge-cutoff-prose.md (path relative to the skill directory)", - "expected_output": "The three rule-knowledge-cutoff-disclaimer findings (lines 1, 3 and 5) are reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. The fixture is documentation ABOUT knowledge cutoffs, so the rule fires on prose describing the concept rather than disclaiming on the document's own behalf. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.", + "prompt": "/ai-slop:audit docs/knowledge-cutoff-prose.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/knowledge-cutoff-prose.md` (path relative to the skill directory) in as `docs/knowledge-cutoff-prose.md`, and commit it, so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", + "expected_output": "The staged copy's three rule-knowledge-cutoff-disclaimer findings (lines 1, 3 and 5) are reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. The fixture is documentation ABOUT knowledge cutoffs, so the rule fires on prose describing the concept rather than disclaiming on the document's own behalf. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.", "files": ["evals/fixtures/knowledge-cutoff-prose.md"], "expectations": [ "Recognizes prose ABOUT knowledge cutoffs as the recorded false-positive class rather than deleting the sentences", "Recommends the in-file ignore marker or a config exclusion, citing the calibration record", - "Does NOT propose weakening or disabling the shipped rule to make this corpus pass" + "Does NOT propose weakening or disabling the shipped rule to make this corpus pass", + "Recommends the exemption without editing anything — neither the staged copy nor the committed fixture is rewritten on a bare audit" ] }, { From 50104376455886728c6600966f75d3533d67507a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 05:37:59 +0000 Subject: [PATCH 3/8] test(ai-slop): give the eval staging commit an explicit git identity The staging preamble said "git init a temp directory ... and commit it", but git init inherits user.name/user.email from ~/.gitconfig or the GIT_* env, and neither is guaranteed. This suite's own unit tests pin HOME to an empty tmpdir for isolation (detect.test.sh), and a fresh container has no seeded identity. Under that condition the preamble dies at git commit with "Please tell me who you are" (exit 128) BEFORE the audit runs, so all seven staged cases error out instead of grading anything - the same "scenario cannot produce the graded input" failure class this release exists to remove. All seven preambles now commit with the identity inline: git -c user.name=eval -c user.email=eval@local commit -m stage Staging still stops at a commit rather than at git add. git ls-files already reports a staged file as tracked, so either form satisfies the persistence gate, but these cases model a consuming repo and a real one has history: the repo-wide ordering reads git log --since=90.days --name-only for change frequency, which returns nothing in a repo with no commits. Harmless for today's single-file cases, and invisible in every current expectation, which is why it is worth not building in. Verified by executing each preamble as written under an empty HOME with GIT_AUTHOR_*/GIT_COMMITTER_* unset: all seven commit, leave a HEAD, report the target tracked, and emit their declared finding counts (4, 4, 4, 1, 1, 1, 3). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g --- plugins/ai-slop/CHANGELOG.md | 14 ++++++++++++++ plugins/ai-slop/skills/audit/evals/evals.json | 14 +++++++------- 2 files changed, 21 insertions(+), 7 deletions(-) diff --git a/plugins/ai-slop/CHANGELOG.md b/plugins/ai-slop/CHANGELOG.md index f5682157d..9e14eff78 100644 --- a/plugins/ai-slop/CHANGELOG.md +++ b/plugins/ai-slop/CHANGELOG.md @@ -25,6 +25,20 @@ name a committed fixture instead. because nothing was owed, not because the fetch failed. Staging fixes both at once: the rewrite lands on a throwaway copy, and the copy is tracked, so persistence is genuinely owed. The staged copies were re-measured — identical rules, lines and densities at the new path. +- **The staging commit sets its identity inline**, `git -c user.name=eval -c user.email=eval@local + commit -m stage`. `git init` inherits `user.name`/`user.email` from `~/.gitconfig` or the `GIT_*` + environment, and neither is guaranteed: this suite's own unit tests pin `HOME` to an empty tmpdir + for isolation (`detect.test.sh`), and a fresh container has no seeded identity either. Without the + flags the preamble dies at `git commit` with "Please tell me who you are" (exit 128) *before the + audit runs*, so all seven staged cases error out instead of grading — the same "scenario cannot + produce the graded input" failure this release exists to remove. Verified by executing each + preamble as written under an empty `HOME` with `GIT_AUTHOR_*`/`GIT_COMMITTER_*` unset. +- Staging stops at a commit rather than at `git add`, though `git ls-files` already reports a staged + file as tracked and either form would satisfy the persistence gate. These cases model a *consuming + repo*, and a real one has history: the audit's repo-wide ordering reads + `git log --since=90.days --name-only` for change frequency, which returns nothing in a repo with + no commits. That divergence is harmless for today's single-file cases and invisible in every + current expectation, which is exactly why it is worth not building in. - **This reverses 0.1.0's no-fixtures decision, which was recorded in `detect.test.sh`'s header.** That decision holds for the *unit* suite, whose fixtures are still built inline in a tmpdir. It does not survive contact with the eval suite: an eval case is graded against a deterministic diff --git a/plugins/ai-slop/skills/audit/evals/evals.json b/plugins/ai-slop/skills/audit/evals/evals.json index bbda75379..99a6d6fd5 100644 --- a/plugins/ai-slop/skills/audit/evals/evals.json +++ b/plugins/ai-slop/skills/audit/evals/evals.json @@ -4,7 +4,7 @@ { "id": 1, "name": "bare-invocation-read-only", - "prompt": "/ai-slop:audit docs/report-only.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it, so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", + "prompt": "/ai-slop:audit docs/report-only.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", "expected_output": "A findings report for the staged copy `docs/report-only.md` — four findings across four rules (rule-filler-phrases on line 3, rule-significance-inflation on line 6, rule-em-dash and rule-chatbot-artifacts both on line 8), each with its file:line location and fired condition, plus the Summary rows — with no file modified. The fix flow is offered but not run.", "files": ["evals/fixtures/report-only.md"], "expectations": [ @@ -17,7 +17,7 @@ { "id": 2, "name": "fix-runs-semantic-guard", - "prompt": "/ai-slop:audit fix docs/fix-guarded-rewrite.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/fix-guarded-rewrite.md` (path relative to the skill directory) in as `docs/fix-guarded-rewrite.md`, and commit it, so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", + "prompt": "/ai-slop:audit fix docs/fix-guarded-rewrite.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/fix-guarded-rewrite.md` (path relative to the skill directory) in as `docs/fix-guarded-rewrite.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", "expected_output": "The staged copy's four findings — two em dashes (lines 3 and 7) and two filler phrases ('It is worth noting that' on line 3, 'Due to the fact that' on line 4) — are rewritten in one per-file pass, the before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs over the rewritten copy and the findings file is re-emitted (the target is tracked, so persistence is owed) so no stale findings survive.", "files": ["evals/fixtures/fix-guarded-rewrite.md"], "expectations": [ @@ -43,7 +43,7 @@ { "id": 4, "name": "persist-refuses-without-contract", - "prompt": "/ai-slop:audit docs/report-only.md, with the detector-findings contract URL unreachable — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it, so the audit examines a tracked file and a findings file is owed.", + "prompt": "/ai-slop:audit docs/report-only.md, with the detector-findings contract URL unreachable — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit examines a tracked file and a findings file is owed.", "expected_output": "The audit report is delivered with the staged copy's four findings intact, but no findings file is written: the target is tracked, so persistence is owed, and the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.", "files": ["evals/fixtures/report-only.md"], "expectations": [ @@ -56,7 +56,7 @@ { "id": 5, "name": "rubric-findings-reach-the-report-not-the-findings-file", - "prompt": "/ai-slop:audit docs/rubric-boundary.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/rubric-boundary.md` (path relative to the skill directory) in as `docs/rubric-boundary.md`, and commit it, so the audit target is tracked and a findings file is owed.", + "prompt": "/ai-slop:audit docs/rubric-boundary.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/rubric-boundary.md` (path relative to the skill directory) in as `docs/rubric-boundary.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is tracked and a findings file is owed.", "expected_output": "Both layers report on the staged copy: the em dash on line 7 appears as its single script finding, and the promotional register on lines 3-5 ('a breathtaking, must-visit surface sitting right in the heart of the platform') as a rubric finding citing its catalog entry. The target is tracked, so a findings file IS written, and it carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.", "files": ["evals/fixtures/rubric-boundary.md"], "expectations": [ @@ -69,7 +69,7 @@ { "id": 6, "name": "fix-never-swaps-one-tell-for-another", - "prompt": "/ai-slop:audit fix docs/em-dash-substitution.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/em-dash-substitution.md` (path relative to the skill directory) in as `docs/em-dash-substitution.md`, and commit it, so the rewrite lands on a tracked, disposable copy and the committed fixture stays byte-identical.", + "prompt": "/ai-slop:audit fix docs/em-dash-substitution.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/em-dash-substitution.md` (path relative to the skill directory) in as `docs/em-dash-substitution.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrite lands on a tracked, disposable copy and the committed fixture stays byte-identical.", "expected_output": "The staged copy's single finding — the em dash on line 5, in 'The parser accepts empty input — a change from 0.3.' — is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.", "files": ["evals/fixtures/em-dash-substitution.md"], "expectations": [ @@ -82,7 +82,7 @@ { "id": 7, "name": "triads-collapse-toward-the-strongest-item", - "prompt": "/ai-slop:audit fix docs/triads.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/triads.md` (path relative to the skill directory) in as `docs/triads.md`, and commit it, so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", + "prompt": "/ai-slop:audit fix docs/triads.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/triads.md` (path relative to the skill directory) in as `docs/triads.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", "expected_output": "The staged copy trips rule-rule-of-three as one density finding (3 hits in 69 words, 43.5/1000 against a 3.0 threshold), and all three triads reach the fix flow together. The two rhetorical triads — 'fast, simple, and reliable' and 'clean, tight, and portable' — collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration 'user, team, and local' is KEPT intact, with that judgment stated: dropping one names a resolution order that does not exist.", "files": ["evals/fixtures/triads.md"], "expectations": [ @@ -95,7 +95,7 @@ { "id": 8, "name": "known-false-positive-routes-to-config-not-a-rewrite", - "prompt": "/ai-slop:audit docs/knowledge-cutoff-prose.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/knowledge-cutoff-prose.md` (path relative to the skill directory) in as `docs/knowledge-cutoff-prose.md`, and commit it, so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", + "prompt": "/ai-slop:audit docs/knowledge-cutoff-prose.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/knowledge-cutoff-prose.md` (path relative to the skill directory) in as `docs/knowledge-cutoff-prose.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", "expected_output": "The staged copy's three rule-knowledge-cutoff-disclaimer findings (lines 1, 3 and 5) are reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. The fixture is documentation ABOUT knowledge cutoffs, so the rule fires on prose describing the concept rather than disclaiming on the document's own behalf. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.", "files": ["evals/fixtures/knowledge-cutoff-prose.md"], "expectations": [ From 0f5fec477b59049efb0e17808095f6027f8146b1 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 05:54:10 +0000 Subject: [PATCH 4/8] test(ai-slop): pin the detector config cascade in the eval staging preamble The staging preamble isolated git identity but not the config cascade. detect.sh resolves REPO_ROOT from CLAUDE_PROJECT_DIR before falling back to git rev-parse --show-toplevel, then loads that root's .claude/ai-slop.json, and rule disablement is not path-scoped. A maintainer running these evals from a session rooted at this repo therefore inherits its disabled_rules - rule-em-dash, rule-curly-artifacts, rule-emoji-formatting - and those apply to the /tmp scratch repo too. Measured with only CLAUDE_PROJECT_DIR changed: case 1: 4 -> 3 findings (em dash L8 vanishes) case 2: 4 -> 2 (both em-dash findings vanish) case 4: 4 -> 3 case 5: 1 -> 0 (the em dash IS the single script finding) case 6: 1 -> 0 (the case is entirely that one em dash) cases 7, 8: unaffected Cases 5 and 6 fail silently: nothing is found, nothing is asserted, and the expectations pass vacuously. Same golden-answer-the-scenario-cannot-produce class as the rest of this release, arriving through config rather than prose. All seven fixture-backed prompts now point CLAUDE_PROJECT_DIR at the scratch directory and HOME at an empty one, so the cascade resolves to shipped defaults. Applied to 7 and 8 as well: their rules are not disabled today, but disabled_rules can change and uniformity is cheaper than a per-case rule. The knowledge that this isolation is what keeps the fixtures measurable was already recorded in detect.test.sh's header comment; it now lives in the artifact that needs it. Prompts only - no expected_output, files, expectations, narration, or fixture byte changed, and cases 3 and 9 stay byte-identical to dc5ca9ad. Re-ran all seven preambles literally under the pinned cascade: every rule id, line and density figure still matches its expected_output, case 7 included. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g --- plugins/ai-slop/CHANGELOG.md | 10 ++++++++++ plugins/ai-slop/skills/audit/evals/evals.json | 14 +++++++------- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/plugins/ai-slop/CHANGELOG.md b/plugins/ai-slop/CHANGELOG.md index 9e14eff78..9f1b78115 100644 --- a/plugins/ai-slop/CHANGELOG.md +++ b/plugins/ai-slop/CHANGELOG.md @@ -33,6 +33,16 @@ name a committed fixture instead. audit runs*, so all seven staged cases error out instead of grading — the same "scenario cannot produce the graded input" failure this release exists to remove. Verified by executing each preamble as written under an empty `HOME` with `GIT_AUTHOR_*`/`GIT_COMMITTER_*` unset. +- **The preamble pins the config cascade as well as the identity**, pointing `CLAUDE_PROJECT_DIR` at + the scratch directory and `HOME` at an empty one. `detect.sh` resolves `CLAUDE_PROJECT_DIR` ahead + of `git rev-parse --show-toplevel` and then loads that root's `.claude/ai-slop.json`, and rule + disablement is not path-scoped — so a maintainer running these evals from a session rooted at this + repo inherits its `disabled_rules`, and the finding a case grades is silently deleted from a + scratch repo in `/tmp`. Measured with only `CLAUDE_PROJECT_DIR` changed: case 1 drops 4 findings to + 3, case 2 drops 4 to 2, case 4 drops 4 to 3, and cases 5 and 6 drop to **zero**, where the em dash + *is* the whole case and the expectations then pass vacuously. That is the same golden-answer-the- + scenario-cannot-produce failure as the rest of this release, arriving through config rather than + through prose. - Staging stops at a commit rather than at `git add`, though `git ls-files` already reports a staged file as tracked and either form would satisfy the persistence gate. These cases model a *consuming repo*, and a real one has history: the audit's repo-wide ordering reads diff --git a/plugins/ai-slop/skills/audit/evals/evals.json b/plugins/ai-slop/skills/audit/evals/evals.json index 99a6d6fd5..5c194d61b 100644 --- a/plugins/ai-slop/skills/audit/evals/evals.json +++ b/plugins/ai-slop/skills/audit/evals/evals.json @@ -4,7 +4,7 @@ { "id": 1, "name": "bare-invocation-read-only", - "prompt": "/ai-slop:audit docs/report-only.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", + "prompt": "/ai-slop:audit docs/report-only.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", "expected_output": "A findings report for the staged copy `docs/report-only.md` — four findings across four rules (rule-filler-phrases on line 3, rule-significance-inflation on line 6, rule-em-dash and rule-chatbot-artifacts both on line 8), each with its file:line location and fired condition, plus the Summary rows — with no file modified. The fix flow is offered but not run.", "files": ["evals/fixtures/report-only.md"], "expectations": [ @@ -17,7 +17,7 @@ { "id": 2, "name": "fix-runs-semantic-guard", - "prompt": "/ai-slop:audit fix docs/fix-guarded-rewrite.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/fix-guarded-rewrite.md` (path relative to the skill directory) in as `docs/fix-guarded-rewrite.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", + "prompt": "/ai-slop:audit fix docs/fix-guarded-rewrite.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/fix-guarded-rewrite.md` (path relative to the skill directory) in as `docs/fix-guarded-rewrite.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", "expected_output": "The staged copy's four findings — two em dashes (lines 3 and 7) and two filler phrases ('It is worth noting that' on line 3, 'Due to the fact that' on line 4) — are rewritten in one per-file pass, the before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs over the rewritten copy and the findings file is re-emitted (the target is tracked, so persistence is owed) so no stale findings survive.", "files": ["evals/fixtures/fix-guarded-rewrite.md"], "expectations": [ @@ -43,7 +43,7 @@ { "id": 4, "name": "persist-refuses-without-contract", - "prompt": "/ai-slop:audit docs/report-only.md, with the detector-findings contract URL unreachable — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit examines a tracked file and a findings file is owed.", + "prompt": "/ai-slop:audit docs/report-only.md, with the detector-findings contract URL unreachable — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit examines a tracked file and a findings file is owed.", "expected_output": "The audit report is delivered with the staged copy's four findings intact, but no findings file is written: the target is tracked, so persistence is owed, and the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.", "files": ["evals/fixtures/report-only.md"], "expectations": [ @@ -56,7 +56,7 @@ { "id": 5, "name": "rubric-findings-reach-the-report-not-the-findings-file", - "prompt": "/ai-slop:audit docs/rubric-boundary.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/rubric-boundary.md` (path relative to the skill directory) in as `docs/rubric-boundary.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is tracked and a findings file is owed.", + "prompt": "/ai-slop:audit docs/rubric-boundary.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/rubric-boundary.md` (path relative to the skill directory) in as `docs/rubric-boundary.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is tracked and a findings file is owed.", "expected_output": "Both layers report on the staged copy: the em dash on line 7 appears as its single script finding, and the promotional register on lines 3-5 ('a breathtaking, must-visit surface sitting right in the heart of the platform') as a rubric finding citing its catalog entry. The target is tracked, so a findings file IS written, and it carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.", "files": ["evals/fixtures/rubric-boundary.md"], "expectations": [ @@ -69,7 +69,7 @@ { "id": 6, "name": "fix-never-swaps-one-tell-for-another", - "prompt": "/ai-slop:audit fix docs/em-dash-substitution.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/em-dash-substitution.md` (path relative to the skill directory) in as `docs/em-dash-substitution.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrite lands on a tracked, disposable copy and the committed fixture stays byte-identical.", + "prompt": "/ai-slop:audit fix docs/em-dash-substitution.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/em-dash-substitution.md` (path relative to the skill directory) in as `docs/em-dash-substitution.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrite lands on a tracked, disposable copy and the committed fixture stays byte-identical.", "expected_output": "The staged copy's single finding — the em dash on line 5, in 'The parser accepts empty input — a change from 0.3.' — is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.", "files": ["evals/fixtures/em-dash-substitution.md"], "expectations": [ @@ -82,7 +82,7 @@ { "id": 7, "name": "triads-collapse-toward-the-strongest-item", - "prompt": "/ai-slop:audit fix docs/triads.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/triads.md` (path relative to the skill directory) in as `docs/triads.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", + "prompt": "/ai-slop:audit fix docs/triads.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/triads.md` (path relative to the skill directory) in as `docs/triads.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", "expected_output": "The staged copy trips rule-rule-of-three as one density finding (3 hits in 69 words, 43.5/1000 against a 3.0 threshold), and all three triads reach the fix flow together. The two rhetorical triads — 'fast, simple, and reliable' and 'clean, tight, and portable' — collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration 'user, team, and local' is KEPT intact, with that judgment stated: dropping one names a resolution order that does not exist.", "files": ["evals/fixtures/triads.md"], "expectations": [ @@ -95,7 +95,7 @@ { "id": 8, "name": "known-false-positive-routes-to-config-not-a-rewrite", - "prompt": "/ai-slop:audit docs/knowledge-cutoff-prose.md — run it inside a scratch git repo staged first: `git init` a temp directory, copy `evals/fixtures/knowledge-cutoff-prose.md` (path relative to the skill directory) in as `docs/knowledge-cutoff-prose.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", + "prompt": "/ai-slop:audit docs/knowledge-cutoff-prose.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/knowledge-cutoff-prose.md` (path relative to the skill directory) in as `docs/knowledge-cutoff-prose.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", "expected_output": "The staged copy's three rule-knowledge-cutoff-disclaimer findings (lines 1, 3 and 5) are reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. The fixture is documentation ABOUT knowledge cutoffs, so the rule fires on prose describing the concept rather than disclaiming on the document's own behalf. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.", "files": ["evals/fixtures/knowledge-cutoff-prose.md"], "expectations": [ From 1adb33a7db9276b89a80c521b58d1baa90c56c80 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 06:15:33 +0000 Subject: [PATCH 5/8] test(ai-slop): make the eval staging preamble an executable shell sequence Three residual defects, all the same class: the preamble described staging in prose and left the mechanism to the reader. A. The cascade pin was phrased as a one-time export, but the harness re-supplies CLAUDE_PROJECT_DIR on every tool call and shell state does not survive between them. A model that staged in one call and ran the detector in the next got the disabled rule back and measured 0 findings on case 5. The preamble now runs in ONE invocation, says why, and instructs setting CLAUDE_PROJECT_DIR/HOME on every later command rather than exporting once. B. The preamble never issued git add, so a literal executor produced an untracked target - inverting the tracked-file branch cases 4 and 5 exist to grade. The sequence now stages before committing. C. The scratch repo was built without clearing inherited git environment, violating the invariant scripts/check-fixture-git-isolation.sh already enforces for *.sh/*.py fixtures. Under an ambient GIT_DIR the init and the stage commit landed in the CALLER's repository and the scratch directory got no .git at all. That gate cannot see prose inside evals.json, so the eval suite now honors the rule itself: the sequence opens with unset GIT_DIR GIT_WORK_TREE GIT_CONFIG. Also repairs a hyphenated compound broken across a line wrap in the 0.3.0 entry, which rendered as "golden-answer-the- scenario-cannot-produce". Verified by extracting each command verbatim from the committed JSON and running it under a hostile environment - HOME holding a config that disables rule-em-dash, CLAUDE_PROJECT_DIR pointed at this repo, and an ambient GIT_DIR pointed at this repo's .git. All seven: commit exit 0, HEAD created, target tracked, scratch dir owns its .git, caller HEAD unchanged, and the declared finding counts (4, 4, 4, 1, 1, 1, 3). Detector re-run in a separate tool call from staging still reports them; relying on ambient env instead reports 0, which is the failure A describes. Prompts only - no expected_output, expectations, files, narration, name or skill_name changed, cases 3 and 9 stay byte-identical to dc5ca9ad, and no fixture byte moved. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g --- plugins/ai-slop/CHANGELOG.md | 18 +++++++++++++++--- plugins/ai-slop/skills/audit/evals/evals.json | 14 +++++++------- 2 files changed, 22 insertions(+), 10 deletions(-) diff --git a/plugins/ai-slop/CHANGELOG.md b/plugins/ai-slop/CHANGELOG.md index 9f1b78115..e6fc17297 100644 --- a/plugins/ai-slop/CHANGELOG.md +++ b/plugins/ai-slop/CHANGELOG.md @@ -40,9 +40,21 @@ name a committed fixture instead. repo inherits its `disabled_rules`, and the finding a case grades is silently deleted from a scratch repo in `/tmp`. Measured with only `CLAUDE_PROJECT_DIR` changed: case 1 drops 4 findings to 3, case 2 drops 4 to 2, case 4 drops 4 to 3, and cases 5 and 6 drop to **zero**, where the em dash - *is* the whole case and the expectations then pass vacuously. That is the same golden-answer-the- - scenario-cannot-produce failure as the rest of this release, arriving through config rather than - through prose. + *is* the whole case and the expectations then pass vacuously. That is the same failure as the rest + of this release — a golden answer the scenario cannot produce — arriving through config rather + than through prose. +- **The preamble is an explicit shell sequence, not prose**, because three things a prose + instruction leaves to the reader all reintroduce the same failure. It runs in ONE invocation and + says why: the harness re-supplies `CLAUDE_PROJECT_DIR` on every tool call and shell state does not + survive between them, so a pin phrased as a one-time export is gone by the time the detector runs + and the disabled rule silently returns. It issues `git add` before the commit, which the prose + version never did — a literal executor got `nothing to commit` and an untracked target, inverting + exactly the tracked-file branch cases 4 and 5 exist to grade. And it opens with + `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG`, the invariant `scripts/check-fixture-git-isolation.sh` + already enforces for `*.sh`/`*.py` fixtures: without it, an ambient `GIT_DIR` sends `git init` and + the stage commit into the *caller's* repository and leaves the scratch directory with no `.git` at + all. That gate cannot see prose inside `evals.json`, so the eval suite had to honor the rule on its + own. - Staging stops at a commit rather than at `git add`, though `git ls-files` already reports a staged file as tracked and either form would satisfy the persistence gate. These cases model a *consuming repo*, and a real one has history: the audit's repo-wide ordering reads diff --git a/plugins/ai-slop/skills/audit/evals/evals.json b/plugins/ai-slop/skills/audit/evals/evals.json index 5c194d61b..07208f0f4 100644 --- a/plugins/ai-slop/skills/audit/evals/evals.json +++ b/plugins/ai-slop/skills/audit/evals/evals.json @@ -4,7 +4,7 @@ { "id": 1, "name": "bare-invocation-read-only", - "prompt": "/ai-slop:audit docs/report-only.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", + "prompt": "/ai-slop:audit docs/report-only.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/report-only.md\" docs/report-only.md; git add docs/report-only.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", "expected_output": "A findings report for the staged copy `docs/report-only.md` — four findings across four rules (rule-filler-phrases on line 3, rule-significance-inflation on line 6, rule-em-dash and rule-chatbot-artifacts both on line 8), each with its file:line location and fired condition, plus the Summary rows — with no file modified. The fix flow is offered but not run.", "files": ["evals/fixtures/report-only.md"], "expectations": [ @@ -17,7 +17,7 @@ { "id": 2, "name": "fix-runs-semantic-guard", - "prompt": "/ai-slop:audit fix docs/fix-guarded-rewrite.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/fix-guarded-rewrite.md` (path relative to the skill directory) in as `docs/fix-guarded-rewrite.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", + "prompt": "/ai-slop:audit fix docs/fix-guarded-rewrite.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/fix-guarded-rewrite.md\" docs/fix-guarded-rewrite.md; git add docs/fix-guarded-rewrite.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", "expected_output": "The staged copy's four findings — two em dashes (lines 3 and 7) and two filler phrases ('It is worth noting that' on line 3, 'Due to the fact that' on line 4) — are rewritten in one per-file pass, the before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs over the rewritten copy and the findings file is re-emitted (the target is tracked, so persistence is owed) so no stale findings survive.", "files": ["evals/fixtures/fix-guarded-rewrite.md"], "expectations": [ @@ -43,7 +43,7 @@ { "id": 4, "name": "persist-refuses-without-contract", - "prompt": "/ai-slop:audit docs/report-only.md, with the detector-findings contract URL unreachable — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/report-only.md` (path relative to the skill directory) in as `docs/report-only.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit examines a tracked file and a findings file is owed.", + "prompt": "/ai-slop:audit docs/report-only.md, with the detector-findings contract URL unreachable — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/report-only.md\" docs/report-only.md; git add docs/report-only.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the audit examines a tracked file and a findings file is owed.", "expected_output": "The audit report is delivered with the staged copy's four findings intact, but no findings file is written: the target is tracked, so persistence is owed, and the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.", "files": ["evals/fixtures/report-only.md"], "expectations": [ @@ -56,7 +56,7 @@ { "id": 5, "name": "rubric-findings-reach-the-report-not-the-findings-file", - "prompt": "/ai-slop:audit docs/rubric-boundary.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/rubric-boundary.md` (path relative to the skill directory) in as `docs/rubric-boundary.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is tracked and a findings file is owed.", + "prompt": "/ai-slop:audit docs/rubric-boundary.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/rubric-boundary.md\" docs/rubric-boundary.md; git add docs/rubric-boundary.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the audit target is tracked and a findings file is owed.", "expected_output": "Both layers report on the staged copy: the em dash on line 7 appears as its single script finding, and the promotional register on lines 3-5 ('a breathtaking, must-visit surface sitting right in the heart of the platform') as a rubric finding citing its catalog entry. The target is tracked, so a findings file IS written, and it carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.", "files": ["evals/fixtures/rubric-boundary.md"], "expectations": [ @@ -69,7 +69,7 @@ { "id": 6, "name": "fix-never-swaps-one-tell-for-another", - "prompt": "/ai-slop:audit fix docs/em-dash-substitution.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/em-dash-substitution.md` (path relative to the skill directory) in as `docs/em-dash-substitution.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrite lands on a tracked, disposable copy and the committed fixture stays byte-identical.", + "prompt": "/ai-slop:audit fix docs/em-dash-substitution.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/em-dash-substitution.md\" docs/em-dash-substitution.md; git add docs/em-dash-substitution.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the rewrite lands on a tracked, disposable copy and the committed fixture stays byte-identical.", "expected_output": "The staged copy's single finding — the em dash on line 5, in 'The parser accepts empty input — a change from 0.3.' — is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.", "files": ["evals/fixtures/em-dash-substitution.md"], "expectations": [ @@ -82,7 +82,7 @@ { "id": 7, "name": "triads-collapse-toward-the-strongest-item", - "prompt": "/ai-slop:audit fix docs/triads.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/triads.md` (path relative to the skill directory) in as `docs/triads.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", + "prompt": "/ai-slop:audit fix docs/triads.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/triads.md\" docs/triads.md; git add docs/triads.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", "expected_output": "The staged copy trips rule-rule-of-three as one density finding (3 hits in 69 words, 43.5/1000 against a 3.0 threshold), and all three triads reach the fix flow together. The two rhetorical triads — 'fast, simple, and reliable' and 'clean, tight, and portable' — collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration 'user, team, and local' is KEPT intact, with that judgment stated: dropping one names a resolution order that does not exist.", "files": ["evals/fixtures/triads.md"], "expectations": [ @@ -95,7 +95,7 @@ { "id": 8, "name": "known-false-positive-routes-to-config-not-a-rewrite", - "prompt": "/ai-slop:audit docs/knowledge-cutoff-prose.md — run it inside a scratch git repo staged first: `git init` a temp directory, point `CLAUDE_PROJECT_DIR` at that directory and `HOME` at an empty one so the detector's config cascade resolves to shipped defaults instead of the surrounding repo's `.claude/ai-slop.json` (which disables `rule-em-dash` here), copy `evals/fixtures/knowledge-cutoff-prose.md` (path relative to the skill directory) in as `docs/knowledge-cutoff-prose.md`, and commit it with an explicit identity (`git -c user.name=eval -c user.email=eval@local commit -m stage`, since a fresh `HOME` carries no `user.email`), so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", + "prompt": "/ai-slop:audit docs/knowledge-cutoff-prose.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/knowledge-cutoff-prose.md\" docs/knowledge-cutoff-prose.md; git add docs/knowledge-cutoff-prose.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", "expected_output": "The staged copy's three rule-knowledge-cutoff-disclaimer findings (lines 1, 3 and 5) are reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. The fixture is documentation ABOUT knowledge cutoffs, so the rule fires on prose describing the concept rather than disclaiming on the document's own behalf. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.", "files": ["evals/fixtures/knowledge-cutoff-prose.md"], "expectations": [ From d5448e42a84fe74875aeb5d97d5da8ec67f86dcd Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 06:42:16 +0000 Subject: [PATCH 6/8] test(ai-slop): drop the eval staging apparatus for the sibling convention The staging preamble solved a problem this repo does not have. These eval suites are not executable by `claude plugin eval`: the CLI consumes `/**/case.yaml` or `prompt.md` + `graders/*.md`, and this repo has 200 evals.json files, zero of that layout, no manifest declaring experimental.evals, and only lint scripts reading evals.json. Nothing runs a prompt. The prompts are a written specification of expected skill behavior. So every environment property the preamble controlled - config cascade, git identity, `git add`, GIT_DIR isolation, the skill-dir path - was a property of executing a prompt that nothing executes. Six failure modes in five review rounds came from that machinery, in a shape none of the eighteen sibling fixture-backed suites uses. All seven cases now name the fixture through files[] and in prose, matching mcp-tools:audit and docs-hygiene:compress: "evals/fixtures/.md relative to the skill directory". One constraint survives independent of any harness question, and it is the original Codex finding: the fix flow rewrites in place, so cases 2, 6 and 7 say to work on a copy and leave the committed fixture byte-identical. Prose, no mechanism. Cases 4 and 5 state their persistence premise rather than constructing it, and their expectations now grade the skill's decision. expected_output for cases 1, 2, 6, 7 and 8 is restored verbatim to dc5ca9ad - the measured text from before the apparatus. Fixtures are untouched and re-measured under the shipped-defaults condition detect.test.sh itself uses: every rule id, line and density figure still matches. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g --- plugins/ai-slop/CHANGELOG.md | 68 ++++++------------- plugins/ai-slop/skills/audit/evals/evals.json | 53 +++++++-------- 2 files changed, 47 insertions(+), 74 deletions(-) diff --git a/plugins/ai-slop/CHANGELOG.md b/plugins/ai-slop/CHANGELOG.md index e6fc17297..dba43d6b3 100644 --- a/plugins/ai-slop/CHANGELOG.md +++ b/plugins/ai-slop/CHANGELOG.md @@ -14,53 +14,27 @@ name a committed fixture instead. `triads.md` (rule-of-three at 3 hits in 69 words) and `knowledge-cutoff-prose.md` (the recorded false-positive class). Every case's `expected_output` now names the rules, lines and fired thresholds the detector actually emits, measured rather than asserted. -- **Every fixture-backed case stages its target before auditing it**: `git init` a temp directory, - copy the fixture in as `docs/.md`, commit it, and point the invocation at that copy. A case - that named the fixture directly was self-corrupting in two ways. The `fix` cases (2, 6, 7) rewrite - each flagged line in place, so the first run remediated the committed fixture and the second run - graded already-fixed input, where the declared findings no longer fire. And the persistence - expectations (cases 2, 4, 5) grade a branch conditioned on the audit having "examined tracked - files" — an installed plugin's fixture is not tracked in the consuming repo, so the branch was - unreachable and case 4's contract-fetch refusal passed for the wrong reason: nothing was written - because nothing was owed, not because the fetch failed. Staging fixes both at once: the rewrite - lands on a throwaway copy, and the copy is tracked, so persistence is genuinely owed. The staged - copies were re-measured — identical rules, lines and densities at the new path. -- **The staging commit sets its identity inline**, `git -c user.name=eval -c user.email=eval@local - commit -m stage`. `git init` inherits `user.name`/`user.email` from `~/.gitconfig` or the `GIT_*` - environment, and neither is guaranteed: this suite's own unit tests pin `HOME` to an empty tmpdir - for isolation (`detect.test.sh`), and a fresh container has no seeded identity either. Without the - flags the preamble dies at `git commit` with "Please tell me who you are" (exit 128) *before the - audit runs*, so all seven staged cases error out instead of grading — the same "scenario cannot - produce the graded input" failure this release exists to remove. Verified by executing each - preamble as written under an empty `HOME` with `GIT_AUTHOR_*`/`GIT_COMMITTER_*` unset. -- **The preamble pins the config cascade as well as the identity**, pointing `CLAUDE_PROJECT_DIR` at - the scratch directory and `HOME` at an empty one. `detect.sh` resolves `CLAUDE_PROJECT_DIR` ahead - of `git rev-parse --show-toplevel` and then loads that root's `.claude/ai-slop.json`, and rule - disablement is not path-scoped — so a maintainer running these evals from a session rooted at this - repo inherits its `disabled_rules`, and the finding a case grades is silently deleted from a - scratch repo in `/tmp`. Measured with only `CLAUDE_PROJECT_DIR` changed: case 1 drops 4 findings to - 3, case 2 drops 4 to 2, case 4 drops 4 to 3, and cases 5 and 6 drop to **zero**, where the em dash - *is* the whole case and the expectations then pass vacuously. That is the same failure as the rest - of this release — a golden answer the scenario cannot produce — arriving through config rather - than through prose. -- **The preamble is an explicit shell sequence, not prose**, because three things a prose - instruction leaves to the reader all reintroduce the same failure. It runs in ONE invocation and - says why: the harness re-supplies `CLAUDE_PROJECT_DIR` on every tool call and shell state does not - survive between them, so a pin phrased as a one-time export is gone by the time the detector runs - and the disabled rule silently returns. It issues `git add` before the commit, which the prose - version never did — a literal executor got `nothing to commit` and an untracked target, inverting - exactly the tracked-file branch cases 4 and 5 exist to grade. And it opens with - `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG`, the invariant `scripts/check-fixture-git-isolation.sh` - already enforces for `*.sh`/`*.py` fixtures: without it, an ambient `GIT_DIR` sends `git init` and - the stage commit into the *caller's* repository and leaves the scratch directory with no `.git` at - all. That gate cannot see prose inside `evals.json`, so the eval suite had to honor the rule on its - own. -- Staging stops at a commit rather than at `git add`, though `git ls-files` already reports a staged - file as tracked and either form would satisfy the persistence gate. These cases model a *consuming - repo*, and a real one has history: the audit's repo-wide ordering reads - `git log --since=90.days --name-only` for change frequency, which returns nothing in a repo with - no commits. That divergence is harmless for today's single-file cases and invisible in every - current expectation, which is exactly why it is worth not building in. +- **A case names its fixture through `files[]` and in prose, the way every sibling suite does** — + `mcp-tools:audit` and `docs-hygiene:compress` both read "`evals/fixtures/.md` relative to the + skill directory", and none of the eighteen fixture-backed suites here builds a repository to audit + in. These prompts are a *specification* of expected skill behavior, not a script: this repo has 200 + `evals.json` files and zero of the `case.yaml` / `prompt.md` + `graders/` layout `claude plugin + eval` consumes, no manifest declares `experimental.evals`, and the only things that read + `evals.json` are lint scripts. Nothing executes a prompt, so a prompt must be readable by a human + or an agent working by hand, and environment control belongs nowhere in it. If this repo ever + adopts the CLI's format, per-case setup has a first-class home there — a `scaffold_script` run + under `--scaffold`. +- **Cases 2, 6 and 7 tell the reader to work on a copy.** They invoke `fix`, and the fix flow + rewrites each flagged line in place, so running one by hand against the committed fixture + remediates it and dirties the repo — and a later run then grades already-fixed input, where the + declared findings no longer fire. One sentence in the prompt, no mechanism. +- **Cases 4 and 5 state their premise instead of constructing it.** Both grade the persistence step, + which `SKILL.md` gates on the audit having "examined tracked files", so each prompt says the + audited file is tracked in the repo under audit and the expectations grade the skill's *decision*: + that it treats persistence as applicable, fetches the producer contract first, and — case 4 — + refuses to write when that fetch fails, rather than refusing because the target was out of tracked + space. A case cannot verify real repository state, and pretending otherwise is what made case 4 + pass for the wrong reason. - **This reverses 0.1.0's no-fixtures decision, which was recorded in `detect.test.sh`'s header.** That decision holds for the *unit* suite, whose fixtures are still built inline in a tmpdir. It does not survive contact with the eval suite: an eval case is graded against a deterministic diff --git a/plugins/ai-slop/skills/audit/evals/evals.json b/plugins/ai-slop/skills/audit/evals/evals.json index 07208f0f4..c42ada9c9 100644 --- a/plugins/ai-slop/skills/audit/evals/evals.json +++ b/plugins/ai-slop/skills/audit/evals/evals.json @@ -4,27 +4,27 @@ { "id": 1, "name": "bare-invocation-read-only", - "prompt": "/ai-slop:audit docs/report-only.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/report-only.md\" docs/report-only.md; git add docs/report-only.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", - "expected_output": "A findings report for the staged copy `docs/report-only.md` — four findings across four rules (rule-filler-phrases on line 3, rule-significance-inflation on line 6, rule-em-dash and rule-chatbot-artifacts both on line 8), each with its file:line location and fired condition, plus the Summary rows — with no file modified. The fix flow is offered but not run.", + "prompt": "/ai-slop:audit evals/fixtures/report-only.md relative to the skill directory.", + "expected_output": "A findings report for the fixture — four findings across four rules (rule-filler-phrases on line 3, rule-significance-inflation on line 6, rule-em-dash and rule-chatbot-artifacts both on line 8), each with its file:line location and fired condition, plus the Summary rows — with no file modified. The fix flow is offered but not run.", "files": ["evals/fixtures/report-only.md"], "expectations": [ - "Runs detect.sh on the staged copy and reports findings with file:line locations and the fired condition per finding", + "Runs detect.sh on the target and reports findings with file:line locations and the fired condition per finding", "Reports all four rules the fixture trips rather than a subset — filler phrase, significance inflation, em dash, and chat-turn residue", - "Does NOT edit the staged copy, the committed fixture, or any other file on bare invocation", + "Does NOT edit the fixture or any other file on bare invocation", "Mentions the fix action as the explicit follow-up rather than applying rewrites" ] }, { "id": 2, "name": "fix-runs-semantic-guard", - "prompt": "/ai-slop:audit fix docs/fix-guarded-rewrite.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/fix-guarded-rewrite.md\" docs/fix-guarded-rewrite.md; git add docs/fix-guarded-rewrite.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", - "expected_output": "The staged copy's four findings — two em dashes (lines 3 and 7) and two filler phrases ('It is worth noting that' on line 3, 'Due to the fact that' on line 4) — are rewritten in one per-file pass, the before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs over the rewritten copy and the findings file is re-emitted (the target is tracked, so persistence is owed) so no stale findings survive.", + "prompt": "/ai-slop:audit fix evals/fixtures/fix-guarded-rewrite.md relative to the skill directory. Work on a copy so the committed fixture stays byte-identical.", + "expected_output": "The fixture's four findings — two em dashes (lines 3 and 7) and two filler phrases ('It is worth noting that' on line 3, 'Due to the fact that' on line 4) — are rewritten in one per-file pass, the before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs and the findings file is re-emitted so no stale findings survive.", "files": ["evals/fixtures/fix-guarded-rewrite.md"], "expectations": [ - "Applies rewrites only because fix was explicitly requested, and only to the staged copy — `evals/fixtures/fix-guarded-rewrite.md` under the skill directory is byte-identical after the run", + "Applies rewrites only because fix was explicitly requested", "Verifies the rewritten file with a fresh-context semantic-diff subagent and reverts flagged hunks", "Preserves every claim the fixture makes — that the cache invalidates on write, that the reader never sees a stale entry, that writes are rare, and that the policy is least-recently-used", - "Re-runs the detector and re-emits the findings file after the last fixed file, because the audited target is tracked in the scratch repo" + "Re-runs the detector and re-emits the findings file after the last fixed file" ] }, { @@ -43,66 +43,65 @@ { "id": 4, "name": "persist-refuses-without-contract", - "prompt": "/ai-slop:audit docs/report-only.md, with the detector-findings contract URL unreachable — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/report-only.md\" docs/report-only.md; git add docs/report-only.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the audit examines a tracked file and a findings file is owed.", - "expected_output": "The audit report is delivered with the staged copy's four findings intact, but no findings file is written: the target is tracked, so persistence is owed, and the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.", + "prompt": "/ai-slop:audit evals/fixtures/report-only.md relative to the skill directory, with the detector-findings contract URL unreachable. Treat the audited file as tracked in the repo under audit.", + "expected_output": "The audit report is delivered with the fixture's four findings intact, but no findings file is written: the audit examined a tracked file, so persistence applies, and the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.", "files": ["evals/fixtures/report-only.md"], "expectations": [ "Delivers the human-facing findings report regardless", + "Treats persistence as applicable, since the audit examined a tracked file, rather than passing over the step as out of scope", + "Fetches the producer contract before writing anything, per persist-findings.md", "Does NOT write a findings file when the contract fetch fails", - "States that persistence was refused because the contract was unreachable, rather than silently skipping it", - "Attributes the refusal to the failed contract fetch and not to the target sitting outside tracked space — the staged copy is committed, so the persistence branch is genuinely reached" + "States that persistence was refused because the contract was unreachable, rather than silently skipping it, and does not attribute the refusal to the target being out of tracked space" ] }, { "id": 5, "name": "rubric-findings-reach-the-report-not-the-findings-file", - "prompt": "/ai-slop:audit docs/rubric-boundary.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/rubric-boundary.md\" docs/rubric-boundary.md; git add docs/rubric-boundary.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the audit target is tracked and a findings file is owed.", - "expected_output": "Both layers report on the staged copy: the em dash on line 7 appears as its single script finding, and the promotional register on lines 3-5 ('a breathtaking, must-visit surface sitting right in the heart of the platform') as a rubric finding citing its catalog entry. The target is tracked, so a findings file IS written, and it carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.", + "prompt": "/ai-slop:audit evals/fixtures/rubric-boundary.md relative to the skill directory. Treat the audited file as tracked in the repo under audit.", + "expected_output": "Both layers report: the em dash on line 7 appears as the fixture's single script finding, and the promotional register on lines 3-5 ('a breathtaking, must-visit surface sitting right in the heart of the platform') as a rubric finding citing its catalog entry. The audit examined a tracked file, so persistence applies, and the findings file carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.", "files": ["evals/fixtures/rubric-boundary.md"], "expectations": [ "Reports the promotional-register tell as a rubric finding, quoting the offending text and naming the catalog entry it comes from", "Does NOT report the promotional wording as a script finding — the fixture's promotional words sit outside the mechanical vocabulary, so the detector's only finding is the em dash", - "Writes the findings file (the staged copy is tracked) and places NO rubric finding in it — only the em-dash script finding appears there", + "Treats persistence as applicable, since the audit examined a tracked file, rather than passing over the step as out of scope", + "Puts NO rubric finding in the findings file — only the em-dash script finding belongs there", "States the V1 boundary explicitly rather than silently omitting the rubric verdict from the file" ] }, { "id": 6, "name": "fix-never-swaps-one-tell-for-another", - "prompt": "/ai-slop:audit fix docs/em-dash-substitution.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/em-dash-substitution.md\" docs/em-dash-substitution.md; git add docs/em-dash-substitution.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the rewrite lands on a tracked, disposable copy and the committed fixture stays byte-identical.", - "expected_output": "The staged copy's single finding — the em dash on line 5, in 'The parser accepts empty input — a change from 0.3.' — is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.", + "prompt": "/ai-slop:audit fix evals/fixtures/em-dash-substitution.md relative to the skill directory. Work on a copy so the committed fixture stays byte-identical.", + "expected_output": "The fixture's single finding — the em dash on line 5, in 'The parser accepts empty input — a change from 0.3.' — is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.", "files": ["evals/fixtures/em-dash-substitution.md"], "expectations": [ "Reads reference/rewrite-guide.md before applying the rewrite, per the fix flow's first step", "The rewritten line contains no em dash, en dash, parenthesis, or spaced hyphen standing in for the original punctuation", - "Preserves the sentence's claim — the version fact survives the rewrite", - "Rewrites only the staged copy — `evals/fixtures/em-dash-substitution.md` under the skill directory is byte-identical after the run" + "Preserves the sentence's claim — the version fact survives the rewrite" ] }, { "id": 7, "name": "triads-collapse-toward-the-strongest-item", - "prompt": "/ai-slop:audit fix docs/triads.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/triads.md\" docs/triads.md; git add docs/triads.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the rewrites land on a tracked, disposable copy and the committed fixture stays byte-identical.", - "expected_output": "The staged copy trips rule-rule-of-three as one density finding (3 hits in 69 words, 43.5/1000 against a 3.0 threshold), and all three triads reach the fix flow together. The two rhetorical triads — 'fast, simple, and reliable' and 'clean, tight, and portable' — collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration 'user, team, and local' is KEPT intact, with that judgment stated: dropping one names a resolution order that does not exist.", + "prompt": "/ai-slop:audit fix evals/fixtures/triads.md relative to the skill directory. Work on a copy so the committed fixture stays byte-identical.", + "expected_output": "The fixture trips rule-rule-of-three as one density finding (3 hits in 69 words, 43.5/1000 against a 3.0 threshold), and all three triads reach the fix flow together. The two rhetorical triads — 'fast, simple, and reliable' and 'clean, tight, and portable' — collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration 'user, team, and local' is KEPT intact, with that judgment stated: dropping one names a resolution order that does not exist.", "files": ["evals/fixtures/triads.md"], "expectations": [ "Collapses the two rhetorical triads toward one item rather than preserving all three in a reworded form", "Keeps the 'user, team, and local' triad because each element is load-bearing, and says why it was kept", - "Does not treat every three-item enumeration as a tell — enumerating three actual things is not slop", - "Rewrites only the staged copy — `evals/fixtures/triads.md` under the skill directory is byte-identical after the run" + "Does not treat every three-item enumeration as a tell — enumerating three actual things is not slop" ] }, { "id": 8, "name": "known-false-positive-routes-to-config-not-a-rewrite", - "prompt": "/ai-slop:audit docs/knowledge-cutoff-prose.md — stage it first in a scratch git repo, in ONE shell invocation because exports do not survive between tool calls: `unset GIT_DIR GIT_WORK_TREE GIT_CONFIG; d=$(mktemp -d); cd \"$d\"; git init -q; mkdir -p docs home; cp \"$CLAUDE_SKILL_DIR/evals/fixtures/knowledge-cutoff-prose.md\" docs/knowledge-cutoff-prose.md; git add docs/knowledge-cutoff-prose.md; git -c user.name=eval -c user.email=eval@local commit -qm stage; echo \"$d\"` (the fixture path is relative to the skill directory), then run every later command from that directory with `CLAUDE_PROJECT_DIR= HOME=/home` set on the command itself rather than exported once — that keeps the detector's config cascade on shipped defaults instead of the surrounding repo's `.claude/ai-slop.json`, which disables `rule-em-dash` here — so the audit target is a tracked, disposable copy rather than the fixture under the skill directory.", - "expected_output": "The staged copy's three rule-knowledge-cutoff-disclaimer findings (lines 1, 3 and 5) are reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. The fixture is documentation ABOUT knowledge cutoffs, so the rule fires on prose describing the concept rather than disclaiming on the document's own behalf. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.", + "prompt": "/ai-slop:audit evals/fixtures/knowledge-cutoff-prose.md relative to the skill directory.", + "expected_output": "The three rule-knowledge-cutoff-disclaimer findings (lines 1, 3 and 5) are reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. The fixture is documentation ABOUT knowledge cutoffs, so the rule fires on prose describing the concept rather than disclaiming on the document's own behalf. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.", "files": ["evals/fixtures/knowledge-cutoff-prose.md"], "expectations": [ "Recognizes prose ABOUT knowledge cutoffs as the recorded false-positive class rather than deleting the sentences", "Recommends the in-file ignore marker or a config exclusion, citing the calibration record", - "Does NOT propose weakening or disabling the shipped rule to make this corpus pass", - "Recommends the exemption without editing anything — neither the staged copy nor the committed fixture is rewritten on a bare audit" + "Does NOT propose weakening or disabling the shipped rule to make this corpus pass" ] }, { From b263df3f293a520db47692ff8e17080d29ef8909 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 06:46:00 +0000 Subject: [PATCH 7/8] test(ai-slop): grade the work-on-a-copy instruction in the fix cases Cases 2, 6 and 7 told the reader to work on a copy and then checked nothing. An instruction a case states but never checks is the same defect this release removes, in miniature. Each now carries one expectation asserting the outcome: the committed fixture under evals/fixtures/ is byte-identical after the run, and the rewrites land on a copy. How the copy gets made is the reader's business - pinning the mechanism is what the earlier staging preamble did wrong. Expectations only. No prompt, expected_output, files, narration, name or fixture byte changes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g --- plugins/ai-slop/CHANGELOG.md | 5 ++++- plugins/ai-slop/skills/audit/evals/evals.json | 9 ++++++--- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/plugins/ai-slop/CHANGELOG.md b/plugins/ai-slop/CHANGELOG.md index dba43d6b3..d2ab396f4 100644 --- a/plugins/ai-slop/CHANGELOG.md +++ b/plugins/ai-slop/CHANGELOG.md @@ -27,7 +27,10 @@ name a committed fixture instead. - **Cases 2, 6 and 7 tell the reader to work on a copy.** They invoke `fix`, and the fix flow rewrites each flagged line in place, so running one by hand against the committed fixture remediates it and dirties the repo — and a later run then grades already-fixed input, where the - declared findings no longer fire. One sentence in the prompt, no mechanism. + declared findings no longer fire. One sentence in the prompt and one expectation per case, both + about the outcome rather than the mechanism: the committed fixture is byte-identical after the run, + and how the copy gets made is the reader's business. An instruction a case states but never checks + is the same defect in miniature as the one this release removes. - **Cases 4 and 5 state their premise instead of constructing it.** Both grade the persistence step, which `SKILL.md` gates on the audit having "examined tracked files", so each prompt says the audited file is tracked in the repo under audit and the expectations grade the skill's *decision*: diff --git a/plugins/ai-slop/skills/audit/evals/evals.json b/plugins/ai-slop/skills/audit/evals/evals.json index c42ada9c9..51e4df73e 100644 --- a/plugins/ai-slop/skills/audit/evals/evals.json +++ b/plugins/ai-slop/skills/audit/evals/evals.json @@ -24,7 +24,8 @@ "Applies rewrites only because fix was explicitly requested", "Verifies the rewritten file with a fresh-context semantic-diff subagent and reverts flagged hunks", "Preserves every claim the fixture makes — that the cache invalidates on write, that the reader never sees a stale entry, that writes are rare, and that the policy is least-recently-used", - "Re-runs the detector and re-emits the findings file after the last fixed file" + "Re-runs the detector and re-emits the findings file after the last fixed file", + "Leaves `evals/fixtures/fix-guarded-rewrite.md` byte-identical after the run — the rewrites land on a copy" ] }, { @@ -77,7 +78,8 @@ "expectations": [ "Reads reference/rewrite-guide.md before applying the rewrite, per the fix flow's first step", "The rewritten line contains no em dash, en dash, parenthesis, or spaced hyphen standing in for the original punctuation", - "Preserves the sentence's claim — the version fact survives the rewrite" + "Preserves the sentence's claim — the version fact survives the rewrite", + "Leaves `evals/fixtures/em-dash-substitution.md` byte-identical after the run — the rewrite lands on a copy" ] }, { @@ -89,7 +91,8 @@ "expectations": [ "Collapses the two rhetorical triads toward one item rather than preserving all three in a reworded form", "Keeps the 'user, team, and local' triad because each element is load-bearing, and says why it was kept", - "Does not treat every three-item enumeration as a tell — enumerating three actual things is not slop" + "Does not treat every three-item enumeration as a tell — enumerating three actual things is not slop", + "Leaves `evals/fixtures/triads.md` byte-identical after the run — the rewrites land on a copy" ] }, { From 079368938b06338748d922fd6507ccfea1b925d8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 06:59:54 +0000 Subject: [PATCH 8/8] test(ai-slop): assert case 5 actually writes the findings file Case 5's remaining persistence expectations were negative or procedural: no rubric finding in the file, and persistence treated as applicable. Neither requires a file to exist. context/persist-findings.md permits reporting without writing when the destination cannot be proven outside tracked space, so a run could call persistence applicable, write nothing, and satisfy both - green while asserting nothing, which is the shape this release removes. Adds the positive half: the findings file is written, and a run that reports the findings and persists nothing does not satisfy the case. The rubric boundary now constrains a file that must exist. Not a regression from dc5ca9ad, which was also negative-only, but the assertion is gradeable under the specification design and so was worth having. Case 5 expectations only. Every prompt and expected_output in the file is byte-identical to b263df3f, and cases 1-4 and 6-9 are untouched. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g --- plugins/ai-slop/CHANGELOG.md | 5 ++++- plugins/ai-slop/skills/audit/evals/evals.json | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/plugins/ai-slop/CHANGELOG.md b/plugins/ai-slop/CHANGELOG.md index d2ab396f4..65515e0ba 100644 --- a/plugins/ai-slop/CHANGELOG.md +++ b/plugins/ai-slop/CHANGELOG.md @@ -37,7 +37,10 @@ name a committed fixture instead. that it treats persistence as applicable, fetches the producer contract first, and — case 4 — refuses to write when that fetch fails, rather than refusing because the target was out of tracked space. A case cannot verify real repository state, and pretending otherwise is what made case 4 - pass for the wrong reason. + pass for the wrong reason. Case 5 also asserts the positive half — that the findings file is + actually written — because `context/persist-findings.md` permits reporting without writing when the + destination cannot be proven outside tracked space, so a negative-only case would be satisfied by a + run that persists nothing at all. - **This reverses 0.1.0's no-fixtures decision, which was recorded in `detect.test.sh`'s header.** That decision holds for the *unit* suite, whose fixtures are still built inline in a tmpdir. It does not survive contact with the eval suite: an eval case is graded against a deterministic diff --git a/plugins/ai-slop/skills/audit/evals/evals.json b/plugins/ai-slop/skills/audit/evals/evals.json index 51e4df73e..af2437b71 100644 --- a/plugins/ai-slop/skills/audit/evals/evals.json +++ b/plugins/ai-slop/skills/audit/evals/evals.json @@ -65,6 +65,7 @@ "Reports the promotional-register tell as a rubric finding, quoting the offending text and naming the catalog entry it comes from", "Does NOT report the promotional wording as a script finding — the fixture's promotional words sit outside the mechanical vocabulary, so the detector's only finding is the em dash", "Treats persistence as applicable, since the audit examined a tracked file, rather than passing over the step as out of scope", + "Writes the findings file, on the stated premise that the audited file is tracked — a run that reports the findings and persists nothing does not satisfy this case", "Puts NO rubric finding in the findings file — only the em-dash script finding belongs there", "States the V1 boundary explicitly rather than silently omitting the rubric verdict from the file" ]