test(ai-slop): eval the judgment layer, the half a shell test cannot reach (0.2.1)

plugins/ai-slop/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "ai-slop",
-  "version": "0.2.0",
+  "version": "0.2.1",
   "description": "Detects and removes AI-writing tells (slop) in checked-in markdown prose: em dashes, emoji formatting, AI vocabulary, negative parallelisms, chatbot phrases, filler, stacked hedging, citation artifacts, and the rest of a catalog distilled from Wikipedia's Signs of AI writing. Read-only audit by default with a deterministic detector plus a judgment rubric; an explicit fix action rewrites findings behind a semantic-diff guard. Findings conform to the detector-findings convention so the review fanout fix relay can consume them.",
   "author": {
     "name": "Melodic Software",

plugins/ai-slop/CHANGELOG.md

@@ -1,5 +1,34 @@
 # Changelog
 
+## [0.2.1]
+
+Eval coverage for the layer a shell test cannot reach. The deterministic side already had 86 cases
+over all 15 script rules; the judgment side had none, which is the half that only a model performs.
+
+- **Five new `audit` evals** (4 → 9, in line with sibling audit skills): the rubric layer reports
+  but never enters the findings file; a `fix` never swaps an em dash for a parenthesis or en dash
+  (the rewrite guide's substitution guardrail, and the tell most likely to be violated silently
+  because the swap looks like a fix); triads collapse toward the strongest item rather than being
+  repunctuated; the recorded knowledge-cutoff false-positive class routes to a marker or config
+  rather than a rewrite, and never to weakening the rule; exemptions are named with their cause.
+- **Two new `setup` evals** (3 → 5) covering the `_comment` rationale key added in 0.2.0: it is a
+  documented annotation rather than unknown-key drift, and disabling a rule records why alongside
+  the trade-off against `em_dash_allowed_paths`.
+- Every eval case carries `narration: true`. The four original cases named a prose path that
+  resolves nowhere, which the skill-quality Q4 check warns on unless the case declares itself
+  narrative.
+- Two eval scenarios corrected in review, both cases of a golden answer the scenario could not
+  produce: the rubric-boundary case used promotional words that are themselves in the mechanical
+  vocabulary list, so a second script finding fired and contradicted its own "one script finding"
+  answer (measured: 3 hits, density 142.9/1000); and the triad case demanded a load-bearing triad
+  be kept while supplying only rhetorical ones.
+- The triad case needed a second correction, caught in review after the first: its load-bearing
+  example used multi-word items ("project settings"), which `rule-rule-of-three`'s ERE
+  (`[A-Za-z]+, [A-Za-z]+, and [A-Za-z]+`) requires to be single tokens, so the detector never
+  surfaced it and the fix flow had nothing to judge. Each scenario is now verified by running the
+  detector over it — the triad case measures 3 hits at 60.0/1000 words, with all three triads
+  reaching the finding.
+
 ## [0.2.0]
 
 - Added a set of catalog entries inspired by

plugins/ai-slop/skills/audit/evals/evals.json

@@ -7,6 +7,7 @@
       "prompt": "/ai-slop:audit docs/example.md",
       "expected_output": "A findings report for the target file — per-rule findings with file:line locations, fired thresholds, and a summary — with no file modified. The fix flow is offered but not run.",
       "files": [],
+      "narration": true,
       "expectations": [
         "Runs detect.sh on the target and reports findings with file:line locations and the fired condition per finding",
         "Does NOT edit the target file or any other file on bare invocation",
@@ -19,6 +20,7 @@
       "prompt": "/ai-slop:audit fix docs/example.md",
       "expected_output": "The file's findings are rewritten per-file (em dashes reworded, stock phrases deflated), each file's before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs and the findings file is re-emitted so no stale findings survive.",
       "files": [],
+      "narration": true,
       "expectations": [
         "Applies rewrites only because fix was explicitly requested",
         "Verifies every rewritten file with a fresh-context semantic-diff subagent and reverts flagged hunks",
@@ -31,6 +33,7 @@
       "prompt": "/ai-slop:audit",
       "expected_output": "A repo-wide audit over tracked markdown, chunked via --paths-file/--offset/--limit, with instruction surfaces (CLAUDE.md, AGENTS.md, rules, SKILL.md files, READMEs) reported first, then high-change-frequency files; excluded paths from config are named as declined, never silently skipped.",
       "files": [],
+      "narration": true,
       "expectations": [
         "Audits the repo's tracked markdown when no target is given, using the chunk affordances rather than a per-file loop",
         "Orders the report by instruction-surface impact first, then change frequency",
@@ -43,11 +46,77 @@
       "prompt": "/ai-slop:audit docs/example.md (with the detector-findings contract URL unreachable)",
       "expected_output": "The audit report is delivered, but no findings file is written: the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.",
       "files": [],
+      "narration": true,
       "expectations": [
         "Delivers the human-facing findings report regardless",
         "Does NOT write a findings file when the contract fetch fails",
         "States that persistence was refused because the contract was unreachable, rather than silently skipping it"
       ]
+    },
+    {
+      "id": 5,
+      "name": "rubric-findings-reach-the-report-not-the-findings-file",
+      "prompt": "/ai-slop:audit docs/example.md — the file contains promotional register (\"a breathtaking, must-visit dashboard sitting in the heart of the stack\") and one em dash. The promotional words are deliberately outside the mechanical vocabulary list, so the detector matches the em dash and nothing else.",
+      "expected_output": "Both layers report: the em dash appears as the single script finding, and the promotional tone as a rubric finding citing its catalog entry. The persisted findings file carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.",
+      "files": [],
+      "narration": true,
+      "expectations": [
+        "Reports the promotional-register tell as a rubric finding, quoting the offending text and naming the catalog entry it comes from",
+        "Does NOT place any rubric finding in the persisted findings file — only the script finding appears there",
+        "States the V1 boundary explicitly rather than silently omitting the rubric verdict from the file"
+      ]
+    },
+    {
+      "id": 6,
+      "name": "fix-never-swaps-one-tell-for-another",
+      "prompt": "/ai-slop:audit fix docs/example.md — the file's only finding is a single em dash in \"the parser accepts empty input — a change from 0.3.\"",
+      "expected_output": "The em dash is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.",
+      "files": [],
+      "narration": true,
+      "expectations": [
+        "Reads reference/rewrite-guide.md before applying the rewrite, per the fix flow's first step",
+        "The rewritten line contains no em dash, en dash, parenthesis, or spaced hyphen standing in for the original punctuation",
+        "Preserves the sentence's claim — the version fact survives the rewrite"
+      ]
+    },
+    {
+      "id": 7,
+      "name": "triads-collapse-toward-the-strongest-item",
+      "prompt": "/ai-slop:audit fix docs/example.md — the file trips rule-of-three at 3 hits on two kinds of triad. Rhetorical: \"the tool is fast, simple, and reliable\" and \"it reads clean, tight, and portable\". Load-bearing: \"the cascade layers are user, team, and local\" — a complete set the reader needs, where dropping one names a resolution order that does not exist. All three match the density rule's ERE, so all three reach the fix flow as one finding.",
+      "expected_output": "The two rhetorical triads collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration is KEPT intact, with that judgment stated — enumerating three actual things is not a tell.",
+      "files": [],
+      "narration": true,
+      "expectations": [
+        "Collapses rhetorical triads toward one item rather than preserving all three in a reworded form",
+        "Keeps a triad when each element is load-bearing, and says why it was kept",
+        "Does not treat every three-item enumeration as a tell — enumerating three actual things is not slop"
+      ]
+    },
+    {
+      "id": 8,
+      "name": "known-false-positive-routes-to-config-not-a-rewrite",
+      "prompt": "/ai-slop:audit docs/example.md — the file is documentation ABOUT model knowledge cutoffs, so rule-knowledge-cutoff-disclaimer fires on prose that is describing the concept rather than disclaiming on the document's own behalf.",
+      "expected_output": "The finding is reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.",
+      "files": [],
+      "narration": true,
+      "expectations": [
+        "Recognizes prose ABOUT knowledge cutoffs as the recorded false-positive class rather than deleting the sentence",
+        "Recommends the in-file ignore marker or a config exclusion, citing the calibration record",
+        "Does NOT propose weakening or disabling the shipped rule to make this corpus pass"
+      ]
+    },
+    {
+      "id": 9,
+      "name": "declined-exemptions-are-named-with-their-cause",
+      "prompt": "/ai-slop:audit — the consuming repo's .claude/ai-slop.json disables rule-em-dash and excludes two paths, and three more files carry in-file ignore markers.",
+      "expected_output": "The report states which rules were disabled, names each declined file with its cause (excluded glob vs file marker), and gives the per-rule declined counts from the detector's Summary rows — so an exemption is always visible as a decision, never as an absence of findings.",
+      "files": [],
+      "narration": true,
+      "expectations": [
+        "Names the disabled rule(s) explicitly rather than reporting zero findings for them without comment",
+        "Reports each declined file alongside its cause, distinguishing a config exclusion from an in-file marker",
+        "Carries the per-rule declined counts into the report instead of dropping them"
+      ]
     }
   ]
 }

plugins/ai-slop/skills/setup/evals/evals.json

@@ -7,6 +7,7 @@
       "prompt": "/ai-slop:setup check",
       "expected_output": "The current effective configuration is reported — each key with the layer supplying it (user-global, team, local overlay), absent layers named, drift flagged (unknown keys, dead globs, unsupported em-dash threshold) — and no file is written.",
       "files": [],
+      "narration": true,
       "expectations": [
         "Reports the effective config per key with the winning layer named",
         "Flags drift such as unknown keys or an unsupported em-dash threshold key",
@@ -19,6 +20,7 @@
       "prompt": "/ai-slop:setup apply — user asks to exempt docs/style-guide.md from the em-dash rule",
       "expected_output": "A diff of .claude/ai-slop.json adding docs/style-guide.md to em_dash_allowed_paths is shown and explicitly confirmed before writing. The user-global and .local.json layers are named as self-service options, never written by the skill.",
       "files": [],
+      "narration": true,
       "expectations": [
         "Shows the proposed team-layer diff and waits for explicit confirmation before writing",
         "Writes ONLY .claude/ai-slop.json, never the user-global or local overlay layers",
@@ -31,11 +33,38 @@
       "prompt": "/ai-slop:setup apply — user asks to raise the em-dash threshold repo-wide",
       "expected_output": "The skill explains the em-dash rule is zero-tolerance by design with per-document exemption as the supported mechanism, offers em_dash_allowed_paths for the documents that genuinely need em dashes, and does not invent a threshold key for the rule.",
       "files": [],
+      "narration": true,
       "expectations": [
         "Explains the zero-tolerance default and that exemption is per-document, not per-threshold",
         "Offers em_dash_allowed_paths (or disabled_rules with its trade-off named) as the supported paths",
         "Does not write a nonexistent em-dash threshold key"
       ]
+    },
+    {
+      "id": 4,
+      "name": "comment-key-is-not-drift",
+      "prompt": "/ai-slop:setup check — the repo's .claude/ai-slop.json carries a _comment key recording why rule-em-dash is disabled.",
+      "expected_output": "The rationale key is reported as a documented annotation, not flagged as an unknown key. JSON has no comment syntax and the Keys table lists _comment for exactly this purpose, so a config that records its reasoning is conforming.",
+      "files": [],
+      "narration": true,
+      "expectations": [
+        "Does NOT flag _comment as unknown-key drift",
+        "Still flags genuinely unknown keys in the same file, so the allowance is scoped to _comment rather than disabling the drift check",
+        "Surfaces the recorded rationale when reporting the disabled rule, so a reader sees why the exemption exists"
+      ]
+    },
+    {
+      "id": 5,
+      "name": "disabling-a-rule-records-its-reason",
+      "prompt": "/ai-slop:setup apply — user asks to disable rule-em-dash repo-wide because the house style uses em dashes deliberately.",
+      "expected_output": "The team-layer diff adds rule-em-dash to disabled_rules AND records the reason in _comment, with the measured scale of the exemption named. The trade-off is stated plainly: disabling turns the rule off for new files too, where em_dash_allowed_paths would not.",
+      "files": [],
+      "narration": true,
+      "expectations": [
+        "Writes the reason alongside the exemption rather than the exemption alone",
+        "Names the trade-off between disabled_rules and em_dash_allowed_paths so the choice is informed",
+        "Does not change the plugin's shipped default — the exemption lives in the consuming repo's config only"
+      ]
     }
   ]
 }