ai-slop: back the audit evals with real fixture files instead of prose scenarios

[kyle-sexton]

Surfaced by #3039, where the same defect class occurred three times in one PR and was caught by review bots rather than by any gate.

The problem

ai-slop's eval cases describe their input in prose inside the prompt string:

/ai-slop:audit fix docs/example.md — the file trips rule-of-three on "the tool is fast, simple, and reliable"…

Nothing verifies that the described input actually produces the finding the expected_output grades. It drifted three times:

1. The rubric-boundary case (id 5) claimed the detector matched "nothing else", but its promotional wording used vibrant/groundbreaking/nestled — all in DEFAULT_VOCAB, and exactly the density rule's 3-hit floor. Measured 142.9/1000 words, so a second script finding fired and contradicted the case's own answer.
2. The triad case (id 7) demanded a load-bearing triad be kept while supplying only rhetorical ones, leaving that criterion untestable.
3. The fix for (2) used multi-word items ("project settings"), but rule-rule-of-three's ERE is [A-Za-z]+, [A-Za-z]+, and [A-Za-z]+ — three single tokens — so the detector never surfaced the load-bearing triad and the fix flow had nothing to judge.

All three share one shape: a golden answer the scenario cannot produce. Prose describing a fixture can disagree with the detector; a committed fixture cannot.

The fix

The eval schema already supports it. plugins/skill-quality/reference/evals.schema.json defines files as "list of fixture file paths required by this case", and CI already grades fixture consumption:

Every file under a skill's evals/fixtures/ must be consumed by a grader (.github/workflows/ci.yml:664)

So the change is to ship plugins/ai-slop/skills/audit/evals/fixtures/*.md and reference them from files, rather than narrating their contents. The four original cases plus the five added in #3039 all currently use files: [] with narration: true.

Why this plugin specifically

ai-slop's evals are unusual: the thing the model is graded on is downstream of a deterministic detector run. For most skills a prose scenario is fine because the skill's behavior is the only variable. Here the scenario has to satisfy a regex the eval author does not have in front of them, which is exactly the coupling that broke three times.

A fixture also makes the eval self-verifying: detect.sh <fixture> shows what the case actually produces, which is how each scenario was eventually validated by hand in #3039.

The caveat that has to be argued, not assumed

0.1.0 deliberately shipped no fixture files, and said why in detect.test.sh:2-4:

fixtures are built inline in a tmpdir, so the plugin ships no slop samples for the audit to trip over

Fixtures deliberately containing slop would need an exclusion — this plugin's own catalog.md already sits in excluded_paths for exactly that reason (it quotes the tells it detects). So this issue proposes reversing a recorded decision, and the reversal needs an argument: either the fixtures live under a path the repo-wide audit declines, or .claude/ai-slop.json gains an entry for them.

Note the consuming config is currently clean on this point — the dead scripts/fixtures/** glob that used to name a never-created directory was removed in 0.2.0. Any exclusion added here should name the real fixture path rather than reinstating a speculative one.

Related

• test(ai-slop): eval the judgment layer, the half a shell test cannot reach (0.2.1) #3039 — the three defects and their fixes.
• feat(ai-slop): unslop-inspired catalog additions, plus the corrections and dogfood config that make them true (0.2.0) #3031 — the 0.2.0 integration.