test(ai-slop): eval the judgment layer, the half a shell test cannot reach (0.2.1)

[kyle-sexton]

No linked issue

Summary

ai-slop ships two detection layers. The deterministic one has 86 shell cases covering all 15 script rules, tiers, roster agreement, exemptions, and portability. The judgment one — 11 rubric tells, plus the rewrite-guide.md the fix flow reads at step 1 — had zero coverage. That is the half only a model performs, so it is the half evals exist for.

This adds 7 eval cases (audit 4 → 9, setup 3 → 5), bringing ai-slop in line with sibling audit skills (7, 9, and 14).

Fix

audit, five new cases, each tied to a documented behavior that no shell test can assert:

• rubric-findings-reach-the-report-not-the-findings-file — the V1 relay boundary. A rubric verdict has no crosswalk row to look a tier up from, so it reaches the human report only. Silently dropping it from the file and silently including it are both failures, and only a model can tell them apart.
• fix-never-swaps-one-tell-for-another — the substitution guardrail. An em dash must resolve to a comma, a period, or a restructured sentence, never a parenthesis or en dash. This is the guardrail most likely to be violated silently, because the swap looks like a fix.
• triads-collapse-toward-the-strongest-item — a rule-of-three fix collapses toward one item rather than being repunctuated into a three-item list, and keeps a triad whose elements are each load-bearing, saying why.
• known-false-positive-routes-to-config-not-a-rewrite — prose about model knowledge cutoffs trips rule-knowledge-cutoff-disclaimer. The catalog's calibration record already names this class; the recorded answer is a marker or config exclusion with a stated reason, never weakening the shipped rule to make one corpus pass.
• declined-exemptions-are-named-with-their-cause — a disabled rule and an excluded path must read as decisions, not as an absence of findings.

setup, two new cases covering the _comment rationale key introduced in 0.2.0: it is a documented annotation rather than unknown-key drift (while genuinely unknown keys still flag), and disabling a rule records why alongside the trade-off against em_dash_allowed_paths.

Every case now carries narration: true. The four original cases named docs/example.md, which resolves nowhere — a live skill-quality Q4 warning that the flag exists to answer.

Verification

• check-jsonschema against plugins/skill-quality/reference/evals.schema.json — both files ok.
• scripts/check-changelog-parity.sh --check and --check-bump origin/main — both pass at 0.2.1.
• plugins/ai-slop/skills/audit/scripts/detect.test.sh — 86 cases pass, unchanged (this PR adds no rules).
• node scripts/generate-catalog.mjs — catalog already in sync.
• markdownlint-cli2 and typos clean on the touched files.

Dogfood context that motivated the selection. A repo-wide run on main now reports 174 findings across 1214 files, and sampling them shows the plugin predicting its own false-positive profile: 16 of 21 rule-emoji-formatting findings are ❌/✅ semantic wrong-right markers (a documentation convention here, not decoration), 8 rule-knowledge-cutoff-disclaimer findings are the class the calibration record already names, and 72 rule-curly-artifacts findings concentrate in just 7 files as genuine paste residue. Two of the new evals encode exactly the judgment those findings require.

Related

• Refs feat(ai-slop): unslop-inspired catalog additions, plus the corrections and dogfood config that make them true (0.2.0) #3031 — the 0.2.0 integration this covers the judgment half of.
• The de-slop pass over those 174 findings is deliberately NOT in this PR: a large share are the repo's own conventions, so the next step there is config tuning rather than editing prose, and it touches ~100 unrelated documents.

---

Generated by Claude Code

[claude]

Claude finished @kyle-sexton's task in 44s —— View job

---

Security review: no security-relevant surface

• Gather context (PR diff, changed files)
• Apply skip gate check
• Review changed files for security-relevant surface
• Post final findings

Skip gate triggered (condition 2): this change has no security-relevant surface.

All four changed files are non-executable data/docs:

• plugins/ai-slop/.claude-plugin/plugin.json — version bump 0.2.0 → 0.2.1 only.
• plugins/ai-slop/CHANGELOG.md — prose changelog entry for the bump.
• plugins/ai-slop/skills/audit/evals/evals.json — 5 new eval fixtures (prompt/expected_output/expectations strings) plus narration: true flags on existing cases.
• plugins/ai-slop/skills/setup/evals/evals.json — 2 new eval fixtures, same shape.

No scripts, no GitHub Actions workflow files, no config-parsing code, no dependency pins, and no runtime logic are touched — this PR only adds declarative eval test cases (consumed by a model-driven eval harness, not executed as code) and bumps documentation/version metadata. There is no injection surface, no credential handling, no authorization logic, and no supply-chain change here for this review lane to assess.

No security findings to report.

[claude]

Claude finished @kyle-sexton's task in 2m 27s —— View job

---

Code review complete

• Gather context (PR diff, CLAUDE.md, REVIEW.md, changed files)
• Review plugins/ai-slop/skills/audit/evals/evals.json (5 new cases)
• Review plugins/ai-slop/skills/setup/evals/evals.json (2 new cases)
• Review plugins/ai-slop/CHANGELOG.md and plugin.json version bump
• Cross-check against evals.schema.json and sibling skill conventions
• Post findings

Scope: This is the CI code-review lane (/review:code-review); no security-lane workflow found in this repo's .github/workflows, but the diff is eval-fixture/JSON/markdown only with no security surface either way.

No blocking or important findings. This is a tightly-scoped, well-verified eval-coverage PR and every claim it makes checks out against the source it cites:

• Case rubric-findings-reach-the-report-not-the-findings-file (audit id 5) matches the V1 relay boundary stated verbatim in persist-findings.md lines 57–58, and its promotional-language example text ("vibrant", "groundbreaking", "nestled", "in the heart of") is drawn directly from the catalog's own examples for rule-promotional-language (v1: rubric) at catalog.md lines 116–121.
• Case fix-never-swaps-one-tell-for-another (audit id 6) matches the substitution guardrail in rewrite-guide.md lines 17–19.
• Case triads-collapse-toward-the-strongest-item (audit id 7) matches the same file's triad rule at line 24–25. It refers to the rule informally as rule-of-three; the shipped id is rule-rule-of-three, but that shorthand is the existing convention elsewhere in this repo (SKILL.md:80, CHANGELOG.md:81), so this is consistent rather than a new inconsistency.
• Case known-false-positive-routes-to-config-not-a-rewrite (audit id 8) matches the calibration record for rule-knowledge-cutoff-disclaimer at catalog.md lines 59–61.
• Setup cases comment-key-is-not-drift / disabling-a-rule-records-its-reason (ids 4–5) match the _comment key's documented behavior in setup/SKILL.md (Keys table + guidance).
• narration: true addition: confirmed docs/example.md does not resolve anywhere in the repo, so the flag is correctly applied and matches the schema's Q4 opt-out semantics (evals.schema.json).
• Eval id sequences are contiguous and non-duplicated in both files (audit 1–9, setup 1–5); JSON structure in both files is well-formed.
• plugin.json version bump (0.2.0 → 0.2.1) has a matching ## [0.2.1] CHANGELOG.md entry, consistent with the existing heading convention.
• Resulting case counts (audit 9, setup 5) are in line with sibling skills of similar scope in this repo.

No inline comments were needed since no findings met the high-signal bar (this lane excludes style nits, pre-existing issues, and anything a linter/schema check already covers — all of which the author's stated verification already ran).

[github-actions]

Last security-reviewed head: 79bbeca56adeca374a78a092904c178d2a51a3ff. On the next push, the relevance gate compares only the commits since this SHA; delete this comment to force a full re-review.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4cd44383d0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

Claude has reviewed this PR 3 times. The lane skips further automatic reviews after 5; deleting this comment resets the count.

[claude]

Claude finished @kyle-sexton's task in 3m 54s —— View job

---

Code review complete

• Gather context (PR diff, CLAUDE.md, changed files)
• Review plugins/ai-slop/skills/audit/evals/evals.json (5 new cases)
• Review plugins/ai-slop/skills/setup/evals/evals.json (2 new cases)
• Independently reproduce Codex's two flagged findings against detect.sh
• Post findings

Scope: This is the CI code-review lane (/review:code-review); the diff is eval-fixture/JSON/markdown only (no security surface).

Context: the latest commit (4c6a474) already fixed the two scenarios Codex flagged in its earlier pass (the vocabulary collision in eval id 5, and the untestable load-bearing triad in eval id 7). I reproduced both original issues and confirmed they're resolved as described.

One finding — the id 7 fix is incomplete (inline comment posted on evals/evals.json:85):

The load-bearing triad added in 4c6a474 to fix eval triads-collapse-toward-the-strongest-item — "the layers are managed policy, project settings, and the local overlay" — still does not trip rule-rule-of-three. That rule is v1: script-only (catalog.md:188-194), detected purely by the density ERE at detect.sh:79: [A-Za-z]+, [A-Za-z]+, and [A-Za-z]+, which requires three single-token items. "project settings" and "the local overlay" are two/three-word phrases, so the pattern never matches:

$ printf 'The layers are managed policy, project settings, and the local overlay.\n' | grep -oE '[A-Za-z]+, [A-Za-z]+, and [A-Za-z]+'
(no output, exit 1)

(The two rhetorical triads in the same eval do match correctly.) This means the detector still never surfaces the load-bearing triad as a finding for the fix flow to judge on — the same "golden answer the scenario cannot produce" class of bug 4c6a474 set out to fix, just not fully closed for this eval. It also means the CHANGELOG's claim that "Both scenarios now contain exactly what their expectations grade" isn't quite true yet. Suggested fix: reword the load-bearing triad to use single-word items (verified against the same ERE) while it still reads as a genuine complete/load-bearing set.

No other findings — the rest of the new audit and setup eval cases check out against their cited sources (rewrite-guide.md's substitution/triad guardrails, the knowledge-cutoff calibration record, _comment's documented behavior in setup/SKILL.md), and the narration: true additions correctly address the docs/example.md unresolved-path warning.

[claude]

Claude finished @kyle-sexton's task in 49s —— View job

---

Security review: no security-relevant surface

• Gather context (PR diff, changed files)
• Apply skip gate check
• Review changed files for security-relevant surface
• Post final findings

Skip gate triggered (condition 2): this change has no security-relevant surface.

Re-checked against the current head (4c6a4742a35ea8a4e9cee1f2c2a85fc60d03ba1b), which adds one commit since the prior review (4cd4438): 4c6a474 tweaks two eval fixtures in plugins/ai-slop/skills/audit/evals/evals.json (swapping vocabulary-list words in the rubric-boundary case, adding a load-bearing triad to the triad case) and appends a matching entry to CHANGELOG.md — addressing the two P2 review comments from Codex. Full PR diff against origin/main remains the same four files as previously reviewed:

• plugins/ai-slop/.claude-plugin/plugin.json — version bump only.
• plugins/ai-slop/CHANGELOG.md — prose changelog entries.
• plugins/ai-slop/skills/audit/evals/evals.json — declarative eval fixtures (prompt/expected_output/expectations strings).
• plugins/ai-slop/skills/setup/evals/evals.json — declarative eval fixtures, same shape.

No scripts, no GitHub Actions workflow files, no config-parsing code, no dependency pins, and no runtime logic are touched — still only declarative eval test cases (consumed by a model-driven eval harness, not executed as code) plus documentation/version metadata. There is no injection surface, no credential handling, no authorization logic, and no supply-chain change here for this review lane to assess.

No security findings to report.

[claude]

Claude finished @kyle-sexton's task in 2m 17s —— View job

---

Code review complete

• Gather context (PR diff since last-reviewed head 4c6a4742a35ea8a4e9cee1f2c2a85fc60d03ba1b)
• Apply skip gate check
• Review the new commit against /review:code-review criteria
• Post findings

Scope: CI code-review lane. Diff since last review is one commit, 79bbeca, touching only plugins/ai-slop/skills/audit/evals/evals.json (eval id 7) and CHANGELOG.md — the change that addresses the P2 finding both Codex and I raised in the prior pass (the load-bearing triad's multi-word items never tripped rule-rule-of-three).

Verified independently (not just restating the author's claim): re-ran the actual detection regex from detect.sh:79 — [A-Za-z]+, [A-Za-z]+, and [A-Za-z]+ — against the three triads now in the prompt:

fast, simple, and reliable
clean, tight, and portable
user, team, and local

All three match (single-word items throughout), giving 3 hits — which also clears DENSITY_MIN_HITS=3 (detect.sh:75). The eval now mechanically reaches the finding it grades on, closing the gap flagged earlier.

No blocking or important findings. This is a narrowly-scoped, correctly-verified follow-up fix:

• The reworded load-bearing triad ("the cascade layers are user, team, and local") preserves the semantic point (a complete set where dropping one element implies a nonexistent resolution order) while satisfying the mechanical ERE — the fix doesn't just chase the regex at the expense of the scenario's intent.
• expected_output and the expectations array were updated consistently with the reworded prompt (no stale reference to "layers" language left behind).
• CHANGELOG's new entry accurately describes the second correction and cites the same ERE and threshold this review independently confirmed.
• No changes to eval ids 5, 6, 8, 9, or the setup evals in this commit — those were already reviewed in prior passes and are unaffected here.

No inline comments needed — nothing in this commit meets the high-signal bar for a blocking/important finding.

[claude]

Claude finished @kyle-sexton's task in 1m 14s —— View job

---

Security review: no security-relevant surface

• Gather context (PR diff, changed files, HEAD SHA)
• Invoke /review:security-review and follow it exactly
• Post final findings

Skip gate triggered (condition 2): this change has no security-relevant surface.

Re-checked against the current head (79bbeca), which adds one commit since the prior review (4c6a474): 79bbeca reworks the load-bearing triad in eval id 7 (plugins/ai-slop/skills/audit/evals/evals.json) — swapping "the layers are managed policy, project settings, and the local overlay" for "the cascade layers are user, team, and local" so all three items are single tokens — and appends a matching CHANGELOG.md entry recording the correction. Independently verified the reworded triad now matches rule-rule-of-three's ERE:

$ printf 'The cascade layers are user, team, and local.\n' | grep -oE '[A-Za-z]+, [A-Za-z]+, and [A-Za-z]+'
user, team, and local

Full PR diff against origin/main is still the same four files reviewed previously:

• plugins/ai-slop/.claude-plugin/plugin.json — version bump only.
• plugins/ai-slop/CHANGELOG.md — prose changelog entries.
• plugins/ai-slop/skills/audit/evals/evals.json — declarative eval fixtures (prompt/expected_output/expectations strings).
• plugins/ai-slop/skills/setup/evals/evals.json — declarative eval fixtures, same shape.

No scripts, no GitHub Actions workflow files, no config-parsing code, no dependency pins, and no runtime logic are touched — still only declarative eval test cases (consumed by a model-driven eval harness, not executed as code) plus documentation/version metadata. There is no injection surface, no credential handling, no authorization logic, and no supply-chain change here for this review lane to assess.

No security findings to report.