diff --git a/.claude/ai-slop.json b/.claude/ai-slop.json index a016b3069..fd73462da 100644 --- a/.claude/ai-slop.json +++ b/.claude/ai-slop.json @@ -1,8 +1,9 @@ { - "_comment": "Three rules are disabled here, each for a measured, repo-specific reason; the shipped defaults stay neutral and this is the consuming repo's own decision, which is what the config layer is for. (1) rule-em-dash: this marketplace's house style uses em dashes deliberately, measured 34,999 matching lines across 1229 tracked markdown files (2026-08-19), which is a style choice, not a defect backlog. Disabling beats enumerating em_dash_allowed_paths, which would need near-total path coverage to be useful. (2) rule-curly-artifacts: punctuation glyphs were already ruled out of scope by the repo owner, twice in writing, and the rule's yield is almost entirely the material those rulings protect. Measured 2026-08-20 over 1214 tracked markdown files: 72 findings, of which 39 are quoted Pat Pattison book text, 32 are a verbatim pasted external Codex review, and exactly 1 was genuine paste residue (a stray curly apostrophe in plugins/disk-hygiene/README.md, fixed rather than suppressed in the same change as this entry). The rulings: plugins/songwriting/context/pat-pattison/research/book-references.md ('Do not sweep, measure, audit, or open work items on punctuation glyphs' - a session was spent on exactly that before the owner ruled it out of scope; it also notes the four books disagree with each other, so there is no fidelity axis to restore toward) and the songwriting CHANGELOG entry 'PUNCTUATION GLYPHS ARE NOT A FIDELITY AXIS - DO NOT SWEEP FOR THEM', which records a prior session normalizing 470 lines before the ruling landed. (3) rule-emoji-formatting: measured 2026-08-20 over the same corpus, all 21 findings are SEMANTIC markers and none is decoration - 7 wrong/right teaching pairs in docs-hygiene extract-ssot citation-form.md and planning interview loop.md, 4 coaching pairs in the pat-pattison coaching-protocol.md, 9 warning headings across kindle-dedrm README.md, the songwriting CHANGELOG and book-references.md, and 1 severity-legend marker in REVIEW.md that belongs to the same table as its two siblings. The rule is doing its job; this repo simply uses these glyphs as vocabulary rather than ornament. In-file block markers were the first choice and were rejected on cost: the marked files span four plugins, and the changelog-parity gate treats any edit under plugins// as version reuse, so silencing cosmetic audit noise would have published four plugin releases. A per-rule path exemption is the instrument that actually fits all three of these entries, and the config layer supports it only for rule-em-dash today; generalizing it is left to its own change. Revisit any of the three if a corpus of genuine LLM residue ever lands here.", + "_comment": "Three rules are disabled here, each for a measured, repo-specific reason; the shipped defaults stay neutral and this is the consuming repo's own decision, which is what the config layer is for. (1) rule-em-dash: this marketplace's house style uses em dashes deliberately, measured 34,999 matching lines across 1229 tracked markdown files (2026-08-19), which is a style choice, not a defect backlog. Disabling beats enumerating em_dash_allowed_paths, which would need near-total path coverage to be useful. (2) rule-curly-artifacts: punctuation glyphs were already ruled out of scope by the repo owner, twice in writing, and the rule's yield is almost entirely the material those rulings protect. Measured 2026-08-20 over 1214 tracked markdown files: 72 findings, of which 39 are quoted Pat Pattison book text, 32 are a verbatim pasted external Codex review, and exactly 1 was genuine paste residue (a stray curly apostrophe in plugins/disk-hygiene/README.md, fixed rather than suppressed in the same change as this entry). The rulings: plugins/songwriting/context/pat-pattison/research/book-references.md ('Do not sweep, measure, audit, or open work items on punctuation glyphs' - a session was spent on exactly that before the owner ruled it out of scope; it also notes the four books disagree with each other, so there is no fidelity axis to restore toward) and the songwriting CHANGELOG entry 'PUNCTUATION GLYPHS ARE NOT A FIDELITY AXIS - DO NOT SWEEP FOR THEM', which records a prior session normalizing 470 lines before the ruling landed. (3) rule-emoji-formatting: measured 2026-08-20 over the same corpus, all 21 findings are SEMANTIC markers and none is decoration - 7 wrong/right teaching pairs in docs-hygiene extract-ssot citation-form.md and planning interview loop.md, 4 coaching pairs in the pat-pattison coaching-protocol.md, 9 warning headings across kindle-dedrm README.md, the songwriting CHANGELOG and book-references.md, and 1 severity-legend marker in REVIEW.md that belongs to the same table as its two siblings. The rule is doing its job; this repo simply uses these glyphs as vocabulary rather than ornament. In-file block markers were the first choice and were rejected on cost: the marked files span four plugins, and the changelog-parity gate treats any edit under plugins// as version reuse, so silencing cosmetic audit noise would have published four plugin releases. A per-rule path exemption is the instrument that actually fits all three of these entries, and the config layer supports it only for rule-em-dash today; generalizing it is left to its own change. Revisit any of the three if a corpus of genuine LLM residue ever lands here. Separately, two excluded_paths entries name files that CONTAIN the tells on purpose and would otherwise be scanned as if they were defects: reference/catalog.md quotes the tells it catalogues, and skills/audit/evals/fixtures/** are the eval suite's committed slop samples (#3041 - a prose scenario cannot be checked against the detector and disagreed with it three times, so the input is a file now). Excluding the fixture path rather than marking each fixture with an in-file ai-slop-ignore-file marker is what keeps them measurable: a file marker declines the file unconditionally, including under the empty HOME + CLAUDE_PROJECT_DIR isolation the detector's own tests use, so `detect.sh ` would print nothing and the eval author would be back to trusting prose. An excluded_paths glob is a layer of THIS repo's config, so that same isolation lifts it and the fixture reports its real findings.", "excluded_paths": [ "plugins/*/skills/*/vendor/**", - "plugins/ai-slop/skills/audit/reference/catalog.md" + "plugins/ai-slop/skills/audit/reference/catalog.md", + "plugins/ai-slop/skills/audit/evals/fixtures/**" ], "disabled_rules": ["rule-em-dash", "rule-curly-artifacts", "rule-emoji-formatting"] } diff --git a/plugins/ai-slop/.claude-plugin/plugin.json b/plugins/ai-slop/.claude-plugin/plugin.json index a0510efd2..c0b4c5a68 100644 --- a/plugins/ai-slop/.claude-plugin/plugin.json +++ b/plugins/ai-slop/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "ai-slop", - "version": "0.2.2", + "version": "0.3.0", "description": "Detects and removes AI-writing tells (slop) in checked-in markdown prose: em dashes, emoji formatting, AI vocabulary, negative parallelisms, chatbot phrases, filler, stacked hedging, citation artifacts, and the rest of a catalog distilled from Wikipedia's Signs of AI writing. Read-only audit by default with a deterministic detector plus a judgment rubric; an explicit fix action rewrites findings behind a semantic-diff guard. Findings conform to the detector-findings convention so the review fanout fix relay can consume them.", "author": { "name": "Melodic Software", diff --git a/plugins/ai-slop/CHANGELOG.md b/plugins/ai-slop/CHANGELOG.md index 4985baa3d..65515e0ba 100644 --- a/plugins/ai-slop/CHANGELOG.md +++ b/plugins/ai-slop/CHANGELOG.md @@ -1,5 +1,62 @@ # Changelog +## [0.3.0] + +The audit eval cases described their input in prose. Nothing checked that the described input +produced the finding the case graded, and it drifted from the detector three times in one PR +(#3041) — each time a golden answer the scenario could not produce. Seven of the nine cases now +name a committed fixture instead. + +- **Six eval fixtures ship under `skills/audit/evals/fixtures/`**, referenced from each case's + `files` array: `report-only.md` (four rules on one file), `fix-guarded-rewrite.md` (two em dashes + and two filler phrases), `rubric-boundary.md` (one script finding, promotional register the + mechanical rules deliberately miss), `em-dash-substitution.md` (a single em dash), + `triads.md` (rule-of-three at 3 hits in 69 words) and `knowledge-cutoff-prose.md` (the recorded + false-positive class). Every case's `expected_output` now names the rules, lines and fired + thresholds the detector actually emits, measured rather than asserted. +- **A case names its fixture through `files[]` and in prose, the way every sibling suite does** — + `mcp-tools:audit` and `docs-hygiene:compress` both read "`evals/fixtures/.md` relative to the + skill directory", and none of the eighteen fixture-backed suites here builds a repository to audit + in. These prompts are a *specification* of expected skill behavior, not a script: this repo has 200 + `evals.json` files and zero of the `case.yaml` / `prompt.md` + `graders/` layout `claude plugin + eval` consumes, no manifest declares `experimental.evals`, and the only things that read + `evals.json` are lint scripts. Nothing executes a prompt, so a prompt must be readable by a human + or an agent working by hand, and environment control belongs nowhere in it. If this repo ever + adopts the CLI's format, per-case setup has a first-class home there — a `scaffold_script` run + under `--scaffold`. +- **Cases 2, 6 and 7 tell the reader to work on a copy.** They invoke `fix`, and the fix flow + rewrites each flagged line in place, so running one by hand against the committed fixture + remediates it and dirties the repo — and a later run then grades already-fixed input, where the + declared findings no longer fire. One sentence in the prompt and one expectation per case, both + about the outcome rather than the mechanism: the committed fixture is byte-identical after the run, + and how the copy gets made is the reader's business. An instruction a case states but never checks + is the same defect in miniature as the one this release removes. +- **Cases 4 and 5 state their premise instead of constructing it.** Both grade the persistence step, + which `SKILL.md` gates on the audit having "examined tracked files", so each prompt says the + audited file is tracked in the repo under audit and the expectations grade the skill's *decision*: + that it treats persistence as applicable, fetches the producer contract first, and — case 4 — + refuses to write when that fetch fails, rather than refusing because the target was out of tracked + space. A case cannot verify real repository state, and pretending otherwise is what made case 4 + pass for the wrong reason. Case 5 also asserts the positive half — that the findings file is + actually written — because `context/persist-findings.md` permits reporting without writing when the + destination cannot be proven outside tracked space, so a negative-only case would be satisfied by a + run that persists nothing at all. +- **This reverses 0.1.0's no-fixtures decision, which was recorded in `detect.test.sh`'s header.** + That decision holds for the *unit* suite, whose fixtures are still built inline in a tmpdir. It + does not survive contact with the eval suite: an eval case is graded against a deterministic + detector run, so its scenario has to satisfy an ERE the eval author does not have in front of + them. A committed fixture cannot disagree with the detector; prose describing one can, and did. +- Cases 3 and 9 keep `narration: true`. They grade repo-wide flow and consuming-repo config, not + file content, so there is nothing for a fixture to pin. +- The fixtures carry real tells, so a repo auditing its own tree has to decline them. Prefer an + `excluded_paths` glob over an in-file `ai-slop-ignore-file` marker: a file marker declines + unconditionally, including under the empty `HOME` + `CLAUDE_PROJECT_DIR` isolation, so + `detect.sh ` would print nothing and the eval author would be back to trusting prose. + An `excluded_paths` entry is a config layer, and that isolation lifts it. This repo's own + `.claude/ai-slop.json` carries the glob as the worked example. +- Consuming repos need no exclusion of their own: the audit scans `git ls-files '*.md'`, and an + installed plugin's files are not tracked in the repo that installs it. + ## [0.2.2] The in-file suppression the fix flow and the catalog both tell operators to reach for now works in diff --git a/plugins/ai-slop/skills/audit/evals/evals.json b/plugins/ai-slop/skills/audit/evals/evals.json index 62a9f74d6..af2437b71 100644 --- a/plugins/ai-slop/skills/audit/evals/evals.json +++ b/plugins/ai-slop/skills/audit/evals/evals.json @@ -4,27 +4,28 @@ { "id": 1, "name": "bare-invocation-read-only", - "prompt": "/ai-slop:audit docs/example.md", - "expected_output": "A findings report for the target file — per-rule findings with file:line locations, fired thresholds, and a summary — with no file modified. The fix flow is offered but not run.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit evals/fixtures/report-only.md relative to the skill directory.", + "expected_output": "A findings report for the fixture — four findings across four rules (rule-filler-phrases on line 3, rule-significance-inflation on line 6, rule-em-dash and rule-chatbot-artifacts both on line 8), each with its file:line location and fired condition, plus the Summary rows — with no file modified. The fix flow is offered but not run.", + "files": ["evals/fixtures/report-only.md"], "expectations": [ "Runs detect.sh on the target and reports findings with file:line locations and the fired condition per finding", - "Does NOT edit the target file or any other file on bare invocation", + "Reports all four rules the fixture trips rather than a subset — filler phrase, significance inflation, em dash, and chat-turn residue", + "Does NOT edit the fixture or any other file on bare invocation", "Mentions the fix action as the explicit follow-up rather than applying rewrites" ] }, { "id": 2, "name": "fix-runs-semantic-guard", - "prompt": "/ai-slop:audit fix docs/example.md", - "expected_output": "The file's findings are rewritten per-file (em dashes reworded, stock phrases deflated), each file's before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs and the findings file is re-emitted so no stale findings survive.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit fix evals/fixtures/fix-guarded-rewrite.md relative to the skill directory. Work on a copy so the committed fixture stays byte-identical.", + "expected_output": "The fixture's four findings — two em dashes (lines 3 and 7) and two filler phrases ('It is worth noting that' on line 3, 'Due to the fact that' on line 4) — are rewritten in one per-file pass, the before/after pair is verified by a fresh-context semantic-diff subagent that reverts semantic loss or ambiguity, then the detector re-runs and the findings file is re-emitted so no stale findings survive.", + "files": ["evals/fixtures/fix-guarded-rewrite.md"], "expectations": [ "Applies rewrites only because fix was explicitly requested", - "Verifies every rewritten file with a fresh-context semantic-diff subagent and reverts flagged hunks", - "Re-runs the detector and re-emits the findings file after the last fixed file" + "Verifies the rewritten file with a fresh-context semantic-diff subagent and reverts flagged hunks", + "Preserves every claim the fixture makes — that the cache invalidates on write, that the reader never sees a stale entry, that writes are rare, and that the policy is least-recently-used", + "Re-runs the detector and re-emits the findings file after the last fixed file", + "Leaves `evals/fixtures/fix-guarded-rewrite.md` byte-identical after the run — the rewrites land on a copy" ] }, { @@ -43,64 +44,66 @@ { "id": 4, "name": "persist-refuses-without-contract", - "prompt": "/ai-slop:audit docs/example.md (with the detector-findings contract URL unreachable)", - "expected_output": "The audit report is delivered, but no findings file is written: the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit evals/fixtures/report-only.md relative to the skill directory, with the detector-findings contract URL unreachable. Treat the audited file as tracked in the repo under audit.", + "expected_output": "The audit report is delivered with the fixture's four findings intact, but no findings file is written: the audit examined a tracked file, so persistence applies, and the skill reports that the producer contract could not be fetched and refuses to invent a destination, per persist-findings.md.", + "files": ["evals/fixtures/report-only.md"], "expectations": [ "Delivers the human-facing findings report regardless", + "Treats persistence as applicable, since the audit examined a tracked file, rather than passing over the step as out of scope", + "Fetches the producer contract before writing anything, per persist-findings.md", "Does NOT write a findings file when the contract fetch fails", - "States that persistence was refused because the contract was unreachable, rather than silently skipping it" + "States that persistence was refused because the contract was unreachable, rather than silently skipping it, and does not attribute the refusal to the target being out of tracked space" ] }, { "id": 5, "name": "rubric-findings-reach-the-report-not-the-findings-file", - "prompt": "/ai-slop:audit docs/example.md — the file contains promotional register (\"a breathtaking, must-visit dashboard sitting in the heart of the stack\") and one em dash. The promotional words are deliberately outside the mechanical vocabulary list, so the detector matches the em dash and nothing else.", - "expected_output": "Both layers report: the em dash appears as the single script finding, and the promotional tone as a rubric finding citing its catalog entry. The persisted findings file carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit evals/fixtures/rubric-boundary.md relative to the skill directory. Treat the audited file as tracked in the repo under audit.", + "expected_output": "Both layers report: the em dash on line 7 appears as the fixture's single script finding, and the promotional register on lines 3-5 ('a breathtaking, must-visit surface sitting right in the heart of the platform') as a rubric finding citing its catalog entry. The audit examined a tracked file, so persistence applies, and the findings file carries that one script finding ONLY — the rubric verdict never enters it, because a rubric tell has no crosswalk row to look a tier up from.", + "files": ["evals/fixtures/rubric-boundary.md"], "expectations": [ "Reports the promotional-register tell as a rubric finding, quoting the offending text and naming the catalog entry it comes from", - "Does NOT place any rubric finding in the persisted findings file — only the script finding appears there", + "Does NOT report the promotional wording as a script finding — the fixture's promotional words sit outside the mechanical vocabulary, so the detector's only finding is the em dash", + "Treats persistence as applicable, since the audit examined a tracked file, rather than passing over the step as out of scope", + "Writes the findings file, on the stated premise that the audited file is tracked — a run that reports the findings and persists nothing does not satisfy this case", + "Puts NO rubric finding in the findings file — only the em-dash script finding belongs there", "States the V1 boundary explicitly rather than silently omitting the rubric verdict from the file" ] }, { "id": 6, "name": "fix-never-swaps-one-tell-for-another", - "prompt": "/ai-slop:audit fix docs/example.md — the file's only finding is a single em dash in \"the parser accepts empty input — a change from 0.3.\"", - "expected_output": "The em dash is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit fix evals/fixtures/em-dash-substitution.md relative to the skill directory. Work on a copy so the committed fixture stays byte-identical.", + "expected_output": "The fixture's single finding — the em dash on line 5, in 'The parser accepts empty input — a change from 0.3.' — is resolved into a comma, a period, or a restructured sentence. It is NOT replaced with a parenthesis, an en dash, or a spaced hyphen: rewrite-guide.md's substitution guardrail names those as the same interruption wearing a different mark.", + "files": ["evals/fixtures/em-dash-substitution.md"], "expectations": [ "Reads reference/rewrite-guide.md before applying the rewrite, per the fix flow's first step", "The rewritten line contains no em dash, en dash, parenthesis, or spaced hyphen standing in for the original punctuation", - "Preserves the sentence's claim — the version fact survives the rewrite" + "Preserves the sentence's claim — the version fact survives the rewrite", + "Leaves `evals/fixtures/em-dash-substitution.md` byte-identical after the run — the rewrite lands on a copy" ] }, { "id": 7, "name": "triads-collapse-toward-the-strongest-item", - "prompt": "/ai-slop:audit fix docs/example.md — the file trips rule-of-three at 3 hits on two kinds of triad. Rhetorical: \"the tool is fast, simple, and reliable\" and \"it reads clean, tight, and portable\". Load-bearing: \"the cascade layers are user, team, and local\" — a complete set the reader needs, where dropping one names a resolution order that does not exist. All three match the density rule's ERE, so all three reach the fix flow as one finding.", - "expected_output": "The two rhetorical triads collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration is KEPT intact, with that judgment stated — enumerating three actual things is not a tell.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit fix evals/fixtures/triads.md relative to the skill directory. Work on a copy so the committed fixture stays byte-identical.", + "expected_output": "The fixture trips rule-rule-of-three as one density finding (3 hits in 69 words, 43.5/1000 against a 3.0 threshold), and all three triads reach the fix flow together. The two rhetorical triads — 'fast, simple, and reliable' and 'clean, tight, and portable' — collapse toward their single strongest item rather than being reworded into three-item lists with different punctuation. The cascade enumeration 'user, team, and local' is KEPT intact, with that judgment stated: dropping one names a resolution order that does not exist.", + "files": ["evals/fixtures/triads.md"], "expectations": [ - "Collapses rhetorical triads toward one item rather than preserving all three in a reworded form", - "Keeps a triad when each element is load-bearing, and says why it was kept", - "Does not treat every three-item enumeration as a tell — enumerating three actual things is not slop" + "Collapses the two rhetorical triads toward one item rather than preserving all three in a reworded form", + "Keeps the 'user, team, and local' triad because each element is load-bearing, and says why it was kept", + "Does not treat every three-item enumeration as a tell — enumerating three actual things is not slop", + "Leaves `evals/fixtures/triads.md` byte-identical after the run — the rewrites land on a copy" ] }, { "id": 8, "name": "known-false-positive-routes-to-config-not-a-rewrite", - "prompt": "/ai-slop:audit docs/example.md — the file is documentation ABOUT model knowledge cutoffs, so rule-knowledge-cutoff-disclaimer fires on prose that is describing the concept rather than disclaiming on the document's own behalf.", - "expected_output": "The finding is reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.", - "files": [], - "narration": true, + "prompt": "/ai-slop:audit evals/fixtures/knowledge-cutoff-prose.md relative to the skill directory.", + "expected_output": "The three rule-knowledge-cutoff-disclaimer findings (lines 1, 3 and 5) are reported, and the recommended remedy is the in-file marker or a config exclusion — not a rewrite. The fixture is documentation ABOUT knowledge cutoffs, so the rule fires on prose describing the concept rather than disclaiming on the document's own behalf. This is the false-positive class the catalog's calibration record names for that rule, and the recorded answer is an exemption with a stated reason rather than weakening the rule.", + "files": ["evals/fixtures/knowledge-cutoff-prose.md"], "expectations": [ - "Recognizes prose ABOUT knowledge cutoffs as the recorded false-positive class rather than deleting the sentence", + "Recognizes prose ABOUT knowledge cutoffs as the recorded false-positive class rather than deleting the sentences", "Recommends the in-file ignore marker or a config exclusion, citing the calibration record", "Does NOT propose weakening or disabling the shipped rule to make this corpus pass" ] diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/em-dash-substitution.md b/plugins/ai-slop/skills/audit/evals/fixtures/em-dash-substitution.md new file mode 100644 index 000000000..117bb1bab --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/em-dash-substitution.md @@ -0,0 +1,8 @@ +# Parser release notes + +## 0.4 + +The parser accepts empty input — a change from 0.3. Callers that relied on the +old rejection path have to check for the empty case themselves. + +The token table grew by two entries. Both of them are punctuation. diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/fix-guarded-rewrite.md b/plugins/ai-slop/skills/audit/evals/fixtures/fix-guarded-rewrite.md new file mode 100644 index 000000000..37b4ef297 --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/fix-guarded-rewrite.md @@ -0,0 +1,8 @@ +# Cache invalidation + +It is worth noting that the cache invalidates on write — the reader never sees +a stale entry. Due to the fact that writes are rare, the extra round trip costs +little. + +The eviction policy is least-recently-used — a choice made when the working set +was small, and one the team has not revisited since. diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/knowledge-cutoff-prose.md b/plugins/ai-slop/skills/audit/evals/fixtures/knowledge-cutoff-prose.md new file mode 100644 index 000000000..b76df12ed --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/knowledge-cutoff-prose.md @@ -0,0 +1,9 @@ +# What a knowledge cutoff is + +A model's knowledge cutoff is the date after which its training corpus contains +nothing. Documentation that explains the concept has to name it, which is why +this page writes "knowledge cutoff" as its subject rather than as a disclaimer +about this page's own currency. + +Readers sometimes confuse the cutoff with the release date. They are different +dates, and the gap between them is often months wide. diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/report-only.md b/plugins/ai-slop/skills/audit/evals/fixtures/report-only.md new file mode 100644 index 000000000..77164b491 --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/report-only.md @@ -0,0 +1,9 @@ +# Retry helper + +In order to retry a failed request, the helper doubles its wait between +attempts and gives up after five tries. + +The helper plays a crucial role in keeping the queue drained during an outage. + +I hope this helps — the defaults live in `config.toml` if you want to change +them. diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/rubric-boundary.md b/plugins/ai-slop/skills/audit/evals/fixtures/rubric-boundary.md new file mode 100644 index 000000000..738ef2509 --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/rubric-boundary.md @@ -0,0 +1,8 @@ +# Metrics dashboard + +The metrics dashboard is a breathtaking, must-visit surface sitting right in the +heart of the platform, and the team behind it has poured real craft into every +panel. Anyone who cares about latency will want to spend an afternoon here. + +Panels refresh every thirty seconds — the interval is fixed and not yet +configurable. diff --git a/plugins/ai-slop/skills/audit/evals/fixtures/triads.md b/plugins/ai-slop/skills/audit/evals/fixtures/triads.md new file mode 100644 index 000000000..91cd25f7c --- /dev/null +++ b/plugins/ai-slop/skills/audit/evals/fixtures/triads.md @@ -0,0 +1,14 @@ +# Configuration cascade + +## Why the tool reads well + +The tool is fast, simple, and reliable. The maintainers intend to keep it that +way. + +The generated output reads clean, tight, and portable, which is the whole point +of the generator. + +## The cascade + +The cascade layers are user, team, and local. Resolution runs in that order, +and dropping any one of them names an order that does not exist. diff --git a/plugins/ai-slop/skills/audit/scripts/detect.test.sh b/plugins/ai-slop/skills/audit/scripts/detect.test.sh index 219b2059e..c31168f48 100755 --- a/plugins/ai-slop/skills/audit/scripts/detect.test.sh +++ b/plugins/ai-slop/skills/audit/scripts/detect.test.sh @@ -1,7 +1,13 @@ #!/usr/bin/env bash -# Self-contained tests for detect.sh (no external test lib; fixtures are built -# inline in a tmpdir, so the plugin ships no slop samples for the audit to trip -# over). Per the shell-test-helpers convention, assertion helpers are local. +# Self-contained tests for detect.sh (no external test lib; THIS suite's +# fixtures are built inline in a tmpdir, so no unit-test slop corpus sits in +# the tree). The eval suite is the deliberate exception: evals/fixtures/ holds +# committed slop samples, because a prose scenario cannot be checked against +# the detector and drifted from it three times (#3041). This repo keeps its own +# audit off them with an excluded_paths glob in .claude/ai-slop.json, and they +# stay measurable through the same config isolation this suite uses — an empty +# HOME and CLAUDE_PROJECT_DIR give the shipped defaults back. +# Per the shell-test-helpers convention, assertion helpers are local. set -uo pipefail # Fixture git isolation: an inherited GIT_DIR/GIT_WORK_TREE/GIT_CONFIG would