test(ai-slop): back the audit evals with committed fixtures (0.3.0) - #3063
Conversation
The audit eval cases described their input in prose, and nothing checked that the described input produced the finding the case graded. It drifted from the detector three times in one PR — each time a golden answer the scenario could not produce. Seven of the nine cases now name a committed fixture under skills/audit/evals/fixtures/ via files[], and every expected_output names the rules, lines and fired thresholds the detector actually emits, measured with detect.sh under an empty HOME and CLAUDE_PROJECT_DIR rather than asserted. Cases 3 and 9 keep narration: true — they grade repo-wide flow and consuming-repo config, not file content. This reverses 0.1.0's no-fixtures decision, which was recorded in detect.test.sh's header and is updated in place. The decision still holds for the unit suite, whose fixtures stay inline in a tmpdir; it does not survive the eval suite, where a case is graded against a deterministic detector run and its scenario has to satisfy an ERE the eval author cannot see. The fixtures carry real tells, so this repo's own audit declines them through an excluded_paths glob rather than an in-file marker: a file marker declines unconditionally, including under the config isolation the detector's tests use, which would leave detect.sh <fixture> printing nothing and the eval author back to trusting prose. Closes #3041 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g
|
Claude finished @kyle-sexton's task in 4m 15s —— View job Code review — PR #3063Lane: Verification method: this lane's Bash access is restricted to
Also checked: Result: no findings. Every |
|
Claude finished @kyle-sexton's task in 59s —— View job Security Review
No security issues found. This PR's changed surface is entirely test/config data, not executable logic:
No GitHub Actions workflows, authentication/authorization logic, injection surfaces, or secret-handling code are touched by this PR. |
|
Last security-reviewed head: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dc5ca9ade3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Claude has reviewed this PR 5 times. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
…g them The fixture-backed cases pointed the invocation straight at the committed fixture, which reintroduced the failure class #3041 was meant to close. The fix cases (2, 6, 7) rewrite each flagged line in place, so the first run remediated the committed fixture and the second run graded already-fixed input, where the declared findings no longer fire. The persistence expectations (2, 4, 5) grade a branch conditioned on the audit having examined tracked files. An installed plugin's fixture is not tracked in the consuming repo, so the branch was unreachable, and case 4's contract-fetch refusal passed for the wrong reason: nothing was written because nothing was owed, not because the fetch failed. Every fixture-backed case now stages its target first — git init a temp directory, copy the fixture in as docs/<name>.md, commit it, audit the copy. The rewrite lands on a throwaway file and the target is tracked, so the persistence branch is genuinely reached. Staged copies re-measured: identical rules, lines and densities at the new path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g
|
Claude finished @kyle-sexton's task in 59s —— View job Security Review
No security issues found. Skip gate: the previous run on this PR (head Both files are non-executable:
No scripts, GitHub Actions workflows, config/permission surfaces, or detector logic ( |
|
Claude finished @kyle-sexton's task in 3m 50s —— View job Code review — PR #3063Lane:
Scope of this pass: the new commit ( 1 finding posted inline ( Everything else in |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
The staging preamble said "git init a temp directory ... and commit it", but git init inherits user.name/user.email from ~/.gitconfig or the GIT_* env, and neither is guaranteed. This suite's own unit tests pin HOME to an empty tmpdir for isolation (detect.test.sh), and a fresh container has no seeded identity. Under that condition the preamble dies at git commit with "Please tell me who you are" (exit 128) BEFORE the audit runs, so all seven staged cases error out instead of grading anything - the same "scenario cannot produce the graded input" failure class this release exists to remove. All seven preambles now commit with the identity inline: git -c user.name=eval -c user.email=eval@local commit -m stage Staging still stops at a commit rather than at git add. git ls-files already reports a staged file as tracked, so either form satisfies the persistence gate, but these cases model a consuming repo and a real one has history: the repo-wide ordering reads git log --since=90.days --name-only for change frequency, which returns nothing in a repo with no commits. Harmless for today's single-file cases, and invisible in every current expectation, which is why it is worth not building in. Verified by executing each preamble as written under an empty HOME with GIT_AUTHOR_*/GIT_COMMITTER_* unset: all seven commit, leave a HEAD, report the target tracked, and emit their declared finding counts (4, 4, 4, 1, 1, 1, 3). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g
|
Claude finished @kyle-sexton's task in 1m 19s —— View job Security Review
No security issues found. Skip gate: the previous run on this PR (head Both changed files are non-executable, and neither introduces new security-relevant surface:
No scripts, GitHub Actions workflows, config/permission surfaces, or detector logic ( |
|
Claude finished @kyle-sexton's task in 1m 18s —— View job Code review — PR #3063Lane:
Scope of this pass: the new commit ( Verification:
Result: no findings. The P2 finding from the prior review pass is resolved correctly and consistently across all 7 affected cases; nothing new introduced by this commit. |
…eamble The staging preamble isolated git identity but not the config cascade. detect.sh resolves REPO_ROOT from CLAUDE_PROJECT_DIR before falling back to git rev-parse --show-toplevel, then loads that root's .claude/ai-slop.json, and rule disablement is not path-scoped. A maintainer running these evals from a session rooted at this repo therefore inherits its disabled_rules - rule-em-dash, rule-curly-artifacts, rule-emoji-formatting - and those apply to the /tmp scratch repo too. Measured with only CLAUDE_PROJECT_DIR changed: case 1: 4 -> 3 findings (em dash L8 vanishes) case 2: 4 -> 2 (both em-dash findings vanish) case 4: 4 -> 3 case 5: 1 -> 0 (the em dash IS the single script finding) case 6: 1 -> 0 (the case is entirely that one em dash) cases 7, 8: unaffected Cases 5 and 6 fail silently: nothing is found, nothing is asserted, and the expectations pass vacuously. Same golden-answer-the-scenario-cannot-produce class as the rest of this release, arriving through config rather than prose. All seven fixture-backed prompts now point CLAUDE_PROJECT_DIR at the scratch directory and HOME at an empty one, so the cascade resolves to shipped defaults. Applied to 7 and 8 as well: their rules are not disabled today, but disabled_rules can change and uniformity is cheaper than a per-case rule. The knowledge that this isolation is what keeps the fixtures measurable was already recorded in detect.test.sh's header comment; it now lives in the artifact that needs it. Prompts only - no expected_output, files, expectations, narration, or fixture byte changed, and cases 3 and 9 stay byte-identical to dc5ca9a. Re-ran all seven preambles literally under the pinned cascade: every rule id, line and density figure still matches its expected_output, case 7 included. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g
|
Claude finished @kyle-sexton's task in 1m 7s —— View job Security Review
No security issues found. Skip gate: the previous run on this PR (head Both changed files are non-executable, and neither introduces new security-relevant surface:
No scripts, GitHub Actions workflows, config/permission surfaces, or detector logic ( Note: my working tree shows an uncommitted, unrelated local modification to |
|
Claude finished @kyle-sexton's task in 2m 37s —— View job Code review — PR #3063Lane:
Scope of this pass: the new commit ( Verification, checked against the actual fixtures and detector code rather than taken on claim:
Also confirmed: the config-cascade ordering description in the CHANGELOG entry ( Result: no findings. This closes a real, subtle bug (an ambient session env var silently deleting the graded finding from an otherwise-isolated scratch repo) with a correct, narrowly-scoped fix, consistent with the pattern of the prior three review rounds on this PR. |
…uence Three residual defects, all the same class: the preamble described staging in prose and left the mechanism to the reader. A. The cascade pin was phrased as a one-time export, but the harness re-supplies CLAUDE_PROJECT_DIR on every tool call and shell state does not survive between them. A model that staged in one call and ran the detector in the next got the disabled rule back and measured 0 findings on case 5. The preamble now runs in ONE invocation, says why, and instructs setting CLAUDE_PROJECT_DIR/HOME on every later command rather than exporting once. B. The preamble never issued git add, so a literal executor produced an untracked target - inverting the tracked-file branch cases 4 and 5 exist to grade. The sequence now stages before committing. C. The scratch repo was built without clearing inherited git environment, violating the invariant scripts/check-fixture-git-isolation.sh already enforces for *.sh/*.py fixtures. Under an ambient GIT_DIR the init and the stage commit landed in the CALLER's repository and the scratch directory got no .git at all. That gate cannot see prose inside evals.json, so the eval suite now honors the rule itself: the sequence opens with unset GIT_DIR GIT_WORK_TREE GIT_CONFIG. Also repairs a hyphenated compound broken across a line wrap in the 0.3.0 entry, which rendered as "golden-answer-the- scenario-cannot-produce". Verified by extracting each command verbatim from the committed JSON and running it under a hostile environment - HOME holding a config that disables rule-em-dash, CLAUDE_PROJECT_DIR pointed at this repo, and an ambient GIT_DIR pointed at this repo's .git. All seven: commit exit 0, HEAD created, target tracked, scratch dir owns its .git, caller HEAD unchanged, and the declared finding counts (4, 4, 4, 1, 1, 1, 3). Detector re-run in a separate tool call from staging still reports them; relying on ambient env instead reports 0, which is the failure A describes. Prompts only - no expected_output, expectations, files, narration, name or skill_name changed, cases 3 and 9 stay byte-identical to dc5ca9a, and no fixture byte moved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g
|
Claude finished @kyle-sexton's task in 1m 14s —— View job Security Review
No security issues found. Skip gate: the previous run on this PR (head Both changed files are non-executable, and neither introduces new security-relevant surface:
No scripts, GitHub Actions workflows, config/permission surfaces, fixtures, or detector logic ( |
|
Claude finished @kyle-sexton's task in 3m 22s —— View job Code review — PR #3063Lane:
Scope of this pass: the new commit ( 1 finding posted inline ( Everything else in the diff (the |
…tion The staging preamble solved a problem this repo does not have. These eval suites are not executable by `claude plugin eval`: the CLI consumes `<eval dir>/**/case.yaml` or `prompt.md` + `graders/*.md`, and this repo has 200 evals.json files, zero of that layout, no manifest declaring experimental.evals, and only lint scripts reading evals.json. Nothing runs a prompt. The prompts are a written specification of expected skill behavior. So every environment property the preamble controlled - config cascade, git identity, `git add`, GIT_DIR isolation, the skill-dir path - was a property of executing a prompt that nothing executes. Six failure modes in five review rounds came from that machinery, in a shape none of the eighteen sibling fixture-backed suites uses. All seven cases now name the fixture through files[] and in prose, matching mcp-tools:audit and docs-hygiene:compress: "evals/fixtures/<name>.md relative to the skill directory". One constraint survives independent of any harness question, and it is the original Codex finding: the fix flow rewrites in place, so cases 2, 6 and 7 say to work on a copy and leave the committed fixture byte-identical. Prose, no mechanism. Cases 4 and 5 state their persistence premise rather than constructing it, and their expectations now grade the skill's decision. expected_output for cases 1, 2, 6, 7 and 8 is restored verbatim to dc5ca9a - the measured text from before the apparatus. Fixtures are untouched and re-measured under the shipped-defaults condition detect.test.sh itself uses: every rule id, line and density figure still matches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g
|
Claude finished @kyle-sexton's task in 1m 6s —— View job Security Review
No security issues found. Skip gate: the previous run on this PR (head Both changed files are non-executable, and this commit removes rather than adds any shell-executable content:
No scripts, GitHub Actions workflows, config/permission surfaces, fixtures, or detector logic ( Note (unrelated to this PR): the working tree at review time carries an uncommitted local modification to |
Cases 2, 6 and 7 told the reader to work on a copy and then checked nothing. An instruction a case states but never checks is the same defect this release removes, in miniature. Each now carries one expectation asserting the outcome: the committed fixture under evals/fixtures/ is byte-identical after the run, and the rewrites land on a copy. How the copy gets made is the reader's business - pinning the mechanism is what the earlier staging preamble did wrong. Expectations only. No prompt, expected_output, files, narration, name or fixture byte changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g
|
Claude finished @kyle-sexton's task in 49s —— View job Security Review
No security issues found. Skip gate: the previous run on this PR (head Both changed files are non-executable, and this commit is purely additive to grading criteria — no new mechanism, no new executable content:
No scripts, GitHub Actions workflows, config/permission surfaces, fixtures, or detector logic ( Note (unrelated to this PR, unchanged from prior reviews): the working tree at review time still carries an uncommitted local modification to |
Case 5's remaining persistence expectations were negative or procedural: no rubric finding in the file, and persistence treated as applicable. Neither requires a file to exist. context/persist-findings.md permits reporting without writing when the destination cannot be proven outside tracked space, so a run could call persistence applicable, write nothing, and satisfy both - green while asserting nothing, which is the shape this release removes. Adds the positive half: the findings file is written, and a run that reports the findings and persists nothing does not satisfy the case. The rubric boundary now constrains a file that must exist. Not a regression from dc5ca9a, which was also negative-only, but the assertion is gradeable under the specification design and so was worth having. Case 5 expectations only. Every prompt and expected_output in the file is byte-identical to b263df3, and cases 1-4 and 6-9 are untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g
|
Claude finished @kyle-sexton's task in 43s —— View job Security Review
No security issues found. Skip gate: the previous run on this PR (head Both changed files are non-executable, and this commit only strengthens an existing grading expectation — no new mechanism, path, or executable content:
No scripts, GitHub Actions workflows, config/permission surfaces, fixtures, or detector logic ( Note (unrelated to this PR, unchanged from prior reviews): the working tree at review time still carries an uncommitted local modification to |
…tes them T2 (Codex P2, agreed). The audit skill is the normal entry point that recommends remediation after a run, and it still told users to keep this plugin's prose findings away from `review:fanout fix` because routing them there "retires the findings without fixing them". That is now flatly false: the crosswalk declares `/ai-slop:audit fix` as the remediation owner for the other fourteen rules, so the relay hands those rows over instead of retiring them. Leaving step 6 in place would have made the new route unreachable through the documented flow while the contract advertised it, and handed the model directly contradictory instructions. Both stale passages corrected: - Step 6 now recommends `review:fanout fix` for the whole file when the operator is already running a fix pass, and this skill's own `fix` directly when they are not, and names the one condition that changes the answer -- the relay can only hand the rows over when /ai-slop:audit is available in that session, and surfaces them otherwise. - The Purpose paragraph drew a real distinction in the wrong place. The narrowness is about what the relay APPLIES, not what it ROUTES: rule-utm-params is still the only row the relay applies itself, and the other fourteen are now routed rather than left unrouted. It keeps the true half -- the cleanup route never loads this skill's rewrite guide, which is exactly why the declaration exists. No behavior, script, rule, or eval changed: this is the producer half of a claim the consumer now honors. ai-slop 0.3.1 (0.3.0 is published on main via #3063, so amending it was not available). review stays 0.26.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… skill (0.26.0) (#3068) Closes #3033 ## Summary The `detector-findings` convention lets a producer emit a findings file that `review:fanout`'s `fix` action consumes. The relay then decided **how** to remediate — and for `ai-slop:audit` that decision was wrong for 14 of its 15 rules. Those rows are prose-style findings. They classify as **cleanup** by content, and `fix-pass-mode.md` Step 4 hands the whole cleanup class to `/simplify` — a **code**-simplification skill that, in its own words, "rediscovers cleanups from the working-tree diff — it does NOT read the findings files", and which never loads `plugins/ai-slop/skills/audit/reference/rewrite-guide.md` where that plugin's rewrite discipline lives. Step 5 then retired the file anyway. A relay pass over an `ai-slop:audit` findings file reported a clean run while applying at most `rule-utm-params`, the one genuinely auto-applicable rule. #3031 narrowed the plugin's own claim to say exactly that, which was honest and left the gap. This teaches the contract to let a producer name the skill that owns its findings' remediation, and teaches the relay to honor it — **without a new field, and without any producer changing what it emits.** ### The question settled first: does an existing disposition already cover this? Asked before anything was written, because a third disposition an existing one already covers is pure cost. **It does not**, on two independent grounds: 1. **Off-site is a statement about the SITE, and these repairs are at `Location`.** `docs/conventions/detector-findings/README.md` "When the remediation is not at `Location`" binds its producer obligation to "a rule whose remediation can lie **outside `Location`'s file**", and both limbs of the consumer's trigger in `fix-pass-mode.md` "Step 2" are site limbs — the `Action` names a different file, or the contract declares the rule off-site. `testing:audit`'s own adopter row states the contrast: its `Location` "IS the remediation site". Claiming off-site to reach the disposition would assert something false about where the fix goes, the same defect the doc already forbids in retargeting `Location`. And it routes to **surface-only**, which trades a wrong apply for no apply rather than closing the gap. 2. **`Auto-applicable: No` has no path to the route that actually misapplies these rows.** Step 4's surface-instead-of-auto-applying fence sits under its **correctness-class** heading. Cleanup-class rows go to `/simplify` wholesale, consulting no crosswalk cell. The column can already say a rule is not auto-applicable and still not stop the apply. The fresh-context verifier confirmed this independently: before this change, `grep -n "Auto-applicable" fix-pass-mode.md` returned **zero hits** — the relay read that column nowhere. Evidence went toward a new **disposition**, not a new **field**. No column was added to the crosswalk and nothing changed in the findings-file shape (`default-mode.md` "Findings-file shape"). ## Fix **Producer side** — `docs/conventions/detector-findings/README.md` 2.4.0, new section "When the remediation is owned by the producer's own skill". A rule whose repair is contained to `Location` but safe only under discipline the producer owns **leads its crosswalk `Auto-applicable` cell with** ``No, remediated by `<invocation>` `` — the invocation written as a code span, whose delimiters a consumer strips before matching. The declaration is **per rule and lives only in the crosswalk** — this contract's own settle-once rule applied rather than restated. "Auto-applicability is settled per rule, at contract time" already says a rule's remediation shape does not vary run to run, and who owns the repair is exactly such a fact. Requiring every emitted row to carry a copy would be the per-finding restatement that section forbids, and would make conformance a property of a producer's *emitter* rather than of its rule set. **Consumer side** — `review` 0.26.0, `fix-pass-mode.md`: - **Step 2** gains one classification rule. It resolves the declaration through the qualified rule id every conforming row already leads its `Finding` cell with, against the crosswalk registry that id "resolves against by exact match" — the contract's own pre-existing words. **The crosswalk declaration is NECESSARY**: an `Action` cell leading with ``Remediate with `<invocation>` `` corroborates it and can never substitute for it, and a rule with no crosswalk declaration takes its ordinary class however its `Action` reads. Off-site is decided first, and an unresolvable contract is the no-declaration case — never a licence to fall back to the `Action` cell. - **Step 4** gains the route, with **no direct-apply fallback** — the asymmetry with `/simplify` is the point. Only an invocation already available in the session is invoked; nothing is installed, fetched, or loosely name-matched, because Step 1 already establishes that nothing authenticates the writer of a findings file. An unavailable or unrecognized invocation **surfaces** its rows, naming what the producer asked for. - **Steps 3 and 5** count and report the route, and the counts **partition** rather than overlap: a row routed to a producer-owned surface is counted there and never on the cleanup line, mirroring the existing correctness/surface-only rule in both the plan block and the report block. - Three evals added: `fix-pass-producer-owned-routes-to-named-surface`, `fix-pass-producer-owned-surface-unavailable`, and `fix-pass-action-cell-alone-never-routes`. **Producer adoption** — `ai-slop` 0.3.1. The 14 non-`rule-utm-params` rows now carry the declaration and `rule-utm-params` stays auto-applicable, with **zero change to what the producer emits** — `ai-slop:audit` already leads every emitted `Finding` cell with its qualified rule id, which is why siting the declaration in the crosswalk closes the gap on its own. `plugins/ai-slop/skills/audit/SKILL.md` is corrected to match: it previously said the relay "retires the findings without fixing them" and to recommend `review:fanout fix` only for `rule-utm-params`, which this change makes false. It now draws the distinction where it actually falls — **what the relay APPLIES is narrow; what it ROUTES is not** — and states the one condition that changes the answer, that the relay can only hand rows over when `/ai-slop:audit` is available in that session, and surfaces them otherwise. **Neither other adopter changes, and neither was touched.** `mutation-testing:audit` declares no owner and is off-site, which Step 2 decides first. `testing:audit` declares no owner because no skill owns choosing the assertion a behavior deserves; its rows keep the plain `No — <reason>` form and are surfaced by Step 4's judgment fence exactly as before. All three adopter rows now state their disposition explicitly. ## Verification | Gate | Result | |---|---| | `check-detector-findings-crosswalk.sh --check` + its self-test | PASS — 22 rule rows, every disposition argued | | `check-changelog-parity.sh` `--check` / `--check-bump origin/main` / `--check-order` / `--check-preserved origin/main` | PASS (83 changelogs newest-first; 64 headings compared) | | `check-changed-skills.sh origin/main` | `audit: PASS — 0 errors`, `fanout: PASS — 0 errors` | | `check-stale-base-overlap.sh --check origin/main` | up to date with `origin/main` | | `check-contract-clause-coverage.py` | PASS | | `check-contract-slice-prune.sh --check` / `--check-diff origin/main` | PASS | | `check-cross-plugin-source-drift.sh --check` | PASS | | `check-evals-quality.sh` on both changed eval sets | PASS — 0 warnings | | `markdownlint-cli2` on every changed markdown file | PASS — 0 issues | | `detect.test.sh` (regression check) | all 92 cases passed | | `node scripts/generate-catalog.mjs` | Catalog already in sync; `docs/CATALOG.md` unchanged | ### Fresh-context verification Separate opus verifiers, in fresh context and told to judge the final state rather than any description of it, reviewed this. The first found **three** real defects, all fixed: the `mutation-testing:audit` adopter row misdescribed two of its own `Auto-applicable` cells; the declared lead was written as a code span in the shipped cells but bare in the contract, so a consumer implementing it literally would match nothing; and the contract-reachability argument was evidence about the **producer's** session rather than the consumer's. Its full verdict is on [#3033](#3033 (comment)). ### Review findings, all fixed - **P1 — routing on an `Action` cell alone.** The original text let the untrusted half of the pair authorize a route by itself. Since nothing authenticates a findings file's writer, any component able to write one could have named any already-installed skill and handed it arbitrary rows, with effects bounded neither by `Location` nor by Step 2. Now the crosswalk declaration is necessary and the `Action` cell only corroborates — the trust boundary being that the crosswalk lives in the consuming repo's own docs, **outside** the artifact being consumed, while the `Action` cell is inside it. **Availability is not authentication.** - **P2 — ambiguous plan counts.** The cleanup and producer-owned lines now partition rows explicitly, in both the plan and the report, so a plan can no longer print `Cleanup-class (14) → /simplify` for rows that never reach `/simplify`. - **P2 — the producer steering operators off the new route.** `ai-slop`'s `SKILL.md`, corrected above. ## Related - #3031 narrowed `ai-slop`'s relay claim to what was then true, and left this gap open on purpose. - #3041 / #3063 backed the `ai-slop` audit evals with committed fixtures; `ai-slop` 0.3.1 here builds on the 0.3.0 that shipped there. - `docs/conventions/detector-findings/README.md` — "The severity crosswalk", the `Auto-applicable` column, "When the remediation is not at `Location`" (the precedent this is argued against, whose rejected remediation-target column is why no column was added here), and the Adopters table. - `plugins/review/skills/fanout/context/fix-pass-mode.md` — the consumer algorithm this teaches to honor the declaration. ### Follow-up, deliberately not in this PR Two `ai-slop` items are tracked separately and nothing here depends on either. `emit-findings.sh` writes `date: …T05-45-10Z` — hyphens where ISO-8601 needs colons in the time portion; Step 1 classes that UNREADABLE and fails **open**, so it costs at most one extra pass and drops nothing. And `docs/upstream/cursor-pstack.md` carries no `unslop` row, although it calls itself the single source of truth for everything derived from that upstream collection and its recheck trigger only covers rows its table names. Merge ordering, disclosed rather than closed: the relay's only contract URL points at `main`, so the new crosswalk cells are not visible to a live relay until this merges. --------- Co-authored-by: Claude <noreply@anthropic.com>
…on gaps (#3097) Four corrections surfaced by adversarial verification on #3063 and #3068. emit-findings.sh wrote date: with hyphens where ISO-8601 needs colons in the time portion, so the value parsed nowhere. The consumer classes an unparseable date as UNREADABLE and fails open, so it cost one extra pass and dropped nothing, but the staleness check degraded silently. The bug was guarded by a passing test asserting the hyphenated form; the replacement is discriminating, failing against the old emitter output. Two live passages described the relay's cleanup route as handing the class wholesale to /simplify, omitting the branch that applies rows directly when /simplify is absent. Two further occurrences sit in published changelog entries and are left as written, with the correction noted in the new entry instead. A contract fence sentence read unconditionally while binding the correctness class only; now scoped, with the class-independent half preserved verbatim. It predates #3068 by six days. ai-slop eval case 2 grades findings-file persistence but lacked the tracked-file premise its siblings carry. Premise added. Whether tracked-ness is a substantive gate on the fix flow's re-emit step is recorded as an open question with the seam named, rather than closed by an argument the cited text does not support. ai-slop 0.3.3, detector-findings 2.4.1.
…enerally The paragraph asserted that nothing executes an eval prompt, and told authors not to assume a runner would stage anything. Both overreach. This repo's own surfaces are scoped more carefully: plugins/evals/skills/design/SKILL.md and that plugin's README say no MARKETPLACE command runs model-graded evals, and name Anthropic's skill-creator as a runner a consumer may have installed — the runner belonging to this very format. Left as written, the guidance would have pushed authors toward cases that are incompatible with skill-creator's staging, which is the opposite of the intent. The corrected text keeps what is true and load-bearing — nothing here runs these, so a case must be followable by hand — and replaces the false half with the actual rule the #3063 apparatus violated: declare fixtures in files[] and reference them by their documented path, rather than hand-rolling staging in the prompt string. A prompt that builds its own workspace is neither followable by hand nor compatible with the runner that would otherwise stage it. Found by review on #3102. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… eval`'s (#3102) The Evals section already stated that no first-party command executes model-graded evals, but never said which format the 203 shipped suites are in or how it relates to the `claude plugin eval` command — so a reader meeting that command reasonably reads them as drift. They are not: the shipped format is skill-creator's, which is why the schema notes upstream's `assertions` naming, and it is the ecosystem-wide shape. The command consumes a different layout and is early access, refusing to run, so adopting it would trade a corpus CI checks for one nobody here can execute. Adoption stays deferred behind the same tracker as the runner. The authoring consequence is scoped precisely, after review caught an overclaim in the first draft: nothing in THIS marketplace or THIS CI executes a prompt, so a case must be followable by hand — but a consumer with skill-creator installed can run these suites, and it stages a case's files[]. So declare fixtures in files[] rather than hand-rolling staging in the prompt string. A prompt that builds its own workspace is neither followable by hand nor compatible with the runner that would otherwise stage it. Also reconciles write-for-agents-brief.md, whose acceptance criterion named a `claude plugin eval` suite as its settling artifact while the skill shipped with the house format. Found while diagnosing #3063, where three rounds of work built machinery to make an eval prompt literally executable before establishing what actually runs one.
2 participants
Closes #3041
Summary
ai-slop's audit eval cases described their input in prose inside thepromptstring. Nothing checked that the described input produced the finding the case graded, and it disagreed with the detector three times in one PR — each time a golden answer the scenario could not produce. Seven of the nine cases now name a committed fixture instead.Fix
Six fixtures ship under
plugins/ai-slop/skills/audit/evals/fixtures/, referenced from each case'sfilesarray:report-only.mdfix-guarded-rewrite.mdrubric-boundary.mdem-dash-substitution.mdtriads.mdrule-rule-of-three, density 43.5/1000 (3 hits in 69 words, threshold 3.0)knowledge-cutoff-prose.mdrule-knowledge-cutoff-disclaimerfindings (L1, L3, L5), all in the recorded false-positive classEvery
expected_outputnames the rules, lines and fired thresholds the detector emits, measured rather than asserted. Case 7's three triads are all single-token, so the load-bearing one genuinely reaches the fix flow — the third of the three original defects was a fix whose multi-word items never matched the rule's ERE.A fixture is named the way every sibling suite names one:
files[]plus a prose prompt pointing atevals/fixtures/<name>.mdrelative to the skill directory, the same formmcp-tools:auditanddocs-hygiene:compressuse. Eighteen other fixture-backed suites do this and none builds a repository to audit in.Three
fixcases carry a graded copy instruction.fixrewrites in place, so cases 2, 6 and 7 say to work on a copy, and each asserts the committed fixture is left byte-identical after the run — graded on the outcome, not on how the copy is made.Cases 4 and 5 state their tracked-file premise ("treat the audited file as tracked in the repo under audit") rather than constructing repository state for it.
SKILL.mdgates findings-file persistence on the audit having "examined tracked files", so trackedness is a precondition for reaching the branch, not the claim under test — what those cases grade is the decision: that persistence applies, that the producer contract is fetched before anything is written, that the findings file is written, and that a refusal is attributed to an unreachable contract rather than to the target being out of tracked space.Cases 3 and 9 keep
narration: true: they grade repo-wide flow and consuming-repo config, not file content, so there is nothing for a fixture to pin.The 0.1.0 no-fixtures decision is reversed, in the header that recorded it. It still holds for the unit suite (fixtures stay inline in a tmpdir); it does not survive the eval suite, where a case is graded downstream of a deterministic detector run and its scenario has to satisfy an ERE the eval author cannot see.
Exclusion via
excluded_paths, not an in-file marker. A file marker declines unconditionally — including under the emptyHOME+CLAUDE_PROJECT_DIRisolation the detector's own tests use — sodetect.sh <fixture>would print nothing and the eval author would be back to trusting prose. Anexcluded_pathsentry is a config layer, and that isolation lifts it. This repo's.claude/ai-slop.jsoncarries the glob as the worked example, with the reasoning in_comment.Consuming repos need no exclusion of their own: the audit scans
git ls-files '*.md', and an installed plugin's files are not tracked in the repo that installs it.Why the prompts are a specification and not a script
Three intermediate commits built staging into the
promptstrings —git inita scratch repo, an identity-scoped commit, pinnedCLAUDE_PROJECT_DIRandHOME,unset GIT_DIR. That apparatus is gone, and the reason is worth recording so it is not rebuilt.claude plugin eval --helpruns<eval dir>/**/case.yamlorprompt.md+graders/*.md. This repo has zero of that layout against 200evals.jsonfiles, no plugin manifest declaresexperimental.evals, and the only things that readevals.jsonare lint scripts. Nothing executes these prompts. Every defect the staging apparatus accumulated — a missing git identity, a leaked config cascade,$CLAUDE_SKILL_DIRnot being an exported variable, an unspecified working directory,GIT_DIRsurviving into the graded commands, and aVAR=… detect.shform that cannot match the skill's own literal-prefixallowed-toolsgrant — was a property of executing something nothing executes.If this repo ever adopts that layout, staging has a documented home: a case's
scaffold_script, run under--scaffold. It never belonged inside a prompt string.Verification
Every fixture was measured against the shipped defaults (
HOMEandCLAUDE_PROJECT_DIRpinned to an empty dir, confirmed via--show-configreporting(none; bundled defaults)) before its case was written — the table above is that output, not a description of it.detect.sh <fixture>× 6, isolated configexpected_outputexactlytriads.mdfigures recomputed from sourcedetect.sh evals/fixtures/with this repo's configcause=excluded-glob)scripts/check-orphaned-fixtures.sh --checkCONSUMEDcheck-evals-quality.shon this filecheck-jsonschemavsevals.schema.jsondetect.test.shcheck-changelog-parity.sh --check-bump origin/maincheck-changed-skills.sh origin/mainshellcheck detect.test.shgit init,mktemp,unset GIT,CLAUDE_*_DIR,scratch,docs/, …)Fresh-context adversarial passes reviewed each head with instructions to refute rather than confirm, and found the defects the automated lanes did not. The final pass confirms the apparatus is gone with no residue, all six fixtures reconcile figure-for-figure, nothing was weakened against the pre-apparatus baseline, and every CHANGELOG claim re-derives.
Related