Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion plugins/claude-config/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
"name": "claude-config",
"version": "0.9.0",
"version": "0.9.1",
"description": "Four audit skills for a repo's Claude Code configuration: audit (settings.json / .mcp.json / hooks / plugins / permissions drift), audit-automation-gaps (evidence-gated verdicts on automation gaps), audit-permission-grants (allow-rule / allowed-tools grants for auto-mode durability and portability), and audit-instructions (locally-owned instruction surfaces vs current model capability — proposes removals/rewrites of instructions the model no longer needs).",
"author": {
"name": "Melodic Software",
Expand Down
10 changes: 10 additions & 0 deletions plugins/claude-config/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,16 @@
All notable changes to the `claude-config` plugin are documented here. Format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning.

## [0.9.1]

### Added

- **`audit-instructions` eval: `step-list-culled-not-preserved`.** Exercises check I8's
step-list nuance: a mechanical numbered procedure is culled to intent plus hard constraints
(genuine ordering, safety gates, external contracts kept) rather than preserved verbatim.
Absorbed from the superseded `audit-model-fit` suite (its C2 analog), per the follow-up
material recorded when 0.9.0 removed that skill.

## [0.9.0]

### Added
Expand Down
14 changes: 14 additions & 0 deletions plugins/claude-config/skills/audit-instructions/evals/evals.json
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,20 @@
"Demotes a proposal the verifier defends to info or drops it, rather than presenting it as a confident removal",
"Does not present a refuted or behavioral-tier removal as an error or an applied change"
]
},
{
"id": 6,
"name": "step-list-culled-not-preserved",
"prompt": "/claude-config:audit-instructions one of my skills spells out a 9-step numbered procedure for opening a PR: 1. run git status to see what changed; 2. run git diff to review the changes; 3. stage the files with git add; 4. write a commit message; 5. run the test suite and stop if it fails; 6. commit; 7. push the branch; 8. title the PR per Conventional Commits (CI rejects other titles); 9. open the PR with gh pr create. Should a capable model keep all the steps?",
"expected_output": "Flags the mechanical numbered procedure as an I8 model-era finding (over-prescriptive step list) and proposes culling it to the intent plus the hard constraints, removing the narration steps (status/diff/add/message/commit/push — a procedure a current model already sequences correctly) while keeping the steps that encode a real safety gate or external contract: run-tests-and-stop-on-failure (step 5) and the Conventional Commits PR-title requirement CI enforces (step 8). It does not preserve all 9 steps verbatim. Because I8 is behavioral-tier, the cull reaches the report as a verify-pass-vetted behavioral proposal, not an unverified confident removal.",
"files": [],
"expectations": [
"Identifies the numbered step list as an I8 over-prescriptive finding",
"Proposes culling to intent plus hard constraints rather than keeping all 9 steps verbatim",
"Keeps the test-gate step (run tests, stop on failure) and the CI-enforced Conventional Commits PR-title step as load-bearing",
"Culls narration steps (git status/diff/add/commit/push) that a current model sequences correctly from intent",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Cover every culled step in this eval

Fresh evidence since the prior comment: the full 9-step list is now present, but this cull assertion still only requires removing status/diff/add/commit/push. A response can preserve step 4 (write a commit message) and step 9 (open the PR with gh pr create) while satisfying the objectively-verifiable expectations, even though line 73 names only steps 5 and 8 as load-bearing; that leaves the case unable to catch a partial step-list trim regression.

Useful? React with 👍 / 👎.

"Surfaces the cull as a behavioral proposal that passed the fresh-context verify pass, not an unverified confident removal"
]
}
]
}
Loading