diff --git a/plugins/claude-memory/.claude-plugin/plugin.json b/plugins/claude-memory/.claude-plugin/plugin.json index 4d141e557..e052d40bd 100644 --- a/plugins/claude-memory/.claude-plugin/plugin.json +++ b/plugins/claude-memory/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "claude-memory", - "version": "0.5.2", + "version": "0.5.3", "description": "Keeps a repo's Claude Code memory layer healthy and under your control, against criteria derived from official Claude Code documentation. The audit skill checks the instruction/memory layer (CLAUDE.md, CLAUDE.local.md, .claude/rules/, auto-memory) with a deterministic script-backed spine plus judgment-tier checks. The stateless skill inspects, disables, and (confirm-gated) purges Claude-written auto memory across all settings scopes.", "author": { "name": "Melodic Software", diff --git a/plugins/claude-memory/CHANGELOG.md b/plugins/claude-memory/CHANGELOG.md index c9ee1670d..99ec07dc7 100644 --- a/plugins/claude-memory/CHANGELOG.md +++ b/plugins/claude-memory/CHANGELOG.md @@ -3,6 +3,39 @@ All notable changes to the `claude-memory` plugin are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning. +## [0.5.3] + +### Added + +- **`audit` gains C9, a check that a project CLAUDE.md states the repo's exact build and test + commands** (claude-memory 0.5.2 → 0.5.3; criteria 1.3.0 → 1.4.0). It is the only CLAUDE.md check + that looks for *missing* content — C4 asks whether an instruction that exists is concrete enough + to verify, C5 whether it should have been cut, and neither asks whether the commands are there at + all. Official memory guidance lists build and test commands first among what project memory is + for, and `/init` populates them by analyzing the codebase, so without the statement they are + inferred every session rather than read. Two severities, following C6 and C7's pattern of heading + a check at its higher branch: FAIL for a stated command the repo's own manifest does not have, + which is C7's wrong-reference class and worse than an absent one (Claude runs it and the check + fails for the wrong reason); WARN for a command that is absent or given only as prose naming the + tool ("we use pytest" is not a command). Never flags a repo that genuinely has no build or test + step. Scoped to project CLAUDE.md — skipped for CLAUDE.local.md and personal files, which are not + repo-scoped. + + **A step 0 keeps the check honest against its own source.** The memory page states both halves of + a tension: project memory is for "build and test commands", while the same page's + CLAUDE.md-vs-auto-memory table puts "Build commands" in the *auto memory* column. So the check + asks first whether the commands are stated on any loaded surface — nested CLAUDE.md, path-scoped + rule, or auto memory — and treats a yes as a C3 placement question rather than a C9 finding. The + requirement is that the commands be reachable, not that they sit in one file. Without this, C9 + would flag a repo for following the other half of the page it cites. + + Boundary with C7 stated explicitly so a run does not double-report: C7 owns file paths, version + pins, and counts; C9 owns whether the command itself runs. Backed by a new "Build and test + commands" section in `reference/official-guidance.md` carrying the sourced quotes the skill's own + determinism contract requires, and by a new eval covering the wrong-command FAIL, the step-0 + carve-out, and the no-double-report rule. + The applicability ranges in `SKILL.md` and `context/audit.md` move to C9 accordingly. + ## [0.5.2] ### Fixed diff --git a/plugins/claude-memory/skills/audit/SKILL.md b/plugins/claude-memory/skills/audit/SKILL.md index c71667d24..b8eb1beba 100644 --- a/plugins/claude-memory/skills/audit/SKILL.md +++ b/plugins/claude-memory/skills/audit/SKILL.md @@ -62,7 +62,7 @@ operator can weigh it against current official prompting guidance. The checklist at [reference/criteria.md](reference/criteria.md) is codified, not a subjective rubric. Its **deterministic spine** (C1 line budget, M1 index size, the script-backed M2 index integrity and RD1 orphan-rule checks) yields byte-identical findings on the same repo state; its **judgment tier** -(C2-C8, R1-R4, M3-M4) applies fixed criteria with model reading, so findings vary in wording though +(C2-C9, R1-R4, M3-M4) applies fixed criteria with model reading, so findings vary in wording though not in criteria — label those "judgment candidate" in the report. Criteria derive from official Claude Code documentation (sourced quotes in [reference/official-guidance.md](reference/official-guidance.md)); refresh both via the `update` action. diff --git a/plugins/claude-memory/skills/audit/context/audit.md b/plugins/claude-memory/skills/audit/context/audit.md index 5471f9501..e81ed016c 100644 --- a/plugins/claude-memory/skills/audit/context/audit.md +++ b/plugins/claude-memory/skills/audit/context/audit.md @@ -28,7 +28,7 @@ For each file found, record: path, line count, visible line count (excluding HTM Read [../reference/criteria.md](../reference/criteria.md), then execute every applicable check against each discovered file. Apply by entity type: -- **C1-C8**: CLAUDE.md and CLAUDE.local.md +- **C1-C9**: CLAUDE.md and CLAUDE.local.md (C9 is project-scoped — skip for CLAUDE.local.md) - **R1-R4**: `.claude/rules/` files - **C7/R3 (currency)**: version pins and counts are checked against the repo's own pin files (`global.json`, `.nvmrc`, `.python-version`, `.mcp.json`, or ecosystem equivalents). File-path-existence diff --git a/plugins/claude-memory/skills/audit/evals/evals.json b/plugins/claude-memory/skills/audit/evals/evals.json index 83636aa91..316795778 100644 --- a/plugins/claude-memory/skills/audit/evals/evals.json +++ b/plugins/claude-memory/skills/audit/evals/evals.json @@ -84,6 +84,18 @@ "States that @path imports do NOT reduce context because imported files load at launch, and does not propose an import split as a saving", "Names a load-deferring destination (skill or path-scoped rule) for the only-sometimes-relevant Terraform walkthrough, justified by relevance rather than by the line budget, and states the post-compaction cost of that destination alongside the recommendation" ] + }, + { + "id": 8, + "name": "build-test-commands-present-and-reachable", + "prompt": "Audit my memory health. My root CLAUDE.md says \"we use pnpm and vitest\" but doesn't spell out the invocations. It also says to build with `pnpm run compile`, though package.json only defines `build`, `test`, and `lint`. Separately, our actual build and test commands are already recorded in auto memory from when you worked them out.", + "expected_output": "C9 FAILs the `pnpm run compile` line, because package.json defines no `compile` script and a wrong command is worse than an absent one. It does NOT WARN that build and test commands are missing, because step 0 finds them stated on another loaded surface (auto memory), which makes their location a C3 placement question rather than a C9 finding. The bare \"we use pnpm and vitest\" prose is identified as naming tools without invocations. The wrong command is reported once, under C9 rather than also under C7, since C7 owns file paths, version pins, and counts while C9 owns command correctness.", + "files": [], + "expectations": [ + "FAILs the `pnpm run compile` command because package.json does not define that script", + "Does NOT report build/test commands as absent, because step 0 finds them on another loaded surface (auto memory) and routes location to C3", + "Reports the wrong command once under C9 and does not double-report it under C7" + ] } ] } diff --git a/plugins/claude-memory/skills/audit/reference/criteria.md b/plugins/claude-memory/skills/audit/reference/criteria.md index 12a782df4..8f0da7284 100644 --- a/plugins/claude-memory/skills/audit/reference/criteria.md +++ b/plugins/claude-memory/skills/audit/reference/criteria.md @@ -1,7 +1,7 @@ # Memory Health Criteria -Version: 1.3.0 -Last updated: 2026-07-25 +Version: 1.4.0 +Last updated: 2026-08-04 Source: Official Claude Code docs (code.claude.com/docs/en/memory, code.claude.com/docs/en/best-practices, code.claude.com/docs/en/sub-agents, code.claude.com/docs/en/skills) This file defines every check the audit runs. Each check has a severity, description, and instructions @@ -204,6 +204,48 @@ instead)." **Why**: Prefer deterministic enforcement over documentation — when a guideline can become a compile-time or runtime check, that is the stronger default. +### C9: Build and Test Commands Present [FAIL] + +**What**: Does a project CLAUDE.md state the repo's exact build and test commands, and are the +commands it states correct? + +The only CLAUDE.md check that looks for missing or wrong content rather than surplus — C4 asks +whether an instruction that exists is concrete, C5 whether it should have been cut. Applies to +project CLAUDE.md only; skip for CLAUDE.local.md and for personal (`~/.claude/CLAUDE.md`) files, +which are not repo-scoped. + +**How to check**: + +0. First ask whether the commands are stated on another loaded surface — a nested CLAUDE.md, a + path-scoped rule, or auto memory. If they are, this is a C3 placement question, not a C9 + finding for ABSENCE: do not WARN that CLAUDE.md omits them. The carve-out suppresses only the + absence branch — any command CLAUDE.md itself still states goes through steps 2-3 regardless, + because a stale stated command misleads whether or not a correct one exists elsewhere +1. Look for the repo's build and test invocations stated as runnable commands +2. Verify each stated command against the repo's own manifest or task runner (`package.json` + scripts, `Makefile`, `*.csproj`, `pyproject.toml`, or ecosystem equivalent) +3. FAIL for a stated command that does not exist there — worse than an absent one: Claude runs it + and the check fails for the wrong reason +4. WARN if either command is absent, or if present only as prose naming the tool without the + invocation ("we use pytest" is not a command) +5. Do not flag a repo that has no build or test step; flag only a missing statement of one that exists + +**Boundary with C7.** C7 owns *references* — file paths, version pins, counts. C9 owns *commands*. +A wrong build command is not a C7 finding today, because a command is none of the three things C7 +checks. Report a wrong command under C9 only, and do not double-report it. + +**Why**: Official docs list "build and test commands" first among what project memory is for +(code.claude.com/docs/en/memory), and `/init` populates them by analyzing the codebase — so without +the statement, they are inferred every session rather than read. This check fires on a CLAUDE.md +that exists but omits them. Absent commands make every verification loop start by guessing how to +run the check. + +**Counter-evidence, and why step 0 exists**: the same page's CLAUDE.md-vs-auto-memory table puts +"Build commands" in the *auto memory* column's "Use for" cell, against CLAUDE.md's "Coding +standards, workflows, project architecture". The page states both, so the honest reading is that +the commands must be *reachable*, not that they must sit in CLAUDE.md specifically. Step 0 is what +keeps this check from flagging a repo that followed the other half of the same page. + --- ## Checks for .claude/rules/ files diff --git a/plugins/claude-memory/skills/audit/reference/official-guidance.md b/plugins/claude-memory/skills/audit/reference/official-guidance.md index cdbdb3deb..718dc891f 100644 --- a/plugins/claude-memory/skills/audit/reference/official-guidance.md +++ b/plugins/claude-memory/skills/audit/reference/official-guidance.md @@ -61,6 +61,20 @@ Official include/exclude table (code.claude.com/docs/en/best-practices): | Developer environment quirks (required env vars) | File-by-file descriptions of the codebase | | Common gotchas or non-obvious behaviors | Self-evident practices like "write clean code" | +## Build and test commands + +> "Create this file and add instructions that apply to anyone working on the project: build and test commands, coding standards, architectural decisions, naming conventions, and common workflows." +> — code.claude.com/docs/en/memory, "Project memory" + +Build and test commands lead the list of what project memory is for. The inference cost of omitting +them is stated on the same page, in what `/init` does instead: + +> "Claude analyzes your codebase and creates a file with build commands, test instructions, and project conventions it discovers." +> — code.claude.com/docs/en/memory + +So a project CLAUDE.md that omits them leaves those commands to be discovered per session rather +than read. Backs C9. + ## @import syntax > "CLAUDE.md files can import additional files using `@path/to/import` syntax. Imported files are expanded and loaded into context at launch alongside the CLAUDE.md that references them." diff --git a/plugins/playbooks/.claude-plugin/plugin.json b/plugins/playbooks/.claude-plugin/plugin.json index b6a4f269d..eb02773e3 100644 --- a/plugins/playbooks/.claude-plugin/plugin.json +++ b/plugins/playbooks/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "playbooks", - "version": "0.6.15", + "version": "0.6.16", "description": "Doctrine and knowledge playbooks as on-demand skills, plus a maintainer-facing update skill. boris — Boris Cherny's Claude Code workflow tips (howborisusesclaudecode.com); skill-authoring — Anthropic's internal skill-authoring playbook; fable-5 — Claude Fable 5's operating doctrine (self-authored, no upstream). The boris and skill-authoring packs vendor a verbatim upstream baseline; /playbooks:update drift-checks and syncs those baselines centrally (maintainers).", "author": { "name": "Melodic Software", diff --git a/plugins/playbooks/CHANGELOG.md b/plugins/playbooks/CHANGELOG.md index 024b63ec0..fb350505d 100644 --- a/plugins/playbooks/CHANGELOG.md +++ b/plugins/playbooks/CHANGELOG.md @@ -4,6 +4,46 @@ All notable changes to the `playbooks` plugin are recorded here. The `version` i `.claude-plugin/plugin.json` is the delivery vehicle — a consumer receives a change only after that version increases. +## [0.6.16] + +### Added + +- **`skill-authoring`: a spoke on verification loops in skills** (playbooks 0.6.15 → 0.6.16), at + `reference/verification-loops-in-skills.md`, reached from one new SKILL.md section. It answers + three questions the upstream playbook leaves open once a skill's job is checking work rather than + producing it. + + **Three creation routes, not two.** Anthropic's verification-loops blog post offers hand-writing + and the `skill-creator` plugin. The platform's skill-authoring best-practices page documents a + third — ask Claude directly — and explicitly disclaims needing a dedicated skill-writing skill. + The spoke ranks it ahead of the plugin the post reaches for first, on the narrow ground that it + needs no install — not on any claim that the plugin is undocumented. Creation via `skill-creator`, + including the interview flow, is documented first-party by that plugin's own README and + `SKILL.md`, which carries an "Interview and Research" step; the harness *skills page* is what + covers only the eval loop. + + **The plugin invocation is written namespaced, for a narrower reason than it appears.** The post + shows a bare `/skill-creator`. Both the plugin-namespaced and directory-scoped forms bare-resolve; + the difference is that the plugin one is **conditional** — the bare name also invokes the skill + unless another command already uses that name, and a plugin copy and a same-named original both + stay reachable rather than one overriding the other. So the qualified form is preferred because it + is unconditional, not because the bare one fails. Recorded as current behavior: before v2.1.216 a + frontmatter `name` replaced the whole command name. + + **Shadowing is a documented third route the post omits.** The post rules bundled and + plugin-managed skills off-limits for embedding a check, leaving chaining as the only alternative. + A same-name skill at project or personal level *replaces* a bundled one. The spoke presents it + with its actual semantics — replace, not extend, so you inherit the whole behavior and stop + receiving upstream improvements — which is the trade against chaining. + + **Embed-failure diagnosis leads with the documented cause.** When an appended check silently does + not run, the platform's answer is insufficient prominence or wording, and a linked step may need a + more explicit reference. The post instead attributes it to the skill's description or earlier + instructions; no reference page states that, so it is carried as a second hypothesis. Leading with + it sends readers to the frontmatter when the documented cause is usually the body. The spoke also + separates this failure from a skill that never surfaced at all, which is a different failure with + a different remedy (`/discipline:use-your-skills`). + ## [0.6.15] ### Fixed diff --git a/plugins/playbooks/skills/skill-authoring/SKILL.md b/plugins/playbooks/skills/skill-authoring/SKILL.md index 06fe6e7f2..78af46c3a 100644 --- a/plugins/playbooks/skills/skill-authoring/SKILL.md +++ b/plugins/playbooks/skills/skill-authoring/SKILL.md @@ -165,4 +165,15 @@ and `shell:` conventions we pin, see [`reference/precompute-context.md`](referen --- +## Verification loops in skills (Melodic Software addition) + +When the skill's job is *checking* work rather than producing it, three questions the playbook above +leaves open: which of the three routes creates the skill (and why the namespaced invocation is the +one that resolves unconditionally), how to attach a check to a bundled or plugin-managed skill you +cannot edit (shadow versus chain), and how to diagnose an embedded check that silently does not run +— documented prominence causes first, the blog's description diagnosis second. See +[`reference/verification-loops-in-skills.md`](reference/verification-loops-in-skills.md). + +--- + Source: [@trq212's March 17, 2026 post](https://x.com/trq212/status/2033949937936085378) diff --git a/plugins/playbooks/skills/skill-authoring/reference/verification-loops-in-skills.md b/plugins/playbooks/skills/skill-authoring/reference/verification-loops-in-skills.md new file mode 100644 index 000000000..20d07c44a --- /dev/null +++ b/plugins/playbooks/skills/skill-authoring/reference/verification-loops-in-skills.md @@ -0,0 +1,108 @@ +# Verification loops in skills + +Locally-owned Melodic Software guidance (not part of the upstream playbook). It covers three +questions the playbook leaves open once a skill's job is *checking* work: which route creates the +skill, how to attach a check to a skill you do not own, and what to do when an embedded check +silently does not run. + +It does not restate skill syntax, frontmatter, or invocation rules — the authoritative references +are [Skills](https://code.claude.com/docs/en/skills) (harness) and +[Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices) +(platform). Read those for the schema. + +Provenance, as this file uses the term: a claim is called **vendor-claimed** here when Anthropic's +[verification-loops blog post](https://claude.com/blog/building-verification-loops-in-claude-code-with-skills) +states it and the harness and platform reference pages do not, checked 2026-08-04. That is a local +reading convention for this file, not a repo-wide marker. Treat such lines as vendor guidance worth +adopting as convention, not as documented harness behavior. First-party sources outside those two +reference properties — a plugin's own README, for instance — are cited where they settle a point and +named as what they are. + +## Three routes to create the skill, not two + +| Route | Status | Use it when | +|---|---|---| +| **Hand-write `SKILL.md`** | Documented end to end — locations, frontmatter, walkthrough ([Skills](https://code.claude.com/docs/en/skills)) | Default. You know the shape you want. | +| **Ask Claude directly** | Documented. The platform states Claude generates a properly structured `SKILL.md` natively and explicitly disclaims needing a dedicated skill-writing skill ([Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices)) | You want a draft from a description, with no plugin dependency. | +| **`skill-creator` plugin** | Creation, including the interview flow, is documented first-party by the plugin's own README and `SKILL.md`, which carries an "Interview and Research" step. The harness *skills page* covers only its eval loop ([Skills — run evals with skill-creator](https://code.claude.com/docs/en/skills#run-evals-with-skill-creator)) | You want the plugin to interview you and elicit the procedure. | + +The blog reaches for the plugin first. The middle route needs no install, so prefer it before adding +a dependency — not because the plugin is undocumented, but because a dependency should earn itself. + +### Write the invocation namespaced + +Write the plugin route `/skill-creator:skill-creator` rather than the bare `/skill-creator` the blog +shows — but for a narrower reason than it first appears. + +**Both forms bare-resolve. The difference is that one is conditional:** + +- **Plugin namespace** (`plugin-name:skill-name`): the qualified form always works, and the bare name + *also* invokes the skill **unless another command already uses that name**. Where a name is taken, + the bare token keeps belonging to the incumbent and the namespaced form becomes the plugin skill's + only command — which is why namespacing means plugin skills cannot collide, and why a plugin copy + and a same-named original both stay reachable rather than one overriding the other + ([Skills — how a skill gets its command name](https://code.claude.com/docs/en/skills#how-a-skill-gets-its-command-name), + [Plugins](https://code.claude.com/docs/en/plugins)). Note this is current behavior: before + v2.1.216 a frontmatter `name` replaced the whole command name. +- **Directory-scoped namespace** (`apps/web:deploy`): the bare name resolves to the project-root + variant, and the qualified form reaches the nested one + ([Skills — where skills live](https://code.claude.com/docs/en/skills#where-skills-live)). + +So the bare plugin form is not wrong — it is **contingent on no other command claiming the name**, +which is a condition you do not control and cannot see from inside your own repo. Write the +qualified form because it is unconditional, not because the bare one fails. + +## Attaching a check to a skill you do not own + +Editing the producing skill's body is the simplest way to make a check fire automatically — but only +where you own the file. Two cases where you do not, and they have different answers: + +- **Plugin-managed skills.** Edits are lost: the plugin root is replaced on update. Do not edit. +- **Bundled skills.** The blog calls these off-limits and offers chaining as the only alternative. + **That is incomplete.** A same-name skill at project or personal level *replaces* a bundled one — + a `code-review` skill in `.claude/skills/` replaces the bundled `/code-review` + ([Skills](https://code.claude.com/docs/en/skills)). + +Shadowing **replaces, it does not extend**. You inherit maintenance of the whole behavior, and you +stop receiving upstream improvements to the bundled version. That is the trade against chaining, +which leaves the original intact and adds a wrapper around it. Pick shadowing when you want the +bundled behavior *changed*; pick chaining when you want it *followed by* something. + +"Chaining" names three different things across first-party sources — the blog's sense (one skill's +body invoking another at its end), the harness's sense (several skills invoked in one user message, +[Slash commands](https://code.claude.com/docs/en/commands)), and the platform's combining of Skills +for one multi-step task ([Agent Skills overview](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview)). +Say which you mean. + +## When the embedded step does not run + +Verify an embed by running the producing skill and confirming the added step actually fires — on +**real work, not a test scenario**, which is the platform's own instruction and the sharper form of +the blog's "invoke it on a new task" +([Skill authoring best practices — "Develop Skills iteratively with Claude"](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices)). +A contrived case exercises the step you are watching for and hides the salience problem that only +shows up when the skill is competing with a real task's context. + +When the step does not fire, work the documented diagnosis first: + +1. **Prominence and wording** — the platform's own answer. A rule the skill states but Claude skips + is treated as not prominent enough or not strong enough: reorganize so it stands out, strengthen + the language, or restructure the surrounding section (same page and section). +2. **Reference not followed** — if the step lives in a linked file rather than inline, the link + itself may need to be more explicit or prominent (same page, "Observe how Claude navigates + Skills"). +3. **Description or earlier instructions not pulling the check in** — *vendor-claimed*. The blog + attributes a non-firing embed to the skill's description or its earlier instructions. No + reference page states this diagnosis; it is a second hypothesis, not the first move. + +Leading with (3) misdirects: it sends you to the frontmatter when the documented cause is usually the +body. Work 1 and 2, then 3. + +**Do not confuse this with a skill that never surfaced at all.** An appended step that did not run is +a skill that *did* load and skipped an instruction. A skill that did not trigger is a different +failure with more than one owner: a description that does not match how the work is phrased is +skill-authoring QA (`/skill-quality:check`, if installed), a listing entry dropped by the shared +description budget is a configuration question (`/claude-config:audit`, if installed), and the habit +of consulting the listing at all has its own corrector (`/discipline:use-your-skills`, if +installed). Different failure, different remedy — and each diagnostic resolves only where its +plugin is present.