From ea729272709288dc9aab13e46a7ce44e604c8270 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 12 Jul 2026 00:14:36 -0400 Subject: [PATCH 1/3] feat(codebase-audit): add repo-wide drift-audit plugin Author + ship the codebase-audit plugin (category: quality) from medley's in-repo skill. Verifies factual claims in docs/config/code/architecture against reality via per-file subagent fan-out with independent validation. De-couples all medley path/convention assumptions: audit dimensions and their primary/verification/example-claim targets now flow through a tracked .claude/codebase-audit.md config seam (extensibility contract v2.1 seam 2, additive user-global -> team -> local overlay resolution), written by a re-runnable /codebase-audit:setup skill per the convention-resolution ladder. Refs melodic-software/medley#1376 --- .claude-plugin/marketplace.json | 6 + README.md | 1 + .../codebase-audit/.claude-plugin/plugin.json | 12 + plugins/codebase-audit/README.md | 85 ++++++ .../skills/codebase-audit/SKILL.md | 246 ++++++++++++++++++ .../context/discovery-method.md | 85 ++++++ .../skills/codebase-audit/evals/evals.json | 66 +++++ .../reference/audit-checklist.md | 60 +++++ .../reference/category-playbook.md | 38 +++ .../codebase-audit/templates/checklist.md | 19 ++ plugins/codebase-audit/skills/setup/SKILL.md | 53 ++++ .../skills/setup/evals/evals.json | 41 +++ .../skills/setup/templates/config-template.md | 78 ++++++ 13 files changed, 790 insertions(+) create mode 100644 plugins/codebase-audit/.claude-plugin/plugin.json create mode 100644 plugins/codebase-audit/README.md create mode 100644 plugins/codebase-audit/skills/codebase-audit/SKILL.md create mode 100644 plugins/codebase-audit/skills/codebase-audit/context/discovery-method.md create mode 100644 plugins/codebase-audit/skills/codebase-audit/evals/evals.json create mode 100644 plugins/codebase-audit/skills/codebase-audit/reference/audit-checklist.md create mode 100644 plugins/codebase-audit/skills/codebase-audit/reference/category-playbook.md create mode 100644 plugins/codebase-audit/skills/codebase-audit/templates/checklist.md create mode 100644 plugins/codebase-audit/skills/setup/SKILL.md create mode 100644 plugins/codebase-audit/skills/setup/evals/evals.json create mode 100644 plugins/codebase-audit/skills/setup/templates/config-template.md diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index aa1c979d9..757f39dd3 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -210,6 +210,12 @@ "source": "./plugins/implementation", "category": "development", "tags": ["implementation", "build", "lint", "testing", "tdd", "e2e", "verification", "skill"] + }, + { + "name": "codebase-audit", + "source": "./plugins/codebase-audit", + "category": "quality", + "tags": ["audit", "drift", "documentation", "configuration", "architecture", "verification", "claims", "skill"] } ] } diff --git a/README.md b/README.md index 435c5409a..b7b72b81f 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,7 @@ Browse and manage with `/plugin`. To refresh after updates: `/plugin marketplace | [`planning`](plugins/planning) | Skills | Pre-implementation planning pipeline of six skills: `/planning:brainstorm` (cheapest→most-ambitious candidate divergence), `/planning:prd` (three-tier product requirements), `/planning:interview` (depth-first Q&A locking a task contract into a PLAN.md Brief), `/planning:design` (collaborative type/contract/topology exploration with a binary handoff gate), `/planning:devils-advocate` (evidence-backed adversarial stress-testing), and `/planning:architect` (structured implementation plans with blast radius, parallelism analysis, and a user approval gate). | | [`review-toolkit`](plugins/review-toolkit) | Agents + Skills | Code-review toolkit: six read-only reviewer agents (code quality, security, architecture, doc drift, build/test/lint, CI-log audit) plus two orchestration skills — `/review-toolkit:quality-gate` (single-lens checkpoint with eight modes) and `/review-toolkit:code-review-fanout` (multi-surface fan-out normalized into one severity-ranked findings report, with a findings-driven fix pass). | | [`implementation`](plugins/implementation) | Skills | Implementation-stage toolkit of ten skills: `implement` / `implement-dispatch` (inline vs orchestrated plan execution with TDD cadence, divergence detection, and phase-boundary handoffs), `build` / `lint` (polyglot ecosystem-detecting verification with consumer-convention overrides), `test-write` / `test-plan` / `test-diagnose` / `test-e2e` (authoring, coverage-gap analysis, failure diagnosis, live E2E evidence), and `verify-changes` / `verify-improvement` (outcome verification and baseline-vs-after improvement measurement). | +| [`codebase-audit`](plugins/codebase-audit) | Skills | Repo-wide drift audit that verifies a codebase's factual claims against reality: `/codebase-audit:codebase-audit` runs an eight-phase per-file subagent fan-out over configurable documentation/configuration/code-quality/architecture dimensions, independently validates each finding, and fixes or reports in a severity-rated table; `/codebase-audit:setup` interviews and writes the tracked `.claude/codebase-audit.md` audit-target config. | Install one: `/plugin install @melodic-software`. diff --git a/plugins/codebase-audit/.claude-plugin/plugin.json b/plugins/codebase-audit/.claude-plugin/plugin.json new file mode 100644 index 000000000..12e21260b --- /dev/null +++ b/plugins/codebase-audit/.claude-plugin/plugin.json @@ -0,0 +1,12 @@ +{ + "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", + "name": "codebase-audit", + "version": "0.1.0", + "description": "Repo-wide drift audit between docs, config, code, and architecture: verifies every factual claim against reality via parallel subagent fan-out, severity-rates findings, and fixes or presents for review. Audit dimensions are configurable through a tracked .claude/codebase-audit.md config file written by the setup skill.", + "author": { + "name": "Melodic Software", + "email": "info@melodicsoftware.com" + }, + "license": "MIT", + "keywords": ["audit", "drift", "documentation", "configuration", "architecture", "verification", "claims", "skill"] +} diff --git a/plugins/codebase-audit/README.md b/plugins/codebase-audit/README.md new file mode 100644 index 000000000..194542065 --- /dev/null +++ b/plugins/codebase-audit/README.md @@ -0,0 +1,85 @@ +# codebase-audit + +A Claude Code plugin for repo-wide drift auditing: it verifies that a codebase's **factual claims** — +in docs, config, code, and architecture notes — still match reality. Every claim is checked against +ground truth via a parallel per-file subagent fan-out, findings are severity-rated, and the audit +either fixes them or presents them for review. + +Distinct from diff/PR review (which judges a change) and from Claude Code configuration audits (which +check `settings.json` / hooks / permissions): this plugin verifies whether the repo's own written +claims about itself are true. + +| Skill | What it does | +|---|---| +| `/codebase-audit:codebase-audit` | Runs the audit — prime conventions, fan out claim-extraction per file, independently validate, severity-rate, then fix or report. | +| `/codebase-audit:setup` | Configures the audit for this repo — interviews the user, infers targets from the layout, and writes the tracked `.claude/codebase-audit.md` config. | + +## The audit + +Eight phases (0–7): prime the repo's conventions, discover via per-file fan-out, independently +validate each finding (a separate agent re-verifies — never self-review), categorize and present in a +severity-rated table with a verified-non-issues proof-of-thoroughness list, then — unless +`--review-only` — fix in priority order, verify against the repo's own gates, self-review, and +retrospect. + +```shell +/codebase-audit:codebase-audit # audit every configured dimension (scope-gated) +/codebase-audit:codebase-audit docs/ --docs-only # one dimension, scoped to a subtree +/codebase-audit:codebase-audit README.md --review-only # scoped, present findings, no fixes +``` + +Dimension filters (`--docs-only`, `--code-only`, `--config-only`, `--arch-only`) are mutually +exclusive. A scope path narrows the file set. An unscoped whole-repo run is gated — the skill +requires a scope, a filter, or explicit confirmation before fanning out, because a full fan-out spans +every doc/config/source file. + +## Configurable audit dimensions + +What the audit reads and how it verifies claims is **not baked in** — it comes from the consuming +repo's tracked config, resolved additively across three layers: + +1. `~/.claude/codebase-audit.md` — user-global base (optional) +2. `.claude/codebase-audit.md` — team config (tracked) +3. `.claude/codebase-audit.local.md` — personal overlay (gitignored) + +Each dimension declares `primary-sources` (globs where claims live), `verification-sources` (globs +where claims are checked against ground truth), and `example-claims` (concrete `{ claim, verify-via }` +rows that teach the extraction pass what drift looks like in THIS repo). The four bundled dimensions +are `documentation`, `configuration`, `code-quality`, and `architecture`; the config may tune their +globs, remove a dimension, or add custom ones. + +When no config is present, the audit infers targets from the repo layout, uses them, and offers to +persist the inference via `/codebase-audit:setup` — so the next run is deterministic. It never +hardcodes a repo's layout. + +```shell +/codebase-audit:setup # interview + write .claude/codebase-audit.md (re-runnable) +``` + +Add `.claude/*.local.*` to your `.gitignore` so personal overlays stay out of version control while +team config stays tracked. + +## Consumer conventions + +Phase 0 reads the consuming repo's own `CLAUDE.md` / `AGENTS.md` / `.claude/rules/` to learn what +"correct" looks like — a claim contradicting those conventions is a finding; one following them is a +verified non-issue. Nothing project-specific is baked into the plugin. + +## Install + +```shell +/plugin marketplace add melodic-software/claude-code-plugins +/plugin install codebase-audit@melodic-software +``` + +## Configuration + +No `userConfig` — audit targets flow through the tracked `.claude/codebase-audit.md` config seam +above (written by `/codebase-audit:setup`). No hooks, no MCP servers, no bundled scripts, no network +calls of its own (Phase 2 may use whatever documentation-research tools your setup provides). State: +the audit reads and writes only the consumer's own files under the scope you give it. + +## License + +MIT (SPDX-License-Identifier: MIT). See the LICENSE file at the root of the +melodic-software/claude-code-plugins repository. diff --git a/plugins/codebase-audit/skills/codebase-audit/SKILL.md b/plugins/codebase-audit/skills/codebase-audit/SKILL.md new file mode 100644 index 000000000..4d08aea31 --- /dev/null +++ b/plugins/codebase-audit/skills/codebase-audit/SKILL.md @@ -0,0 +1,246 @@ +--- +name: codebase-audit +description: "Audit a codebase for drift between docs, config, code, and architecture. Verifies every factual claim against reality via parallel subagent fan-out, severity-rates findings, auto-fixes or presents for review. Use when: 'audit codebase', 'check for drift', 'verify docs', 'full audit'. Flags: `--review-only` (present findings, no auto-fix), `--docs-only`, `--code-only`, `--config-only`, `--arch-only`." +argument-hint: "[scope] [--review-only] [--docs-only|--code-only|--config-only|--arch-only]" +user-invocable: true +disable-model-invocation: false +--- + +## Pre-computed context + +Current branch: !`git branch --show-current 2>/dev/null || echo "unknown"` +Working tree status: !`git status --porcelain 2>/dev/null | head -20 || echo "clean"` +Changed files (staged+unstaged): !`git diff --name-only HEAD 2>/dev/null || echo "none"` + +## Variables + +Arguments: `$ARGUMENTS` + +## Argument Parsing + +Parse `$ARGUMENTS` for: + +- **Scope** (optional): directory or file path to limit the audit (default: entire repo) +- **`--review-only`**: present findings for user approval before fixing; do NOT auto-fix +- **Dimension filters** (optional, mutually exclusive): + - `--docs-only`: only documentation checks + - `--code-only`: only code-quality checks + - `--config-only`: only configuration checks + - `--arch-only`: only architecture checks + +If no filter is specified, audit every active dimension — the Phase 1 per-file fan-out makes +dimension order irrelevant (each file gets its own subagent). Enumerate `primary-sources` across all +active dimensions and dispatch per file. + +## Model auto-invoke default + +When the model invokes this skill without explicit user authorization to auto-fix, treat the run as +**`--review-only`**. Phases 4–7 (auto-fix) run only when the user explicitly requests fixes or omits +`--review-only` with clear fix intent. + +--- + +## Adapting to your environment (graceful degrade) + +This skill is self-contained. Where a phase names an adjacent capability — a build/verify skill, a +code-simplification pass, a quality-gate review — treat it as optional: if your setup provides an +equivalent skill or tool, use it; otherwise follow the inline guidance, which stands on its own. + +Scope boundary with adjacent audit lanes: this skill verifies **factual claims** in docs/config +against code state. Claude Code configuration files (`settings.json`, `.mcp.json`, hooks, +permissions) and automation-landscape gap analysis are different lanes — when the +`claude-config-audit` plugin is installed, route those to `/claude-config-audit:settings-audit` and +`/claude-config-audit:automation-deep-dive`; otherwise state they are out of scope rather than +running claim-extraction over them. + +--- + +## Audit dimensions & targets (tracked config seam) + +Per-dimension audit targets — `primary-sources` (where claims live), `verification-sources` (where +to verify), and `example-claims` (illustrative `{ claim, verify-via }` rows for the claim-extraction +pass) — come from the consuming repo's tracked config, resolved additively across three layers: + +1. `~/.claude/codebase-audit.md` (user-global, optional) +2. `.claude/codebase-audit.md` (team, tracked) +3. `.claude/codebase-audit.local.md` (personal overlay, gitignored) + +The four bundled dimensions are `documentation`, `configuration`, `code-quality`, and +`architecture`; the config may tune their globs, remove a dimension, or add custom ones. Settle +targets by this ladder: + +1. **Config present → use it.** +2. **Absent → infer from the repo** (doc dirs, build manifests, source/test roots, CI workflows), + then **persist the inference** by offering to run `/codebase-audit:setup` — so the next run is + deterministic. +3. **Cannot infer → ask the user**, and offer to persist the answer via setup. +4. **Otherwise → safe generic defaults**: documentation = `docs/**/*.md` + `README.md` + any + agent-instruction files; the other dimensions require inference or config — skip a dimension + you cannot ground rather than guessing. + +Never hardcode a repo layout; read a declared value, infer-and-record, or ask. + +## Emit checklist + +For any audit run (Phases 0-7), copy +`${CLAUDE_PLUGIN_ROOT}/skills/codebase-audit/templates/checklist.md` into wherever the consuming +repo keeps working task notes (or keep it in-response). Tick each phase as completed. Phases 4-7 may +SKIP per `--review-only` mode. + +--- + +## Phase 0: Prime Context + +Before auditing, load what "correct" looks like in this repo: + +1. **Read the consuming repo's `CLAUDE.md` / `AGENTS.md` and `.claude/rules/` files** (where + present) — conventions, naming rules, enforcement expectations. +2. **Resolve the audit config** per the dimension seam above; read the convention files its + `verification-sources` name. + +These define the lens through which findings are evaluated. A claim contradicting repo conventions +is a finding; one following them is a verified non-issue. You cannot make that judgment without +reading conventions first. + +--- + +## Phase 1: Discover + +The goal is exhaustive verification, not sampling. Every factual claim in every relevant file must +be checked against reality. The most common audit failure is skipping items — thoroughness beats +speed. + +Discovery runs as a **parallel subagent fan-out — one agent per primary-source file**, NOT a single +sequential pass (fresh context per file ≈ 2× claim coverage and ~4× drift caught; a single context +skips claims as it fills — the #1 audit failure). Each agent applies the claim-extraction method +(read top-to-bottom → extract every factual claim → verify each independently → record), fenced per +the scope-fencing rules in [context/discovery-method.md](context/discovery-method.md). + +**Scope first (MANDATORY — cost gate):** require a `[scope]` or dimension filter (`--docs-only` +etc.) for large targets; if the enumerated list exceeds ~20 files, confirm with the user before +dispatching. Never fan out the whole repo unprompted — an unscoped run across every doc/config/ +source file costs millions of tokens. + +Full method — claim-extraction steps, the verify-ALL-claims-on-a-line rule, +enumerate/scope/dispatch/collect detail, and the per-finding report format — in +[context/discovery-method.md](context/discovery-method.md). Dimension-specific claim guidance: +[reference/audit-checklist.md](reference/audit-checklist.md). + +--- + +## Phase 2: Validate & Enrich + +Re-read each finding to confirm accuracy. This is the false-positive gate — every finding must +survive scrutiny before being reported. + +**Validate independently, not by self-review.** When Phase 1 ran as a fan-out, dispatch the +false-positive gate as a SEPARATE subagent that re-verifies each finding against the source of +truth — do NOT let the discovering agent grade its own findings. A model re-checking its own work +rubber-stamps it; an independent agent re-reading the doc claim AND the actual code catches both +false positives and miscategorized-but-correct claims. Fence each validator to read-only (its +findings' files + verification-sources). + +### External research (required, not optional) + +Use available documentation-research tools (MCP docs servers, library-docs lookers-up, web search) +to validate findings involving: + +- **Best-practice claims** — is the documented pattern the current recommended approach? +- **Library API claims** — does the method/class/parameter exist in the current version? +- **Configuration behavior** — does the setting do what the docs say? + +Cross-reference research results against the repo's conventions (loaded in Phase 0). External +consensus matters, but repo conventions are the primary lens — a pattern unusual industry-wide may +be intentionally chosen here. + +### False-positive prevention + +**If uncertain, it is NOT a finding.** Ambiguous items go to `needs-review` and are separated from +confirmed findings in output. The cost of a false positive (eroding trust in the audit) exceeds the +cost of missing a marginal issue (catchable next run). + +### Tag findings with confidence + +- **verified**: confirmed by reading source files AND (where applicable) external research +- **likely**: strong evidence, one piece ambiguous — still reported as a finding +- **needs-review**: requires human judgment — separated from confirmed findings in the output + +--- + +## Phase 3: Categorize & Present + +### Group findings per [reference/category-playbook.md](reference/category-playbook.md) + +Fix order matters — see the playbook for why: Config Drift → Missing Enforcement → Code Quality → +Doc Drift. + +### Output format + +Use this exact table with consistent `error`/`warning`/`info` severity: + +| # | Severity | Category | File:Line | Description | Verification | +|---|----------|----------|-----------|-------------|-------------| +| 1 | error | doc-drift | `:` | Doc claims suppression includes rule X but actual list is `Y;Z` | Read `:` | + +### Required sections after the findings table + +1. **Verified non-issues** — every claim checked that turned out correct. This is the thoroughness + proof. Include at least as many verified items as findings. +2. **Drift patterns** — group related findings and identify root causes (e.g., "7 findings trace to + a registration refactor where code was updated but docs weren't") +3. **Fix priority** — recommended fix order per the category playbook +4. **Enforcement escalation** — for each finding, what automated enforcement (formatter, linter, + analyzer, type check, test, git hook, CI gate) could catch this class automatically? + +### Zero-findings outcome + +If the audit finds no discrepancies, report a clean bill of health: + +- Present the **verified non-issues** list as proof of thoroughness (this is the whole point — + showing what was checked) +- State explicitly: "No findings. All claims verified as correct." +- Do NOT invent findings to justify the audit. A clean codebase is the goal, not a guaranteed list + of issues. +- Skip Phases 4-6 (nothing to fix). Proceed directly to Phase 7 (Retrospective) with scope/coverage + observations. + +### Review-only gate + +**If `--review-only`**: present the full report and **STOP**. +**If autonomous**: present the summary count and continue to Phase 4. + +--- + +## Phase 4: Implement / Fix + +Execute fixes in priority order: Config Drift → Missing Enforcement → Code Quality → Doc Drift. + +For code changes: TDD (write/update a failing test first, implement, verify). +For config/doc changes: apply the change, verify the repo's build still passes. + +After all fixes, run a simplification pass over the changed files (via a code-simplification skill +when your setup provides one, otherwise a manual read-through for reuse and altitude cleanups). + +--- + +## Phase 5: Verify + +Run the consuming repo's own verification commands — build, tests, linters — as documented in its +`CLAUDE.md` / contributing docs (or a verify/build skill when installed). + +If any gate fails: diagnose, fix, re-verify. Max 3 iterations. + +--- + +## Phase 6: Review + +Self-review: every planned fix applied, no regressions, new tests cover new behavior, docs match +code, cross-references valid. + +--- + +## Phase 7: Retrospective + +Summary table, enforcement escalation recommendations, process observations (recurring patterns, +healthy areas, scope suggestions — including config gaps worth persisting via +`/codebase-audit:setup`). diff --git a/plugins/codebase-audit/skills/codebase-audit/context/discovery-method.md b/plugins/codebase-audit/skills/codebase-audit/context/discovery-method.md new file mode 100644 index 000000000..97560de66 --- /dev/null +++ b/plugins/codebase-audit/skills/codebase-audit/context/discovery-method.md @@ -0,0 +1,85 @@ +# codebase-audit — Phase 1 discovery method + +The full claim-extraction + per-file fan-out method. The SKILL.md Phase 1 keeps the goal + the +scope-first cost gate; this file carries the method detail, execution steps, scope-fencing, and the +finding-report format. + +## How discovery works + +For each active dimension, follow the **claim-extraction method**: + +1. **Read the file top to bottom.** +2. **Extract every factual claim** — anything asserting something about the codebase (file paths, + tool names, package names, configuration values, convention descriptions, ranges, counts). +3. **Verify each claim** — read the file or run the command that confirms or denies it. +4. **Record the result** — either it matches (verified passing) or it doesn't (finding). + +Not a checklist to browse. Read every line. The audit config's per-dimension `example-claims` rows +(`{ claim: "If a doc asserts X", verify-via: "" }`) are the ones the agent will +actually see in THIS repo — apply the same pattern to every claim encountered: extract the claim, +identify the verification path, run it. + +**Critical: verify ALL claims, not just some.** A single line or bullet list often contains multiple +factual claims. Finding one issue on a line does NOT mean other claims on that line are correct — +verify each one independently. For example: + +- A tools list ("tool A, tool B, tool C, tool D") contains separate claims — verify ALL of them + against the package manifest, not just the first two. +- A dependency rule ("Layer A → Layer B only") is a verifiable claim — read the actual build + manifest to confirm, even if another claim on the same line already has an issue. +- A line with both a file reference AND a factual assertion needs both verified separately. + +## Dimension-specific guidance + +Read [../reference/audit-checklist.md](../reference/audit-checklist.md) for the kinds of claims to +watch for in each dimension. The checklist is a guide for **what to look for**, not a list to check +off. The primary method is always: read the file, extract claims, verify each one. + +## Execution: fan out one subagent per file + +Discovery runs as a **parallel subagent fan-out — one agent per primary-source file**, NOT a single +sequential pass. A single context cannot exhaustively verify many files at once: it skips claims as +context fills (the #1 audit failure named above). One dedicated subagent per file keeps each file's +verification in a fresh, focused context — empirically ~2× the claim coverage and ~4× the drift +caught vs a single sequential agent. + +1. **Enumerate** — expand the active dimensions' `primary-sources` globs (from the resolved audit + config) to a concrete file list. `verification-sources` are the read-only ground-truth set every + agent may consult. +2. **Scope first (MANDATORY — cost gate)** — a full unscoped run fans out across every doc/config/ + source file (hundreds of thousands of tokens). REQUIRE a `[scope]` or dimension filter + (`--docs-only` etc.) for large targets. If the enumerated list exceeds ~20 files, confirm scope + with the user before dispatching. Never fan out the whole repo unprompted. +3. **Dispatch** — one subagent per file. Each agent's ALLOWED surface = its ONE assigned file + (read) + all verification-sources (read-only) — verification-sources stay read-only ALLOWED even + when the same path also appears as another dimension's primary-source. FORBIDDEN = every other + primary-source file (except those doubling as verification-sources, per the read-only exception), + any write, any git op. Each agent applies the claim-extraction method above to its file and + returns findings + a verified-count. Use repo-relative paths only (never absolute machine + paths — agents must audit the current worktree). Throttle in waves (≤~16 concurrent); lower-tier + worker models are sufficient and dodge burst overload. +4. **Collect** — aggregate per-file findings + verified counts into the Phase 2 input. + +Because each agent owns one file, the "complete one dimension before the next" sequencing is moot — +there is no shared context to thin out, so dimension order does not matter. + +**Background / unattended variant:** the same per-file fan-out can run as a saved background workflow +(background execution, same-session resume, rerunnable script) instead of in-session subagents — same +discovery, different executor. Reach for the workflow form only when you want a fire-and-forget +periodic audit you can walk away from; the in-session fan-out above is the default for an interactive +audit you are actively driving. + +## What to report + +Report **every discrepancy** found, no matter how small. Also report what you verified as correct — +this proves the audit was thorough and didn't skip files. + +For each finding: + +``` +- file: +- category: +- severity: +- description: +- verification: +``` diff --git a/plugins/codebase-audit/skills/codebase-audit/evals/evals.json b/plugins/codebase-audit/skills/codebase-audit/evals/evals.json new file mode 100644 index 000000000..39074f463 --- /dev/null +++ b/plugins/codebase-audit/skills/codebase-audit/evals/evals.json @@ -0,0 +1,66 @@ +{ + "skill_name": "codebase-audit", + "evals": [ + { + "id": 1, + "name": "scope-cost-gate-unscoped-repo", + "prompt": "/codebase-audit:codebase-audit\n\nAudit the whole codebase for drift between docs, config, code, and architecture.", + "expected_output": "The skill recognizes that an unscoped whole-repo run would fan out one subagent per doc/config/source file (a very large token cost), so it does NOT immediately dispatch discovery. It requires a scope or dimension filter, or confirms scope with the user, before fanning out.", + "files": [], + "expectations": [ + "Does NOT immediately fan out discovery subagents across the whole repository", + "Requires a scope or dimension filter, or asks the user to confirm scope, before dispatching discovery", + "States the cost rationale — a full unscoped fan-out spans every doc/config/source file at very large token cost" + ] + }, + { + "id": 2, + "name": "review-only-gate-stops", + "prompt": "/codebase-audit:codebase-audit README.md --docs-only --review-only", + "expected_output": "Runs claim-extraction scoped to the single named file, presents the full findings report, then STOPS at the review-only gate without applying any fixes (Phases 4-7 skipped).", + "files": [], + "expectations": [ + "Presents the findings report and STOPS without applying any fixes", + "Does not modify, edit, or write any files", + "Scopes the audit to the named file only, not the whole repository" + ] + }, + { + "id": 3, + "name": "resolve-config-or-infer-and-persist", + "prompt": "/codebase-audit:codebase-audit --docs-only", + "expected_output": "Resolves the tracked audit config (.claude/codebase-audit.md and overlays) for the documentation dimension. When it is present, uses its primary/verification sources; when absent, infers targets from the repo layout and offers to persist the inference via /codebase-audit:setup rather than hardcoding a layout.", + "files": [], + "expectations": [ + "Attempts to resolve the tracked audit config before enumerating targets, rather than assuming a fixed repo layout", + "When config is absent, infers documentation targets from the repo and offers to persist them via the setup skill", + "Does not hardcode a specific repository's directory layout" + ] + }, + { + "id": 4, + "name": "scoped-run-output-contract", + "prompt": "/codebase-audit:codebase-audit README.md --docs-only", + "expected_output": "Runs the scoped audit and presents findings in the exact required table with error/warning/info severity, followed by the required sections — verified non-issues, drift patterns, fix priority, and enforcement escalation — with each finding confidence-tagged.", + "files": [], + "expectations": [ + "Presents findings in the required table with Severity, Category, File:Line, Description, and Verification columns using error/warning/info severity", + "Includes a 'Verified non-issues' section listing claims that were checked and found correct", + "Includes an enforcement-escalation assessment naming which automated check could catch each finding", + "Tags findings by confidence (verified / likely / needs-review) and separates needs-review items from confirmed findings" + ] + }, + { + "id": 5, + "name": "clean-scope-no-invented-findings", + "prompt": "/codebase-audit:codebase-audit package.json --config-only\n\nBe thorough — I want to be sure nothing is drifting.", + "expected_output": "Verifies each factual claim in the named config file against reality. When everything checks out, it reports a clean bill of health with the verified non-issues list as proof of thoroughness, and does not fabricate marginal findings to justify the run.", + "files": [], + "expectations": [ + "Does not fabricate findings — reports only discrepancies actually verified against source", + "Presents a verified non-issues list showing what was checked", + "When no discrepancies are found, states explicitly that all claims verified as correct rather than padding with marginal issues" + ] + } + ] +} diff --git a/plugins/codebase-audit/skills/codebase-audit/reference/audit-checklist.md b/plugins/codebase-audit/skills/codebase-audit/reference/audit-checklist.md new file mode 100644 index 000000000..b1f4f66ac --- /dev/null +++ b/plugins/codebase-audit/skills/codebase-audit/reference/audit-checklist.md @@ -0,0 +1,60 @@ +# Audit Discovery Guide + +A guide for what kinds of claims to look for in each dimension. The primary method is always +**exhaustive claim verification**: read every line, extract factual claims, verify each one. This +guide helps you recognize claims you might otherwise skip. + +The four bundled dimensions (documentation / configuration / code-quality / architecture) are +universal. Per-dimension `primary-sources`, `verification-sources`, and `example-claims` for THIS +repo come from the resolved audit config (`.claude/codebase-audit.md` and its overlays). + +## Documentation Claims to Verify + +When reading a doc file, these are claim TYPES that commonly drift: + +- **File/directory references** — "see ``", "in ``" → does that file/directory exist? +- **Package/tool names** — "uses tool X for Y" → is it in the package manifest? Commented out or + active? Referenced by any project? **Check EVERY tool in a bullet list, not just some** — verify + all entries individually. +- **Ranges and counts** — "rules `NNN-MMM`", "N categories", "M libraries" → count + actual items. +- **Dependency claims** — "Layer A → Layer B only", "Module X has ZERO deps" → read the actual + build manifest. +- **Convention descriptions** — "each module exposes ``" → does the actual code follow this + pattern? +- **Tool/installation claims** — "Tool X installed" → run the equivalent ` list` or + check. +- **Suppression claims** — "Rule X is suppressed globally" → read the actual disabled-rules list in + build config, check ALL entries not just the first. +- **Test class/location claims** — "Test class T in project P" → grep for the class, check what + project it's in. +- **Status claims** — "Steps 1-2 implemented, 3-6 planned" → check if any "planned" items are + actually done. +- **API surface claims** — "Type T has methods M1, M2, M3" → verify each method actually exists in + source. Method-level claims are high-risk because methods get renamed, removed, or never + implemented while docs persist. + +## Configuration Claims to Verify + +- **Every documented setting** vs actual file content — don't just spot-check, compare line by line. +- **Undocumented settings** — settings in config files no doc mentions. +- **Cross-file consistency** — does lint config match the convention narrative? Does build config + match the build-config narrative? +- **Section scoping** — config files with sectioned scope (e.g. `.editorconfig` `[*.cs]`) can break + scope unexpectedly. +- **Suppression rationale** — are suppressions documented and justified? + +## Code Quality to Check + +- **SOLID violations** — SRP, OCP, DIP especially (concrete deps, god classes). +- **DRY violations** — duplicated logic across 3+ files. +- **Missing test coverage** — non-trivial libraries without test projects. +- **Assertion library consistency** — all test projects using the same library? +- **Pattern compliance** — do implementations match documented conventions? + +## Architecture to Check + +- **Dependency direction** — read build-manifest files, verify against documented rules. +- **Enforcement gaps** — rules in docs that could be analyzers or tests but aren't. +- **Naming conventions** — interfaces, namespaces, test classes following docs. +- **Analyzer coverage** — help-link URLs valid? identifier range consistent across all docs? diff --git a/plugins/codebase-audit/skills/codebase-audit/reference/category-playbook.md b/plugins/codebase-audit/skills/codebase-audit/reference/category-playbook.md new file mode 100644 index 000000000..9d2ea3e54 --- /dev/null +++ b/plugins/codebase-audit/skills/codebase-audit/reference/category-playbook.md @@ -0,0 +1,38 @@ +# Category Playbook + +## Why fix order matters + +Config issues affect build tooling — fix first. Enforcement prevents regression — add before +refactoring. Docs reflect all changes — update last. + +## Categories + +### Config Drift + +Config files that don't match their documentation or each other. +**Fix:** Update config or docs (config is the behavioral source of truth). + +### Missing Enforcement + +Rules in docs that could be caught automatically. +**Fix:** Implement the check (analyzer, test, formatter/linter rule, git hook, CI gate). + +### Code Quality + +SOLID/DRY violations, missing abstractions, domain-model issues. +**Fix:** Refactor with TDD. + +### Doc Drift + +Documentation that doesn't match code reality. +**Fix:** Update docs to match code. Mark aspirational content as "planned". + +## Priority: Config Drift → Missing Enforcement → Code Quality → Doc Drift + +## Severity (use these consistently — never HIGH/MEDIUM/LOW) + +| Severity | Definition | +|----------|-----------| +| `error` | Incorrect claim, broken reference, contradicts reality, will mislead | +| `warning` | Undocumented setting, inconsistency, stale content | +| `info` | Minor, cosmetic, or improvement opportunity | diff --git a/plugins/codebase-audit/skills/codebase-audit/templates/checklist.md b/plugins/codebase-audit/skills/codebase-audit/templates/checklist.md new file mode 100644 index 000000000..de8d58983 --- /dev/null +++ b/plugins/codebase-audit/skills/codebase-audit/templates/checklist.md @@ -0,0 +1,19 @@ +# codebase-audit Checklist + +Copy into wherever the consuming repo keeps working task notes. Tick as each phase completes. + +## Phases + +- [ ] Phase 0: Prime context — read the repo's conventions (`CLAUDE.md` / `AGENTS.md` / `.claude/rules/`); resolve the audit config and read its verification-sources +- [ ] Phase 1: Discover — per-file subagent fan-out over active dimensions' primary-sources (scope-gated) +- [ ] Phase 2: Validate & enrich — independent re-verification of each finding against source of truth; external research where a claim needs it +- [ ] Phase 3: Categorize & present — findings table with error/warning/info severity + verified non-issues, drift patterns, fix priority, enforcement escalation +- [ ] Phase 4: Implement / fix — apply fixes in priority order (respect `--review-only`) +- [ ] Phase 5: Verify — run the repo's own build/test/lint gates on each fix +- [ ] Phase 6: Review — self-review: fixes applied, no regressions, docs match code, refs valid +- [ ] Phase 7: Retrospective — summary, enforcement recommendations, config-gap observations + +## Skip criteria + +- Phases 4-7 SKIPPED in `--review-only` mode (audit produces findings; user/separate session implements) +- Phase 5 fix-verify pair may iterate per-fix vs batch — depends on blast radius diff --git a/plugins/codebase-audit/skills/setup/SKILL.md b/plugins/codebase-audit/skills/setup/SKILL.md new file mode 100644 index 000000000..f9624a7df --- /dev/null +++ b/plugins/codebase-audit/skills/setup/SKILL.md @@ -0,0 +1,53 @@ +--- +name: setup +description: "Configure the codebase-audit plugin for this repository: interview the user, infer audit targets from the repo layout, and write the tracked .claude/codebase-audit.md config file. Use when: 'set up codebase-audit', 'configure the audit', 'codebase-audit setup', the audit skill reports missing or thin config, or audit dimensions need tuning. Re-runnable — safe to invoke again to reconfigure." +argument-hint: "(no arguments — interactive interview)" +user-invocable: true +disable-model-invocation: false +--- + +## Purpose + +Write (or update) the consuming repo's tracked audit-dimension config at `.claude/codebase-audit.md` +so `/codebase-audit:codebase-audit` runs deterministically instead of re-inferring targets every run. +Idempotent: re-running reads the existing config and offers updates rather than overwriting blind. + +## Task + +1. **Read existing config first.** If `.claude/codebase-audit.md` exists, load it and present a + short summary (dimensions present, glob counts, example-claim counts). The interview then + proposes changes against that baseline; nothing is dropped without the user confirming. +2. **Explore the repo to draft defaults.** Before asking anything, infer candidates: + - **documentation** primary-sources: doc directories (`docs/`, `README.md`), agent-instruction + files (`AGENTS.md`, `CLAUDE.md`), ADR directories, convention docs. + - **configuration** primary-sources: build config, lint config, CI workflows, git-hook config — + detected from what actually exists (e.g. `Directory.Build.props`, `pyproject.toml`, + `package.json`, `.github/workflows/`, `lefthook.yml`, `.editorconfig`). + - **code-quality** primary-sources: source roots; verification-sources: test roots. + - **architecture** primary-sources: dependency manifests + architecture docs; + verification-sources: analyzers / architecture tests where present. +3. **Interview, one decision at a time.** Present each dimension's drafted globs with a + recommendation; let the user accept, edit, or remove the dimension. Offer custom dimensions + last ("anything else this repo should audit as its own lane?"). +4. **Draft example-claims.** For each accepted dimension, read one or two representative + primary-source files and propose 2–4 concrete `{ claim, verify-via }` rows drawn from real + sentences in them. Concrete rows teach the discovery pass what drift looks like in THIS repo — + the highest-value part of the config. The user approves or edits each row. +5. **Write the config.** Materialize `.claude/codebase-audit.md` following the structure in + [templates/config-template.md](templates/config-template.md) (replace every placeholder comment + with real values; drop unused placeholder rows). Confirm the file is tracked, not ignored. +6. **Offer the overlay convention.** Personal overrides go in `.claude/codebase-audit.local.md`; + recommend the consumer add `.claude/*.local.*` to `.gitignore` if not already covered. A + user-global base at `~/.claude/codebase-audit.md` is also honored. Layers resolve + user-global → team → local overlay, additively. + +## Output + +A tracked `.claude/codebase-audit.md` in the consuming repo, plus a one-paragraph summary of what +was written and how to re-run this setup to reconfigure. + +## What this skill does NOT do + +- Run an audit — that is `/codebase-audit:codebase-audit`. +- Write machine-local state — configuration lives in the consumer's tracked file, never in the + plugin directory or the plugin data directory. diff --git a/plugins/codebase-audit/skills/setup/evals/evals.json b/plugins/codebase-audit/skills/setup/evals/evals.json new file mode 100644 index 000000000..30e458d0b --- /dev/null +++ b/plugins/codebase-audit/skills/setup/evals/evals.json @@ -0,0 +1,41 @@ +{ + "skill_name": "setup", + "evals": [ + { + "id": 1, + "name": "infers-and-interviews-before-writing", + "prompt": "/codebase-audit:setup", + "expected_output": "Explores the repo to draft candidate audit dimensions (doc dirs, build/lint/CI config, source and test roots), then interviews the user one dimension at a time with a recommendation before writing anything. It does not write the config until the user has confirmed the dimensions.", + "files": [], + "expectations": [ + "Explores the repo to draft candidate primary/verification sources before asking", + "Interviews the user per dimension with a recommendation, rather than writing blind", + "Writes the config only after the user confirms the dimensions" + ] + }, + { + "id": 2, + "name": "idempotent-reads-existing-config", + "prompt": "/codebase-audit:setup", + "expected_output": "When .claude/codebase-audit.md already exists, reads it, summarizes the current dimensions, and proposes changes against that baseline rather than overwriting it blind. Nothing is dropped without the user confirming.", + "files": [], + "expectations": [ + "Detects and reads an existing .claude/codebase-audit.md before proposing changes", + "Summarizes the current config and proposes edits against that baseline", + "Does not drop or overwrite existing dimensions without user confirmation" + ] + }, + { + "id": 3, + "name": "writes-tracked-config-not-plugin-dir", + "prompt": "/codebase-audit:setup\n\nJust write a sensible default config for this repo.", + "expected_output": "Materializes the config at the consumer's tracked .claude/codebase-audit.md, confirms it is tracked (not gitignored), and offers the .claude/*.local.* overlay/gitignore convention. It never writes configuration into the plugin directory or a plugin data directory.", + "files": [], + "expectations": [ + "Writes the config to the consumer's tracked .claude/codebase-audit.md", + "Offers the .claude/*.local.* personal-overlay and gitignore convention", + "Does not write configuration into the plugin directory or plugin data directory" + ] + } + ] +} diff --git a/plugins/codebase-audit/skills/setup/templates/config-template.md b/plugins/codebase-audit/skills/setup/templates/config-template.md new file mode 100644 index 000000000..f101313fc --- /dev/null +++ b/plugins/codebase-audit/skills/setup/templates/config-template.md @@ -0,0 +1,78 @@ +# codebase-audit configuration + +Tracked audit-dimension configuration for the codebase-audit plugin. Each `##` section below is one +audit dimension. The four bundled dimensions (`documentation`, `configuration`, `code-quality`, +`architecture`) may be tuned or removed here; new `##` sections add custom dimensions. Personal +overlays go in `.claude/codebase-audit.local.md` (gitignored); a user-global base may live at +`~/.claude/codebase-audit.md`. Layers resolve user-global → this file → local overlay, additively. + +Each dimension carries: + +- **primary-sources** — glob patterns for files where factual claims live (docs, conventions, + ADRs, status pages). Discovery reads these top-to-bottom and extracts claims. +- **verification-sources** — glob patterns for files where claims are verified against ground + truth (build config, manifests, source, tests). +- **example-claims** — optional `{ claim, verify-via }` rows teaching the claim-extraction pass + what drifts in THIS repo. The more concrete, the better the audit. + +## documentation + +**primary-sources:** + +- `docs/**/*.md` +- `README.md` +- + +**verification-sources:** + +- + +**example-claims:** + +- claim: "" + - verify-via: "" + +## configuration + +**primary-sources:** + +- + +**verification-sources:** + +- + +**example-claims:** + +- claim: "" + - verify-via: "" + +## code-quality + +**primary-sources:** + +- + +**verification-sources:** + +- + +**example-claims:** + +- claim: "" + - verify-via: "" + +## architecture + +**primary-sources:** + +- + +**verification-sources:** + +- + +**example-claims:** + +- claim: "" + - verify-via: "" From 0bbd07b93dc0dda6fc6aa57f847d4b185ade48e4 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 12 Jul 2026 00:27:14 -0400 Subject: [PATCH 2/3] fix(codebase-audit): address review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Use ${CLAUDE_PLUGIN_ROOT} on all internal skill file references so they resolve from the plugin cache in a consumer repo (matches diagnose/ review-toolkit pattern; checklist ref already did). - Spec additive merge semantics when a dimension name spans config layers (union globs, concat+dedupe example-claims, empty lists = explicit opt-out). - Cross-file dimensions (code-quality/architecture) must list source roots + manifests in verification-sources so discovery agents can read peer files under the read-only fence — wired in setup defaults, template, and the discovery fence note (Codex P2). - Soften Phase 2 "required" to "required when tooling exists" + needs-review degrade path for missing research tools. - Fix Phase 7 label (retrospective, not auto-fix); precomputed-context clean- tree fallback (blank vs misleading "clean"); add verified-non-issues eval expectation; drop the underdefined background-workflow paragraph. Refs melodic-software/medley#1376 --- .../skills/codebase-audit/SKILL.md | 37 +++++++++++++------ .../context/discovery-method.md | 18 +++++---- .../skills/codebase-audit/evals/evals.json | 3 +- plugins/codebase-audit/skills/setup/SKILL.md | 11 ++++-- .../skills/setup/templates/config-template.md | 4 ++ 5 files changed, 50 insertions(+), 23 deletions(-) diff --git a/plugins/codebase-audit/skills/codebase-audit/SKILL.md b/plugins/codebase-audit/skills/codebase-audit/SKILL.md index 4d08aea31..ca1979e3a 100644 --- a/plugins/codebase-audit/skills/codebase-audit/SKILL.md +++ b/plugins/codebase-audit/skills/codebase-audit/SKILL.md @@ -9,7 +9,7 @@ disable-model-invocation: false ## Pre-computed context Current branch: !`git branch --show-current 2>/dev/null || echo "unknown"` -Working tree status: !`git status --porcelain 2>/dev/null | head -20 || echo "clean"` +Working tree status: !`git status --porcelain 2>/dev/null | head -20 || echo "(unavailable)"` Changed files (staged+unstaged): !`git diff --name-only HEAD 2>/dev/null || echo "none"` ## Variables @@ -35,8 +35,8 @@ active dimensions and dispatch per file. ## Model auto-invoke default When the model invokes this skill without explicit user authorization to auto-fix, treat the run as -**`--review-only`**. Phases 4–7 (auto-fix) run only when the user explicitly requests fixes or omits -`--review-only` with clear fix intent. +**`--review-only`**. Phases 4–6 (auto-fix) and Phase 7 (retrospective) run only when the user +explicitly requests fixes or omits `--review-only` with clear fix intent. --- @@ -66,8 +66,16 @@ pass) — come from the consuming repo's tracked config, resolved additively acr 3. `.claude/codebase-audit.local.md` (personal overlay, gitignored) The four bundled dimensions are `documentation`, `configuration`, `code-quality`, and -`architecture`; the config may tune their globs, remove a dimension, or add custom ones. Settle -targets by this ladder: +`architecture`; the config may tune their globs, remove a dimension, or add custom ones. + +**Merge semantics when the same dimension name appears in two layers:** additive by default — the +later layer's `primary-sources` and `verification-sources` globs UNION with the earlier layer's (not +replace), and `example-claims` concatenate with duplicate `claim` text collapsed. A layer removes an +inherited dimension by declaring it with empty source lists (an explicit opt-out), never by silent +omission. This keeps a personal overlay purely additive to team config unless it deliberately zeroes a +dimension out. + +Settle targets by this ladder: 1. **Config present → use it.** 2. **Absent → infer from the repo** (doc dirs, build manifests, source/test roots, CI workflows), @@ -114,7 +122,7 @@ Discovery runs as a **parallel subagent fan-out — one agent per primary-source sequential pass (fresh context per file ≈ 2× claim coverage and ~4× drift caught; a single context skips claims as it fills — the #1 audit failure). Each agent applies the claim-extraction method (read top-to-bottom → extract every factual claim → verify each independently → record), fenced per -the scope-fencing rules in [context/discovery-method.md](context/discovery-method.md). +the scope-fencing rules in [`${CLAUDE_PLUGIN_ROOT}/skills/codebase-audit/context/discovery-method.md`](context/discovery-method.md). **Scope first (MANDATORY — cost gate):** require a `[scope]` or dimension filter (`--docs-only` etc.) for large targets; if the enumerated list exceeds ~20 files, confirm with the user before @@ -123,8 +131,9 @@ source file costs millions of tokens. Full method — claim-extraction steps, the verify-ALL-claims-on-a-line rule, enumerate/scope/dispatch/collect detail, and the per-finding report format — in -[context/discovery-method.md](context/discovery-method.md). Dimension-specific claim guidance: -[reference/audit-checklist.md](reference/audit-checklist.md). +[`${CLAUDE_PLUGIN_ROOT}/skills/codebase-audit/context/discovery-method.md`](context/discovery-method.md). +Dimension-specific claim guidance: +[`${CLAUDE_PLUGIN_ROOT}/skills/codebase-audit/reference/audit-checklist.md`](reference/audit-checklist.md). --- @@ -140,10 +149,10 @@ rubber-stamps it; an independent agent re-reading the doc claim AND the actual c false positives and miscategorized-but-correct claims. Fence each validator to read-only (its findings' files + verification-sources). -### External research (required, not optional) +### External research (required when the tooling exists) -Use available documentation-research tools (MCP docs servers, library-docs lookers-up, web search) -to validate findings involving: +When your setup provides documentation-research tools (MCP docs servers, library-docs lookers-up, web +search), using them is REQUIRED — not optional — to validate findings involving: - **Best-practice claims** — is the documented pattern the current recommended approach? - **Library API claims** — does the method/class/parameter exist in the current version? @@ -153,6 +162,10 @@ Cross-reference research results against the repo's conventions (loaded in Phase consensus matters, but repo conventions are the primary lens — a pattern unusual industry-wide may be intentionally chosen here. +**Graceful degrade when no research tool is available:** do not skip the claim and do not guess. +Verify whatever the local repo can confirm, then confidence-tag the externally-unverifiable part as +`needs-review` (below) so it surfaces for human judgment rather than being asserted or dropped. + ### False-positive prevention **If uncertain, it is NOT a finding.** Ambiguous items go to `needs-review` and are separated from @@ -169,7 +182,7 @@ cost of missing a marginal issue (catchable next run). ## Phase 3: Categorize & Present -### Group findings per [reference/category-playbook.md](reference/category-playbook.md) +### Group findings per [`${CLAUDE_PLUGIN_ROOT}/skills/codebase-audit/reference/category-playbook.md`](reference/category-playbook.md) Fix order matters — see the playbook for why: Config Drift → Missing Enforcement → Code Quality → Doc Drift. diff --git a/plugins/codebase-audit/skills/codebase-audit/context/discovery-method.md b/plugins/codebase-audit/skills/codebase-audit/context/discovery-method.md index 97560de66..654db35dd 100644 --- a/plugins/codebase-audit/skills/codebase-audit/context/discovery-method.md +++ b/plugins/codebase-audit/skills/codebase-audit/context/discovery-method.md @@ -31,7 +31,7 @@ verify each one independently. For example: ## Dimension-specific guidance -Read [../reference/audit-checklist.md](../reference/audit-checklist.md) for the kinds of claims to +Read [`${CLAUDE_PLUGIN_ROOT}/skills/codebase-audit/reference/audit-checklist.md`](../reference/audit-checklist.md) for the kinds of claims to watch for in each dimension. The checklist is a guide for **what to look for**, not a list to check off. The primary method is always: read the file, extract claims, verify each one. @@ -58,17 +58,21 @@ caught vs a single sequential agent. returns findings + a verified-count. Use repo-relative paths only (never absolute machine paths — agents must audit the current worktree). Throttle in waves (≤~16 concurrent); lower-tier worker models are sufficient and dodge burst overload. + + **Peer files a claim must be checked against are read via `verification-sources`, not the fence + exception.** A cross-file claim — DRY duplication across N files, a dependency-direction rule, an + architecture boundary — can only be validated by reading peer files, and the fence forbids the + *other* primary-source files. So for any dimension whose claims are cross-file (notably + `code-quality` and `architecture`), the config MUST list the relevant source roots and dependency + manifests in `verification-sources` as well (they may also be `primary-sources` — the read-only + exception covers the overlap). If a cross-file dimension's `verification-sources` omit the peer + files its claims reference, those findings are systematically missed — the setup skill wires this + in by default. 4. **Collect** — aggregate per-file findings + verified counts into the Phase 2 input. Because each agent owns one file, the "complete one dimension before the next" sequencing is moot — there is no shared context to thin out, so dimension order does not matter. -**Background / unattended variant:** the same per-file fan-out can run as a saved background workflow -(background execution, same-session resume, rerunnable script) instead of in-session subagents — same -discovery, different executor. Reach for the workflow form only when you want a fire-and-forget -periodic audit you can walk away from; the in-session fan-out above is the default for an interactive -audit you are actively driving. - ## What to report Report **every discrepancy** found, no matter how small. Also report what you verified as correct — diff --git a/plugins/codebase-audit/skills/codebase-audit/evals/evals.json b/plugins/codebase-audit/skills/codebase-audit/evals/evals.json index 39074f463..903672132 100644 --- a/plugins/codebase-audit/skills/codebase-audit/evals/evals.json +++ b/plugins/codebase-audit/skills/codebase-audit/evals/evals.json @@ -22,7 +22,8 @@ "expectations": [ "Presents the findings report and STOPS without applying any fixes", "Does not modify, edit, or write any files", - "Scopes the audit to the named file only, not the whole repository" + "Scopes the audit to the named file only, not the whole repository", + "Includes the required 'Verified non-issues' section even in review-only mode" ] }, { diff --git a/plugins/codebase-audit/skills/setup/SKILL.md b/plugins/codebase-audit/skills/setup/SKILL.md index f9624a7df..7550740e1 100644 --- a/plugins/codebase-audit/skills/setup/SKILL.md +++ b/plugins/codebase-audit/skills/setup/SKILL.md @@ -23,9 +23,14 @@ Idempotent: re-running reads the existing config and offers updates rather than - **configuration** primary-sources: build config, lint config, CI workflows, git-hook config — detected from what actually exists (e.g. `Directory.Build.props`, `pyproject.toml`, `package.json`, `.github/workflows/`, `lefthook.yml`, `.editorconfig`). - - **code-quality** primary-sources: source roots; verification-sources: test roots. + - **code-quality** primary-sources: source roots; verification-sources: test roots **plus the + same source roots** — a cross-file DRY/SOLID claim is validated by reading peer source files, + which a discovery agent can only read when they are in `verification-sources` (the fence forbids + the other primary-source files). Omitting them makes cross-file findings unreachable. - **architecture** primary-sources: dependency manifests + architecture docs; - verification-sources: analyzers / architecture tests where present. + verification-sources: analyzers / architecture tests where present **plus the dependency + manifests and source roots** — dependency-direction and boundary claims need peer manifests + readable, for the same fence reason. 3. **Interview, one decision at a time.** Present each dimension's drafted globs with a recommendation; let the user accept, edit, or remove the dimension. Offer custom dimensions last ("anything else this repo should audit as its own lane?"). @@ -34,7 +39,7 @@ Idempotent: re-running reads the existing config and offers updates rather than sentences in them. Concrete rows teach the discovery pass what drift looks like in THIS repo — the highest-value part of the config. The user approves or edits each row. 5. **Write the config.** Materialize `.claude/codebase-audit.md` following the structure in - [templates/config-template.md](templates/config-template.md) (replace every placeholder comment + [`${CLAUDE_PLUGIN_ROOT}/skills/setup/templates/config-template.md`](templates/config-template.md) (replace every placeholder comment with real values; drop unused placeholder rows). Confirm the file is tracked, not ignored. 6. **Offer the overlay convention.** Personal overrides go in `.claude/codebase-audit.local.md`; recommend the consumer add `.claude/*.local.*` to `.gitignore` if not already covered. A diff --git a/plugins/codebase-audit/skills/setup/templates/config-template.md b/plugins/codebase-audit/skills/setup/templates/config-template.md index f101313fc..95691b37d 100644 --- a/plugins/codebase-audit/skills/setup/templates/config-template.md +++ b/plugins/codebase-audit/skills/setup/templates/config-template.md @@ -56,6 +56,8 @@ Each dimension carries: **verification-sources:** - +- **example-claims:** @@ -71,6 +73,8 @@ Each dimension carries: **verification-sources:** - +- **example-claims:** From 5e3f53c340680f00e71ac595e94ac5e944acbac0 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 12 Jul 2026 00:31:47 -0400 Subject: [PATCH 3/3] docs(codebase-audit): scope zero-findings Phase 7 to non-review-only runs Refs melodic-software/medley#1376 --- plugins/codebase-audit/skills/codebase-audit/SKILL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plugins/codebase-audit/skills/codebase-audit/SKILL.md b/plugins/codebase-audit/skills/codebase-audit/SKILL.md index ca1979e3a..edbf7ea7c 100644 --- a/plugins/codebase-audit/skills/codebase-audit/SKILL.md +++ b/plugins/codebase-audit/skills/codebase-audit/SKILL.md @@ -214,8 +214,8 @@ If the audit finds no discrepancies, report a clean bill of health: - State explicitly: "No findings. All claims verified as correct." - Do NOT invent findings to justify the audit. A clean codebase is the goal, not a guaranteed list of issues. -- Skip Phases 4-6 (nothing to fix). Proceed directly to Phase 7 (Retrospective) with scope/coverage - observations. +- Skip Phases 4-6 (nothing to fix). If not `--review-only`, proceed to Phase 7 (Retrospective) with + scope/coverage observations. ### Review-only gate