diff --git a/plugins/claude-config/.claude-plugin/plugin.json b/plugins/claude-config/.claude-plugin/plugin.json index 50d1b88a3..f8bf8594f 100644 --- a/plugins/claude-config/.claude-plugin/plugin.json +++ b/plugins/claude-config/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "claude-config", - "version": "0.33.1", + "version": "0.34.0", "description": "Eight configuration-health skills (plus setup) for a repo's Claude Code configuration: audit (settings.json / .mcp.json / hooks / plugins / permissions drift), audit-automation-gaps (evidence-gated verdicts on automation gaps), audit-permission-grants (allow-rule / allowed-tools grants for auto-mode durability and portability), audit-permission-state (which settings scopes exist and what rules each one holds — managed policy, user-global, project, local, and the pre-v2.1.211 start-directory copy), audit-instructions (locally-owned instruction surfaces vs current model capability — proposes removals/rewrites of instructions the model no longer needs, and detects cross-surface instruction conflicts), audit-prompting-postures (the additive lane — posture guidance the prompting guide says a component's purpose needs but the component does not carry), audit-pass (one coordinated, ordered, resumable pass over a named target — three-scope inventory, run-time-derived exclusion set, stable finding identity, suppression memory, resume, one human gate — delegating every check to the plugin that owns it), and unhobble (the empirical bare-baseline experiment: reversibly strip a repo's standing instructions, log real stumbles against the current model, re-add only what evidence earns).", "author": { "name": "Melodic Software", diff --git a/plugins/claude-config/CHANGELOG.md b/plugins/claude-config/CHANGELOG.md index 226d09a40..cb7f3d07e 100644 --- a/plugins/claude-config/CHANGELOG.md +++ b/plugins/claude-config/CHANGELOG.md @@ -3,6 +3,78 @@ All notable changes to the `claude-config` plugin are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning. +## [0.34.0] + +### Added + +- **`audit` can now enumerate the hooks an enabled plugin ships.** It read settings-declared hooks + only, and three of its own surfaces said so in prose — `context/procedures.md` called the hook + question "a question to answer, not a lookup", and `reference/required-permissions.md` carried a + "Fail open where no hook inventory was taken" clause that, on most runs, resolved to hedging every + Category B finding. That gap was load-bearing in two places: Category D writes rules for + `${CLAUDE_PLUGIN_ROOT}` / `${CLAUDE_PLUGIN_DATA}` placeholders that only ever appear in a + plugin-provided hook, and Category B's third baseline narrowing turns on whether such a hook is + live. New `scripts/check-hook-coverage.sh` (with `.test.sh`) resolves each enabled plugin through + the installed-plugin registry — so no version-directory ordering is inferred — and reads its hook + config in all four documented shapes: `hooks/hooks.json`, a `hooks` path, an array of paths, and an + inline object in `plugin.json`. It also reports `disableAllHooks`, `allowManagedHooksOnly`, and + `strictPluginOnlyCustomization`, because a hook a setting has already switched off is not coverage. + The exit code is the contract: `0` complete, `1` partial with the unenumerated sources named, `2` + no inventory. The fail-open posture survives, narrowed to what a partial run could not read — + "could not look" is still never reportable as "looked and found nothing". Read-only: it never runs + a hook, and it never decides whether a hook *covers* a family, which stays Category B's judgment + against its three preconditions. +- **Two positive Category G evals.** The category had no positive case at all — its only two mentions + in `evals.json` were negative assertions inside the scope-filter case. One case now exercises the + headless overflow route, the other the lever-matches-the-roster rule. + +### Changed + +- **Category G is executable now, from either kind of session.** Its only overflow detector was + `/doctor`, which needs an interactive TTY, so the whole category yielded nothing in the harness's + own headless mode while the documented headless route went unmentioned. It now names `--debug` — + *"When the listing exceeds its budget, Claude Code also writes a warning to the debug log"* — as + the headless route, and `/context`'s Skills row as what it actually is: a second *interactive* + reading, not a substitute. A run that could measure nothing reports "not measured", never "no + overflow". +- **Category G states the budget it measures against.** The category named the per-entry cap and both + scaling settings and gave no budget value and no formula, so "overflowed, and by how much?" was + unanswerable by hand. It now carries `skillListingBudgetFraction` default `0.01`, + `SLASH_COMMAND_TOOL_CHAR_BUDGET`'s documented 8,000-character fallback, + `skillListingMaxDescChars` default `1536`, and the `200,000 × 4 × 0.01 = 8,000` arithmetic that + reconciles the first two. +- **Category G's lever list no longer recommends a lever the operator cannot pull.** The ordered + "cheapest first" list put `skillOverrides` second while carrying the caveat that it does not apply + to plugin skills, and never named the substitute upstream prescribes — so on a plugin-heavy roster + the list degenerated to the entry it labels "last resort". Levers are now split by roster origin, + `/plugin` is named for plugin skills, and a roster-composition count is required *before* any lever + is recommended. No per-skill `name-only` state is promised for `/plugin`; no page documents one. +- **Category G has a checklist table, like every other Phase 2 category.** It was the only letter + missing from `audit-checklist.md`, whose heading sequence ran A–F, H, I. The table also points at + `plugins/skill-quality/scripts/check-listing-budget.sh` — the aggregate measurement this + marketplace already ships — as an explicit **in-repo proxy**, with the population difference said + out loud: that script walks a repository's skills roots, while Category G asks about the listing a + consumer's session assembled. Measured cost is stated too (8 skills in 5.98s on Windows), because a + marketplace-wide invocation will not finish inside a default Bash tool timeout. +- **The `destructive-bash-deny` baseline is reported with its fragility, the way the Read deny table + already is.** Eight argument-constraining globs shipped rated `error` when absent with no caveat, + in a file that quotes the permissions page's *"Bash permission patterns that try to constrain + command arguments are fragile"* warning one section earlier against a different table. The concrete + hole is now stated: matching is prefix-anchored, so `Bash(git push --force *)` does not match + `git push origin main --force`. The patterns stay in the baseline — they raise the cost of an + accidental force push — but a finding no longer implies they bound a determined one. +- **The "a `PreToolUse` hook is a speed bump, not a boundary" ranking is scoped to the threat model it + was written for.** It is right for secret exfiltration, where an OS-level boundary exists and the + hook is strictly worse than it. It was stated unscoped, so it also governed destructive git — where + the sandbox's vocabulary is `filesystem.*` paths and `network.*` hosts, with no expression for a + command's arguments, so it does not separate `git push` from `git push --force` to the same remote. + Scoped, not deleted. +- **The read-it-verbatim guard now covers all of Phase 3, not one checklist row.** It was attached to + the Category F env-vars row while Phase 3.1 and Phase 3.4 fetched `settings` and `permissions` with + no such instruction — and `settings` is the page on which a summarizing fetch already reported three + present keys as absent. The guard now also says explicitly that a truncated read supports no finding + in *either* direction. + ## [0.33.1] ### Fixed diff --git a/plugins/claude-config/skills/audit/SKILL.md b/plugins/claude-config/skills/audit/SKILL.md index 399e8fc42..530a225e9 100644 --- a/plugins/claude-config/skills/audit/SKILL.md +++ b/plugins/claude-config/skills/audit/SKILL.md @@ -89,6 +89,31 @@ Record the installed Claude Code version (`claude --version`) — Phase 3.2 comp against it. Then run `bash "${CLAUDE_PLUGIN_ROOT}/skills/audit/scripts/check-structure.sh"` before the table below. +### 1.0 Hook inventory + +Run `bash "${CLAUDE_PLUGIN_ROOT}/skills/audit/scripts/check-hook-coverage.sh"` and keep its output for +the rest of the run. It enumerates settings-declared hooks **and** the hooks shipped by every enabled +plugin — resolved through the installed-plugin registry, so no version-directory guessing is +involved — plus the levers (`disableAllHooks`, `allowManagedHooksOnly`, +`strictPluginOnlyCustomization`) that switch hooks off wholesale. + +Two categories depend on it and neither could take this inventory before: + +- **Category D** writes rules for `${CLAUDE_PLUGIN_ROOT}` / `${CLAUDE_PLUGIN_DATA}` placeholders, which + only ever appear in a plugin-provided hook. +- **Category B's third narrowing** demotes a missing baseline deny to `info` when a *live* `PreToolUse` + hook already blocks the family. That narrowing is only takeable against a real inventory. + +**Read the exit code, not just the table.** `0` means the inventory is complete. `1` means it is +**partial** — some enabled plugin did not resolve, or a hook config did not parse — and the "Not +enumerated" block names which. For anything a partial inventory could not see, keep the conditional +posture in [required-permissions.md](reference/required-permissions.md) "Fail open where the inventory +is incomplete". `2` is fatal (no readable settings scope, or `jq` missing); treat it as no inventory at +all rather than as an empty one. + +The script never runs a hook and never decides whether a hook *covers* a family — that judgment stays +with Category B, against the three preconditions in "Narrowing the baseline". + ### 1.1 JSON validity `Valid JSON: no` blocks further analysis. @@ -126,7 +151,7 @@ Nine categories — names + the question each answers below; **full per-check cr - **D — Hooks**: paths resolve + readable, `timeout` in seconds and sane, valid matchers, quoted path placeholders in shell form, exec-form `command` resolvable on Windows, no duplicates, valid events - **E — Plugins**: static checks (marketplace membership) + live upstream drift detection (`scripts/check-plugin-drift.sh` — ORPHAN/NEW/RENAME modes, auto-fix policy table in the context file) - **F — Environment Variables**: documented/justified vars, secrets in `settings.local.json` only, forward-slash paths -- **G — Skill-listing budget**: `/doctor` overflow check and trim levers (description trimming, `skillOverrides`, budget settings) +- **G — Skill-listing budget**: overflow check by a route this run can actually take (`/doctor` interactively, `--debug` headless), the budget constant it was measured against, and trim levers scoped to the roster's composition (`skillOverrides` reaches project and user skills; plugin skills are managed through `/plugin`) - **H — Model and effort settings**: `effortLevel`, `fallbackModel`, `availableModels`, `enforceAvailableModels` — values the harness accepts into the file but does not apply as written - **I — Deep-link registration**: `disableDeepLinkRegistration` — the one documented value that takes effect, and a visible attempt at an enforcement requirement lodged in a scope that cannot enforce it @@ -136,6 +161,15 @@ Nine categories — names + the question each answers below; **full per-check cr External verification against current documentation. +**Read every page in this phase verbatim, not through a summarizer.** These pages are long — `settings` +and `env-vars` are hundreds of KB — and a summarizing fetch truncates, then reports the rows past the +cutoff as *absent*. That false negative has already been observed on the `settings` page: three keys +reported NOT FOUND that raw `curl` + `grep` found. So for each fetch below, +`curl https://code.claude.com/docs/en/.md` to a file and grep the file, per the +[fetch route](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/conventions/upstream-drift/README.md#reading-the-basis--the-fetch-route). +**A truncated read supports NO finding** — say so and move on, in either direction: neither "the key is +gone" nor "the key is unchanged" is reportable from a read that may have been cut. + ### 3.1 Official docs check Fetch [code.claude.com/docs/en/settings](https://code.claude.com/docs/en/settings) and compare: diff --git a/plugins/claude-config/skills/audit/context/procedures.md b/plugins/claude-config/skills/audit/context/procedures.md index 7e413ef8d..e5a652ec0 100644 --- a/plugins/claude-config/skills/audit/context/procedures.md +++ b/plugins/claude-config/skills/audit/context/procedures.md @@ -65,8 +65,11 @@ added, and neither is mechanical: baseline", whose three preconditions govern: installed and enabled is not enough (`disableAllHooks` and the managed `allowManagedHooksOnly` / `strictPluginOnlyCustomization` levers switch hooks off), a `Bash` hook does not cover the `Read`-pattern family, and one command family's coverage says - nothing about another's. The audit reads settings-declared hooks only and cannot enumerate a - plugin's `hooks/hooks.json`, so this is a question to answer, not a lookup. + nothing about another's. **The inventory half is now a lookup**: Phase 1.0's + `scripts/check-hook-coverage.sh` enumerates settings-declared *and* plugin-declared hooks, + resolving each enabled plugin through the installed-plugin registry. What stays a judgment is + whether an enumerated hook covers *this* family — and where that script exited 1, the sources it + names as unenumerated remain a question, not an absence. 2. **Would the addition suppress a gate the project built on purpose?** Deny and ask rules are evaluated regardless of what a `PreToolUse` hook returns, so adding a deny over a family a project hook escalates to an *ask* replaces the prompt with an outright block and the human loses the diff --git a/plugins/claude-config/skills/audit/context/validation-categories.md b/plugins/claude-config/skills/audit/context/validation-categories.md index 4344edd91..24e2e1a63 100644 --- a/plugins/claude-config/skills/audit/context/validation-categories.md +++ b/plugins/claude-config/skills/audit/context/validation-categories.md @@ -51,6 +51,14 @@ Load the audit checklist alongside these: [audit-checklist.md](../reference/audi ## Category D: Hooks +**The inventory this category checks is Phase 1.0's**, from +`scripts/check-hook-coverage.sh` — settings-declared hooks *and* every enabled plugin's own hook +config, resolved through the installed-plugin registry. That matters for two of the rules below: +`${CLAUDE_PLUGIN_ROOT}` and `${CLAUDE_PLUGIN_DATA}` only ever appear in a plugin-provided hook, so +before Phase 1.0 existed those rules were written against a surface the audit never opened. Where the +script exited 1, say which sources went unenumerated rather than reporting the inventory as the +complete set. + - All hook script paths resolve to existing files on disk - Scripts are readable (not permission-denied) - `timeout` is a seconds value — flag a recognizably millisecond-scale figure (a round thousands @@ -131,16 +139,50 @@ bash "${CLAUDE_PLUGIN_ROOT}/skills/audit/scripts/fix-plugin-drift.sh" --yes ## Category G: Skill-listing budget -- **Overflow check** — if `/doctor` reports dropped skill descriptions, the skill listing has exceeded - its budget and the least-invoked skills' trigger keywords are silenced (names still resolve; - auto-invocation degrades silently). `/doctor` needs an interactive TTY — prompt the user to run it. - Repos with large skill rosters overflow routinely -- **Levers, cheapest first** — trim `description` / `when_to_use` frontmatter (key use case first; - 1,536-char cap per entry), `skillOverrides: { : "name-only" }` in a contributor's - `settings.local.json` (does NOT apply to plugin skills), then `skillListingBudgetFraction` / - `SLASH_COMMAND_TOOL_CHAR_BUDGET` in project settings as a last resort (costs context every turn) -- **Recommend, don't apply the list** — `skillOverrides` is contributor-scoped; surface the candidate - least-invoked skills, leave the actual name-only list to the developer +Row-by-row criteria are in [audit-checklist.md](../reference/audit-checklist.md) "G. Skill-listing +budget". What governs the category: + +- **State the budget, or the finding is not computable.** The listing budget is + `skillListingBudgetFraction` of the model's context window — **default `0.01`, i.e. 1%** — and + `SLASH_COMMAND_TOOL_CHAR_BUDGET` overrides it with a fixed character count, **documented fallback + 8,000 characters**. Each entry's combined `description` + `when_to_use` text is separately capped at + `skillListingMaxDescChars`, **default `1536`**. For a 200K-token window, `200,000 × 4 × 0.01 = 8,000` + characters, which is why the env var's fallback is that number. Without the constant a report can say + "overflowed" but not "by how much", so quote it. All three are upstream-owned: confirm them in Phase + 3 against [settings](https://code.claude.com/docs/en/settings) and + [env-vars](https://code.claude.com/docs/en/env-vars) before publishing a number +- **Overflow check — two routes, and only one survives a headless run.** `/doctor` estimates the + listing's cost and its biggest contributors, and it needs an interactive TTY, so prompt the user to + run it. When this audit runs headless — `-p`, a spawned agent, a background job — use the documented + debug route instead: *"When the listing exceeds its budget, Claude Code also writes a warning to the + debug log, visible with `--debug`"* + ([skills](https://code.claude.com/docs/en/skills), "Skill descriptions are cut short"). Report which + route was taken; a category that names only `/doctor` yields nothing in the harness's own headless + mode. `/context`'s Skills row reports the listing size after the budget is applied — a second + *interactive* reading, not a headless one. Overflow silences the least-invoked skills' trigger + keywords (names still resolve; auto-invocation degrades silently), and repos with large skill rosters + overflow routinely +- **Measure the roster composition before naming a lever.** Count listing entries by origin — plugin + skills, project skills (`.claude/skills/`), user skills (`${CLAUDE_CONFIG_DIR:-~/.claude}/skills/`). + This is the single input that decides which levers exist, and it is cheap. A run that skips it + recommends levers the operator cannot pull +- **Levers, cheapest first — and the ordering depends on that composition:** + - *Any origin* — trim `description` / `when_to_use` at the source, key use case first. Costs nothing + at runtime and is the only lever that helps every roster + - *Project and user skills* — `skillOverrides: { : "name-only" }` in a contributor's + `settings.local.json` + - *Plugin skills* — `skillOverrides` **does not reach them**: *"Does not apply to plugin skills, + which are managed through `/plugin`"* (settings) and *"Plugin skills are not affected by + `skillOverrides`. Manage those through `/plugin` instead"* (skills). So on a plugin-heavy roster + the lever is `/plugin` — disabling a plugin removes its skills from the listing — plus trimming + the descriptions upstream in the plugin that owns them. Neither page documents a per-skill + `name-only` state reachable from `/plugin`, so do not promise one + - *Last resort, any origin* — raise `skillListingBudgetFraction` / `SLASH_COMMAND_TOOL_CHAR_BUDGET` + in project settings. It costs context every turn, which is why it is last here even though the + docs present it first +- **Recommend, don't apply the list** — `skillOverrides` is contributor-scoped and `/plugin` is a + machine-level action; surface the candidate least-invoked skills, leave the actual list to the + developer ## Category H: Model and effort settings diff --git a/plugins/claude-config/skills/audit/evals/evals.json b/plugins/claude-config/skills/audit/evals/evals.json index ce3bb38e0..a7c61ae27 100644 --- a/plugins/claude-config/skills/audit/evals/evals.json +++ b/plugins/claude-config/skills/audit/evals/evals.json @@ -106,14 +106,15 @@ }, { "id": 9, - "name": "no-hook-inventory-states-the-finding-conditionally", + "name": "partial-hook-inventory-hedges-only-what-it-could-not-read", "prompt": "/audit permissions — settings.json permissions.deny is missing the destructive-git patterns. I have no idea what hooks are installed, and I am not going to go look.", - "expected_output": "A Category B report that, having no way to enumerate a plugin's hooks/hooks.json, does not assume hook coverage is absent. Where a baseline deny pattern is missing and no hook inventory was taken, the finding is stated conditionally — if a PreToolUse hook on Bash already blocks this family, the finding is void — and the report says which inventory would settle it. It does not silently assert the patterns are missing.", + "expected_output": "A Category B report that takes the inventory itself rather than hedging by default: it runs Phase 1.0's check-hook-coverage.sh, which enumerates settings-declared and plugin-declared hooks. On a complete inventory (exit 0) each finding is decided against what was actually enumerated. On a partial one (exit 1) the report hedges ONLY for the sources the script named as unenumerated — stating those conditionally, 'if a PreToolUse hook on Bash already blocks this family, this finding is void', and naming the specific unresolved plugin that would settle it — while deciding normally everything the run did enumerate. A missing or partial inventory is never reported as evidence that no hook covers the family.", "files": [], "expectations": [ - "States a missing baseline deny finding conditionally when no hook inventory was taken, rather than asserting it", - "Says what would settle it — an inventory of the installed plugins' hooks on the Bash tool surface", - "Does NOT treat the absence of a hook inventory as evidence that no hook covers the family" + "Runs the hook-coverage enumeration rather than reading 'I am not going to go look' as licence to skip it", + "Reads the script's exit code and states whether the inventory was complete or partial", + "Hedges only the families a partial inventory could not settle, names the specific unenumerated source, and decides the rest", + "Does NOT treat an absent or partial hook inventory as evidence that no hook covers the family" ] }, { @@ -128,6 +129,32 @@ "States why the narrowing did not apply rather than silently omitting it", "Does NOT describe the declared hook as protection while it is suppressed" ] + }, + { + "id": 11, + "name": "category-g-headless-run-uses-the-debug-route", + "prompt": "/audit all — run this headless, I am invoking you with -p from a script and there is no terminal attached.", + "expected_output": "A Category G section that is actually executed rather than deferred. Because /doctor needs an interactive TTY and this run has none, the report uses the documented headless route instead — the budget warning Claude Code writes to the debug log, visible with --debug — and names which route it took. Any overflow finding states the budget it was measured against: skillListingBudgetFraction, default 0.01 of the context window, or SLASH_COMMAND_TOOL_CHAR_BUDGET with its documented 8,000-character fallback, plus the per-entry skillListingMaxDescChars cap of 1536. If no measurement was obtainable at all, the section says 'not measured' and never 'no overflow'.", + "files": [], + "expectations": [ + "Names --debug as the headless overflow route rather than stopping at /doctor's TTY requirement", + "States the budget constant the finding is measured against (fraction default 0.01, or the 8,000-character SLASH_COMMAND_TOOL_CHAR_BUDGET fallback)", + "Reports an unobtainable measurement as 'not measured', never as 'no overflow'", + "Does NOT present /context as a headless route — it is a second interactive one" + ] + }, + { + "id": 12, + "name": "category-g-lever-matches-the-roster-origin", + "prompt": "/audit all — /doctor says my skill listing is over budget. Almost everything I have is from installed plugins. Give me the cheapest fix.", + "expected_output": "A Category G recommendation that first counts the roster by origin — plugin, project, user — and then names only levers reachable for that composition. On a plugin-dominated roster skillOverrides is not one of them: the settings page states it 'Does not apply to plugin skills, which are managed through /plugin'. The report therefore recommends trimming descriptions at their source in the owning plugins and, where a plugin is not worth its listing cost, disabling it through /plugin — without promising a per-skill name-only state reachable from /plugin, which no page documents. Raising skillListingBudgetFraction or SLASH_COMMAND_TOOL_CHAR_BUDGET is offered last, with its per-turn context cost stated.", + "files": [], + "expectations": [ + "Counts the listing roster by origin before recommending any lever", + "Does NOT recommend skillOverrides name-only for plugin skills, and says why it does not reach them", + "Names /plugin and upstream description trimming as the plugin-roster levers", + "Offers raising the budget last and states that it costs context every turn" + ] } ] } diff --git a/plugins/claude-config/skills/audit/reference/audit-checklist.md b/plugins/claude-config/skills/audit/reference/audit-checklist.md index d1ec68290..6372257b2 100644 --- a/plugins/claude-config/skills/audit/reference/audit-checklist.md +++ b/plugins/claude-config/skills/audit/reference/audit-checklist.md @@ -2,9 +2,9 @@ Validation rules organized by category. Each check has a severity, what to look for, and how to verify. -Category G (skill-listing budget) has no table here — its checks are procedural and live in -[context/validation-categories.md](../context/validation-categories.md), which carries every -category's criteria. +Every Phase 2 category has a table here. What *governs* a category — the reasoning and the ordering a +table row cannot carry — lives alongside it in +[context/validation-categories.md](../context/validation-categories.md). ## A. Schema & Structure @@ -118,6 +118,40 @@ taken, the finding is stated conditionally, not asserted. | Secrets are in settings.local.json only | error | settings.local.json is gitignored | | Path-based env vars use forward slashes | info | Windows compatibility | +## G. Skill-listing budget + +Unlike every other category here, G's inputs are not in a settings file this skill already opened — +they are the listing the running harness assembled. So the first two rows are about *taking a +measurement at all*, and the rest only apply once one exists. + +| Check | Severity | How to verify | +| --- | --- | --- | +| Overflow was measured by a route that works in this run's mode | warning | **Interactive**: `/doctor` reports the listing's cost and its biggest contributors, and needs a TTY — prompt the user to run it and paste the output. **Headless** (`-p`, spawned agent, background job): re-launch with `--debug` and look for the budget warning Claude Code writes to the debug log ([skills](https://code.claude.com/docs/en/skills), "Skill descriptions are cut short"). `/context`'s Skills row is a second *interactive* reading only. Name the route in the finding | +| No measurement is reported as "not measured", never as "no overflow" | error | An unmeasured category that reports clean is the failure mode this row exists to block — the same defect as a passing check that never ran | +| Any overflow finding names the budget constant it was measured against | warning | `skillListingBudgetFraction` (**default `0.01`**) × context window × ~4 chars/token, or `SLASH_COMMAND_TOOL_CHAR_BUDGET` when set (**documented fallback 8,000 chars**). Confirm both in Phase 3.1 against [settings](https://code.claude.com/docs/en/settings) and [env-vars](https://code.claude.com/docs/en/env-vars) — they are upstream-owned. Without the constant the report cannot say *how far* over | +| Roster composition counted before any lever is recommended | warning | Count listing entries by origin: plugin skills, project skills (`.claude/skills/`), user skills (`${CLAUDE_CONFIG_DIR:-~/.claude}/skills/`). This decides which levers exist | +| Every recommended lever is reachable for the origin it targets | error | `skillOverrides` reaches project and user skills only — *"Does not apply to plugin skills, which are managed through `/plugin`"* ([settings](https://code.claude.com/docs/en/settings)). On a plugin-heavy roster the levers are `/plugin` and upstream description trimming. Recommending `skillOverrides` for a plugin skill is a lever the operator cannot pull | +| Per-entry text within the per-skill cap | warning | Combined `description` + `when_to_use` ≤ `skillListingMaxDescChars` (**default `1536`**). This is a per-entry cap, independent of the shared budget above | + +### Measuring it in a repository (in-repo proxy, not the real population) + +Where the audited target is a repository that *publishes* skills, this marketplace already ships the +aggregate measurement: `bash plugins/skill-quality/scripts/check-listing-budget.sh ...`, +surfaced as `skill-quality:check`'s `listing-budget` action. Use it — and state plainly what it is and +is not: + +- **It measures a different population.** The script walks *skills roots in a repository*. Category G + is asking about *the listing the consumer's running session assembled*, which is the installed + plugin cache plus that machine's project and user skills. A repository's own roots are a **proxy** + for that, useful when the audited repo is the publisher, and not a substitute for `/doctor` or + `--debug` on the consumer's machine. Never present its number as the consumer's listing size. +- **It is slow enough to matter for how you call it.** Measured on this Windows machine: + `check-listing-budget.sh plugins/claude-config/skills` → 8 skills in **5.98s real**. It scales + per-skill, so a marketplace-wide `plugins/*/skills` run is minutes, not seconds, and will exceed a + default Bash tool timeout (tracked in #2216). Scope it to the roots you need, or run it in the + background — do not make a Category G step depend on a marketplace-wide invocation completing + inline. + ## H. Model and effort settings Each check reads a settings file this skill already opens, and each detects a value the harness diff --git a/plugins/claude-config/skills/audit/reference/required-permissions.md b/plugins/claude-config/skills/audit/reference/required-permissions.md index abab990d2..67d2491f2 100644 --- a/plugins/claude-config/skills/audit/reference/required-permissions.md +++ b/plugins/claude-config/skills/audit/reference/required-permissions.md @@ -84,11 +84,19 @@ Windows is not supported, and the PowerShell tool lists "On Windows, sandboxing among its preview limitations. On a native-Windows workstation the OS-level remedy is unavailable, so do not offer it there as the fix. -**A `PreToolUse` hook on `Bash|PowerShell` is a speed bump, not a boundary.** It can inspect the -command string and deny the call, and a hook exiting 2 blocks a call an *allow* rule would otherwise -have permitted. A decision it returns cannot loosen a deny — see "Interaction with hook-based gates" -below for the precise ordering. But it inspects that same command string, so it inherits the evasion surface of a -Bash deny glob. Rank it below the sandbox and never describe it as protection. +**A `PreToolUse` hook on `Bash|PowerShell` is a speed bump, not a boundary — *against this threat +model*.** It can inspect the command string and deny the call, and a hook exiting 2 blocks a call an +*allow* rule would otherwise have permitted. A decision it returns cannot loosen a deny — see +"Interaction with hook-based gates" below for the precise ordering. But it inspects that same command +string, so it inherits the evasion surface of a Bash deny glob. Rank it below the sandbox and never +describe it as protection. + +**The ranking is scoped to secret exfiltration; it does not carry to destructive-bash-deny.** It holds +here because an OS-level boundary for *reading a file* exists, so something strictly better than the +hook is on the table. Nothing equivalent exists for a destructive git argument: the sandbox's +vocabulary is `filesystem.*` paths and `network.*` hosts, with no expression for a command's +*arguments*, so it cannot separate `git push` from `git push --force` to the same remote. Do not +carry "rank it below the sandbox" into a destructive-git finding — see that section's own note. **Residual risk, stated plainly.** Where no OS-level boundary is available, a deny glob cannot keep a secret from a session that has shell execution. **Directory location is not a boundary**: a @@ -123,6 +131,30 @@ Bash deny patterns for destructive git operations — the universal baseline. | `Bash(git clean -f *)` | Force clean | | `Bash(git clean -fd *)` | Force clean with directories | +**Report this baseline with its fragility, the same way the Read deny table is reported with its +scope.** Every pattern above constrains a command's *arguments*, and the permissions page's own +warning is that *"Bash permission patterns that try to constrain command arguments are fragile"* +([permissions](https://code.claude.com/docs/en/permissions)). A finding that recommends these without +saying so ships the false confidence the `sensitive-file-deny` section refuses to ship. + +**The concrete hole is prefix anchoring, and it is worth stating in the finding.** Matching is +prefix-based: *"`Bash(npm run test *)` matches Bash commands starting with `npm run test`"*, and a +trailing `*` with a space before it *"enforces a word boundary"* (same page). So +`Bash(git push --force *)` matches `git push --force origin main` and does **not** match +`git push origin main --force`, which is the ordinary spelling. Flag-position variants, `--force-with-lease`, +`-f` bundled into another short-flag cluster, and `git push` aliases all pass the same way. These +patterns raise the cost of an accidental force push; they do not bound a determined one. + +**What the ranking is here, and what it is not.** Do not import the Read-deny section's "rank the hook +below the sandbox": the sandbox constrains filesystem paths and network hosts and has no expression +for a command's arguments, so it does not bound `git push --force` at all. Against destructive git +the available controls are the deny globs above and a `PreToolUse` hook, and the honest ordering is +that a hook can parse the command rather than prefix-match it, while inheriting the same +command-string evasion surface. Upstream supports the fragility claim generally; its *"use PreToolUse +hooks"* recommendation on that page is scoped to URL filtering, so do not cite upstream as ranking +the hook above the glob for destructive commands — that reach is ours to argue, not theirs to have +said. + ## ask-rules (Bash ask) Bash patterns that should require confirmation before execution. `git push` is the canonical ask-gate @@ -160,11 +192,12 @@ or from an installed plugin; a plugin-provided hook is no weaker a block than a turns every hook off, and a managed `allowManagedHooksOnly` or `strictPluginOnlyCustomization` suppresses non-exempt hooks outright. A hook a setting has already switched off blocks nothing, so under any of those the narrowing does not apply at all and the finding stands at its unnarrowed - severity. Category D reads and reports all three, so take the reading from there; where that reading - was not taken, the narrowing is **unavailable** rather than assumed clear — an unread lever is not an - unset one. Category D runs *after* Category B, so pull the reading forward or defer the downgrade and - revise the severity once Category D has run; on a scope-filtered run that never reaches Category D - (`/audit permissions`), the narrowing is unavailable unless the operator supplies the state. + severity. **Phase 1.0's `check-hook-coverage.sh` reports all three**, in every scope it could read, + so the reading is available before Category B runs and on a scope-filtered `/audit permissions` run + as well — the ordering problem this bullet used to carry is gone. Where the reading was not taken at + all, the narrowing is **unavailable** rather than assumed clear: an unread lever is not an unset one. + Note the script reads the scopes it can open; a managed-settings layer it cannot read leaves + `allowManagedHooksOnly` unknown, which is a partial reading, not a clear one. - **The hook is on the tool surface the pattern defends.** `destructive-bash-deny` and `ask-rules` are Bash-command families, so a `PreToolUse` hook on `Bash`/`PowerShell` can cover them. `sensitive-file-deny` is a `Read`-pattern family, and a Read deny covers the built-in file tools as @@ -181,11 +214,21 @@ ends if the providing plugin is disabled or uninstalled, that it is narrowable b the hook exposes, and that it is suppressible later by `disableAllHooks`, `allowManagedHooksOnly`, or `strictPluginOnlyCustomization` even where none of them is set today. -**Fail open where no hook inventory was taken.** The audit has no enumeration path over a plugin's -`hooks/hooks.json` (it reads settings-declared hooks only), so on most runs you will not know what is -installed. Do not resolve that by assuming absence. Where no inventory was taken, state the finding as -conditional — "if a `PreToolUse` hook on `Bash` already blocks this family, this finding is void" — -rather than as an assertion, and say which inventory would settle it. +**Take the inventory; fail open only where it is incomplete.** Phase 1.0 runs +`scripts/check-hook-coverage.sh`, which enumerates settings-declared hooks **and** every enabled +plugin's hook config, resolved through the installed-plugin registry. Read its **exit code**, because +that is what tells you which posture you are in: + +- **`0` — complete.** Every enabled plugin resolved. Narrowing 3 is decidable: an absent pattern whose + family no enumerated hook blocks is a genuine finding at full severity, and one a live hook does + block drops to `info` with the residual named. State that the inventory was taken. +- **`1` — partial.** The script's "Not enumerated" block names what it could not read. For families + those sources could plausibly cover, do **not** assume absence: state the finding as conditional — + "if a `PreToolUse` hook on `Bash` already blocks this family, this finding is void" — and name the + specific unresolved plugin or unparsed file that would settle it. Everything the run *did* enumerate + is still decidable; partial is not a blanket licence to hedge. +- **`2` or not run — no inventory.** Treat as partial for every family, and say so. "Could not look" is + never reportable as "looked and found nothing". ## Interaction with hook-based gates diff --git a/plugins/claude-config/skills/audit/scripts/check-hook-coverage.sh b/plugins/claude-config/skills/audit/scripts/check-hook-coverage.sh new file mode 100755 index 000000000..906936269 --- /dev/null +++ b/plugins/claude-config/skills/audit/scripts/check-hook-coverage.sh @@ -0,0 +1,474 @@ +#!/usr/bin/env bash +# Hook inventory for the audit skill (Categories B and D). +# +# WHY THIS EXISTS. Category D writes rules for hooks that use +# ${CLAUDE_PLUGIN_ROOT} / ${CLAUDE_PLUGIN_DATA}, and those placeholders only +# ever appear in a PLUGIN-provided hook. Category B's third baseline narrowing +# demotes a missing deny pattern to `info` when a LIVE PreToolUse hook already +# blocks the family. Both need to know what hooks are installed — and until this +# script, the skill read settings-declared hooks only and three of its own +# surfaces said so in prose. This is the enumeration path those surfaces were +# describing the absence of. +# +# WHAT IT ENUMERATES. +# 1. Settings-declared hooks: project settings.json, project +# settings.local.json, and the user-scope settings.json. +# 2. Plugin-declared hooks: for every plugin enabled in any of those scopes, +# the plugin's own hook config, resolved through the installed-plugin +# registry so no version-directory guessing is involved. +# 3. The suppression levers that can switch hooks off wholesale, because a +# hook that cannot run is not coverage. +# +# WHERE A PLUGIN'S HOOKS LIVE. plugins-reference (fetched 2026-08-12) says +# "Location: `hooks/hooks.json` in plugin root, or inline in plugin.json", and +# the manifest's `hooks` key is `string|array|object` — a path, several paths, +# or an inline config. All four shapes are read here; a plugin whose shape this +# script cannot parse is reported UNREADABLE, never silently as "no hooks". +# +# WHAT IT DOES NOT DO. It does not decide whether a hook covers a permission +# family. That judgment is the audit's, and required-permissions.md "Narrowing +# the baseline" carries the three preconditions it has to apply. This script +# answers only "what is installed", which is the question that previously had no +# answer at all. +# +# Read-only: opens JSON and prints. Never executes a hook command. +# +# Exit codes: +# 0 inventory COMPLETE — every enabled plugin resolved to a real directory +# 1 inventory PARTIAL — at least one enabled plugin could not be resolved or +# read. The audit keeps the conditional-finding posture for the families +# those plugins might cover. Not an error; a stated limit. +# 2 fatal — jq missing, or no settings scope readable at all +# +# Env overrides (the test seam): +# HOOK_COVERAGE_FIXTURE_DIR project root to scan instead of the git toplevel +# HOOK_COVERAGE_INSTALLED_JSON path to installed_plugins.json +# HOOK_COVERAGE_USER_DIR user config dir (else CLAUDE_CONFIG_DIR, else $HOME/.claude) + +set -uo pipefail + +# Every `jq` result that re-enters the shell goes through this. On Git for +# Windows, jq writes stdout in TEXT mode and appends a CR to every line — a +# property of jq's own output stream, NOT of the input file's line endings, so +# it happens even when every fixture is pure LF. Verified here by deleting the +# `tr` and re-running the suite: 17 of 34 checks failed, including cases whose +# fixtures contain no CR at all. Untreated, a plugin key read out of jq is +# `name@marketplace\r`, every registry lookup misses, and the plugin is reported +# as not installed on a machine where it is installed — a silent false negative +# on exactly the axis this script exists to make trustworthy. +jqs() { jq "$@" 2>/dev/null | tr -d '\r'; } + +usage() { + cat <<'EOF' +check-hook-coverage.sh — enumerate the hooks actually installed for this project. + +Reads settings-declared hooks (project, local, user scope) and plugin-declared +hooks (via the installed-plugin registry), plus the levers that suppress hooks +wholesale. Read-only; never runs a hook. + +Usage: + check-hook-coverage.sh [--json] [--help] + + --json emit the inventory as JSON on stdout instead of a table + +Exit: 0 inventory complete; 1 inventory partial (some plugin unresolved); + 2 fatal (jq missing, or no settings scope readable). +EOF +} + +EMIT_JSON=0 +while [[ $# -gt 0 ]]; do + case "$1" in + -h | --help) + usage + exit 0 + ;; + --json) + EMIT_JSON=1 + shift + ;; + *) + echo "ERROR: unknown argument: $1" >&2 + usage >&2 + exit 2 + ;; + esac +done + +if ! command -v jq >/dev/null 2>&1; then + echo "ERROR: jq required (install with: winget install jqlang.jq | apt install jq | brew install jq)" >&2 + exit 2 +fi + +# --- Roots ------------------------------------------------------------------- + +if [[ -n "${HOOK_COVERAGE_FIXTURE_DIR:-}" ]]; then + PROJECT_ROOT="$HOOK_COVERAGE_FIXTURE_DIR" +else + # Consumer project root: the cwd's git toplevel, then Claude Code's exported + # project dir, then cwd. Never the plugin's own install directory. + PROJECT_ROOT="$(git rev-parse --show-toplevel 2>/dev/null | tr -d '\r')" + [[ -n "$PROJECT_ROOT" ]] || PROJECT_ROOT="${CLAUDE_PROJECT_DIR:-$PWD}" +fi + +if [[ -n "${HOOK_COVERAGE_USER_DIR:-}" ]]; then + USER_DIR="$HOOK_COVERAGE_USER_DIR" +elif [[ -n "${CLAUDE_CONFIG_DIR:-}" ]]; then + USER_DIR="$CLAUDE_CONFIG_DIR" +elif [[ -n "${HOME:-}" ]]; then + USER_DIR="$HOME/.claude" +else + USER_DIR="" +fi + +INSTALLED_JSON="${HOOK_COVERAGE_INSTALLED_JSON:-}" +if [[ -z "$INSTALLED_JSON" && -n "$USER_DIR" ]]; then + INSTALLED_JSON="$USER_DIR/plugins/installed_plugins.json" +fi + +SCOPES=() +SCOPE_LABELS=() +add_scope() { + # add_scope