Skip to content

feat(claude-config): extend the audit-instructions catalog with checks I12-I14 - #1349

Merged
kyle-sexton merged 7 commits into
mainfrom
feat/context-engineering-criteria-catalog
Jul 25, 2026
Merged

feat(claude-config): extend the audit-instructions catalog with checks I12-I14#1349
kyle-sexton merged 7 commits into
mainfrom
feat/context-engineering-criteria-catalog

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

No linked issue — this PR is lane L3 of the context-engineering-rightsizing topic effort,
which is tracked in docs/topics/context-engineering-rightsizing/ rather than as a filed GitHub
issue. See Related for the issues this work defers to.

Summary

Folds the context-engineering criteria work into the incumbent audit-instructions catalog
rather than standing up a second one. plugins/claude-config/skills/audit-instructions/reference/criteria.md
already carried the three axes, eleven seeded checks I1–I11, the surface partition, and the source
list, and its consuming skill is already report-only (SKILL.md:25-29, "There is no --fix"), so
"never auto-applied" is inherited rather than built.

Three rows added. Every cited page was re-fetched during this session and quoted verbatim.

Row Tier Severity Surfaces Flags
I12 Stale or misattributed harness-capability claim behavioral warning all an instruction asserting a Claude Code harness behavior the current official page contradicts
I13 Citation form that does not load mechanical warning non-memory an @path written where @ carries no import meaning, so the cited content silently never arrives
I14 Retrieval of an already-loaded surface mechanical info agent definitions, skill bodies an instruction telling a subagent to read a surface its own startup context already contains

Each row carries its must-not-flag cases. Three official sources join the catalog's list:
CLI reference,
subagents,
skills.

Why each row survived the incumbent gate, which eliminated 44 of 47 enumerated candidates (94%):

  • I12 vs I8 — I8's subject is the model (prior-model workarounds); I12's is the product.
    Not covered by claude-memory's C7 (Currency), which checks references, versions and counts
    against the repo, never against a vendor page, and is memory-surface-only.
  • I13 vs check-skill.sh check 5 — check 5 verifies that backtick-cited internal files
    resolve. I13's target is an un-backticked @path on a surface where @ is inert. The
    memory page defines @path/to/import for the CLAUDE.md family; the skills page defines no import
    syntax at all, and instead says supporting files are referenced "so Claude knows what each file
    contains and when to load it"
    .
  • I14 — no existing checker knows what a non-fork subagent's startup context contains. Its
    must-not-flag list is anchored on the same page's statement that "Explore and Plan are the only
    subagents that omit CLAUDE.md and git status. There is no frontmatter field or per-agent setting
    to change which agents skip them"
    .

A fourth candidate was drafted and then eliminated. "Flag a positive-form absolute pinning a
style value the working context already exhibits" is covered on non-memory surfaces by I4
("content the model can derive from the code, standard language conventions it already knows")
and on memory surfaces by claude-memory's C5, which carries the identical official quote.
It is also refuted rather than merely unbacked: the memory page's Specificity section holds
up "'Use 2-space indentation' instead of 'Format code properly'" as the good example — the
exact phrasing the check would have flagged. claude-memory's C4 already cites it.

With that candidate gone, the Default state axis and the widened OPINION gloss drafted
alongside it had no consumer, so they are not shipped — an unused axis a consuming SKILL.md
must then explain is speculative structure. The Authority gloss instead states the rule the gate
actually applied: a candidate whose only backing is a practitioner's claim earns a row when an
official page states the behavior it asserts, and is eliminated when the search finds none.
All
fourteen checks remain ANTHROPIC-DOCS and the axis stays a closed three-value set, so no
existing consumer of the axis breaks.

Version. claude-config 0.9.20.10.0. New backward-compatible checks added to an
existing catalog are a MINOR bump under the semver the plugin's own CHANGELOG.md:3-4 declares;
the repo root CLAUDE.md:47 mandates only that an explicit semver version exist. Catalog header
1.0.01.1.0. origin/main carries 0.9.2 after rebase, so 0.10.0 is unclaimed.

Test plan

Every command below was run from the branch worktree after rebasing onto origin/main at
f4627e7172.

  • plugins/skill-quality/scripts/check-skill.sh audit-instructions
    (CHECK_SKILL_SKILLS_ROOT=plugins/claude-config/skills) — PASS, 0 errors, 0 warnings.
    SKILL.md 166/500 lines; all 6 base-ref trigger phrases preserved; markdownlint clean;
    scripts/instruction-scan.test.sh passed.
  • npx markdownlint-cli2 "plugins/claude-config/**/*.md" — 17 files, 0 errors.
  • scripts/check-changelog-parity.sh --check and --check-bump origin/mainexit 0 (the
    manifest bump has a matching ## [0.10.0] entry).
  • Repo-wide grep for residual eleven / I1–I11 / I6–I11 / I15 / fifteen-check: the only
    survivors are historical CHANGELOG.md entries for 0.9.x, which are correctly immutable.

Source verification. All four official pages were fetched in-session and the quoted lines read
off the live fetch, not carried forward from an earlier agent:

Page Used by Verified verbatim
cli-reference I12 "Print read-only installation and settings diagnostics from the terminal without starting a session … For the in-session setup checkup that can also apply fixes, run /doctor."
memory I13, and the I15 elimination "CLAUDE.md files can import additional files using @path/to/import syntax"; "wrap it in backticks"; "'Use 2-space indentation' instead of 'Format code properly'"
skills I13 "Reference supporting files from SKILL.md so Claude knows what each file contains and when to load it" — no import syntax defined anywhere on the page
sub-agents I14 "every level of the CLAUDE.md hierarchy the main conversation loads, including ~/.claude/CLAUDE.md, project rules, CLAUDE.local.md, and managed policy files"; "There is no frontmatter field or per-agent setting to change which agents skip them"

scripts/run-plugin-tests.sh claude-config was started locally but did not complete inside the
session's window — the machine was saturated by concurrent work. It is left to CI, which runs the
same gate. Nothing was suppressed or skipped to make a check pass.

Related

…s I12-I14

Folds the context-engineering criteria work into the incumbent
`audit-instructions` catalog rather than standing up a second one. The
catalog already carried the three axes, eleven seeded checks I1-I11, the
surface partition, and the source list; its consuming skill is already
report-only, so "never auto-applied" is inherited, not built.

Three rows added, each cited to a page re-fetched this session:

- **I12** (behavioral / warning / all surfaces) flags an instruction
  asserting a Claude Code *harness* behavior the current official page
  contradicts. The subject is the product, not the model, which is what
  separates it from I8; `claude-memory`'s C7 checks references against
  the repo, never against a vendor page.
- **I13** (mechanical / warning / non-memory) flags an `@path` written
  where `@` carries no import meaning, so the cited content silently
  never arrives. Import syntax is a property of the CLAUDE.md family;
  the skills page defines none. Distinct from `check-skill.sh` check 5,
  which verifies that backtick-cited internal files resolve.
- **I14** (mechanical / info / agent definitions and skill bodies) flags
  an instruction telling a subagent to read a surface its own startup
  context already contains.

Each row carries its must-not-flag cases. Three official sources join
the catalog's list: CLI reference, subagents, skills.

A fourth candidate was drafted and then eliminated by the incumbent
gate. "Flag a positive-form absolute pinning a style value the working
context already exhibits" is covered on non-memory surfaces by I4
("content the model can derive from the code, standard language
conventions it already knows") and on memory surfaces by
`claude-memory`'s C5, which carries the identical official quote. It is
also refuted rather than merely unbacked: the memory page's Specificity
section holds up *"Use 2-space indentation" instead of "Format code
properly"* as the good example -- the exact phrasing the check would
have flagged.

With that candidate gone, the `Default state` axis and the widened
`OPINION` gloss drafted alongside it had no consumer, so they are not
shipped. The `Authority` gloss instead states the rule the gate applied:
a candidate whose only backing is a practitioner's claim earns a row
when an official page states the behavior it asserts, and is eliminated
when the search finds none. All fourteen checks remain `ANTHROPIC-DOCS`
and the axis stays a closed three-value set.

Version: `0.9.2` -> `0.10.0` (new backward-compatible checks are a MINOR
bump under the semver the plugin's own CHANGELOG header declares).
Catalog header `1.0.0` -> `1.1.0`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 2m 43s —— View job


PR Review: feat(claude-config): extend audit-instructions catalog with checks I12-I14

Status: Review complete

Tasks:

  • Read changed files and gather context
  • Review criteria.md additions (I12, I13, I14)
  • Review SKILL.md updates
  • Review plugin.json version bump
  • Review CHANGELOG.md entry
  • Review README.md changes
  • Review evals.json changes
  • Post final review

Overview

The PR is well-structured and internally consistent. Three checks with clear, non-overlapping scope are added to the incumbent catalog. The PR description documents the gate process thoroughly, including the eliminated candidate. No security or correctness blockers. Two items worth discussing before merge:


Finding 1 — "Full catalog I1–I14" in SKILL.md is slightly imprecise for non-memory surfaces

File: plugins/claude-config/skills/audit-instructions/SKILL.md

Observation:
SKILL.md line 50 states:

On non-memory surfaces (skill bodies, agent definitions, prompt-type hooks, output styles) the full catalog I1–I14 applies

But I14's row in criteria.md scopes it to "agent definitions and skill bodies" only — it does not apply to prompt-type hooks or output styles. A model implementing Phase B that reads only SKILL.md for dispatch rules might apply I14 to hook text, producing spurious findings on an output style instructed to read CLAUDE.md.

The existing pattern uses "full catalog" as shorthand with the per-check scope in criteria.md as the authoritative override, but the SKILL.md text could be made precise at low cost:

On **non-memory surfaces** (skill bodies, agent definitions, prompt-type hooks, output styles) the
full catalog I1–I14 applies — no incumbent auditor covers instruction content there. (I14 is
further bounded to agent definitions and skill bodies; see the catalog for per-check surface
applicability.)

This is a low-severity precision nit, not a correctness bug — criteria.md governs if both are read — but SKILL.md is the first document a subagent lane receives.


Finding 2 — No evals for the three new checks

File: plugins/claude-config/skills/audit-instructions/evals/evals.json

Observation:
Six evals ship today; none exercises I12, I13, or I14. The two checks with the clearest boundary conditions are strong eval candidates:

  • I13 must-NOT-flag boundary — a backtick-wrapped `@path/to/file` on a skill body must not be flagged; an unquoted @path/to/file in a skill body prose sentence must be flagged. This boundary is subtle (the backtick is exactly the documented escape), and a future model seeing only the detect clause could miss it.
  • I14 must-NOT-flag for Explore/Plancriteria.md anchors the carve-out on the documented fact that Explore and Plan are the only subagents that skip CLAUDE.md with no per-agent override. An eval that prompts /claude-config:audit-instructions against a delegation prompt that restates CLAUDE.md context specifically for Explore would verify the check doesn't false-positive here.

I12 is behavioral tier and needs a fetch, so it's a weaker eval target (the PR notes this is intentional — I12 stays advice). But I13 and I14 are both mechanical tier, which is exactly what the existing evals for I8 and I9 nuance test. The PR description records I13 as a #445 candidate (deterministic gate), making its correct boundary behavior especially important to lock down before it moves into a CI lane.

Not blocking, since the PR description notes I13/I14 as recorded #445 candidates, implying a follow-up lane would bring its own test coverage. Worth filing a follow-on ticket if one isn't already tracked.


Everything else looks good

  • I12/I13/I14 check bodies: Detect, remediate, must-NOT-flag, and source are each precise and cross-consistent with the surface partition. The I12/I8 and I13/check-5 non-overlap arguments in the PR description are sound.
  • Surface partition paragraph (criteria.md lines 31–32): I6–I12 apply to all surfaces; I13 and I14 name narrower surface sets — matches each check row's Surfaces: field exactly.
  • Version bump 0.9.20.10.0: Correct MINOR bump (backward-compatible addition to an existing catalog). criteria.md header 1.0.01.1.0 is also present and correct.
  • CHANGELOG.md: Both Added and Changed entries are accurate; the Changed entry for the Authority gloss captures the gate rule that now governs the closed set.
  • evals.json eval fix: correct plugin-system inaccuracies and harden references #2 updated from I6–I11 to I6–I12 — consistent with SKILL.md's scope boundary update.
  • README.md "eleven-check" → "fourteen-check" in both the table cell and the prose paragraph — both updated.
  • Three new sources in the source list (CLI reference, subagents, skills) all have live URLs and the PR describes in-session fetch verification for each.
  • Authority gloss widening to state the candidate-gate rule does not change the closed three-value set — clean.

@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 28s —— View job


Security Review — PR #1349

  • Gather PR diff and changed files
  • Review plugin manifest changes
  • Review criteria.md additions (I12–I14)
  • Review SKILL.md and evals changes
  • Post security findings

Findings

No security vulnerabilities found.

This PR is a documentation-only catalog expansion. All six changed files are markdown content or JSON metadata — no scripts, no workflow files, no code execution surfaces.

Coverage checklist:

Vector Assessment
Code execution / command injection No scripts added or modified
Path traversal No file path handling; criteria.md contains natural-language patterns only
Secrets / credentials No secret handling introduced
GitHub Actions No workflow files modified
Supply-chain / dependency pins No dependency changes
Permission widening No permissions: or settings changes
Third-party / untrusted sources The three new source URLs (cli-reference, sub-agents, skills) are all code.claude.com and platform.claude.com — official Anthropic documentation
Automated remediation blast radius Skill is report-only and never auto-applied; new checks I12–I14 inherit that ceiling from the existing skill
I13 @path detection The check is advisory only — flagging inert @path usage on non-memory surfaces cannot trigger any automated action
I14 memory-surface boundary I12 promoted to "all surfaces" in SKILL.md (I6–I12 range) is consistent with its own row declaring Surfaces: all; no boundary inconsistency

Nothing in the changes creates an injection surface, weakens an access boundary, introduces a supply-chain dependency, or expands automated behavior. Security posture is unchanged from pre-PR state.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 118855a1c8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/claude-config/skills/audit-instructions/reference/criteria.md Outdated
Recheck triggers watched three pages while the catalog cites nine. Every
check cites a page in the source list, so the trigger set is now the
source list — a subset left the new harness-behavior rows depending on
pages nothing watched.

I12 treated a page that no longer documents a behavior as evidence the
behavior drifted. Documentation silence is not evidence: pages are
rewritten, condensed and reorganized, and this repository deliberately
keeps empirical tests for behaviors the official pages never specified.
Detection now requires an incompatible statement on the current page or
a failed version-matched reproduction, and silence is called out as a
must-not-flag rather than left implicit.

I14 treated every CLAUDE.md and project rule as already resident, but
the subagent guarantee it cites covers the hierarchy the main
conversation loads. Nested CLAUDE.md files and path-scoped rules load
lazily when work reaches their scope, so an instruction to read a
package-local file before operating there can be doing real work.
Detection is bounded to the startup-loaded set and the lazy-loaded
surfaces are an explicit must-not-flag, with the memory page's own
lazy-loading line carried in the source row as the bound.

Also restates the Authority gloss descriptively: "all fourteen checks
are currently ANTHROPIC-DOCS" is a statement about present contents, not
a rule that would foreclose a future TALK or OPINION row.
@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 2m 15s —— View job


PR Review: feat(claude-config): extend audit-instructions catalog with checks I12-I14

Tasks:

  • Gather context and understand the request
  • Read changed files at HEAD (8faced0e0a)
  • Verify Codex review findings were addressed
  • Review overall PR quality
  • Post final review findings

Codex Findings — All Three Closed in 8faced0

P2-1: Recheck triggers watching only three pagesFixed.
criteria.md lines 12–15 now reads "a change to any page listed under Sources below," with the rationale that each check cites one of those pages — a subset would leave harness-behavior rows depending on pages nothing watches. The trigger is now self-maintaining: new sources automatically become watched.

P2-2: Documentation silence treated as drift evidence (I12)Fixed.
criteria.md lines 180–184 promotes silence to a first-class must-not-flag with the rationale spelled out verbatim: pages are routinely rewritten and condensed, and this repo keeps empirical tests for behaviors the docs never specified. Detection now requires either an explicit incompatible statement on the current page or a failed version-matched reproduction. The fix is precise and correctly aligned with the repo's own test philosophy.

P2-3: I14 treating all CLAUDE.md/rules as startup-residentFixed.
criteria.md lines 221–230 now bounds I14 to the startup-loaded set (root CLAUDE.md, ~/.claude/CLAUDE.md, CLAUDE.local.md, unconditional project rules, managed policy files) and makes lazy-loaded surfaces an explicit must-not-flag with the memory page's own lazy-loading language carried in the source row. The fix is well-scoped and the must-not-flag wording ("when a surface's residency is not established, leave it") provides the right conservative default.


Open Item from Prior Review — SKILL.md I14 Surface Precision

SKILL.md line 50 still reads "the full catalog I1–I14 applies" for all non-memory surfaces, but I14's own row in criteria.md scopes it to "agent definitions and skill bodies" only (not prompt-type hooks or output styles).

Practical risk is low because Phase B explicitly shares criteria.md with every subagent lane ("each sharing reference/criteria.md"), so the lane will see the narrowing when it reads the catalog. The surface partition paragraph in criteria.md also states "I13 and I14 name narrower surface sets in their own rows." A subagent that reads both documents will not false-positive on this. The previous review correctly called it a precision nit rather than a correctness bug. Whether to tighten SKILL.md's language here (adding a parenthetical pointing to per-check surface applicability) is a judgment call — it would be a one-line change and would make SKILL.md independently accurate, but it is not blocking.


Known Gap — No Evals for I12/I13/I14

Still no eval exercising the three new checks; evals.json has six entries covering I1–I11 behavior. As the prior review noted, I13 and I14 are the strongest candidates given their mechanical tier and #445 status. Nothing new to add: the PR description treats this as an acknowledged follow-up and the gap does not block the catalog from being usable.


Everything Else Holds

  • I12 check body is well-formed after the fix. The canonical doctor pair (terminal claude doctor read-only / in-session /doctor applies fixes) is a good concrete anchor for the must-not-flag.
  • I13 check body is precise. The backtick escape (`@path`) as an explicit must-not-flag directly matches the memory page's stated design.
  • I14 check body is precise after the fix. The Explore/Plan carve-out is properly anchored to the documented fact that they are the only subagents that skip CLAUDE.md with no per-agent override.
  • Surface partition paragraph (criteria.md lines 31–32): I6–I12 apply to all surfaces; I13 and I14 name narrower surface sets matches each row's Surfaces: field exactly.
  • CHANGELOG.md 0.10.0 section correctly records both the Added items (I12–I14) and the Changed items from the fix commit (recheck triggers, I12 silence handling, I14 boundary, Authority gloss). The gloss restatement ("all fourteen checks are currently ANTHROPIC-DOCS" as a present-tense statement, not a rule) is a clean way to keep the closed three-value set extensible without widening it.
  • Version bumps (0.9.2 → 0.10.0, catalog 1.0.0 → 1.1.0) are correct MINOR bumps for backward-compatible additions.
  • README.md "eleven-check" → "fourteen-check catalog" and "prompting doctrine" → "prompting and harness doctrine" — both accurate.

Overall: The three Codex findings are properly closed by 8faced0. The PR adds three non-overlapping, well-sourced checks with clean boundary conditions. The SKILL.md I14 precision nit is the only unresolved item, and it carries low practical risk given how criteria.md is delivered to subagent lanes.

@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 24s —— View job


Security Review — PR #1349

Tasks:

  • Load tools
  • Fetch PR diff
  • Review changed files for security issues
  • Post findings

Findings

No security vulnerabilities found.

This PR makes exclusively documentation and metadata changes across six files. Full coverage checklist below.

Vector Assessment
Code execution / command injection No scripts added or modified
GitHub Actions / workflow injection No workflow files touched — .github/ is unchanged
Secrets / credentials No secret handling in any changed file
Path traversal criteria.md contains natural-language detection patterns, not evaluated code or path operations
Supply-chain / dependency pins No dependency files changed
Permission widening No permissions:, allowed_tools, or settings changes
Third-party URLs The three new source URLs (cli-reference, sub-agents, skills) are all official code.claude.com Anthropic documentation
I12 live-fetch surface I12 is behavioral tier — detection requires a fetch + judgment by a human-gated skill, not automated code execution. No fetch is triggered by any file in this PR.
I13 @path detection I13 is advisory text describing a pattern to flag in other files. The skill is report-only; no automated action fires on a finding.
I14 startup-surface detection I14 is advisory text bounded to a startup-loaded file set. The second commit tightens the boundary to exclude lazily-loaded surfaces (nested CLAUDE.md, path-scoped rules) — a more conservative, not more expansive, detection surface.
evals.json content Updated strings are expected-output descriptions for eval assertions — no executable content, no external fetch instructions, no schema changes.
Plugin manifest Version bump 0.9.20.10.0 only — no new permissions, no new MCP servers, no new hook definitions.
Automated remediation blast radius The skill is report-only and never auto-applied; the new checks inherit that ceiling from the existing skill. Nothing in this PR changes that.

Nothing in the changes creates an injection surface, weakens an access boundary, introduces a supply-chain dependency, expands automated behavior, or touches any execution path.


@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8faced0e0a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/claude-config/skills/audit-instructions/reference/criteria.md Outdated
…me forms

The startup list named CLAUDE.local.md without qualification while the
must-not-flag exception covered only nested CLAUDE.md, so an instruction
to open a package-local CLAUDE.local.md was flaggable. Lazy loading is a
property of the location, not the name: both forms load when work
reaches their subdirectory. The startup list now says root-level
explicitly and the exception covers both forms.
@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 2m 34s —— View job


PR Review: feat(claude-config): extend audit-instructions catalog with checks I12-I14

Tasks:

  • Gather context and understand the request
  • Read changed files at HEAD (494b96993c)
  • Review criteria.md additions (I12, I13, I14)
  • Review SKILL.md updates
  • Review plugin.json, CHANGELOG.md, README.md, evals.json
  • Post final review

Review at HEAD 494b96993c

This is the third review cycle. All four findings raised across the two prior cycles are closed. The check below is a fresh read of the current state.


Closed items confirmed

Codex P2-1 (recheck triggers)criteria.md:11–15
The trigger is now "a change to any page listed under Sources below." Self-maintaining: new rows automatically inherit coverage through their source entry. ✓

Codex P2-2 (silence ≠ drift in I12)criteria.md:180–184
Silence is a first-class must-not-flag with the rationale stated inline. Detection now requires an explicit incompatible statement on the current page OR a failed version-matched reproduction. ✓

Codex P2-3 (I14 startup boundary)criteria.md:221–230
Detection names the startup-loaded set explicitly. Lazy-loaded surfaces (nested CLAUDE.md, nested CLAUDE.local.md, path-scoped rules) are a must-not-flag with the location-based rationale. ✓

Codex (post-fix): nested CLAUDE.local.md symmetrycriteria.md:228–230
Both filename forms now share the carve-out, governed by location rather than name. The startup list is also corrected to say root CLAUDE.md and root CLAUDE.local.md, removing the asymmetry. ✓


Remaining items (no new findings)

Precision nit: SKILL.md line 50SKILL.md:49–50

Still reads "full catalog I1–I14 applies" for all non-memory surfaces. I14 is scoped to "agent definitions and skill bodies" — prompt-type hooks and output styles are excluded. criteria.md:32 does carry "I13 and I14 name narrower surface sets in their own rows" and every Phase B lane receives criteria.md directly, so the practical risk remains low. This was correctly characterized as a precision nit in prior reviews and is still the only unresolved item.

One-line fix if desired: Fix this →

No evals for I12/I13/I14evals/evals.json

Still no eval exercising the three new checks. Acknowledged as a follow-up recorded against #445; I13 and I14's backtick-escape boundary and Explore/Plan carve-out remain the strongest eval candidates when that lane lands.


Everything else

  • I12 body: Tier behavioral is correct (requires fetch + judgment). The canonical claude doctor / /doctor pair is the right anchor for both the source and the must-not-flag. Separation from I8 (product vs model) is clear and stated.
  • I13 body: Detect is precisely scoped. Must-not-flag covers the documented backtick escape, package scopes (@anthropic-ai/…), decorators, email, and usernames — no false-positive gaps visible. The contrast between the memory page and the skills page in the source formulation correctly establishes why @ is inert on non-memory surfaces.
  • I14 body: After all three fix commits the check is well-bounded. "Root-level is load-bearing" rationale prevents over-flagging nested files of the same name. Both filename forms in the carve-out are governed by location, which is the right invariant.
  • Surface partition: criteria.md:31–32 ("I6–I12 apply to all surfaces; I13 and I14 name narrower surface sets in their own rows") matches every check row's Surfaces: field exactly.
  • SKILL.md: Memory-surface range updated to I6–I12 ✓; count updated to "fourteen checks I1–I14" ✓.
  • evals.json eval fix: correct plugin-system inaccuracies and harden references #2: I6–I11I6–I12 ✓. Existing evals unaffected.
  • CHANGELOG.md 0.10.0: Added and Changed entries accurately reflect all three commits. The Changed bullet for the Authority gloss captures "a statement about present contents, not a rule" — correctly keeps the closed three-value set extensible.
  • Version bumps: 0.9.2 → 0.10.0 (plugin.json) and 1.0.0 → 1.1.0 (criteria.md header) are correct MINOR bumps for backward-compatible additions.
  • README.md: "eleven-check" → "fourteen-check catalog" and "prompting doctrine" → "prompting and harness doctrine" — both accurate, the latter correctly scoping to harness doctrine since I12 is product-behavior-specific.

Overall: No new blocking findings. The SKILL.md I14 surface precision nit is the only open item, carries low practical risk, and has been called out across two prior review cycles. The PR is clean.

@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 32s —— View job


Security Review — PR #1349

Tasks:

  • Load tools
  • Fetch PR diff
  • Review changed files for security issues
  • Post findings

Findings

No security vulnerabilities found.

This PR is an exclusively documentation and metadata change across six files. Full coverage below.

Vector Assessment
Code execution / command injection No scripts added or modified. The existing instruction-scan.sh reference in SKILL.md is unchanged.
GitHub Actions / workflow injection No .github/ files touched.
Secrets / credentials No secret handling in any changed file.
Path traversal criteria.md contains natural-language detection patterns; no evaluated file path operations.
Supply-chain / dependency pins No dependency files changed.
Permission widening No permissions:, allowed_tools, or settings changes in plugin.json — version field only.
Third-party URLs The three new source URLs (code.claude.com/docs/en/cli-reference, code.claude.com/docs/en/sub-agents, code.claude.com/docs/en/skills) are all official Anthropic documentation.
I12 live-fetch surface I12 is behavioral tier — detection requires a fetch + judgment at skill-execution time, not from any file in this PR. The skill is report-only; no automated action fires on a finding. The must-not-flag on silence further constrains the detection surface.
I13 @path detection Advisory-only text describing an inert-citation pattern to surface in other files. No automated remediation path.
I14 startup-surface detection Advisory-only, bounded by the fix in 494b969 to startup-loaded files only. The CLAUDE.local.md nested-form exclusion applies to both filename forms, scoped by location rather than name — conservative by design.
Plugin manifest (plugin.json) Version bump 0.9.2 → 0.10.0 only. No new permissions, MCP servers, hook definitions, or trust delegation.
Automated remediation blast radius Skill is report-only (SKILL.md:25-29, "There is no --fix"). All three new checks inherit that ceiling — no finding triggers an automated file change.

Nothing in the changes creates an injection surface, weakens an access boundary, introduces a supply-chain dependency, expands automated behavior, or touches any execution path. Security posture is unchanged from pre-PR state.

kyle-sexton added a commit that referenced this pull request Jul 25, 2026
The execution record described lanes by what sat uncommitted in a
worktree, which is unreachable from a clone and therefore not a
citation. Every lane has since published, so each row now names the PR
or issue that carries its result: L2 as #1343, L3 as #1349 with the
UNBACKED-to-OPINION mapping stated so the claim set does not need to
survive as a separate list, L4 as #1385 closed with recovery on #1403,
L1 as #1286 closed with the fold on #1271 and gate defects on #1404.

The source article is named with its URL rather than left as "a
practitioner article", so a reader auditing this ADR's premises can
reach the thing the digests measured. The digests themselves prune with
the contract slice, which is why the pointer replaces them rather than
supplementing them.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 494b96993c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/claude-config/skills/audit-instructions/reference/criteria.md Outdated
Detection compared a harness claim against the current page even when
the claim was scoped to a pinned or supported older release, so an
accurate version-gated instruction could be flagged and rewritten wrong.
That contradicted I12's own remediation, which asks authors to carry a
minimum version with such a claim.

The comparison is now scoped to the version the claim is about, and a
successful version-matched reproduction settles it — newer behavior on
the latest page is then evidence of a version difference rather than of
an inaccurate instruction. A claim carrying no version is about current
behavior and is still measured against the current page.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aee49bfb21

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/claude-config/skills/audit-instructions/SKILL.md
Comment thread plugins/claude-config/skills/audit-instructions/reference/criteria.md Outdated
Resolves the claude-config version and changelog collision: main landed
0.9.3 (setup evals) while this branch was on 0.10.0. Both sections are
kept, 0.10.0 stays the version this branch ships, and the 0.9.3 entry
takes its place beneath it.
@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 3m 12s —— View job


PR Review: feat(claude-config): extend audit-instructions catalog with checks I12-I14

Tasks:

  • Gather context and understand the request
  • Read changed files at HEAD (4f8b860)
  • Review latest Codex findings (SKILL.md discoverability + I13 false-positive concern)
  • Verify previously open items are still addressed
  • Post final review

Review at HEAD 4f8b860

This is the fourth review cycle. All prior findings (Codex P2-1 through P2-4, plus the nested CLAUDE.local.md asymmetry and the version-gated claim issue) are confirmed closed. Fresh read of the current state below.


Latest Codex findings — assessed

Codex SKILL.md:20 — Discoverability gap for I12–I14 ✗ Confirmed P2

SKILL.md:3 (frontmatter description)

With disable-model-invocation: false, the model selects this skill based solely on the description field. The current Use when: list covers only model-era problem shapes:

'after a model upgrade', 'are my instructions holding the model back', 'instructions the model no longer needs', 'too prescriptive', 'audit instructions', 'instruction audit'

None of these phrases match the problem shapes that produce I12–I14 findings. A user asking any of the following would not trigger this skill via model invocation:

  • "do any of my agent definitions tell Claude to re-read CLAUDE.md at startup?" (I14)
  • "does this skill file use broken @path imports?" (I13)
  • "is there a stale harness claim in my instructions?" (I12)

The description: body also describes the skill as finding "instructions current models no longer need" — that framing maps to I1–I11 and says nothing about harness-behavior accuracy (I12), citation form (I13), or startup-context redundancy (I14). Codex correctly identifies that the three new checks are undiscoverable via model invocation.

Suggested one-liner addition to the Use when: list: 'stale harness claim', 'broken @path import', 'redundant read'. Or a brief clause appended to the description: prose: "Also detects stale harness-capability claims, inert @path citations on non-memory surfaces, and redundant startup-context reads."

This is a real usability gap, not a precision nit. Fix this →


Codex criteria.md:211 — I13 false-positive scope ✓ Detect clause guards against it

criteria.md:207–211

Codex argues that Follow @reference/rules.md in a skill body could still work because Claude reads referenced paths on demand, and therefore flagging it is a false positive.

The detect clause already carries the guard that settles this: "in a sentence that treats it as though the file arrives." The check is not triggered by any @path mention — it requires the sentence to be making an assumption about automatic import semantics. A neutral reference (see @reference/rules.md for context) could reasonably survive the check; an instruction that depends on the file being pre-loaded (All decisions must follow the rules in @reference/rules.md) would be flagged for the correct reason: the author assumed auto-import behavior that doesn't exist on skill surfaces.

Codex's suggestion to "limit to prose that explicitly claims automatic import" is materially what the detect clause already does.

One genuinely improvable phrase: "its content silently never arrives" (line 211) overstates the case — Claude can still read the file on demand after seeing the path, so "never arrives" isn't strictly accurate. The more precise claim is that the @ import mechanism is inert, not that the content is unreachable. Tightening to "its content is not auto-imported" or "the @ prefix carries no import effect here" would remove the foothold for this objection without changing the check's scope. This is a wording nit, not a false-positive risk with the current qualifier in place.


Carry-over items (no new state)

Precision nit: SKILL.md line 50SKILL.md:49–50

"Full catalog I1–I14 applies" for all non-memory surfaces, but I14 is bounded to "agent definitions and skill bodies." criteria.md:32 carries the override. This has been called a precision nit across three prior review cycles. Still low practical risk given Phase B lanes receive criteria.md directly. Still the only carry-over precision issue on SKILL.md. Fix this →

No evals for I12/I13/I14 — acknowledged follow-up against #445; I13's backtick-escape boundary and I14's Explore/Plan carve-out remain the strongest candidates when that lane lands.


Everything else holds

  • All previously fixed items confirmed closed: recheck triggers (self-maintaining source-list reference), I12 silence guard, I14 startup boundary, nested CLAUDE.local.md symmetry, version-gated claim precedence. All four commit-level fixes read correctly against the current file state.
  • I12 check body: The canonical claude doctor//doctor pair is well-anchored. The two-condition detect (explicit incompatibility on the page or failed version-matched reproduction) is correct and the silence must-not-flag is first-class.
  • I13 check body: The must-not-flag list (memory surfaces, package scopes, decorators, emails, usernames, backtick-wrapped paths) covers the documented escape cases. The citation-form contrast between the memory page and the skills page in the Source row is tight and well-sourced.
  • I14 check body: The startup-loaded set is named explicitly. Both filename forms share the lazy-loading carve-out, governed by location — the right invariant. The Explore/Plan carve-out is anchored to the documented fact that they're the only subagents that skip CLAUDE.md with no per-agent override.
  • Surface partition (criteria.md:31–32): "I6–I12 apply to all surfaces; I13 and I14 name narrower surface sets in their own rows" matches every row's Surfaces: field exactly.
  • Version bumps: 0.9.2 → 0.10.0 (plugin.json), 1.0.0 → 1.1.0 (criteria.md header) — correct MINOR bumps.
  • CHANGELOG.md 0.10.0: Added and Changed entries accurately reflect all three fix commits.
  • README.md: "eleven-check" → "fourteen-check catalog", "prompting doctrine" → "prompting and harness doctrine" — both accurate.
  • evals.json eval fix: correct plugin-system inaccuracies and harden references #2: I6–I11I6–I12 updated correctly.

Overall: One confirmed P2 (SKILL.md discoverability), one wording nit on I13 ("silently never arrives" overstates), the long-standing SKILL.md I14 surface precision nit. None block the catalog from being correct; the discoverability gap is the most actionable of the three before this merges.

@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 28s —— View job


Security Review — PR #1349

Tasks:

  • Load tools
  • Fetch PR diff
  • Review changed files for security issues
  • Post findings

Findings

No security vulnerabilities found.

This PR is an exclusively documentation and metadata change across six files. Full coverage below.

Vector Assessment
Code execution / command injection No scripts added or modified. The instruction-scan.sh reference in SKILL.md is unchanged and is a read-only script.
GitHub Actions / workflow injection No .github/ files touched.
Secrets / credentials No secret handling in any changed file.
Path traversal criteria.md contains natural-language detection patterns only — no evaluated file path operations.
Supply-chain / dependency pins No dependency files changed.
Permission widening plugin.json changes only the version field (0.9.2 → 0.10.0). No new permissions:, allowed_tools, MCP servers, hook definitions, or trust delegation.
Third-party URLs The three new source URLs (cli-reference, sub-agents, skills) are all official code.claude.com Anthropic documentation.
I12 live-fetch surface I12 is behavioral tier — detection requires a fetch + judgment at skill-execution time, not triggered by any file in this PR. The must-not-flag on documentation silence and version-gated claims further narrows the detection surface.
I13 @path detection Advisory-only text describing an inert-citation pattern to flag in other files. The detect rule is passive natural-language pattern matching; the skill is report-only with no automated remediation path. The Codex false-positive concern (whether @path silently drops content or is read on demand) is a correctness question, not a security surface.
I14 startup-surface detection Advisory-only, bounded after 494b969 to startup-loaded root-level files only. Nested CLAUDE.md / CLAUDE.local.md and path-scoped rules are explicit must-not-flag by location — conservative by design.
evals.json Updated strings are expected-output descriptions for eval assertions. The I6–I11I6–I12 change in eval #2 is a string update only — no executable content, no external fetch instructions, no schema changes.
Automated remediation blast radius The skill is report-only (SKILL.md:25–29, "There is no --fix"). All three new checks inherit that ceiling — no finding triggers an automated file change.

Nothing in the changes creates an injection surface, weakens an access boundary, introduces a supply-chain dependency, expands automated behavior, or touches any execution path. Security posture is unchanged from pre-PR state.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4f8b860641

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/claude-config/skills/audit-instructions/reference/criteria.md Outdated
Comment thread plugins/claude-config/skills/audit-instructions/reference/criteria.md Outdated
The description framed the skill purely as finding instructions the
model no longer needs, and only the description is available during
skill selection. A request about a stale harness claim, a non-loading
@path, or a redundant startup-surface read would not have selected the
fourteen-check catalog that answers it. The description now names those
three shapes and carries their trigger phrases; all six base-ref phrases
are preserved.

I13 treated the citation form itself as content loss, but an inert @path
is still a legible path in the loaded prompt: "follow @reference/rules.md"
works, and warning on it would flag a working instruction. Detection now
requires prose asserting the file has already arrived, and the
distinction is stated as a must-not-flag so a later edit does not widen
it back.
@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 3m 46s —— View job


PR Review: feat(claude-config): extend audit-instructions catalog with checks I12-I14

Tasks:

  • Load tools
  • Read changed files at HEAD (96bcb8c)
  • Verify 96bcb8c fixes (discoverability + I13 narrowing)
  • Assess two new Codex findings from aee49bfb21 review (criteria.md:176, criteria.md:238)
  • Confirm SKILL.md line 50 precision nit status
  • Post final review

Review at HEAD 96bcb8c

This is the fifth review cycle.


96bcb8c fixes — confirmed ✓

Discoverability (Codex SKILL.md:20, 22:39:18Z)SKILL.md:3
The description now includes "plus instructions that misstate Claude Code's own behavior, cite a file in a form that never loads, or re-read a surface already in context" and the Use when: list carries four new phrases: 'stale Claude Code behavior', 'outdated harness claim', 'my @path import is not loading', 'instruction re-reads CLAUDE.md'. All three I12–I14 problem shapes are now reachable via model invocation. ✓

I13 detect narrowing (Codex criteria.md:211, 22:39:18Z)criteria.md:207–219
The detect clause now requires prose that "asserts the file has already arrived" — with two concrete examples inline and the must-not-flag explicitly stating "The finding is the false assumption of automatic loading, not the citation form … When the prose does not say the content already arrived, leave it." The false-positive risk on working @path references (e.g. Follow @reference/rules.md) is eliminated by the qualifier. ✓


Two Codex P2 findings from aee49bfb21 review (22:32:34Z) — not yet addressed

These were raised by Codex on commit aee49bfb21 but were not explicitly addressed by either 4f8b860 or 96bcb8c. They're the only unresolved Codex findings in the thread.

criteria.md:176 — Reproduction preconditions beyond version criteria.md:175–177

I12's aee49bf fix correctly scoped documentation comparison "for the version the claim is about" and added a must-not-flag for version-gated claims. But the failed-reproduction arm still reads "a version-matched reproduction of the asserted behavior fails" without accounting for other stated preconditions a claim might carry (OS, account tier, feature flag, launch mode). A claim like "on macOS, the notification banner appears in the top-right corner" that fails when reproduced on Linux would satisfy the detect clause today — the reproduction matches the version but not the OS.

The severity is partially mitigated because I12 is behavioral tier (findings are proposals, not confident cuts) and the verify pass is supposed to catch false positives. But the detect clause promising "a failed reproduction" without qualifying "under matching conditions" still creates the gap Codex described.

Suggested addition to the must-not-flag list: "A reproduction that matches the version but not another stated applicability condition (OS, account tier, feature flag, launch mode) — mismatched-condition failures are inconclusive, not evidence of drift." Fix this →

criteria.md:238 — Pre-edit reads not covered by the audit carve-out criteria.md:248–253

I14's must-not-flag for agents that "open the file because inspecting it is the work" covers the audit case. It does not cover a subagent instructed to read the root CLAUDE.md before editing it — where the startup-cached version is a snapshot that could be stale if another process has since modified the file. A delegation prompt that includes "read CLAUDE.md, then apply the agreed change" would be flagged today.

Practical risk is lower than the criteria.md:176 finding — this requires a mutation workflow that touches a startup-loaded surface — and I14 is info severity. But the gap is real and the carve-out is phrased around "inspecting" not "modifying." Extending to "an agent whose instruction requires current disk state of the file (to read-before-edit)" would be a narrow, defensible addition. Fix this →


Carry-over items (no new state)

Precision nit: SKILL.md line 50SKILL.md:49–50

"Full catalog I1–I14 applies" for all non-memory surfaces. I14's own row scopes it to "agent definitions and skill bodies" — prompt-type hooks and output styles are excluded. criteria.md:32 carries "I13 and I14 name narrower surface sets in their own rows" and every Phase B lane receives criteria.md directly, so practical risk remains low. This has been called out across four prior review cycles. A one-line parenthetical would make SKILL.md independently accurate. Fix this →

No evals for I12/I13/I14 — acknowledged follow-up recorded against #445. I13's backtick-escape boundary and I14's Explore/Plan carve-out remain the strongest candidates. Not blocking.


Everything else holds

  • All previously closed items confirmed at HEAD: recheck triggers (self-maintaining source-list reference), I12 silence guard, I14 startup boundary, nested CLAUDE.local.md symmetry, version-gated claim precedence, SKILL.md discoverability, I13 detect narrowing. All read correctly at 96bcb8c.
  • I12 check body: Two-condition detect (explicit incompatibility on page OR failed version-matched reproduction), silence as a first-class must-not-flag, version-gated claims carve-out. The canonical claude doctor//doctor pair anchor is tight.
  • I13 check body: The full must-not-flag list (memory surfaces, package scopes, decorators, emails, usernames, backtick-wrapped paths) plus the new "false assumption of automatic loading" framing. No false-positive gaps visible other than the unaddressed preconditions finding above.
  • I14 check body: Startup-loaded set named explicitly. Both filename forms share the lazy-loading carve-out, governed by location. Explore/Plan carve-out is anchored to the documented fact.
  • Surface partition (criteria.md:31–32): "I6–I12 apply to all surfaces; I13 and I14 name narrower surface sets in their own rows" matches every row's Surfaces: field exactly.
  • Version bumps: 0.9.2 → 0.10.0 (plugin.json) and 1.0.0 → 1.1.0 (criteria.md header) — correct MINOR bumps.
  • CHANGELOG.md: Added and Changed entries accurately reflect all commits through 96bcb8c.
  • README.md: Both text updates accurate.
  • evals.json: I6–I11I6–I12 in eval fix: correct plugin-system inaccuracies and harden references #2 is correct; remaining evals unaffected.

Overall: Two genuinely open Codex P2 findings from the aee49bfb21 review cycle were not addressed in 96bcb8c. Both are narrow in scope — I12's preconditions gap is partially mitigated by the behavioral tier and verify pass; I14's pre-edit reads gap is low-severity (info) and limited to mutation workflows. The SKILL.md I14 surface precision nit is five cycles old. None are correctness blockers, but the criteria.md:176 preconditions finding is the most actionable before merge.

@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 26s —— View job


Security Review — PR #1349

Tasks:

  • Load tools
  • Fetch PR diff and review changed files
  • Analyze for security vulnerabilities
  • Post findings

Findings

No security vulnerabilities found.

This PR is exclusively a documentation and metadata expansion across six files. The latest commit (96bcb8c) adds SKILL.md discoverability improvements and tightens I13's detection scope — both conservative, not expansive, changes. Full coverage checklist:

Vector Assessment
Code execution / command injection No scripts added or modified. The instruction-scan.sh reference in SKILL.md is unchanged.
GitHub Actions / workflow injection No .github/ files touched.
Secrets / credentials No secret handling in any changed file.
Path traversal criteria.md contains only natural-language detection patterns — no evaluated path operations.
Supply-chain / dependency pins No dependency files changed.
Permission widening plugin.json changes only the version field (0.9.3 → 0.10.0). No new permissions:, allowed_tools, MCP servers, hook definitions, or trust delegation.
Third-party URLs The three new source URLs (code.claude.com/docs/en/cli-reference, code.claude.com/docs/en/sub-agents, code.claude.com/docs/en/skills) are all official Anthropic documentation.
@path in SKILL.md description The phrase 'my @path import is not loading' appears inside a JSON-encoded frontmatter string — literal text, not parsed as an import directive on a SKILL.md surface (which defines no import syntax). No injection surface.
I12 live-fetch surface I12 is behavioral tier — detection requires fetch + judgment at skill-execution time. No fetch is triggered by any file in this PR. The must-not-flag on silence and version-gated claims further narrows the surface.
I13 @path detection The 96bcb8c fix moves detection off the citation form and onto the false assumption of automatic loading — a more conservative scope than the prior revision. Advisory-only; the skill is report-only with no automated remediation path.
I14 startup-surface detection Advisory-only, bounded to root-level startup-loaded files. Nested CLAUDE.md/CLAUDE.local.md and path-scoped rules are explicit must-not-flag by location — conservative by design.
evals.json content Two string updates (I6-I11 → I6-I12) in expected-output assertions — no executable content, no external fetch instructions, no schema changes.
Automated remediation blast radius The skill is report-only (SKILL.md:25–29, "There is no --fix"). All three new checks inherit that ceiling — no finding triggers an automated file change.

Nothing in the changes creates an injection surface, weakens an access boundary, introduces a supply-chain dependency, expands automated behavior, or touches any execution path. Security posture is unchanged from pre-PR state.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 96bcb8c85c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/claude-config/skills/audit-instructions/reference/criteria.md Outdated
Comment thread plugins/claude-config/skills/audit-instructions/reference/criteria.md Outdated
Comment thread plugins/claude-config/skills/audit-instructions/SKILL.md Outdated
…dings

Five review findings, all cases where a row was broader than the
evidence supporting it.

I12's reproduction arm matched only the version, so replaying a claim
that is conditional on OS, a setting, an account tier, a feature flag or
launch mode could fail for the wrong reason and be read as drift. It now
requires every stated precondition to match, and treats a failure
without them as inconclusive.

I13's remediation changed the citation syntax while leaving the false
assertion in place. Neither a backticked path nor a markdown link
imports anything either, so the agent could still proceed without the
content. The remediation now rewrites the assertion into an explicit
read.

I14 hardcoded ~/.claude/CLAUDE.md, but CLAUDE_CONFIG_DIR moves the whole
config tree — the hardcoded path flags a read that is now necessary and
misses the redundant read of the configured one. It resolves the
variable, matching what Phase A already does. It also exempts any read
where the file is the operation's subject: the startup copy is a
launch-time snapshot, so cutting a pre-edit read patches stale text.

The surface partition claimed the full catalog applies on non-memory
surfaces while I13 and I14 declare narrower sets, so a lane could emit
I14 findings on hooks and output styles the row excludes. Each row's own
declaration now bounds it.

The changelog entry is restructured to describe the checks as they ship
rather than as a sequence of corrections to them.
@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude encountered an error after 2s —— View job


I'll analyze this and get back to you.

@claude

claude Bot commented Jul 25, 2026

Copy link
Copy Markdown

Claude encountered an error after 2s —— View job


I'll analyze this and get back to you.

@github-actions

Copy link
Copy Markdown

Warning

Automated review did not complete — this is an infrastructure failure, not a review verdict.

Treat any Claude comment on this PR (including a placeholder like "I'll analyze this and get back to you") as incomplete, not "no findings."

Re-running the job, or pushing a new commit, will retry the review.

@github-actions

Copy link
Copy Markdown

Warning

Automated security review did not complete — this is an infrastructure failure, not a review verdict.

Treat any Claude comment on this PR (including a placeholder like "I'll analyze this and get back to you") as incomplete, not "no findings."

Re-running the job, or pushing a new commit, will retry the review.

@kyle-sexton
kyle-sexton merged commit 7a08977 into main Jul 25, 2026
26 checks passed
@kyle-sexton
kyle-sexton deleted the feat/context-engineering-criteria-catalog branch July 25, 2026 22:52

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 90ac21f923

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

skill bodies.

- **Detect:** an instruction directing the agent to go read a surface the main conversation loads at
startup and therefore already carries — the **root** `CLAUDE.md`, the user `CLAUDE.md` at the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include the alternate project-memory path in I14

In projects that use ./.claude/CLAUDE.md instead of ./CLAUDE.md, this startup-loaded set omits the active project instructions, so a skill or agent that redundantly reads that file will escape I14. Phase A explicitly inventories both alternatives at SKILL.md:81, so include ./.claude/CLAUDE.md in this detection set as well.

Useful? React with 👍 / 👎.

Comment on lines +257 to +258
- **Must NOT flag:** an instruction to read a surface that is *not* auto-loaded — `AGENTS.md`,
contributing guides, ADRs, CI workflow files, per-ecosystem convention docs. Those are ordinary

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Resolve active imports before exempting supporting documents

When the startup CLAUDE.md imports a supporting file such as @docs/CONTRIBUTING.md, its contents are already loaded at launch—the repository's captured official guidance states this at plugins/claude-memory/skills/audit/reference/official-guidance.md:66. This unconditional exemption therefore misses exactly the redundant retrieval I14 is intended to detect; exempt these documents only when they are not reached through an active startup import, or add resolved imports to the loaded set.

Useful? React with 👍 / 👎.

kyle-sexton added a commit that referenced this pull request Jul 25, 2026
#1349 landed checks I12-I14 as claude-config 0.10.0, so this branch
becomes 0.11.0 and its changelog section sits above that one.

Both branches edited the skill description and argument hint. The merged
frontmatter keeps every trigger phrase from each: the harness-claim,
@path and startup-read vocabulary from I12-I14, and the conflict
vocabulary from this pass, plus the `conflicts` scope in the hint.

Also states that the pre-scan is a priority ordering rather than the
work list. It only reaches directives naming a tool-shaped entity, so
an ordinary pair like "Always run tests before committing" against
"Never run tests before committing" emits nothing at all. Widening the
entity pattern is not the fix -- precision is already 28% on the rows it
does emit -- so the lane reads the in-scope surfaces for pairs the scan
cannot shape-match, and a pass that reports only what the scanner
emitted has not run this check.
kyle-sexton added a commit that referenced this pull request Jul 25, 2026
…g row

Both lanes claimed I12. #1349 landed first with I12 as the stale
harness-capability claim, so this pass's conflict check was pointing at
a row that exists and means something else. It becomes I15.

The row itself was missing, which is the substance of the review
finding: conflict-criteria.md deliberately does not restate the
definition, comparison set, @path and symlink resolution, AGENTS.md
exclusion, remediation-by-scope rules or must-not-flag cases, on the
grounds that the catalog entry owns them -- and the catalog entry did
not exist. I15 now carries all of it, and conflict-criteria.md stays
what it says it is: the adjudication procedure.
kyle-sexton added a commit that referenced this pull request Jul 25, 2026
main reached claude-config 0.10.0 via #1349 (audit-instructions checks
I12-I14), so this branch's 0.11.0 is the next rung and the manifest keeps
it. Both CHANGELOG sections are kept -- 0.11.0 above main's 0.10.0 --
rather than losing a released entry to the resolution.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AdF74i18gkGtHC9fjZyDrU
kyle-sexton added a commit that referenced this pull request Aug 9, 2026
…instructions and setup (#2003)

## Summary

Discharges nine verifier-confirmed review findings against
`claude-config`'s `audit-instructions`
and `setup` skills, and bumps the plugin to `0.21.10` with a changelog
entry.

Seven of the nine are instruction-surface defects in
`audit-instructions` — a skill that audits other
people's instruction surfaces and was violating its own rules on three
of them. One is a scanner
false negative reproduced and fixed with tests. One is a prerequisite
the docs understated by two
skills.

## What changed, per finding

| # | Complaint | Discharge |
|---|---|---|
| 1 | The skill forbids hardcoding `~/.claude`, then hardcodes it |
`SKILL.md:45`, `:75`, `:197`, `:219-222` now resolve against the user
root Phase A establishes |
| 2 | I3 rejects `@path` imports as non-deferring, then names a
`skills:` preload as a valid destination | `criteria.md:196-201` strikes
the preload; only conditional runtime invocation qualifies |
| 3 | A subagent's own `memory` is graded real but never inventoried |
Inventory bullet at `SKILL.md:216-229`; co-residency row at
`conflict-criteria.md:98` |
| 4 | The liveness gate resolved a closed five-input list with no hook
enablement | `SKILL.md:175-185` and `conflict-criteria.md:270-292`
resolve `disableAllHooks` per scope plus `allowManagedHooksOnly` |
| 5 | A nested project memory pair routed to a check that cannot
discover the file | `SKILL.md:413-417` narrows the boundary to
**root-level** project; `.claude/rules/**` deliberately unchanged |
| 6 | `BOUNDARY_ERE` omits `and`, dropping a real conflict | `COORD_ERE`
/ `COORD_HEAD_ERE` at `conflict-scan.sh:131-143`, three new test cases |
| 7 | The requirements list names only `jq`/`curl` | `README.md:163-177`
and `setup/SKILL.md:28-33,46-52,106-108` name `awk`/`sort` across all
three skills that use them |
| 8 | I14's startup set omits `./.claude/CLAUDE.md` |
`criteria.md:612-621` covers both supported root locations |
| 9 | I14's supporting-document exemption ignores startup `@path`
imports | `criteria.md:636-642` resolves imports first, to four hops |

### Finding 6, reproduced

`conflict-scan.sh --count` on the finding's exact strings:

| Case | `origin/main` | this branch |
|---|---|---|
| "Always use `Read` and never use `Bash`" vs "Never use `Read`" | 0 |
**1** |
| "Always use `Read` but never use `Bash`" vs "Never use `Read`" | 1 | 1
|

A **bare** `and` boundary would also return 1 for the first row while
creating a new false negative:
"Never use `Bash` and `Grep`" is one directive over two objects, and
cutting at the coordinator strips
the `never` governing `Grep`. The boundary therefore requires a polarity
token after `and`, and is
consumed asymmetrically — a leading window resumes after the coordinator
alone so that token still
classifies its entity. Case 35 is the must-not-flag test and fails under
a bare-`and` boundary.

### Citations re-verified against the live docs

Every citation this batch introduced was diffed character-for-character
against the raw markdown of
`hooks`, `memory`, and `sub-agents` (fetched 2026-08-08). Three did not
survive and were corrected:

- **`disableAllHooks` has no documented "own settings level and below"
cascade.** The docs say
"Disable all hooks", with exactly one carve-out: set in user, project,
or local settings it cannot
reach managed hooks. The earlier wording invented a directional cascade.
Rewritten.
- **The `skills:` citation was a splice** of the frontmatter-table
sentence and the body sentence,
quotable as neither. Replaced with the real one: "The full content of
each listed skill is injected
  into the subagent's context at startup."
- **Imports recurse "with a maximum depth of four hops"** — the text
pointed at a "hop limit" the page
  never names. Now states the figure and quotes the wording.

Also corrected: the `AGENTS.md` import is *recommended* (a symlink is a
co-equal alternative, and the
import is mandated only on Windows), not prescribed; subagent memory
paths take upstream's
`<name-of-agent>` placeholder; and the auto-memory gate now names what
the subagent actually loses —
"the memory instructions or the memory tool access".

### Also fixed in passing

`conflict-scan.test.sh:378` had unescaped backticks inside a
double-quoted assertion message, so the
suite ran `and` as a command (`and: command not found` on stderr) and
printed the label with a hole in
it. Exit status stayed 0, so no gate saw it. Now single-quoted, matching
the file's existing idiom.

### Deliberately not changed

- `${CLAUDE_CONFIG_DIR:-~/.claude}` — that form *is* the correct
resolver, not a hardcode.
- `~/.claude` inside quoted upstream text (`criteria.md:651`,
`conflict-criteria.md:111`) — altering a
  quotation is a worse defect than the one it would resolve.
- `.claude/rules/**` routing to `claude-memory`'s C6 — C6's rules
discovery is recursive, so nested
  rules do not leak. Finding 5 is narrower than filed.

## Testing

- `conflict-scan.test.sh` — 46/46, no stderr noise
- All 7 `claude-config` plugin tests — PASS
- `check-changelog-parity.sh` `--check` / `--check-bump origin/main` /
`--check-order` — PASS
- `check-changed-skills.sh origin/main` — 4 skills, 0 failed
(`audit-instructions` 281/500 lines)
- `check-shell-portability.sh` / `check-skill-portability.sh` — PASS
- `shellcheck`, `markdownlint-cli2`, `typos` — clean
- `validate-plugins.sh`, `check-contract-slice-prune.sh`,
`check-contract-clause-coverage.py`,
`check-silent-skips.sh`, `check-cross-plugin-source-drift.sh`,
`check-skill-leaf-names.sh`,
  `check-orphaned-fixtures.sh` — PASS

## Related

Discharges review findings filed on #1316, #1343, and #1349.

No linked issue

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant