Skip to content

fix(verification,testing): stop routing live-app verification through the user-invoked-only /verify - #1878

Merged
kyle-sexton merged 4 commits into
mainfrom
fix/corpus-ra10-verification-version-note
Aug 2, 2026
Merged

fix(verification,testing): stop routing live-app verification through the user-invoked-only /verify#1878
kyle-sexton merged 4 commits into
mainfrom
fix/corpus-ra10-verification-version-note

Conversation

@kyle-sexton

Copy link
Copy Markdown
Contributor

Repo-alignment finding RA-10 from the doc-corpus campaign's Phase 3a. The finding as filed was refused; a different, real defect was found underneath it and fixed.

The refusal

RA-10 claimed the >=2.1.145 gate on the bundled /verify note was stale. It is not. Live curl of the raw skills reference (HTTP 200, 73,907 bytes, 933 lines — non-truncated, and the length is recorded because a rendered fetch of a long page can only manufacture a false absence) still states: "All three skills require Claude Code v2.1.145 or later" for /run, /verify, /run-skill-generator.

The triage conflated two orthogonal axes. 2.1.145 is the availability floor. 2.1.215 changed the invoker: "others, including /verify and /code-review, run only when you invoke them... Before v2.1.215, Claude could also run /verify and /code-review on its own." Nothing about an invocation restriction makes a version floor stale.

The real defect, in the opposite direction

The triage asked whether the repo claims auto-invocation, and correctly found it does not. But the repo did something stronger: it instructed the agent to invoke /verify — a surface the model may no longer invoke.

Four live assertion sites now state the opposite: /verify is surfaced as a suggestion for the user to run, never delegated to.

/run is deliberately untouched — upstream restricts only /verify and /code-review, and /run survives as supplementary and agent-invocable at all nine live sites.

The eval gap

The first pass left confirm/evals/evals.json encoding the removed delegation, so the skill would have shipped failing its own graded rubric. Caught by an independent verifier on population completeness — the campaign's most repeated error. Eval 9 is now entailed clause-by-clause by the shipped skill text instead of contradicting it, and its id/name/prompt are unchanged because /run delegation genuinely remains.

Changelog entries for already-shipped versions are deliberately left as written: a released entry is an immutable record of what that version did.

Verification

Two independent verification rounds, rationale withheld. The second re-enumerated from scratch rather than inheriting the first's list: 50 /verify lines repo-wide, 34 unrelated, 12 changelog text, 4 live assertion sites — all four corrected, zero remaining. The agent-invocable wording now shipping inside a graded rubric was grounded empirically (this session's skills listing exposes run while bundled verify and code-review are absent) rather than on upstream's non-exhaustive "including".

CI gates were run directly rather than trusted from the commit message, including the markdownlint pass that the skill-check harness had skipped.

No linked issue

Related

kyle-sexton and others added 2 commits August 2, 2026 07:21
…r-invoked-only /verify

RA-10 alleged that the `≥2.1.145` gate on the bundled `/verify` + `/run` note in
`plugins/verification/skills/confirm/SKILL.md` was stale as an availability
statement, because corpus entry verifblog OQ-01-1 records that Claude Code
v2.1.215 made bundled `/verify` and `/code-review` user-invoked only.

That allegation is refused. A live fetch of
https://code.claude.com/docs/en/skills.md on 2026-08-02 shows the two facts sit
on orthogonal axes: line 45 still states "All three skills require Claude Code
v2.1.145 or later", so the floor is current, while line 25 restricts who may
invoke — "others, including `/verify` and `/code-review`, run only when you
invoke them". v2.1.215 changed the invoker, not the version floor.

The `2.1.145` floor value is therefore unchanged wherever it appears — no site's
version was altered. Two of the four pre-existing mentions outside `.work/` sit
on lines this commit rewrote for the invocation fix, and the two new changelog
entries restate the floor, so the branch carries six mentions against
origin/main's four.

The investigation did surface a real, adjacent defect on the same lines: these
skills instruct the *agent* to invoke `/verify`, a surface the model may no
longer invoke. Six delegation sites named it across the two plugins; the four
that told the skill to reach for it are corrected, and the two that remain now
frame `/verify` as a suggestion for the user to run. `/run` is untouched — the
v2.1.215 sentence names neither it nor `run-skill-generator`, so it stays the
agent-invocable supplementary path.

The `/code-review` half of the same upstream change is deliberately out of
scope: its 23 references live in the `review` plugin and are predominantly
descriptive boundary prose or already user-gated, a distinct population needing
its own assessment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uHDhn2Jfd1N6M1S3AsMSf
…d-only /verify

The preceding commit stopped `/verification:confirm` from delegating live-app
verification to the bundled `/verify`, because Claude Code v2.1.215 made
`/verify` and `/code-review` run only when the user invokes them. Four prose
sites moved; the skill's graded rubric did not.

`plugins/verification/skills/confirm/evals/evals.json` eval 9
(`live-app-delegates-to-bundled-with-fallback`) still encoded the removed
delegation as its pass condition: `expected_output` required "delegates to
/testing:run-e2e (primary for orchestrated apps) or bundled /verify + bundled
/run", and its second expectation required "bundled /verify + /run as
supplementary". Against SKILL.md:112 — "so suggest the user run it rather than
delegating to it — this skill cannot invoke it" — a grader reading both marks
the corrected skill FAIL. The branch carried a skill contradicting its own
rubric; no released version did, since both halves land in unreleased 0.3.6.

Eval 9's `expected_output` and expectations now state what the skill does:
`/testing:run-e2e` primary for orchestrated apps, the bundled `/run`
supplementary and agent-invocable, the bundled `/verify` surfaced as a
suggestion for the user rather than invoked, and the graceful fallback keyed to
`/run` alone rather than the ambiguous "bundled-tool". The case `id`, `name`,
and `prompt` are unchanged — `/run` delegation genuinely remains, so the name
still describes the case and nothing else in the repo cites it.

Population re-enumerated rather than inherited. `git grep -n -I -E
'/verify([^a-zA-Z0-9_:-]|$)' -- .` returns 49 lines: 34 outside the
`verification` and `testing` plugins are unrelated (docs-hygiene's `/verify`
rename examples, `actions/verify.md` path refs, `fix/verify/report` prose), and
of the 15 in-scope lines every one but these two was already corrected or is an
immutable shipped-version CHANGELOG record (testing 0.2.x line 58, verification
lines 68/137/170). Two token-free sweeps closed the residual gap: `git grep -n
-I 'bundled-skills' -- .` returns 3 lines, all already-corrected sites, and `git
grep -n -I 'run-e2e' -- . ':!plugins/testing' ':!plugins/verification'` returns
5 lines, none naming the bundled `/verify`.

No model-graded eval runner exists in this repo, so the regression was proven by
entailment against the shipped text rather than a run. The executable gates CI
runs were executed: `check-jsonschema --schemafile
plugins/skill-quality/reference/evals.schema.json` exits 0 on the edited file,
`scripts/check-changed-skills.sh origin/main` reports both touched skills PASS
with 0 errors, and `scripts/check-cross-plugin-source-drift.sh` exits 0 without
implicating either changed file. The orphaned-fixtures gate is structurally
unaffected — the plugin has no `evals/fixtures/` directory and eval 9's `files`
array is empty and untouched.

The 0.3.6 CHANGELOG entry gains one clause noting the rubric moved with the
behavior; the version bump already on this branch is not re-bumped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uHDhn2Jfd1N6M1S3AsMSf
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 3m 1s —— View job


Review: fix(verification,testing) — stop routing live-app verification through the user-invoked-only /verify

No blocking findings. This is a well-scoped, well-verified fix. Summary of what I checked:

Fresh-docs mandate compliance. This PR touches skill-body instructions that tell the agent whether it may invoke /verify — a contract-surface change under docs/OFFICIAL-DOCS.md's scope. I independently re-fetched https://code.claude.com/docs/en/skills and confirmed both claims verbatim:

  • {/* min-version: 2.1.145 */}All three skills require Claude Code v2.1.145 or later. — the availability floor is genuinely unchanged, so the RA-10 finding's premise (that it was stale) was correctly refused.
  • {/* min-version: 2.1.215 */}...including \/verify` and `/code-review`, run only when you invoke them... Before v2.1.215, Claude could also run `/verify` and `/code-review` on its own.— confirms the real defect this PR fixes: the skill bodies previously instructed the agent to delegate to/verify`, a surface it can no longer invoke.

Internal consistency across the 4 corrected sites (confirm/SKILL.md, verification/README.md, confirm/SKILL.md skill-chaining table, run-e2e/SKILL.md) — all four now consistently frame /verify as a user-run suggestion rather than something the skill delegates to, and /run is correctly left untouched as the agent-invocable supplementary path. A repo-wide grep for remaining delegate...\/verify phrasing turned up nothing else live; the one historical hit (plugins/testing/CHANGELOG.md:58) is a released changelog entry correctly left as written per this repo's established never-rewrite-history precedent.

Eval/rubric alignment. confirm/evals/evals.json eval 9 (live-app-delegates-to-bundled-with-fallback) now matches the corrected SKILL.md clause-by-clause — this closes the eval gap the PR description calls out (the rubric would otherwise have graded the fixed skill as failing).

Versioning. Both plugin.json bumps are correct patch increments (verification 0.3.5→0.3.6, testing 0.3.2→0.3.3) with matching CHANGELOG.md entries in the established Keep a Changelog / prose style used by prior entries in these files.

CI. At review time: pr-title, pr-issue-linkage, claude-security-review, and do-not-merge all passed; ci was still in progress with no failures so far.

Nothing to request changes on. This is exactly the kind of narrow, root-cause fix CLAUDE.md's fresh-docs mandate is meant to produce — a plausible-looking finding was correctly rejected once checked against the live page, and the fix stayed scoped to the actual defect underneath it.

@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 56s —— View job


Security review complete

  • Gather context (PR diff, files changed)
  • Review each changed file for security issues
  • Post findings

No security issues found.

This PR touches only prose/documentation content and version metadata across 8 files:

  • plugins/testing/.claude-plugin/plugin.json, plugins/verification/.claude-plugin/plugin.json — version bumps only (no commands, hooks, or mcpServers entries; no permission or trust-boundary changes).
  • plugins/testing/CHANGELOG.md, plugins/verification/CHANGELOG.md — changelog prose additions.
  • plugins/testing/skills/run-e2e/SKILL.md, plugins/verification/skills/confirm/SKILL.md, plugins/verification/README.md — markdown instruction text rewording which bundled skill (/verify vs /run) the agent may delegate to; no scripts, shell commands, URLs to fetch-and-execute, or credential handling introduced.
  • plugins/verification/skills/confirm/evals/evals.json — updates expected_output/expectations strings for eval case 9 to match the corrected skill text; no executable eval runner logic here, purely graded-rubric prose.

No GitHub Actions workflows, hook scripts, .mcp.json entries, or executable code are part of this diff, so there's no injection surface (command/SQL/path/template), no secret/token handling, and no permission-widening to review. The change is a pure behavioral-instruction correction (stop telling the agent to invoke /verify, which upstream restricted to user-only invocation as of Claude Code v2.1.215) plus a matching eval-rubric fix.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2457dee4d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/verification/skills/confirm/SKILL.md Outdated
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

Claude has reviewed this PR 3 times. The lane skips further automatic reviews after 5; deleting this comment resets the count.

…to v2.1.215

The branch's correction overshot. `plugins/verification/skills/confirm/SKILL.md`
said `/verify` "is user-invoked only ... this skill cannot invoke it" with no
version attached, and `plugins/testing/skills/run-e2e/SKILL.md` said
"[user-invoked only]". Both statements are true for current clients and false
for 2.1.145-2.1.214, where `/verify` exists and Claude may still run it. The
originating finding conflated availability with invocability; an unscoped
restriction re-introduces the same conflation from the opposite side.

Upstream is explicit and version-scoped. A raw `curl` of
https://code.claude.com/docs/en/skills.md on 2026-08-02 returned 73907 bytes /
933 lines; line 25 reads "others, including `/verify` and `/code-review`, run
only when you invoke them" followed by "Before v2.1.215, Claude could also run
`/verify` and `/code-review` on its own", and line 45 reads "All three skills
require Claude Code v2.1.145 or later". Two axes, two versions.

An unscoped statement would be correct only behind a declared support floor, and
this repository declares none. `grep -rniE 'minimum (supported )?(claude code
)?version|min(imum)? claude code|requires claude code|support (floor|window)|
supported versions|oldest supported'` over tracked `.md`/`.json`/`.yml` returns
7 lines, none of which is a repo-wide Claude Code floor: two are the per-claim
`v2.1.206` gate on `/doctor`'s trim, one is this very line 112, one is lint
criteria prose, one is Kindle tooling, one is a decision-matrix bullet, and
`SECURITY.md` "Supported Versions" governs plugin versions, not client versions.
No `plugin.json` or `marketplace.json` carries an `engines` /
`minClaudeCodeVersion` / `requiredVersion` field (`grep -rlE` returns zero
files), and `docs/PLUGIN-PHILOSOPHY.md` / `docs/MIGRATION-PLAYBOOK.md` state
none. An undeclared floor is not a floor, so the claim gets the version.

The repo's own convention agrees: `plugins/claude-config/skills/
audit-instructions/reference/criteria.md:328` — "Where the behavior is
version-gated, carry the minimum version with the claim" — and the neighbouring
`>=2.1.145` availability note on the same lines is exactly that shape. That note
is untouched and still its own axis; the two are now named as separate axes so
the correction cannot be misread as re-merging them.

The *instruction* stays uniform on purpose and is not branched on the client's
version: suggesting the user run `/verify` is correct on every version it exists
on, so no runtime version probe is warranted for a closed window. The behavior
therefore does not regress on pre-2.1.215 clients - it simply does not
autonomously delegate there either, which the changed prose now states rather
than justifying with a false absolute.

Population enumerated, not assumed. `grep -rniE 'user-invoked|user invoked|
agent-invocable|agent invocable|model-invocable|cannot invoke|cannot reach|only
when you invoke'` over tracked `.md`/`.json` returns 113 lines repo-wide,
dominated by unrelated `cannot reach` changelog boilerplate; discriminating on
the bundled `/verify` leaves 6 sites across the two plugins. Three stated the
restriction unscoped and are corrected here (`confirm/SKILL.md:112`,
`run-e2e/SKILL.md:61`, and eval 9's `expected_output` plus its second
expectation in `confirm/evals/evals.json`). The two unreleased CHANGELOG entries
(verification 0.3.6, testing 0.3.3) already named v2.1.215 in-sentence and gain
the scope explicitly; no shipped-version entry was touched. `README.md` made no
invocability claim after the earlier commit removed `/verify` from the fallback
sentence, so it needed nothing.

The `/code-review` half of the same upstream sentence was swept rather than
inherited: `grep -rn '/code-review'` returns 40 lines, all in the `review`
plugin, `playbooks` vendor text, and design docs. None asserts an invocability
restriction in either direction - the `review` surfaces already gate `--fix` /
`--comment` on explicit user opt-in - so there is no parallel overreach to
scope, and the second commit's deliberate scope-out stands.

Gates run on the edited files: `markdownlint-cli2` reports 0 errors across all
four markdown files, `check-jsonschema --schemafile
plugins/skill-quality/reference/evals.schema.json` exits 0 on the eval file,
`scripts/check-changed-skills.sh origin/main` reports both touched skills PASS
with 0 errors (the one warning each is pre-existing description phrasing), and
`scripts/check-cross-plugin-source-drift.sh` exits 0 without implicating any
changed file. No version is re-bumped; 0.3.6 and 0.3.3 remain unreleased.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uHDhn2Jfd1N6M1S3AsMSf
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 42s —— View job


Review: fix(verification,testing) — stop routing live-app verification through the user-invoked-only /verify

  • Diff the new commit (5b74629) against the previously-reviewed state
  • Check whether the new commit addresses the Codex P2 comment (pre-2.1.215 delegation)
  • Re-verify internal consistency and CLAUDE.md fresh-docs compliance
  • Post final review feedback

No blocking findings on the new commit. 5b74629 is a narrow, well-justified refinement on top of the state I reviewed in my prior pass (2457dee).

What changed. The prior commits stated the "/verify is user-invoked only" restriction with no version attached. 5b74629 scopes that claim to v2.1.215+ in the four sites it touches — confirm/SKILL.md#L112, run-e2e/SKILL.md#L61, both CHANGELOG.md entries, and eval 9's expected_output/expectations in confirm/evals/evals.json#L80-L84 — keeping the two version axes (≥2.1.145 availability, ≥2.1.215 invocability) explicitly separate rather than letting the second correction blur back into the first, which was the exact failure mode RA-10 itself was refused for.

On the Codex P2 comment (preserve delegation on pre-2.1.215 clients): the commit deliberately does not implement that suggestion, and explains why in both the commit message and the CHANGELOG.md prose — the instruction stays uniform (suggest, never delegate) across the whole 2.1.145–2.1.214 window on purpose, because nothing in a skill's markdown body can probe the connected client's exact patch version at runtime to branch on it. That's consistent with this repo's existing convention: audit-pass/reference/doctor-handoff.md documents a /doctor version-gated behavior cutover (v2.1.205) the same way — stating the floor and the cutover in prose, not branching agent behavior on a detected version. I'd call this a reasoned rejection of the Codex suggestion rather than a gap: the alternative (conditional delegation) isn't implementable without a version-probe mechanism this repo doesn't have, and the changelog is explicit that this is a deliberate tradeoff, not an oversight.

Consistency re-check. Read all four touched sites plus both plugin.jsons post-commit:

  • confirm/SKILL.md:112 and run-e2e/SKILL.md:61 now state the same two-axis fact (availability floor unchanged at ≥2.1.145; invocability restricted from v2.1.215) without contradiction between the two files.
  • Eval 9's expected_output and second expectations entry were updated to require the scoped claim ("scoping the user-invoked-only restriction to v2.1.215 and later rather than stating it unscoped") — this keeps the rubric aligned with the shipped skill text, closing the same eval-gap risk called out for the prior commit.
  • plugins/verification/.claude-plugin/plugin.json (0.3.6) and plugins/testing/.claude-plugin/plugin.json (0.3.3) versions are unchanged from the prior commit and still match their respective CHANGELOG.md ## [x.y.z] headers — no re-bump needed since both versions remain unreleased until this PR merges.
  • README.md is untouched by this commit, consistent with the commit message's claim that it made no invocability assertion after the earlier commit removed /verify from its fallback sentence (verified in my prior review pass).

Nothing here changes my earlier assessment: this remains a well-scoped, well-verified fix, and the new commit tightens precision without introducing new inconsistencies.

@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 51s —— View job


Security review complete

  • Gather context (git diff against the PR's merge base, changed files)
  • Review each changed file for security issues (injection, auth/access-control, secrets, dangerous GitHub Actions patterns)
  • Post findings

No security issues found.

All 8 changed files in this PR are non-executable content — no code paths, no workflow files, no scripts:

  • plugins/testing/.claude-plugin/plugin.json, plugins/verification/.claude-plugin/plugin.json — version-number bumps only (0.3.20.3.3, 0.3.50.3.6). No commands, hooks, mcpServers, or permission entries touched.
  • plugins/testing/CHANGELOG.md, plugins/verification/CHANGELOG.md — prose changelog additions describing the fix; no executable content.
  • plugins/testing/skills/run-e2e/SKILL.md, plugins/verification/skills/confirm/SKILL.md, plugins/verification/README.md — markdown instruction text. The substantive change re-scopes which bundled Claude Code skill (/verify vs /run) the skill instructs the agent to invoke, per the v2.1.215 user-invoked-only restriction on /verify. This narrows agent-invocable surface (removing an instruction to delegate to /verify) rather than widening it — the opposite direction of a privilege-escalation concern. No shell commands, URLs fetched-and-executed, credential handling, or path construction from untrusted input is introduced.
  • plugins/verification/skills/confirm/evals/evals.json — updates expected_output/expectations strings for eval case 9 to match the corrected skill text; these are graded-rubric prose strings consumed by the eval harness, not executable logic, and contain no injectable content.

No GitHub Actions workflow files, hook scripts, .mcp.json entries, or other executable surfaces are part of this diff. There is no injection surface (command/SQL/path/template), no secret or token handling, and no permission-widening to review. This confirms the earlier automated security-review pass on this PR (commit 2457dee) — the subsequent commit (5b74629, version-scoping the /verify claim to 2.1.215) is likewise prose-only and introduces nothing security-relevant.

Two defects in the preceding commit, both caught by independent review.

`plugins/testing/skills/run-e2e/SKILL.md:61` read "user-invoked only from
v2.1.215 - earlier versions in that window let Claude run it too". No window had
been named: the only prior version on the line is the open-ended `>=2.1.145`
floor. The phrase resolved to nothing, in the exact file the review thread names
as a suspected sibling site. It now matches the wording its `confirm/SKILL.md:112`
sibling already used - "before v2.1.215 Claude could also run it on its own" -
and the trailing clause names the window it means (`>=2.1.145`) instead of
gesturing at an undefined one. No fact changed; the sentence now states the one
it always intended.

The preceding commit message also cited "40 lines" for the `/code-review` sweep.
That number was never produced by a command. Piped through `wc -l`, `grep -rn
'/code-review' --include="*.md" --include="*.json" . | grep -v '^./.git/'`
returns 48, distributed `plugins/review` 31, `plugins/playbooks` 9, `docs` 5,
and one line each in `plugins/verification`, `plugins/source-control`, and
`plugins/docs-hygiene`. The qualitative finding that number supported is
unchanged and still holds: none of the 48 asserts an invocability restriction in
either direction - the `review` surfaces gate `--fix` / `--comment` on explicit
user opt-in, the `playbooks` lines are vendored upstream tip text, and the rest
are boundary or naming prose - so `/code-review` carries no parallel overreach to
version-scope. The earlier commit's separate "23 references" figure counted the
`review` plugin alone at an earlier tree state and is not comparable; 48 is the
repo-wide count at this commit. Recorded here rather than by amending, because
the branch has an open PR and its review record must stay intact.

`markdownlint-cli2` reports 0 errors on the edited file and
`scripts/check-changed-skills.sh origin/main` reports all 5 touched skills PASS
with 0 errors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uHDhn2Jfd1N6M1S3AsMSf
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 17s —— View job


Review: fix(verification,testing) — stop routing live-app verification through the user-invoked-only /verify

  • Diff the new commit (246e228) against the previously-reviewed state (5b74629)
  • Review new changes for correctness and CLAUDE.md alignment
  • Post final review feedback

No blocking findings. 246e228 is a small, self-correcting follow-up — a single line changed in plugins/testing/skills/run-e2e/SKILL.md#L61 — that fixes two defects the commit message itself identifies in the prior commit, both caught by independent review rather than by me.

Defect 1 (dangling antecedent) — verified fixed. The prior wording was "user-invoked only from v2.1.215 — earlier versions in that window let Claude run it too," which referred to an undefined "that window." The new text names the window explicitly and now matches its sibling line verbatim in structure:

  • run-e2e/SKILL.md:61: "...before v2.1.215 Claude could also run it on its own. Suggest the user run it and consume its findings rather than delegating to it, on every version: the suggestion holds across the whole ≥2.1.145 availability window, delegation does not."
  • confirm/SKILL.md:112 (unchanged sibling): "...before v2.1.215 Claude could also run it on its own. Suggest the user run it on every version rather than probing the client's: the suggestion holds across the whole availability window, delegation does not."

Both files now state the same two-axis fact without an unresolved reference. No behavioral claim changed — only the antecedent was restored.

Defect 2 (miscounted /code-review sweep) — independently reproduced, matches exactly. I re-ran the grep the commit message describes (/code-review across *.md and *.json, repo-wide) rather than trusting the commit's arithmetic:

  • Total: 48 occurrences — matches.
  • Distribution: plugins/review 31 (29 .md + 2 .json), plugins/playbooks 9, docs 5, one each in plugins/verification, plugins/source-control, plugins/docs-hygiene — matches exactly.

I also spot-checked plugins/review/skills/quality-gate/context/pr.md:26-34, the site most likely to carry a competing invocability claim: it gates --fix/--comment on explicit opt-in (a mutation gate) and explicitly carves out bare /code-review <target> as ungated read-only. That's a different axis from the v2.1.215 invocation restriction this PR addresses, so the commit's conclusion — no parallel version-scoping needed for /code-review — holds.

Process note, not a finding. The commit correctly avoids amending 2457dee/5b74629 and instead records both corrections in a new commit, consistent with this PR's own stated policy ("a released entry is an immutable record") extended sensibly to an open PR's review record.

CI at review time: pr-title, pr-issue-linkage, do-not-merge passed; ci, claude-security-review, and this claude-review run were still in progress with no failures so far.

Nothing to request changes on — this commit does what it says: two narrow, independently-verified corrections to the immediately preceding commit, no new scope introduced.

@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 53s —— View job


Security review complete

  • Gather context (git diff against the PR's merge base 4bebb89, matches the 8 changed files listed on the PR)
  • Review each changed file for security issues (injection, auth/access-control, secrets, dangerous GitHub Actions patterns)
  • Post findings

No security issues found.

All 8 changed files are non-executable content — no workflow files, scripts, or code paths are part of this diff:

  • plugins/testing/.claude-plugin/plugin.json, plugins/verification/.claude-plugin/plugin.json — version-number bumps only (0.3.20.3.3, 0.3.50.3.6). No commands, hooks, mcpServers, or permission entries touched.
  • plugins/testing/CHANGELOG.md, plugins/verification/CHANGELOG.md — prose changelog additions; no executable content.
  • plugins/testing/skills/run-e2e/SKILL.md#L61, plugins/verification/skills/confirm/SKILL.md, plugins/verification/README.md — markdown instruction text re-scoping which bundled Claude Code skill (/verify vs /run) the agent may delegate to. The substantive change narrows agent-invocable surface (removing an instruction to delegate to /verify, which upstream restricted to user-only invocation as of v2.1.215) rather than widening it — the opposite direction of a privilege-escalation concern. No shell commands, fetched-and-executed URLs, credential handling, or untrusted-input path construction introduced. The one embedded URL (https://code.claude.com/docs/en/skills#bundled-skills) is a plain markdown reference link, not fetched or executed by any script.
  • plugins/verification/skills/confirm/evals/evals.json#L80-L84 — updates expected_output/expectations strings for eval case 9 to match the corrected skill text; these are graded-rubric prose strings consumed by the eval harness, not executable logic, and contain no injectable content.

No GitHub Actions workflow files, hook scripts, .mcp.json entries, or other executable surfaces are part of this diff. There is no injection surface (command/SQL/path/template), no secret or token handling, and no permission-widening to review. This diff is identical in scope to what the two prior automated security-review passes on this PR (commits 2457dee, 5b74629) already covered — the final commit (246e228) only corrects prose wording and a miscount in the description, with no new security-relevant surface.

@kyle-sexton

Copy link
Copy Markdown
Contributor Author

Accepted on the facts, with one deliberate partial. You are right that the shipped statement overreached; I took option 1 — version-scope it — because the repo has no support floor that would license the unscoped form.

Upstream, fetched raw. curl -sSL https://code.claude.com/docs/en/skills.md, HTTP 200, 73,907 bytes, 933 lines. Line 25: "others, including /verify and /code-review, run only when you invoke them", followed by "Before v2.1.215, Claude could also run /verify and /code-review on its own." Line 45: "All three skills require Claude Code v2.1.145 or later." Two axes, two versions — exactly as you read it.

The floor search was a null, so option 2 was not available. A pattern sweep for a repo-wide client floor returns 7 lines, none of which is one: two are the per-claim v2.1.206 gate on /doctor's trim, one is the line under review, and SECURITY.md's "Supported Versions" governs plugin versions, not client versions. No plugin.json or marketplace.json carries an engines / minClaudeCodeVersion / requiredVersion field — zero files. PLUGIN-PHILOSOPHY.md, MIGRATION-PLAYBOOK.md, README.md and CLAUDE.md state none. An undeclared floor is not a floor.

The repo's own convention says the same thing directly — plugins/claude-config/skills/audit-instructions/reference/criteria.md:328: "Where the behavior is version-gated, carry the minimum version with the claim."

What changed. confirm/SKILL.md:112 and run-e2e/SKILL.md:61 now read that /verify shares the >=2.1.145 availability floor "but from v2.1.215 it is user-invoked only — before v2.1.215 Claude could also run it on its own". Eval 9's rubric scopes the restriction rather than shipping "user-invoked only" bare inside a graded expectation. Both unreleased CHANGELOG entries now state the scope explicitly; shipped-version entries are untouched.

The >=2.1.145 note is intact and now explicitly a separate axis. No version value moved. The lines now say in so many words that availability floor and invocability are different axes — conflating them was the original finding's error, and this branch exists to fix it, so the correction had to not re-introduce it.

Where I declined, and why. Your title asks to preserve delegation on pre-2.1.215 clients. I preserved the statement's accuracy but left the instruction uniform rather than branching on client version. That is a decision, not an omission: suggesting the user run /verify is correct on every version /verify exists on, whereas delegation is correct on only part of that window. Branching would require probing claude --version at runtime to buy autonomous delegation across a closed, superseded range — cost with no accuracy gain, and a permanent conditional in a skill body. If you would rather have the version branch, say so and I will add it, but I would want the runtime probe justified on its own merits.

Sibling swept. /code-review sits under the same upstream sentence, so I checked it: 48 references, none asserting an invocability restriction in either direction. No parallel overreach to scope.

@kyle-sexton
kyle-sexton merged commit 42d805e into main Aug 2, 2026
32 checks passed
@kyle-sexton
kyle-sexton deleted the fix/corpus-ra10-verification-version-note branch August 2, 2026 19:33
kyle-sexton added a commit that referenced this pull request Aug 2, 2026
…nnel extraction artifacts (#1879)

Second batch of evidence-forced docpage-digest amendments. `knowledge`
0.10.15 -> 0.10.16. Two rules ship; six items are deliberately held,
each with its reason stated.

## What ships

- **PA-Y — the near-miss disclosure rule, written into the profile.** It
previously lived only inside Ruling 1 Decision A and was re-derived per
unit, which is why undisclosed near-misses are the largest MINOR class
in the slice that measured it. A harness page covering the same subject
without stating the claim's specific rule is a near-miss: the tag
survives, and the row must disclose it by name and line.
- **PA-Z — blog-channel extraction artifacts** (collapsed H1, split
reading-time) with reconstruction guidance. Its deferral condition was
met: a second `claude.com/blog` run reproduced both exactly, re-verified
byte-exact in both slices. The guidance leads with the canonical URL
slug and states what a slug **loses** — word boundaries recover,
punctuation and casing do not — so a slug-recovered title is labelled
reconstructed.
- **PA-AG — verified and closed, no edit.** The `cc-applicable`/`mixed`
boundary rule already in the profile matches what its source asked for.

## What is held, and why — two different reasons

**Five held because the campaign's judgment file holds them.** PA-I
(J-6), PA-Q (J-7), PA-N (J-8), PA-M (J-12) and PA-AD (J-14) were shipped
in the first draft of this branch and then withdrawn. The triage marks
them `evidence-forced`; `PHASE2-JUDGMENT-AMENDMENTS` writes them up as
judgment calls under a header stating "None applied." Two records
disagreed, and the judgment file governs on its own reasoning:

- J-14 says outright it is "held here only because no defect was
demonstrated, so the bar for the applied subset is not met."
- J-7 says it "should be decided **after** Decision-A ordering" — an
escalated, owner-reserved question. Shipping it would have pre-empted a
decision the owner kept for himself.

They are now full rows in the owner-decision artifact, each restating
the triage-vs-judgment conflict rather than resolving it silently.
Between the two readings they would otherwise have landed in neither
place.

**Three held because they change untracked instruments.** PA-B's second
half, PA-P and PA-AK all modify scripts under the memory-tier `.work/`
root, which is never committed and is not part of the shipped plugin.

## Verification

Independently verified twice. The first round FAILED on the five
judgment-held rules. The second confirmed the split is clean: `SKILL.md`
is byte-identical to its 0.10.15 state, grep finds no residue of the
withdrawn rules, and PA-Y reads as a whole rule rather than half of the
graded pair it was originally merged into.

The implementer also ran a second pass over the whole *file* rather than
the diff, and caught three defects in its own first commit — including a
new rule that contradicted one shipped in #1875, which is exactly the
two-rules-for-one-boundary drift this pipeline exists to prevent.

No linked issue

## Related

- Follows #1875 (`knowledge` 0.10.15), which shipped the first
evidence-forced batch.
- Phase 3a self-alignment merged in #1876, #1877, #1878; this repo
conforms before any of it ships outward.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
kyle-sexton added a commit that referenced this pull request Aug 2, 2026
…urfaces (#1880)

Applies the evidence-forced half of the RA-9 extend-or-cite pass to
instruction-audit catalog row **I10** (reasoning-echo). `claude-config`
0.17.0 -> 0.18.0; `criteria.md` 1.4.0 -> 1.5.0.

## What this does NOT do, and why that is the point

**I10 keeps `Model scope: fable-5`. It is not promoted, and `mythos-5`
is not added.**

The campaign triage ranked this promotion its **#1 item**, on the theory
that the extended-thinking platform page is a second, model-independent
source that would satisfy I10's promotion gate. It is not, and the
page's own structure is what settles it:

- `:874` — the H2 reads "Thinking output on Claude Fable 5 **and Claude
Mythos 5**"
- `:876` — names **both** models for the adjacent raw-chain-of-thought
property
- `:887` — names **only** Claude Fable 5 for `stop_details.category:
"reasoning_extraction"`

Eleven lines apart, in one section. The page had a sentence-adjacent
opportunity to widen the refusal and declined it. That is deliberate
scoping, not loose phrasing. Adding `mythos-5` would fabricate scope
from a claim about a different property — exactly what the catalog's own
model-scoping block warns against.

That reasoning is now recorded **in the row itself**, so it is not
re-litigated a fourth time.

## What it does

- **A corroborating second Source** — the extended-thinking page,
independent of the Fable 5 model guide already cited.
- **A concretized Remediate line.** It previously said, abstractly, to
read structured `thinking` blocks or use a send-to-user tool. It now
names the actual surfaces: `Ctrl+O` verbose mode and
`showThinkingSummaries: true`, and `display: "summarized"`.

**Each surface is attributed to the page that actually states it.** The
pass this work came from asserted all three were on the thinking page;
two of them are not — `Ctrl+O` and `showThinkingSummaries` are stated at
`model-config:532`, and only `display: "summarized"` is on the thinking
page. Both pages are therefore added to `## Sources`, because
`criteria.md` carries its own invariant that "the trigger set is the
source set — naming a subset would leave the harness-behavior rows
depending on pages nothing watches."

## Verification

Independently verified by a second model, with the implementer's
rationale withheld. Both pages re-fetched raw (`model-config.md` 83,644
bytes; `thinking.md` 52,769 bytes, byte-identical to the frozen
snapshot), all three surface names confirmed verbatim, and the
promotion-gate facts re-confirmed at both snapshot and live bytes.

**Self-fire check.** Because a catalog row that fires on this repo would
break the campaign's own rule against shipping a consumer check we fail:
the scanner was run against both trees. I10 candidates in `criteria.md`
go 2 -> 4, and all four are inert — `criteria.md` is a skill reference
file, outside the audited population (`CLAUDE.md` / `rules/` / `skills/`
/ `agents/` / `output-styles/` under the user and project roots). No new
class of self-hit is introduced.

Detection is untouched: `instruction-scan.test.sh` reports 46/46, and
this edit changes Source and Remediate only.

## Held back deliberately

The **I8-b promotion**, whose gate genuinely IS met by verbatim
two-guide convergence, is not here. Promoting it makes the row fire on
this repo's own `plugins/review/context/severity.md` — the same work as
triage row RA-2, an open owner decision. Shipping the promotion first
would make the next audit run flag this repository.

No linked issue

## Related

- Phase 3b of the doc-corpus campaign; Phase 3a self-alignment merged in
#1875, #1876, #1877, #1878, #1879.
- Source pass: `RA9-EXTEND-OR-CITE-PASS-2026-08-02.md` (23 rows
adjudicated; the triage had checked 6).

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
kyle-sexton added a commit that referenced this pull request Aug 3, 2026
…y.md enumerable tier criteria (#1881)

Promotes instruction-audit catalog row **I8-b** (conservative-reporting
detection) to unscoped and gives `plugins/review/context/severity.md`
enumerable tier criteria. `claude-config` 0.18.0 -> 0.19.0; `review`
0.15.5 -> 0.16.0; `criteria.md` 1.5.0 -> 1.6.0.

## The promotion

I8-b's gate is MET on its second arm (multiple model guides converge):
the Sonnet 5 prompting guide states all three trigger phrases verbatim
in one sentence (`source.md:140`, "Code review harnesses"), converging
with the Opus 5 guide. Annotated in row I7's met-gate precedent form;
the Sonnet 5 guide URL added to `## Sources` per the
trigger-set-is-the-source-set invariant.

**Source attribution corrected while citing:** "don't nitpick" appears
nowhere in the Opus 5 guide (`grep -cin nitpick` -> 0); that guide
states only the other two phrases (its line 20). The Sonnet 5 guide is
the phrase's only cited home. This strengthens the convergence gate —
three phrases now each attributed to a page that actually contains them.

## #1880's "Held back deliberately" premise was wrong

That PR deferred this promotion because unscoping I8-b would allegedly
make it fire on `severity.md`. It does not, on three independently
verified grounds:

1. **Zero scanner candidates** — the real scanner over `severity.md`
emits one I6 row and no I8-b; the I8-b ERE greps 0 on both trees.
2. **I8-b's own carve-out** (`criteria.md:252-255`) excludes
"severity-based routing where everything is still reported somewhere" —
severity.md classifies findings and withholds none.
3. **Outside the audited population** — SKILL.md Phase A inventories
CLAUDE.md / rules/ / skills/ / agents/ / output-styles/ under user and
project roots plus hook text; `plugins/review/context/` is none of those
(same result #1880 recorded for `criteria.md` itself).

The promotion could have shipped alone. The severity.md work ships here
anyway, **re-founded on its own source**: nine lines below the
three-phrase line, the same Sonnet 5 guide says to "be concrete about
where the bar is rather than using qualitative terms like `important`"
(`source.md:150`) — and `important` is one of severity.md's own tier
names. That is triage row RA-2, a distinct claim from the one I8-b
cites.

## The severity rewrite

Each tier now carries a decidable test instead of a qualitative label;
no finding changes tier. Guards added where the criterion-stating change
could have silently re-tiered:

- The P1-P5 fold explicitly takes precedence for P-scored findings
(otherwise every P3 security finding would have read into the new
CRITICAL test).
- CRITICAL's subsequent-change limb reads "otherwise-correct change", so
cascade architecture violations (break a *correct* future change) stay
CRITICAL while code duplication (bites only through an *incomplete*
future edit) stays IMPORTANT. All eleven tier examples adjudicated
against the new tests — twice, independently.

## Verification

Independently verified by a second model with the implementer's
rationale withheld, across two rounds. Scanner counts replayed from git
refs both rounds: origin/main 23 rows / 6 files, all fenced; working
tree 28 / 7, every addition this branch's own quoting. Planted-positive
check: a constructed three-phrase file emitted I8-b on all three lines
in the same invocation where severity.md emitted zero.
Scanner-vs-git-grep equivalence proven on identical row sets.
`instruction-scan.test.sh` 46/46; markdownlint 0 errors.

**Known citation defect in an immutable commit message:** `3603c8a9b8`
says "eight lines later (`source.md:149`)". Both figures are wrong — the
correct citation is `source.md:150`, ten lines after the three-phrase
line at `source.md:140`. No tracked file carries the wrong number;
recorded here rather than rewriting pushed history.

No linked issue

## Related

- Phase 3b of the doc-corpus campaign; Phase 3a merged in #1875, #1876,
#1877, #1878, #1879, #1880.
- Reverses #1880's "Held back deliberately" reasoning with evidence
(above).
- Triage rows: RA-2 (severity criteria), RA-9/Q6 (I8-b promotion), owner
decision Q9 (bundling).

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant