fix(claude-review): forbid restating the PR author's claimed verification as the review's own - #320
Conversation
…tion as the review's own A posted review (provisioning#235, run 30503910653) claimed its conclusions were "backed by verified evidence (`gh variable list`, `gh api orgs/.../actions/variables`, and direct ci.yml reads)" — commands the review had no tool grant to run, and whose exact strings appear verbatim in the PR body's own Test plan. The review restated the author's self-reported verification as if it had performed it. Adds one instruction to the default prompt: never restate an author's claimed verification as the review's own, and label anything not independently run as author-claimed and unverified. Fixes #310 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RhS3T7ShwJgKTrvk2Mvd3C
|
Claude finished @kyle-sexton's task in 1m 53s —— View job Review: fix(claude-review): forbid restating claimed verification
Scope: single file, +7/-0 — Findings: none blocking.
Not independently verified: the Test plan's |
|
Claude finished @kyle-sexton's task in 51s —— View job Security Review
No security issues found. This PR adds 7 lines of static instructional prose to the
Checked for the security-relevant categories in scope for this lane:
Nothing in this diff falls into logic, trust-boundary, or data-flow risk — it's a same-shape, additive prose edit to an existing prompt string. |
|
Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
…ompts already instruct (#319) ## Summary The default review prompt (`claude-review.yml`'s `prompt` input) tells the model "Use \`gh pr diff\` to identify what changed," but the default `claude-args` `--allowedTools` grants no Bash at all — only the inline-comment MCP tool. On a large PR the model has no sanctioned way to enumerate the diff. `claude-security-review.yml`'s default prompt gives the identical "use \`gh pr diff\`" instruction with the same MCP-only default allowlist, so the security lane hits the same denial. ## Verification Fetched run 30503910653 (melodic-software/provisioning#235, reusable at c136b27, v0.9.1): - The composed SDK options log the effective `allowedTools`: `Glob, Grep, LS, Read, mcp__github_comment__update_claude_comment, mcp__github_ci__get_ci_status, mcp__github_ci__get_workflow_run_details, mcp__github_ci__download_job_log, mcp__github_inline_comment__create_inline_comment, Bash(git add:*), Bash(git commit:*), Bash(<git-push.sh>:*), Bash(git rm:*)` — no `Bash(gh:*)` of any kind. - The result entry for that run records `"permission_denials_count": 4`, consistent with the model attempting and being denied tool calls the prompt instructed it to make. - Confirmed the prompt's own text is unchanged at `gh pr diff` in the current `main` default. - Confirmed via `anthropics/claude-code-action`'s own `docs/security.md` (line 18) that the upstream-recommended pattern for exactly this situation is a scoped grant like `Bash(gh issue view:*)` — not rewriting the prompt to use `git diff`, which would face its own problem: the checkout step uses `fetch-depth: 1` (single commit, no base ref available locally), so `git diff <base>...HEAD` would not work today either even if granted. ## Fix Added `Bash(gh pr diff:*)` to the default `claude-args` `--allowedTools` of both `claude-review.yml` and `claude-security-review.yml`, aligning each lane's grant with what its prompt already instructs (the security lane was this PR's own review finding — same root cause, fixed here rather than deferred). Read-only command, so it adds no exposure for a fork PR's read-only token (fork PRs aren't reviewed at all per this repo's `CLAUDE.md`). `claude-e2e-verify.yml` is deliberately untouched: its default `claude-args` sets no `--allowedTools` at all, a different grant model whose defaults this fix does not own. ## Test plan - `yq -e '.' .github/workflows/claude-review.yml` / `yq -e '.' .github/workflows/claude-security-review.yml` — valid YAML. - `node --test .github/scripts/claude-lane-incident.test.cjs .github/scripts/claude-lane-retry-gate.test.cjs .github/scripts/claude-review-superseded-guard.test.cjs` — 89/89 pass. ## Related - `melodic-software/provisioning`#235 (run 30503910653) — the review run whose composed SDK options supplied the `allowedTools` evidence above. - #310 / #320 — the companion prompt-side defect surfaced by that same review run. Fixes #309 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01RhS3T7ShwJgKTrvk2Mvd3C --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Summary
A posted review (provisioning#235, run 30503910653) closed with: "No findings — this is an accurate, well-scoped correction backed by verified evidence (`gh variable list`, `gh api orgs/.../actions/variables`, and direct `ci.yml` reads). Nothing blocking."
Verification
allowedTools(confirmed in claude-review: default prompt instructs 'gh pr diff' but allowedTools grants no Bash, so large-PR diff enumeration is unsanctioned #309's verification) grants noBash(gh:*)at all, so the review could not have run eitherghcommand itself.gh variable list --repo melodic-software/provisioning→ empty.gh api orgs/melodic-software/actions/variables→ noCI_PESTER_ENABLED.Fix
Added one instruction to the default review prompt: never restate a PR author's own claimed verification as the review's own conclusion; label anything not independently run as author-claimed and unverified. Kept minimal — no other prompt or grant changes (the companion tool-grant gap is #309 / #319).
Test plan
yq -e '.' .github/workflows/claude-review.yml— valid YAML.node --test .github/scripts/claude-lane-incident.test.cjs .github/scripts/claude-lane-retry-gate.test.cjs .github/scripts/claude-review-superseded-guard.test.cjs— 89/89 pass.Related
melodic-software/provisioningfix(link-check,queue-monitor-liveness,tool-version-drift-check): restrict tracking-issue adoption to the workflow's own token author #235 (run 30503910653) — the reviewed PR whose posted review restated the author's claimed verification as its own.Fixes #310
🤖 Generated with Claude Code
https://claude.ai/code/session_01RhS3T7ShwJgKTrvk2Mvd3C