Skip to content

fix(claude-review): forbid restating the PR author's claimed verification as the review's own - #320

Merged
kyle-sexton merged 1 commit into
mainfrom
fix/310-claude-review-verification-claim-hardening
Jul 30, 2026
Merged

fix(claude-review): forbid restating the PR author's claimed verification as the review's own#320
kyle-sexton merged 1 commit into
mainfrom
fix/310-claude-review-verification-claim-hardening

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

A posted review (provisioning#235, run 30503910653) closed with: "No findings — this is an accurate, well-scoped correction backed by verified evidence (`gh variable list`, `gh api orgs/.../actions/variables`, and direct `ci.yml` reads). Nothing blocking."

Verification

  • The reviewed run's effective allowedTools (confirmed in claude-review: default prompt instructs 'gh pr diff' but allowedTools grants no Bash, so large-PR diff enumeration is unsanctioned #309's verification) grants no Bash(gh:*) at all, so the review could not have run either gh command itself.
  • The PR body's own Test plan section (melodic-software/provisioning#235) reads verbatim:
    • gh variable list --repo melodic-software/provisioning → empty.
    • gh api orgs/melodic-software/actions/variables → no CI_PESTER_ENABLED.
  • The posted review's closing sentence names exactly those two commands as "verified evidence" it is claiming credit for, despite having no tool grant to run them. It restated the author's self-reported verification as its own.

Fix

Added one instruction to the default review prompt: never restate a PR author's own claimed verification as the review's own conclusion; label anything not independently run as author-claimed and unverified. Kept minimal — no other prompt or grant changes (the companion tool-grant gap is #309 / #319).

Test plan

  • yq -e '.' .github/workflows/claude-review.yml — valid YAML.
  • node --test .github/scripts/claude-lane-incident.test.cjs .github/scripts/claude-lane-retry-gate.test.cjs .github/scripts/claude-review-superseded-guard.test.cjs — 89/89 pass.

Related

Fixes #310

🤖 Generated with Claude Code

https://claude.ai/code/session_01RhS3T7ShwJgKTrvk2Mvd3C

…tion as the review's own

A posted review (provisioning#235, run 30503910653) claimed its conclusions
were "backed by verified evidence (`gh variable list`, `gh api
orgs/.../actions/variables`, and direct ci.yml reads)" — commands the review
had no tool grant to run, and whose exact strings appear verbatim in the PR
body's own Test plan. The review restated the author's self-reported
verification as if it had performed it.

Adds one instruction to the default prompt: never restate an author's
claimed verification as the review's own, and label anything not
independently run as author-claimed and unverified.

Fixes #310

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RhS3T7ShwJgKTrvk2Mvd3C
@claude

claude Bot commented Jul 30, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 53s —— View job


Review: fix(claude-review): forbid restating claimed verification

  • Gather context (PR body, diff, CLAUDE.md)
  • Review the changed file (.github/workflows/claude-review.yml)
  • Post review findings

Scope: single file, +7/-0 — .github/workflows/claude-review.yml, appending one paragraph to the prompt input's default.

Findings: none blocking.

  • The new paragraph is appended as its own paragraph after the existing "Scope your review…" text, using the same double-blank-line separator convention already used earlier in this same default (compare the pre-existing break between the CLAUDE.md paragraph and the "Scope your review" paragraph a few lines above). Indentation (10 spaces) and the absence of trailing whitespace on the blank lines are consistent with the surrounding block scalar, so the YAML folded-scalar rendering won't be affected.
  • The instruction itself is narrowly scoped and directly addresses the failure mode described in the PR body (a review crediting itself with commands only the PR author ran) without touching triggers, permissions, secrets, or SHA pins — none of CLAUDE.md's "Security ground rules" bullets are implicated by this change.
  • The author's stated decision to leave claude-security-review.yml / claude-e2e-verify.yml prompts untouched (tracked separately as claude-review: default prompt instructs 'gh pr diff' but allowedTools grants no Bash, so large-PR diff enumeration is unsanctioned #309/fix(workflows): grant the gh pr diff Bash command both review-lane prompts already instruct #319) is reasonable — it matches this repo's minimal-diff convention and this PR's own title scopes it to claude-review specifically.

Not independently verified: the Test plan's yq -e '.' .github/workflows/claude-review.yml and node --test ... results are the author's own claims — I don't have tool access in this lane to execute arbitrary commands (yq/node) against the checked-out tree, so I did not re-run them myself. Structurally the YAML change is a straightforward two-line string append inside an existing >- block scalar with matching indentation, which is consistent with it parsing cleanly, but that's a visual read, not an independent run.

@claude

claude Bot commented Jul 30, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 51s —— View job


Security Review

  • Gather context (diff, changed file)
  • Review changes for security issues
  • Post findings

No security issues found.

This PR adds 7 lines of static instructional prose to the prompt input's default value in .github/workflows/claude-review.yml (lines 106-110):

"Never restate the PR author's own claimed verification... as evidence you independently confirmed..."

Checked for the security-relevant categories in scope for this lane:

  • Trigger safety — no change to on:, still workflow_call only; the reject-privileged-triggers tripwire and pull_request-only contract are untouched.
  • Permissions — no change to the permissions: blocks (job or step level); still contents: read / pull-requests: write / id-token: write.
  • Injection surface — the edited prompt input is consumed at line 465 as an anthropics/claude-code-action with: value, not interpolated into a run: shell block, so there's no command-injection surface here regardless of the string's content. The added text is static (no github.* context expressions), so it can't introduce script/template injection either.
  • Secrets/tokens — untouched; no new secret references.
  • Supply chain / pins — no uses: lines touched, no version bumps.
  • Logic/behavior — this only changes agent instructions (what the review prompt tells the model to do/not claim), not any control flow, gating, or trust boundary. It cannot itself introduce an authorization or access-control gap since it carries no code-execution or permission effect.

Nothing in this diff falls into logic, trust-boundary, or data-flow risk — it's a same-shape, additive prose edit to an existing prompt string.

@github-actions

Copy link
Copy Markdown

Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count.

kyle-sexton added a commit that referenced this pull request Jul 30, 2026
…ompts already instruct (#319)

## Summary

The default review prompt (`claude-review.yml`'s `prompt` input) tells
the model "Use \`gh pr diff\` to identify what changed," but the default
`claude-args` `--allowedTools` grants no Bash at all — only the
inline-comment MCP tool. On a large PR the model has no sanctioned way
to enumerate the diff. `claude-security-review.yml`'s default prompt
gives the identical "use \`gh pr diff\`" instruction with the same
MCP-only default allowlist, so the security lane hits the same denial.

## Verification

Fetched run 30503910653 (melodic-software/provisioning#235, reusable at
c136b27, v0.9.1):

- The composed SDK options log the effective `allowedTools`: `Glob,
Grep, LS, Read, mcp__github_comment__update_claude_comment,
mcp__github_ci__get_ci_status, mcp__github_ci__get_workflow_run_details,
mcp__github_ci__download_job_log,
mcp__github_inline_comment__create_inline_comment, Bash(git add:*),
Bash(git commit:*), Bash(<git-push.sh>:*), Bash(git rm:*)` — no
`Bash(gh:*)` of any kind.
- The result entry for that run records `"permission_denials_count": 4`,
consistent with the model attempting and being denied tool calls the
prompt instructed it to make.
- Confirmed the prompt's own text is unchanged at `gh pr diff` in the
current `main` default.
- Confirmed via `anthropics/claude-code-action`'s own `docs/security.md`
(line 18) that the upstream-recommended pattern for exactly this
situation is a scoped grant like `Bash(gh issue view:*)` — not rewriting
the prompt to use `git diff`, which would face its own problem: the
checkout step uses `fetch-depth: 1` (single commit, no base ref
available locally), so `git diff <base>...HEAD` would not work today
either even if granted.

## Fix

Added `Bash(gh pr diff:*)` to the default `claude-args` `--allowedTools`
of both `claude-review.yml` and `claude-security-review.yml`, aligning
each lane's grant with what its prompt already instructs (the security
lane was this PR's own review finding — same root cause, fixed here
rather than deferred). Read-only command, so it adds no exposure for a
fork PR's read-only token (fork PRs aren't reviewed at all per this
repo's `CLAUDE.md`). `claude-e2e-verify.yml` is deliberately untouched:
its default `claude-args` sets no `--allowedTools` at all, a different
grant model whose defaults this fix does not own.

## Test plan

- `yq -e '.' .github/workflows/claude-review.yml` / `yq -e '.'
.github/workflows/claude-security-review.yml` — valid YAML.
- `node --test .github/scripts/claude-lane-incident.test.cjs
.github/scripts/claude-lane-retry-gate.test.cjs
.github/scripts/claude-review-superseded-guard.test.cjs` — 89/89 pass.

## Related

- `melodic-software/provisioning`#235 (run 30503910653) — the review run
whose composed SDK options supplied the `allowedTools` evidence above.
- #310 / #320 — the companion prompt-side defect surfaced by that same
review run.

Fixes #309

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01RhS3T7ShwJgKTrvk2Mvd3C

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton merged commit 94f8cf7 into main Jul 30, 2026
42 of 43 checks passed
@kyle-sexton
kyle-sexton deleted the fix/310-claude-review-verification-claim-hardening branch July 30, 2026 19:50
@kyle-sexton
kyle-sexton restored the fix/310-claude-review-verification-claim-hardening branch August 1, 2026 01:38
@kyle-sexton
kyle-sexton deleted the fix/310-claude-review-verification-claim-hardening branch August 8, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

claude-review: review output restated the PR author's claimed verification as its own evidence

1 participant