fix(guardrails): scope skill-reference-verify's replace_all reconstruction to the lines the Edit wrote - #2153
Conversation
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 71e5f19c45
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Claude finished @kyle-sexton's task in 3m 35s —— View job Security review — PR #2153
Scope: the 4 changed files only ( What the diff does
Injection / trust-boundary analysis
Finding[SUGGESTION] Guardrail can go silent on a genuinely-written reference if a later PostToolUse hook reformats the line first — Confidence: Medium The new witness filter matches by exact line text. If another PostToolUse hook reformats the edited line between the Edit and this hook's read (the exact scenario the surrounding fallback-anchor logic is explicitly built to tolerate, per the comments at This is scoped and low-blast-radius: it only fires under No CRITICAL or IMPORTANT issues found. No command/jq injection, no authorization gap, no secret-handling issue, and no dangerous GitHub Actions pattern in this diff — it doesn't touch any workflow file. |
|
Claude finished @kyle-sexton's task in 5m 58s —— View job Code review
This lane owns correctness/design/tests/maintainability; security findings are left to the dedicated Findings (posted inline)
Verified myself vs. author-claimed
|
|
Claude has reviewed this PR 3 times. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
Independent adversarial verification — PARTIALLY UPHELDThe fix is real and is not a silent no-op. I attacked the four surfaces below and could not break Verified against 1. The premise reversal — genuinely sufficient, not merely apparent on the chosen fixturesThe issue argued won't-fix on "nothing in the payload distinguishes the two." That premise is The written line carries Feeding that captured payload verbatim to both hooks: I also checked the patch is not merely sufficient on a two-line file: a 300-line file with 42 copy 2. The matching strategy — duplicate lines are safe; in-place reformatting is notThe named worry (two identical lines, one written and one not) is the benign direction. I built it: The failure mode that actually bites is the one the rationale does not disclose. Matching by text This is reachable in this marketplace. Two claims need amending, and only the prose:
Worth stating as scope rather than defect: under 3. The supply gap — CLOSED. Supply is PROVEN.This was the stated weakest point, and it is no longer open. No hook-event capture existed on this Guarded against the false negative that would have produced a bogus refutation: the dump file
4. The
|
…owed Review finding on #2153, raised by Codex and confirmed independently by Claude, and reached from the other end by this PR's adversarial verifier. All three are right. This is a false negative I INTRODUCED: the gate made the guard quieter than origin/main in a case where origin/main correctly spoke, which is worse than the false positive the PR set out to remove. Gate 3 required the on-disk physical line to equal a `+` line verbatim. An earlier-ordered PostToolUse hook that reflows whitespace leaves the anchor locatable -- a literal substring search does not care what surrounds it -- while changing the line, so the witness missed and a genuinely written reference was dropped. That undid the very tolerance the per-line fallback three lines above exists to provide. Reachable in this marketplace, not theoretical: markdown-format runs `markdownlint-cli2 --fix` and typos-format runs `typos --write-changes`, both PostToolUse with matcher Write|Edit -- the same event and matcher as this hook -- and matching hooks run in parallel, so it is an ordering-dependent race. Two corrections: 1. Comparison is whitespace-normalized (tabs to spaces, runs collapsed, ends trimmed) on both sides. Whitespace is what formatters move; a DIFFERENT reference's line differs by far more than spacing, so discrimination holds. 2. An abstain rule, which is the half that generalizes. If the witness recognizes NO occurrence at all it is stale rather than discriminating, and the unfiltered set stands. Gate 3 may now only ever REMOVE an occurrence when it can positively identify at least one the call wrote, so a formatter that rewrote more than spacing degrades to exactly the pre-gate behaviour instead of muting. I did NOT take the suggested substring-containment remedy. It fails on this hook's own fixture: the anchor is the bare word `ghost`, which occurs inside the `+` line AND inside the untouched `/alpha:ghost-old` line, so containment is true for both occurrences and the gate stops discriminating -- trading the false negative straight back for the false positive. Also hardened, from the verifier's latent-fragility note: the jq read is now shape-tolerant (`objects`). A non-object `tool_response` previously errored hook::jq_fields into its `|| exit 0` and silenced the WHOLE guard, a risk the pre-gate code did not carry because it never touched the field. Verified across string, number, null, absent, and empty-patch shapes. Supply hedge RETIRED. The verifier captured a live PostToolUse payload (temp settings.json dumping stdin, headless claude 2.1.225): `tool_response` arrives as an OBJECT carrying filePath newString oldString originalFile replaceAll structuredPatch userModified -- an object, not the serialized string PostToolBatch passes, which is the one shape that would have broken the filter. structuredPatch was complete, not truncated, at 42 replacement sites in a 300-line file. Measured, this PR's head before and after: case pre post formatter reflowed the written line no yes unreformatted: untouched ref stays suppressed yes yes two identical lines, one written yes yes liveness: empty target yields nothing no no Row 2 is the point: the fix is not "turn Gate 3 off". Scope now stated rather than implied: under replace_all, a genuine reference on a line the patch reports as CONTEXT is not reported. That is the gate working as designed, but it is broader than "fixes one false positive" and belongs in the disclosure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 4m 21s —— View job Code review (re-run after sync)
SummaryI re-read the full diff against
One new, independently-verified finding below (inline): a Keep-a-Changelog formatting nit in the new No new correctness issues found in [Nit] Missing blank line before the next version heading The new |
|
Claude finished @kyle-sexton's task in 1m 50s —— View job Security review — PR #2153 (re-run at HEAD
|
|
Claude finished @kyle-sexton's task in 4m 23s —— View job Code review (re-run at HEAD
|
|
Claude finished @kyle-sexton's task in 53s —— View job Security review — PR #2153 (re-run at HEAD
|
…ction to the lines the Edit wrote
Partial-edit reconstruction separates an occurrence the call wrote from a
coincidental one by requiring the anchor to occur exactly once -- and
`replace_all` is precisely where that rule is suspended, on the reasoning that
there every occurrence IS the edit's own footprint. It is not. After `ghost`
replaces `setup` everywhere, the `ghost` inside a pre-existing `ghost-old`
matches the anchor too, and the guard named a reference the call never touched.
the two. That holds for `tool_input` and FAILS for `tool_response`, which carries
the Edit tool's structured output: `structuredPatch` marks the lines the call
actually wrote with a leading `+`. Under `replace_all` only, an occurrence is now
kept just when its physical line is one the patch reports as written -- the
suspended uniqueness rule gets an external witness instead of nothing.
Both halves confirmed against pages fetched 2026-08-10, not recall: PostToolUse
input carries `tool_response`, "the result it returned", and that field is "the
tool's structured `Output` object" (code.claude.com/docs/en/hooks); `Output` for
Edit is `FileEditOutput`, whose `structuredPatch` is `Array<{oldStart, oldLines,
newStart, newLines, lines: string[]}>` (code.claude.com/docs/en/agent-sdk/typescript).
Matched by line TEXT, not line number. Numbers are wrong the moment another
PostToolUse hook reformats the file between the write and this read -- the case
the reconstruction fallback already exists for -- and mapping a character offset
back to a line number costs a whole-prefix scan per occurrence, reintroducing the
quadratic term 0.21.0 removed. The residual imprecision runs in the safe
direction: an untouched line whose text duplicates an edited one is kept.
Deliberately inert outside its one case. A multi-line `new_string` is not
filtered -- its anchor extent spans several lines, matches no single patch line,
and filtering would erase every finding rather than narrow them. A payload with
no `tool_response`, and every non-`replace_all` Edit, behaves exactly as before.
Measured, with the SHIPPED test file run against the unmodified origin/main hook:
case pre post
the WRITTEN reference is still reported yes yes
the UNTOUCHED reference is not reported NO yes
genuine multi-site: both refs survive yes yes
liveness: empty target yields nothing yes yes
Liveness is the issue's own technique: the identical payload against a
truncated, empty target is silent, so every finding in the real fixture
demonstrably came from READING THE FILE rather than from the payload text.
Two traps hit and fixed while building this. The first draft of the payload
builder passed the diff lines as jq `--args` positionals; every line starts with
`-`, `+` or a space, jq parsed the leading `-` as an option and died on "Unknown
option -u", the payload came back empty, and the hook went silent -- turning
`assert_absent` GREEN for entirely the wrong reason. Lines now reach jq on stdin.
Second, the positive assertion needle `UNRESOLVED_SKILL: /alpha:ghost` is a
substring of the `/alpha:ghost-old` line it exists to exclude; it now carries its
right boundary instead of relying on the paired absence assertion.
NOT verified: a live PostToolUse payload carrying `structuredPatch` was not
observed -- no hook-event capture existed on this machine to read. Schema is
documented and corroborated by real Edit records in Claude Code's transcript
JSONL. If the field never arrives, the filter never engages and nothing regresses.
Suite: 106 pass, 0 fail.
Closes #2129
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…owed Review finding on #2153, raised by Codex and confirmed independently by Claude, and reached from the other end by this PR's adversarial verifier. All three are right. This is a false negative I INTRODUCED: the gate made the guard quieter than origin/main in a case where origin/main correctly spoke, which is worse than the false positive the PR set out to remove. Gate 3 required the on-disk physical line to equal a `+` line verbatim. An earlier-ordered PostToolUse hook that reflows whitespace leaves the anchor locatable -- a literal substring search does not care what surrounds it -- while changing the line, so the witness missed and a genuinely written reference was dropped. That undid the very tolerance the per-line fallback three lines above exists to provide. Reachable in this marketplace, not theoretical: markdown-format runs `markdownlint-cli2 --fix` and typos-format runs `typos --write-changes`, both PostToolUse with matcher Write|Edit -- the same event and matcher as this hook -- and matching hooks run in parallel, so it is an ordering-dependent race. Two corrections: 1. Comparison is whitespace-normalized (tabs to spaces, runs collapsed, ends trimmed) on both sides. Whitespace is what formatters move; a DIFFERENT reference's line differs by far more than spacing, so discrimination holds. 2. An abstain rule, which is the half that generalizes. If the witness recognizes NO occurrence at all it is stale rather than discriminating, and the unfiltered set stands. Gate 3 may now only ever REMOVE an occurrence when it can positively identify at least one the call wrote, so a formatter that rewrote more than spacing degrades to exactly the pre-gate behaviour instead of muting. I did NOT take the suggested substring-containment remedy. It fails on this hook's own fixture: the anchor is the bare word `ghost`, which occurs inside the `+` line AND inside the untouched `/alpha:ghost-old` line, so containment is true for both occurrences and the gate stops discriminating -- trading the false negative straight back for the false positive. Also hardened, from the verifier's latent-fragility note: the jq read is now shape-tolerant (`objects`). A non-object `tool_response` previously errored hook::jq_fields into its `|| exit 0` and silenced the WHOLE guard, a risk the pre-gate code did not carry because it never touched the field. Verified across string, number, null, absent, and empty-patch shapes. Supply hedge RETIRED. The verifier captured a live PostToolUse payload (temp settings.json dumping stdin, headless claude 2.1.225): `tool_response` arrives as an OBJECT carrying filePath newString oldString originalFile replaceAll structuredPatch userModified -- an object, not the serialized string PostToolBatch passes, which is the one shape that would have broken the filter. structuredPatch was complete, not truncated, at 42 replacement sites in a 300-line file. Measured, this PR's head before and after: case pre post formatter reflowed the written line no yes unreformatted: untouched ref stays suppressed yes yes two identical lines, one written yes yes liveness: empty target yields nothing no no Row 2 is the point: the fix is not "turn Gate 3 off". Scope now stated rather than implied: under replace_all, a genuine reference on a line the patch reports as CONTEXT is not reported. That is the gate working as designed, but it is broader than "fixes one false positive" and belongs in the disclosure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d227a5f to
d0be16b
Compare
MD032/MD022 -- the conflict resolution joined this branch's section directly to main's `## [0.24.1]` heading with no blank line between them. Caught by the hygiene lane, reproduced locally with markdownlint-cli2, and both guardrails CHANGELOGs now lint clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Kyle Sexton <kyle-sexton@users.noreply.github.com>
d0be16b to
1bf70c4
Compare
… resolution Co-authored-by: Kyle Sexton <kyle-sexton@users.noreply.github.com>
|
Warning Automated security review did not complete — this is an infrastructure failure, not a review verdict. Treat any Claude comment on this PR (including a placeholder like "I'll analyze this and get back to you") as incomplete, not "no findings."
The check is green on purpose, and it is not evidence. It certifies that a security pass ran, and this one did not complete — but the cause is outside this PR's control, so merging is deliberately left unblocked rather than locking every merge for the length of the outage. Nothing was reviewed at this head. Where this check is required, it is satisfied without that evidence; a human should review security-sensitive changes here before merging. Re-run the job to retry the review; a new push also retries it only if the caller's Re-running does NOT help for every class:
|
Co-authored-by: Kyle Sexton <kyle-sexton@users.noreply.github.com>
|
Warning Automated review did not complete — this is an infrastructure failure, not a review verdict. Treat any Claude comment on this PR (including a placeholder like "I'll analyze this and get back to you") as incomplete, not "no findings."
Re-run the job to retry the review. A new push re-triggers this lane only if the caller's |
Co-authored-by: Kyle Sexton <kyle-sexton@users.noreply.github.com>
Disposition: fixed, not closed won't-fix
The issue offered three dispositions and flagged the defect as possibly payload-inherent. It is not,
and that is the whole reason this ships as a fix.
Partial-edit reconstruction separates an occurrence the call wrote from a coincidental one by
requiring the anchor to occur exactly once.
replace_allis precisely where that rule is suspended,on the reasoning that there every occurrence is the edit's own footprint. It is not: after
ghostreplaces
setupeverywhere, theghostinside a pre-existingghost-oldmatches the anchor too.The issue's premise — "nothing in the payload distinguishes a
ghostthis call wrote from theghostinside a pre-existingghost-old" — holds fortool_inputand fails fortool_response. The Edit tool's structured output carriesstructuredPatch, which marks thelines the call actually wrote with a leading
+. Underreplace_allonly, an occurrence is nowkept just when its physical line is one the patch reports as written. The suspended uniqueness rule
gets an external witness instead of nothing.
Both halves were confirmed against pages fetched 2026-08-10, per the repo's fresh-docs mandate:
PostToolUseinput "includes bothtool_input, the arguments sent to the tool, andtool_response, the result it returned. The exact schema for both depends on the tool", and thatfield is "the tool's structured
Outputobject" —https://code.claude.com/docs/en/hooks, "PostToolUse input".
Outputfor Edit isFileEditOutput, whosestructuredPatchisArray<{oldStart, oldLines, newStart, newLines, lines: string[]}>—https://code.claude.com/docs/en/agent-sdk/typescript, "Edit".
Why line TEXT and not line numbers
Numbers are wrong the moment another PostToolUse hook reformats the file between the write and this
read — the exact case the reconstruction fallback already exists for. And mapping a character offset
back to a line number costs a whole-prefix scan per occurrence, which would reintroduce the
quadratic term 0.21.0 spent a release removing. Text matching is a hash lookup and survives
renumbering. Its one imprecision is conservative: an untouched line whose text duplicates an edited
one is kept, so the filter can only ever drop findings the payload itself calls untouched.
Deliberately inert outside its one case
new_stringis not filtered: its anchor extent spans several lines, matches nosingle patch line, and filtering would erase every finding rather than narrow them.
tool_response, and every non-replace_allEdit, behaves exactly as before —the filter is inert by construction, not by a flag.
Verification
The table below was produced by running the shipped test file against the unmodified
origin/mainhook in a pristine worktree — not against a separate throwaway fixture. Thatdistinction is load-bearing here; see the traps below.
Liveness, using the technique the issue names: the identical payload is run against a
truncated, empty target file. The hook is silent there, so every finding in the real fixture
demonstrably came from reading the file rather than from the payload text. A filter that merely
echoed
new_stringback would have spoken in both.Suite: 106 pass, 0 fail.
Two traps hit and fixed while building this
Both are recorded because both produced a green assertion for the wrong reason, which is the failure
mode this sweep exists to stop.
--argspositionals. Everyline starts with
-,+or a space; jq parsed the leading-as an option and died onUnknown option -u. The payload came back empty, the hook went silent, andassert_absent "the UNTOUCHED reference is not reported"went green — while testing nothingat all. It was caught only because the paired
assert_containsfailed alongside it. Lines nowreach jq on stdin. This is also why the PRE column above is measured with the shipped test file
rather than the standalone fixture: the two artifacts had already diverged behaviorally once.
UNRESOLVED_SKILL: /alpha:ghostis a substring of the/alpha:ghost-oldline it exists to exclude, so it was sound only as long as the paired absence assertion stayed
next to it. It now carries its own right boundary.
Not verified
A live PostToolUse payload carrying
structuredPatchwas not directly observed — no hook-eventcapture existed on the authoring machine to read. The schema is documented (above, fetched today)
and corroborated by real Edit records in Claude Code's own transcript JSONL, which carry
originalFileand astructuredPatchwith+/-/-prefixed lines. If the field never arrives ina hook payload, the filter never engages and nothing regresses — but a reviewer with a hook-event
capture could close this gap in one grep.
Adversarial verification status
A fresh-context adversarial verifier was spawned for this PR and did not return a verdict before
the authoring session ended — the machine was saturated by concurrent agents and every spawned
verifier stalled inside a long test sweep. Treat this PR as carrying the author's own evidence only.
What partially substitutes for it, and why it is not nothing: the PRE column in the table above was
produced by running the shipped test file against the unmodified
origin/mainhook in apristine worktree, which is precisely the headline check such a verifier performs. What is still
unverified by a second party is the "can you break it" attack surface and the payload-supply
question called out below.
Closes #2129
Related
guardrailsfor fix(guardrails): block-dangerous-git still clears an unsafe --force-with-lease via payload cwd and env -S #2124, and fix(guardrails): block-convention-violation's effective_dir scans every word for -C #2113's branch bumps to the same0.24.0; whichever of the three merges later will need to re-bump.