Skip to content

fix(guardrails): scope skill-reference-verify's replace_all reconstruction to the lines the Edit wrote - #2153

Merged
kyle-sexton merged 6 commits into
mainfrom
fix/skill-reference-verify-replace-all
Aug 12, 2026
Merged

fix(guardrails): scope skill-reference-verify's replace_all reconstruction to the lines the Edit wrote#2153
kyle-sexton merged 6 commits into
mainfrom
fix/skill-reference-verify-replace-all

Conversation

@kyle-sexton

Copy link
Copy Markdown
Contributor

Disposition: fixed, not closed won't-fix

The issue offered three dispositions and flagged the defect as possibly payload-inherent. It is not,
and that is the whole reason this ships as a fix.

Partial-edit reconstruction separates an occurrence the call wrote from a coincidental one by
requiring the anchor to occur exactly once. replace_all is precisely where that rule is suspended,
on the reasoning that there every occurrence is the edit's own footprint. It is not: after ghost
replaces setup everywhere, the ghost inside a pre-existing ghost-old matches the anchor too.

The issue's premise — "nothing in the payload distinguishes a ghost this call wrote from the
ghost inside a pre-existing ghost-old" — holds for tool_input and fails for
tool_response.
The Edit tool's structured output carries structuredPatch, which marks the
lines the call actually wrote with a leading +. Under replace_all only, an occurrence is now
kept just when its physical line is one the patch reports as written. The suspended uniqueness rule
gets an external witness instead of nothing.

Both halves were confirmed against pages fetched 2026-08-10, per the repo's fresh-docs mandate:

  • PostToolUse input "includes both tool_input, the arguments sent to the tool, and
    tool_response, the result it returned. The exact schema for both depends on the tool", and that
    field is "the tool's structured Output object" —
    https://code.claude.com/docs/en/hooks, "PostToolUse input".
  • Output for Edit is FileEditOutput, whose structuredPatch is
    Array<{oldStart, oldLines, newStart, newLines, lines: string[]}>
    https://code.claude.com/docs/en/agent-sdk/typescript, "Edit".

Why line TEXT and not line numbers

Numbers are wrong the moment another PostToolUse hook reformats the file between the write and this
read — the exact case the reconstruction fallback already exists for. And mapping a character offset
back to a line number costs a whole-prefix scan per occurrence, which would reintroduce the
quadratic term 0.21.0 spent a release removing. Text matching is a hash lookup and survives
renumbering. Its one imprecision is conservative: an untouched line whose text duplicates an edited
one is kept, so the filter can only ever drop findings the payload itself calls untouched.

Deliberately inert outside its one case

  • A multi-line new_string is not filtered: its anchor extent spans several lines, matches no
    single patch line, and filtering would erase every finding rather than narrow them.
  • A payload with no tool_response, and every non-replace_all Edit, behaves exactly as before —
    the filter is inert by construction, not by a flag.

Verification

The table below was produced by running the shipped test file against the unmodified
origin/main hook
in a pristine worktree — not against a separate throwaway fixture. That
distinction is load-bearing here; see the traps below.

assertion pre post discriminates?
the WRITTEN reference is still reported pass pass no — must not regress
the UNTOUCHED reference is not reported FAIL pass yes
genuine multi-site: both refs survive the filter pass pass no — proves no findings lost
liveness: empty target yields nothing at all pass pass no — see below

Liveness, using the technique the issue names: the identical payload is run against a
truncated, empty target file. The hook is silent there, so every finding in the real fixture
demonstrably came from reading the file rather than from the payload text. A filter that merely
echoed new_string back would have spoken in both.

Suite: 106 pass, 0 fail.

Two traps hit and fixed while building this

Both are recorded because both produced a green assertion for the wrong reason, which is the failure
mode this sweep exists to stop.

  1. The first draft of the payload builder passed the diff lines as jq --args positionals. Every
    line starts with -, + or a space; jq parsed the leading - as an option and died on
    Unknown option -u. The payload came back empty, the hook went silent, and
    assert_absent "the UNTOUCHED reference is not reported" went green — while testing nothing
    at all. It was caught only because the paired assert_contains failed alongside it. Lines now
    reach jq on stdin. This is also why the PRE column above is measured with the shipped test file
    rather than the standalone fixture: the two artifacts had already diverged behaviorally once.
  2. The positive needle UNRESOLVED_SKILL: /alpha:ghost is a substring of the /alpha:ghost-old
    line it exists to exclude, so it was sound only as long as the paired absence assertion stayed
    next to it. It now carries its own right boundary.

Not verified

A live PostToolUse payload carrying structuredPatch was not directly observed — no hook-event
capture existed on the authoring machine to read. The schema is documented (above, fetched today)
and corroborated by real Edit records in Claude Code's own transcript JSONL, which carry
originalFile and a structuredPatch with +/-/ -prefixed lines. If the field never arrives in
a hook payload, the filter never engages and nothing regresses — but a reviewer with a hook-event
capture could close this gap in one grep.

Adversarial verification status

A fresh-context adversarial verifier was spawned for this PR and did not return a verdict before
the authoring session ended — the machine was saturated by concurrent agents and every spawned
verifier stalled inside a long test sweep. Treat this PR as carrying the author's own evidence only.

What partially substitutes for it, and why it is not nothing: the PRE column in the table above was
produced by running the shipped test file against the unmodified origin/main hook in a
pristine worktree, which is precisely the headline check such a verifier performs. What is still
unverified by a second party is the "can you break it" attack surface and the payload-supply
question called out below.

Closes #2129

Related

@cursor

cursor Bot commented Aug 10, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 71e5f19c45

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/guardrails/hooks/skill-reference-verify.sh Outdated
@claude

claude Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 3m 35s —— View job


Security review — PR #2153

  • Gather context (git diff origin/main...HEAD, changed files)
  • Review plugins/guardrails/hooks/skill-reference-verify.sh for logic/injection/trust-boundary issues
  • Review test file and manifest/changelog changes
  • Post findings

Scope: the 4 changed files only (plugins/guardrails/hooks/skill-reference-verify.sh, .test.sh, plugin.json, CHANGELOG.md). No GitHub Actions workflow is touched by this PR, so there's nothing in scope for the zizmor advisory lane (unpinned actions, dangerous triggers, permissions, template injection) — this review focuses on the hook's own logic and trust boundaries.

What the diff does

skill-reference-verify.sh gains a new jq-extracted field, EDIT_WROTE_LINES, pulled from tool_response.structuredPatch (Edit's own structured output, produced by the harness — not attacker/model-controlled text). Under replace_all only, a reconstructed occurrence is now kept only if its physical line text is one the patch reports as written, closing the false-positive from #2129 (a coincidental pre-existing occurrence getting reported as edited).

Injection / trust-boundary analysis

  • The new jq filter (plugins/guardrails/hooks/skill-reference-verify.sh#L84-86) is a static, hardcoded filter string; hook::jq_fields pipes $INPUT to jq over stdin, never interpolates payload content into the jq program text, and uses -j/NUL-delimited output parsing. No jq or shell injection surface here, consistent with the rest of the file.
  • EDIT_WROTE_LINES values are loaded into wrote["L$wl"]=1 on an associative array (local -A wrote=(), skill-reference-verify.sh#L529-538, consumed at #L596). Associative-array subscripts in bash are treated as literal string keys, not arithmetic expressions — unlike indexed-array subscripts, they are not re-evaluated, so attacker-influenced line text placed into $line/$wl cannot trigger the classic bash "arithmetic-subscript command substitution" injection. Verified against bash's documented subscript-evaluation semantics. No exploitable path found.
  • No eval, no unquoted variable expansion passed to a command, no new subprocess spawned with payload-derived arguments. tool_response.structuredPatch is harness-generated from the real on-disk diff, not something a prompt-injected skill/subagent can forge independently of the file it already wrote (which the pre-existing SCAN_CONTENT path already scans).
  • plugin.json / CHANGELOG.md changes are a version bump and documentation only — no config/permission/secret surface.
  • Test file (skill-reference-verify.test.sh) builds fixtures with jq -n --arg/--argjson (never string-interpolated), consistent with safe practice.

Finding

[SUGGESTION] Guardrail can go silent on a genuinely-written reference if a later PostToolUse hook reformats the line first — Confidence: Medium
plugins/guardrails/hooks/skill-reference-verify.sh#L592-598 (also flagged independently by Codex on this PR)

The new witness filter matches by exact line text. If another PostToolUse hook reformats the edited line between the Edit and this hook's read (the exact scenario the surrounding fallback-anchor logic is explicitly built to tolerate, per the comments at #L117-124 and #L552-555), structuredPatch's recorded + line text no longer equals the current on-disk line text even though gate 1's own anchor-fallback can still relocate the reference. The occurrence is then silently dropped — a false negative on an advisory meant to catch hallucinated/stale /plugin:skill references (which could otherwise reference a typosquatted or nonexistent capability unnoticed).

This is scoped and low-blast-radius: it only fires under replace_all, only when a second hook reformats before this one runs, and the hook's own documented design philosophy throughout this file explicitly treats "stay quiet" as the acceptable failure direction for a detect-then-judge advisory (e.g. #L574-578), so this isn't a new failure class introduced by the PR so much as an extension of an accepted trade-off into one more case. Given the file's stated design intent, this is a judgment call for the maintainer rather than a blocking issue — flagging for visibility since it does marginally reduce guardrail coverage.

No CRITICAL or IMPORTANT issues found. No command/jq injection, no authorization gap, no secret-handling issue, and no dangerous GitHub Actions pattern in this diff — it doesn't touch any workflow file.

@claude

claude Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 5m 58s —— View job


Code review

  • Gather context (PR diff, changed files, CLAUDE.md/REVIEW.md)
  • Review skill-reference-verify.sh changes (correctness, error handling, maintainability)
  • Review skill-reference-verify.test.sh changes (test coverage)
  • Evaluate the open Codex review comment (downstream-formatting edge case)
  • Post inline findings and summary

This lane owns correctness/design/tests/maintainability; security findings are left to the dedicated claude-security-review workflow present in this repo, per REVIEW.md.

Findings (posted inline)

  1. [Important — correctness] skill-reference-verify.sh:596 — Codex's review comment on this line is correct, and I traced the mechanism independently to confirm it's real rather than hypothetical: Gate 3's line-text witness is compared against the current on-disk line, but structuredPatch's + lines are captured at edit time. If an earlier-firing PostToolUse hook reformats the same physical line (without touching the edited substring) before this hook runs, the anchor still locates — via the very fallback path built to survive exactly that reformatting — but the witness lookup misses, and a genuinely-written reference is silently dropped. That's narrower than the "one imprecision" the surrounding comment (lines 446-455) documents, and it contradicts the reformatting-tolerance invariant the fallback three lines above Gate 3 exists for. I proposed a substring-containment match as a more resilient alternative, or explicitly documenting it as a second accepted imprecision. Impact is bounded since this hook is advisory-only (a missed advisory, not a wrong one), but the PR's own verification table doesn't exercise this ordering, so "no regression" shouldn't be read as covering it.
  2. [Suggestion — test coverage] skill-reference-verify.test.sh:400 — the four new cases don't cover the scenario in finding 1 (on-disk line text diverging from the patch's captured + line due to incidental reformatting). Suggested a fifth case for it.

Verified myself vs. author-claimed

  • I read the full diff (git diff origin/main...HEAD) and traced the jq extraction (skill-reference-verify.sh:83-86), the hook::jq_fields CR-stripping (hook-utils.sh:672-687), and the Gate 3 filter logic (skill-reference-verify.sh:524-602) by hand.
  • I checked a CRLF-related concern of my own (whether EDIT_WROTE_LINES gets the same \r-stripping as content) and confirmed it does — hook::jq_fields strips CR from every field it returns, so that's not a bug.
  • I could not execute the test suite in this environment (script execution requires approval this lane doesn't have) — the PR's "106 pass, 0 fail" and the pre/post verification table are author-claimed and unverified by me.
  • No test-file variable collisions or structural issues found in the new test block.
  • Version bump 0.24.0 → 0.26.0 (skipping 0.25.0) is already called out by the author as an expected multi-branch collision with fix(guardrails): block-dangerous-git still clears an unsafe --force-with-lease via payload cwd and env -S #2124/fix(guardrails): block-convention-violation's effective_dir scans every word for -C #2113; no action needed from this review.

Comment thread plugins/guardrails/hooks/skill-reference-verify.sh Outdated
Comment thread plugins/guardrails/hooks/skill-reference-verify.test.sh
@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown

Claude has reviewed this PR 3 times. The lane skips further automatic reviews after 5; deleting this comment resets the count.

@kyle-sexton

Copy link
Copy Markdown
Contributor Author

Independent adversarial verification — PARTIALLY UPHELD

The fix is real and is not a silent no-op. I attacked the four surfaces below and could not break
the behaviour. What I did break is one load-bearing sentence of the rationale and the completeness of
the disclosure — both fixable by editing prose, neither a reason to hold the code.

Verified against pull/2153/head, whose skill-reference-verify.sh and .test.sh are byte-identical
to the tree I ran (diff -q clean on both), so everything below is a run against this PR's code.


1. The premise reversal — genuinely sufficient, not merely apparent on the chosen fixtures

The issue argued won't-fix on "nothing in the payload distinguishes the two." That premise is
correctly reversed, and I confirmed it against a payload I captured rather than against the
fixtures the PR chose:

=== structuredPatch, from a LIVE PostToolUse capture ===
[{"oldStart":1,"oldLines":2,"newStart":1,"newLines":2,"lines":[
  "-Run `/alpha:setup` now.","+Run `/alpha:ghost` now.",
  " Legacy `/alpha:ghost-old` stays."]}]

The written line carries +; the pre-existing ghost-old line carries a leading space. The
information the issue said was gone is present, and it is present in the exact shape the filter reads.

Feeding that captured payload verbatim to both hooks:

--- origin/main ---  UNRESOLVED_SKILL: /alpha:ghost
                     UNRESOLVED_SKILL: /alpha:ghost-old
--- this PR     ---  UNRESOLVED_SKILL: /alpha:ghost

I also checked the patch is not merely sufficient on a two-line file: a 300-line file with 42 copy
sites under replace_all produced 42 + lines against 42 occurrences on disk — complete, not
truncated, at that size.

2. The matching strategy — duplicate lines are safe; in-place reformatting is not

The named worry (two identical lines, one written and one not) is the benign direction. I built it:
the untouched duplicate is kept, so the filter errs toward reporting. That matches the PR's own
claim and costs nothing.

The failure mode that actually bites is the one the rationale does not disclose. Matching by text
survives renumbering but not an in-place rewrite:

D2  a formatter re-indented the written line
--- origin/main ---  ghost-one, ghost-two
--- this PR     ---  ghost-two          <- lost a genuinely WRITTEN reference

D3  a formatter trimmed trailing whitespace off the written line
--- origin/main ---  ghost-one
--- this PR     ---  <SILENT>           <- lost the only finding

This is reachable in this marketplace. markdown-format runs markdownlint-cli2 --fix "$FILE"
(markdown-format.sh:882) and typos-format runs typos --write-changes (typos-format.sh:268) —
both PostToolUse, both matcher Write|Edit, the same event and matcher as skill-reference-verify.
Per hooks.md:403 all matching hooks run in parallel, so it is an ordering-dependent race, not a
systematic loss.

Two claims need amending, and only the prose:

  • The stated reason for text over line numbers is incomplete, not wrong. A --fix pass contains
    both line-count-changing rules (MD022/MD031/MD047 — break numbers) and in-place rewrites
    (MD009/MD010/list indent — break text). Neither key survives a formatter; only the
    number-breaking half is disclosed, which makes text read as the robust choice.
  • "The residual imprecision runs in the safe direction: an untouched line whose text duplicates an
    edited one is kept."
    True as far as it goes — D3 is the other direction, undisclosed. For an
    advisory guard, failing quiet may well be preferable; my objection is the accuracy of the
    disclosure, not the severity.

Worth stating as scope rather than defect: under replace_all, a distinct genuine reference on any
line the patch reports as context is no longer reported at all (ghost-one, ghost-twoghost-one).
Consistent with the acceptance criteria, but broader than "fixes one false positive."

3. The supply gap — CLOSED. Supply is PROVEN.

This was the stated weakest point, and it is no longer open. No hook-event capture existed on this
box, so I made one: a temp settings.json with a PostToolUse/Edit hook dumping stdin, driven by a
real headless session on claude 2.1.225.

=== top-level keys ===   cwd duration_ms effort hook_event_name permission_mode prompt_id
                         session_id tool_input tool_name tool_response tool_use_id transcript_path
=== tool_response type ===  object
=== tool_response keys ===  filePath newString oldString originalFile
                            replaceAll structuredPatch userModified

Guarded against the false negative that would have produced a bogus refutation: the dump file
existed
, so the hook fired — "no dump" (misconfiguration) and "dump present, no structuredPatch"
(real refutation) were distinguishable, and this was neither.

tool_response arrives as an object, not the serialized string PostToolBatch passes — the
distinction hooks.md:1991 draws, and the one shape that would have broken the jq filter. The PR's
hedge ("if the field never arrives the filter never engages") can be retired; the field arrives.

4. The assert_absent trap — checked, and this one is sound

The diff adds exactly one absence assertion, and it has a paired positive on the same $OUT, with
no reassignment between them:

assert_contains "replace_all + patch → the WRITTEN reference is still reported" \
  "$OUT" 'UNRESOLVED_SKILL: /alpha:ghost (no such skill'
assert_absent   "replace_all + patch → the UNTOUCHED reference is not reported" \
  "$OUT" "/alpha:ghost-old"

The needle boundary is real, not assumed — against actual origin/main output the excluded line reads
UNRESOLVED_SKILL: /alpha:ghost-old (no such skill, which cannot satisfy ghost (no. Both assertions
pass on this PR, so the hook demonstrably spoke in the run where ghost-old is absent. This absence
assertion cannot be green-for-nothing.
The other assert_absent/assert_silent calls in the file
are pre-existing and untouched by this diff.

Suite delta, both trees, origin/main hook confirmed pristine and test files byte-identical:

origin/main  PASS=105 FAIL=1   <- FAIL: "the UNTOUCHED reference is not reported"
this PR      PASS=106 FAIL=0

Exactly one of the six new assertions discriminates — which is what the PR's own table claims. The
other five pass on both trees: four are legitimate controls (including the empty-target liveness
proof) and are correctly non-discriminating, not padding.

Also attacked, and survived

Only one call site into collect_overlapping_spans — no gate-3 bypass path. CRLF is neutralized
twice over: the live capture shows patch lines carry no trailing \r, and hook::jq_fields:684
CR-strips every value; hand-injected trailing CRs changed nothing. origin/main == this PR on empty
structuredPatch, absent tool_response, replace_all: false with a patch present,
\ No newline at end of file, and multi-hunk (the union across hunks is correct).

One latent fragility, not reachable: a string-valued tool_response makes the new jq expression
error, hook::jq_fields return 1, and || exit 0 silence the hook entirely while origin/main still
reports. The matcher is Write|Edit, tool failures route to the separate PostToolUseFailure event,
and PostToolBatch is not registered — so I could not construct a live path to it. Worth knowing only
because origin/main was shape-agnostic here (it never touched the field) and this PR is not.


Read-versus-run inventory

Verified with runs — the live PostToolUse capture and its object-typed tool_response;
structuredPatch completeness at 42/42 sites; the captured payload replayed verbatim against both
hooks; the 105/1 vs 106/0 suite delta with the pristine-base precondition checked; the duplicate-line
case; D2/D3 in-place-reformat losses; context-line scope narrowing; empty structuredPatch, absent
tool_response, replace_all: false, no-newline-at-EOF, CRLF, injected trailing CRs, multi-hunk
union; the string-tool_response silencing and its unreachability (matcher, event routing,
registration); the assert_absent pairing and needle boundary against real base output; the
markdownlint-cli2 --fix and typos --write-changes invocations and their hook registrations.

Read but not run — the CHANGELOG prose and version bump; the header comment block; the
RECONSTRUCT_MAX_SPANS cap interaction with the new continue (cap tests pass on both trees, but I
did not probe whether the changed budget accounting alters which findings survive at the cap).

Neither — behaviour on any Claude Code version other than 2.1.225; any platform other than
Windows/Git Bash; the skill-level consumers of this hook's advisory output.

What I did NOT test

  • Multi-line new_string passthrough. My fixture went <SILENT> on both trees — a non-result,
    not a confirmation. The PR makes an explicit claim that this shape is passed through unfiltered;
    it remains unverified by me.
  • A structuredPatch truncation ceiling. Proven complete at 42 sites / 300 lines; I did not search
    for a threshold above that. A very large replace_all could in principle truncate and silently drop.
  • A live formatter-vs-verify interleaving. I established same-event/same-matcher registration and
    the parallel-execution doc statement, but D2/D3 used hand-built payloads simulating post-format disk
    state rather than a captured race.
  • The Write path beyond the suite's existing Write cases — the new filter is Edit +
    replace_all-gated.
  • Whether "won't-fix" was the better disposition. The fix demonstrably works, so I attacked the
    premise empirically instead of arguing the judgement call.

Recommendation: the behaviour is merge-worthy. Before merge, amend the two prose claims — the
text-vs-numbers justification is half the picture, and the "safe direction" sentence should disclose
the drop-a-written-finding direction that D3 demonstrates. The supply hedge can be deleted outright;
the field is confirmed delivered.

kyle-sexton added a commit that referenced this pull request Aug 10, 2026
…owed

Review finding on #2153, raised by Codex and confirmed independently by Claude,
and reached from the other end by this PR's adversarial verifier. All three are
right. This is a false negative I INTRODUCED: the gate made the guard quieter
than origin/main in a case where origin/main correctly spoke, which is worse than
the false positive the PR set out to remove.

Gate 3 required the on-disk physical line to equal a `+` line verbatim. An
earlier-ordered PostToolUse hook that reflows whitespace leaves the anchor
locatable -- a literal substring search does not care what surrounds it -- while
changing the line, so the witness missed and a genuinely written reference was
dropped. That undid the very tolerance the per-line fallback three lines above
exists to provide. Reachable in this marketplace, not theoretical: markdown-format
runs `markdownlint-cli2 --fix` and typos-format runs `typos --write-changes`,
both PostToolUse with matcher Write|Edit -- the same event and matcher as this
hook -- and matching hooks run in parallel, so it is an ordering-dependent race.

Two corrections:

1. Comparison is whitespace-normalized (tabs to spaces, runs collapsed, ends
   trimmed) on both sides. Whitespace is what formatters move; a DIFFERENT
   reference's line differs by far more than spacing, so discrimination holds.

2. An abstain rule, which is the half that generalizes. If the witness recognizes
   NO occurrence at all it is stale rather than discriminating, and the unfiltered
   set stands. Gate 3 may now only ever REMOVE an occurrence when it can
   positively identify at least one the call wrote, so a formatter that rewrote
   more than spacing degrades to exactly the pre-gate behaviour instead of muting.

I did NOT take the suggested substring-containment remedy. It fails on this
hook's own fixture: the anchor is the bare word `ghost`, which occurs inside the
`+` line AND inside the untouched `/alpha:ghost-old` line, so containment is true
for both occurrences and the gate stops discriminating -- trading the false
negative straight back for the false positive.

Also hardened, from the verifier's latent-fragility note: the jq read is now
shape-tolerant (`objects`). A non-object `tool_response` previously errored
hook::jq_fields into its `|| exit 0` and silenced the WHOLE guard, a risk the
pre-gate code did not carry because it never touched the field. Verified across
string, number, null, absent, and empty-patch shapes.

Supply hedge RETIRED. The verifier captured a live PostToolUse payload (temp
settings.json dumping stdin, headless claude 2.1.225): `tool_response` arrives as
an OBJECT carrying filePath newString oldString originalFile replaceAll
structuredPatch userModified -- an object, not the serialized string
PostToolBatch passes, which is the one shape that would have broken the filter.
structuredPatch was complete, not truncated, at 42 replacement sites in a
300-line file.

Measured, this PR's head before and after:

  case                                            pre  post
  formatter reflowed the written line             no   yes
  unreformatted: untouched ref stays suppressed   yes  yes
  two identical lines, one written                yes  yes
  liveness: empty target yields nothing           no   no

Row 2 is the point: the fix is not "turn Gate 3 off".

Scope now stated rather than implied: under replace_all, a genuine reference on a
line the patch reports as CONTEXT is not reported. That is the gate working as
designed, but it is broader than "fixes one false positive" and belongs in the
disclosure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 4m 21s —— View job


Code review (re-run after sync)

  • Gather context (git diff origin/main...HEAD, changed files, prior review threads)
  • Review the whitespace-normalization + abstain-rule fix in skill-reference-verify.sh (addresses the Codex/Claude false-negative finding)
  • Review the four new test cases in skill-reference-verify.test.sh
  • Check CHANGELOG.md / plugin.json version consistency
  • Post findings

Summary

I re-read the full diff against origin/main (not the earlier review's diff — the branch has since picked up bae11bb, the fix for the false-negative Codex and I both flagged). Traced by hand, not re-stating anyone's claimed test output:

  • anchor_line() (skill-reference-verify.sh:493-503) correctly factors out the physical-line computation via bash's dynamic scoping (writes into the caller's hs/he/line), and it's only ever called from within reconstruct_partial_edit's own frame — no leakage into an unexpected scope.
  • norm_ws() (skill-reference-verify.sh:518-523)'s collapse loop (while [[ "$s" == *" "* ]]; do s="${s// / }"; done) is not quadratic on a pathological run of spaces the way it looks — each pass is a non-overlapping global substitution that halves the run length, so a run of N spaces collapses in O(log N) passes / O(N) total work, not O(N²). No perf regression here.
  • The abstain rule (skill-reference-verify.sh:657-664) is scoped per anchor, inside the for anchor in "${anchors[@]}" loop — finer-grained than an all-or-nothing global abstain would be. Traced through: if anchor A's occurrences all miss the witness (formatter rewrote more than whitespace) but anchor B's occurrences hit, A abstains to its unfiltered set while B still gets filtered. That's a better property than what either review thread asked for.
  • Walked the five new replall-* fixtures (skill-reference-verify.test.sh:355-431) by hand against the jq extraction and the gate logic — REFLOW and STALE correctly exercise the whitespace-normalize and abstain paths respectively, and the positive/negative assertion pairing (ghost vs ghost-old) is sound against real hook output, not just against each other.
  • I could not execute the suite myself (no shell-execution approval in this lane) — the "106/110 pass" / pre-post table numbers remain author-claimed, unverified by me, consistent with what the prior code-review pass already disclosed.

One new, independently-verified finding below (inline): a Keep-a-Changelog formatting nit in the new CHANGELOG.md entry — confirmed by reading the raw file, not asserted.

No new correctness issues found in skill-reference-verify.sh beyond what Codex/Claude/the author already worked through in this thread; that fix looks sound on this independent pass.


[Nit] Missing blank line before the next version heading
CHANGELOG.md:62-63

The new ## [0.26.0] entry runs directly into ## [0.24.1] with no blank line separating them — every other version heading in this file (## [0.24.0], ## [0.23.1], …) is preceded by a blank line. This repo's .markdownlint-cli2.jsonc doesn't disable MD022 (blanks-around-headings), and CI's "Lint markdown" step runs that config against the repo (continue-on-error: true, so it won't block the build, but it'll show as a lint failure on this PR). One-line fix: insert a blank line before ## [0.24.1].
· branch fix/skill-reference-verify-replace-all

@claude

claude Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 1m 50s —— View job


Security review — PR #2153 (re-run at HEAD 47819a6)

  • Gather diff for latest commits (bae11bb fix + merge 47819a6)
  • Review skill-reference-verify.sh changes since last security pass (whitespace-normalized witness + abstain rule)
  • Review test file additions
  • Post findings

Scope: the 4 changed files (plugins/guardrails/hooks/skill-reference-verify.sh, .test.sh, plugin.json, CHANGELOG.md). No workflow file is touched, so nothing is in scope for the zizmor advisory lane here either.

What changed since the last security pass on this PR

Commit bae11bb fixes the false-negative Codex/Claude flagged (Gate 3 silently dropping a genuinely-written reference when a downstream formatter reflows the edited line before this hook reads it). The fix adds two new pieces to reconstruct_partial_edit:

  • anchor_line() (skill-reference-verify.sh#L493-502) — pure string-slice extraction of the physical line, refactored out of the inline block that existed before this PR. No new operation, same arithmetic.
  • norm_ws() (#L518-523) — whitespace normalization used both when building the wrote[] witness set and when comparing the on-disk line, plus an "abstain" fallback (#L657-664) when the witness matches nothing.

Trust-boundary / injection re-analysis

  • wrote["L$NORM_WS"]=1 (#L599) and its lookup at #L655 remain on the associative array wrote (local -A wrote=(), #L591). Associative-array subscripts are literal string keys in bash, never re-evaluated as arithmetic — the same property that made this construct safe pre-fix still holds after normalization is inserted in front of it. No injection path from attacker/model-controlled line text.
  • norm_ws() and anchor_line() are string-slicing/substitution only (${s//$'\t'/ }, ${s#...}, ${content:off:len}) — no eval, no command substitution of payload-derived text, no new subprocess spawned with untrusted arguments.
  • The jq extraction of structuredPatch (#L83-86) is unchanged by this commit: still a static filter string, $INPUT still reaches jq over stdin via hook::jq_fields, never interpolated into the program text.
  • Test file additions (skill-reference-verify.test.sh#L400-427 and the new replall_patch_json/reflow/stale-witness cases) build fixtures exclusively with jq -n --arg/--argjson and pipe diff lines to jq over stdin rather than as positional args — the PR description itself notes an earlier draft leaked --prefixed lines as jq option flags and got caught by a paired assertion; the shipped version avoids that shape entirely. No injection surface in the test harness.

Availability consideration (not scored as a vulnerability)

EDIT_WROTE_LINES and the wrote[] hash it populates are unbounded by any of the file's existing RECONSTRUCT_MAX_* caps — those all gate $content (the re-read target file) or anchor/occurrence counts, not patch-line count. In practice this isn't a new exposure: structuredPatch size is intrinsically bounded by the edit itself (harness-generated from the actual diff), and the pre-existing RECONSTRUCT_MAX_CHARS gate on the target file already bounds the dominant cost in this function. Not raising this as a finding.

Conclusion

No CRITICAL or IMPORTANT security issues in this commit. The fix is a correctness/logic change (closing a false-negative) built entirely from string comparison and literal associative-array keys — it doesn't add or alter any injection, authorization, secret-handling, or GitHub Actions surface beyond what the two prior review passes on this PR already cleared.

Comment thread plugins/guardrails/CHANGELOG.md
@claude

claude Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 4m 23s —— View job


Code review (re-run at HEAD d227a5f)

  • Gather context (gh pr diff, changed files, full prior review thread)
  • Review skill-reference-verify.sh at current HEAD (Gate 3 witness filter, anchor_line, norm_ws, abstain rule)
  • Review skill-reference-verify.test.sh new cases (WRITTEN / UNTOUCHED / LIVENESS / BOTH / REFLOW / STALE)
  • Check CHANGELOG.md / plugin.json version consistency
  • Post findings

Summary

This is a re-review after the merge and the blank-line-only follow-up commit (d227a5f). The false-negative Codex/Claude flagged earlier in the thread (Gate 3 dropping a genuinely-written reference when a downstream formatter reflows the line) was already fixed by bae11bb and confirmed by the prior review pass on 47819a6 — I re-traced it independently rather than trusting that confirmation, and it holds:

  • anchor_line() (skill-reference-verify.sh:493-502) computes the physical line the same way the pre-refactor inline block did — pure extraction, no behavior change.
  • norm_ws() (#L518-523): I hand-traced the collapse loop against an odd-length run of spaces (e.g. 3) — the first pass leaves one " " pair standing, the while correctly iterates again rather than stopping after one pass. Not quadratic on a pathological run, per the earlier pass's O(log N) analysis.
  • The abstain rule (#L657-664) is scoped per-anchor (inside the for anchor in "${anchors[@]}" loop, keep reset each iteration) — confirmed by reading the loop structure, not just the diff hunk in isolation.
  • Confirmed hook::jq_fields (hook-utils.sh:730-744) CR-strips every returned field uniformly, so EDIT_WROTE_LINES gets the same treatment as content — the witness and the on-disk line are compared on equally-normalized text.
  • plugin.json (0.26.0) and the new CHANGELOG.md heading (## [0.26.0]) agree, and the missing blank line before ## [0.24.1] flagged in the previous pass is fixed in d227a5f.

I could not execute the test suite or shellcheck myself — both require an approval this lane doesn't have in this run — so the "106/110 pass" / pre-post table numbers remain author-claimed and unverified by me, consistent with what every prior pass on this PR has already disclosed.

New finding (posted inline)

[Suggestion — test coverage] skill-reference-verify.test.sh:359 — every new fixture builds a single-hunk structuredPatch array. The jq extraction it's standing in for flattens across all hunk objects ((.tool_response | objects | .structuredPatch)[]?.lines[]?), and that multi-hunk union is currently verified only by the author manually (42/42 sites, already disclosed as author-claimed in the PR body) — nothing in the shipped suite would catch a regression in the flatten. Suggested a sixth case with a two-hunk structuredPatch array. Low severity, cheap to close.

No correctness issues found in this pass beyond what's already been raised and resolved earlier in this thread by Codex and the two prior Claude review rounds.
· branch fix/skill-reference-verify-replace-all

@claude

claude Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 53s —— View job


Security review — PR #2153 (re-run at HEAD d227a5f)

  • Gather diff for latest commit (d227a5f — CHANGELOG blank-line fix)
  • Confirm no security-relevant code changed since the last security pass (47819a6)
  • Post findings

What changed since the last security pass on this PR

Only one commit landed after the previous security review (47819a6): d227a5f, which adds a single blank line in plugins/guardrails/CHANGELOG.md before the ## [0.24.1] heading — exactly the Markdown/MD022 nit flagged in this PR's non-security code-review pass. Confirmed directly:

$ git diff 47819a6..d227a5f --stat
 plugins/guardrails/CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)

No change to skill-reference-verify.sh, skill-reference-verify.test.sh, or plugin.json since the prior security pass. That pass already reviewed the whitespace-normalized witness (norm_ws()) and the per-anchor abstain rule added in bae11bb, and cleared them: no injection surface, wrote[] stays a literal-string-keyed associative array (not re-evaluated as arithmetic), and the jq extraction of structuredPatch remains a static filter string fed via stdin, never interpolated.

Conclusion

No CRITICAL or IMPORTANT security issues. The only change since the last review is a documentation-only blank-line insertion in CHANGELOG.md — no code, config, permission, or trust-boundary surface touched. Nothing new to flag; the security posture established across the prior two review passes on this PR stands unchanged.
· branch fix/skill-reference-verify-replace-all

Comment thread plugins/guardrails/hooks/skill-reference-verify.test.sh
kyle-sexton and others added 2 commits August 11, 2026 23:41
…ction to the lines the Edit wrote

Partial-edit reconstruction separates an occurrence the call wrote from a
coincidental one by requiring the anchor to occur exactly once -- and
`replace_all` is precisely where that rule is suspended, on the reasoning that
there every occurrence IS the edit's own footprint. It is not. After `ghost`
replaces `setup` everywhere, the `ghost` inside a pre-existing `ghost-old`
matches the anchor too, and the guard named a reference the call never touched.

the two. That holds for `tool_input` and FAILS for `tool_response`, which carries
the Edit tool's structured output: `structuredPatch` marks the lines the call
actually wrote with a leading `+`. Under `replace_all` only, an occurrence is now
kept just when its physical line is one the patch reports as written -- the
suspended uniqueness rule gets an external witness instead of nothing.

Both halves confirmed against pages fetched 2026-08-10, not recall: PostToolUse
input carries `tool_response`, "the result it returned", and that field is "the
tool's structured `Output` object" (code.claude.com/docs/en/hooks); `Output` for
Edit is `FileEditOutput`, whose `structuredPatch` is `Array<{oldStart, oldLines,
newStart, newLines, lines: string[]}>` (code.claude.com/docs/en/agent-sdk/typescript).

Matched by line TEXT, not line number. Numbers are wrong the moment another
PostToolUse hook reformats the file between the write and this read -- the case
the reconstruction fallback already exists for -- and mapping a character offset
back to a line number costs a whole-prefix scan per occurrence, reintroducing the
quadratic term 0.21.0 removed. The residual imprecision runs in the safe
direction: an untouched line whose text duplicates an edited one is kept.

Deliberately inert outside its one case. A multi-line `new_string` is not
filtered -- its anchor extent spans several lines, matches no single patch line,
and filtering would erase every finding rather than narrow them. A payload with
no `tool_response`, and every non-`replace_all` Edit, behaves exactly as before.

Measured, with the SHIPPED test file run against the unmodified origin/main hook:

  case                                          pre   post
  the WRITTEN reference is still reported       yes   yes
  the UNTOUCHED reference is not reported       NO    yes
  genuine multi-site: both refs survive         yes   yes
  liveness: empty target yields nothing         yes   yes

Liveness is the issue's own technique: the identical payload against a
truncated, empty target is silent, so every finding in the real fixture
demonstrably came from READING THE FILE rather than from the payload text.

Two traps hit and fixed while building this. The first draft of the payload
builder passed the diff lines as jq `--args` positionals; every line starts with
`-`, `+` or a space, jq parsed the leading `-` as an option and died on "Unknown
option -u", the payload came back empty, and the hook went silent -- turning
`assert_absent` GREEN for entirely the wrong reason. Lines now reach jq on stdin.
Second, the positive assertion needle `UNRESOLVED_SKILL: /alpha:ghost` is a
substring of the `/alpha:ghost-old` line it exists to exclude; it now carries its
right boundary instead of relying on the paired absence assertion.

NOT verified: a live PostToolUse payload carrying `structuredPatch` was not
observed -- no hook-event capture existed on this machine to read. Schema is
documented and corroborated by real Edit records in Claude Code's transcript
JSONL. If the field never arrives, the filter never engages and nothing regresses.

Suite: 106 pass, 0 fail.

Closes #2129

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…owed

Review finding on #2153, raised by Codex and confirmed independently by Claude,
and reached from the other end by this PR's adversarial verifier. All three are
right. This is a false negative I INTRODUCED: the gate made the guard quieter
than origin/main in a case where origin/main correctly spoke, which is worse than
the false positive the PR set out to remove.

Gate 3 required the on-disk physical line to equal a `+` line verbatim. An
earlier-ordered PostToolUse hook that reflows whitespace leaves the anchor
locatable -- a literal substring search does not care what surrounds it -- while
changing the line, so the witness missed and a genuinely written reference was
dropped. That undid the very tolerance the per-line fallback three lines above
exists to provide. Reachable in this marketplace, not theoretical: markdown-format
runs `markdownlint-cli2 --fix` and typos-format runs `typos --write-changes`,
both PostToolUse with matcher Write|Edit -- the same event and matcher as this
hook -- and matching hooks run in parallel, so it is an ordering-dependent race.

Two corrections:

1. Comparison is whitespace-normalized (tabs to spaces, runs collapsed, ends
   trimmed) on both sides. Whitespace is what formatters move; a DIFFERENT
   reference's line differs by far more than spacing, so discrimination holds.

2. An abstain rule, which is the half that generalizes. If the witness recognizes
   NO occurrence at all it is stale rather than discriminating, and the unfiltered
   set stands. Gate 3 may now only ever REMOVE an occurrence when it can
   positively identify at least one the call wrote, so a formatter that rewrote
   more than spacing degrades to exactly the pre-gate behaviour instead of muting.

I did NOT take the suggested substring-containment remedy. It fails on this
hook's own fixture: the anchor is the bare word `ghost`, which occurs inside the
`+` line AND inside the untouched `/alpha:ghost-old` line, so containment is true
for both occurrences and the gate stops discriminating -- trading the false
negative straight back for the false positive.

Also hardened, from the verifier's latent-fragility note: the jq read is now
shape-tolerant (`objects`). A non-object `tool_response` previously errored
hook::jq_fields into its `|| exit 0` and silenced the WHOLE guard, a risk the
pre-gate code did not carry because it never touched the field. Verified across
string, number, null, absent, and empty-patch shapes.

Supply hedge RETIRED. The verifier captured a live PostToolUse payload (temp
settings.json dumping stdin, headless claude 2.1.225): `tool_response` arrives as
an OBJECT carrying filePath newString oldString originalFile replaceAll
structuredPatch userModified -- an object, not the serialized string
PostToolBatch passes, which is the one shape that would have broken the filter.
structuredPatch was complete, not truncated, at 42 replacement sites in a
300-line file.

Measured, this PR's head before and after:

  case                                            pre  post
  formatter reflowed the written line             no   yes
  unreformatted: untouched ref stays suppressed   yes  yes
  two identical lines, one written                yes  yes
  liveness: empty target yields nothing           no   no

Row 2 is the point: the fix is not "turn Gate 3 off".

Scope now stated rather than implied: under replace_all, a genuine reference on a
line the patch reports as CONTEXT is not reported. That is the gate working as
designed, but it is broader than "fixes one false positive" and belongs in the
disclosure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cursor
cursor Bot force-pushed the fix/skill-reference-verify-replace-all branch from d227a5f to d0be16b Compare August 11, 2026 23:41
MD032/MD022 -- the conflict resolution joined this branch's section directly to
main's `## [0.24.1]` heading with no blank line between them. Caught by the
hygiene lane, reproduced locally with markdownlint-cli2, and both guardrails
CHANGELOGs now lint clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-authored-by: Kyle Sexton <kyle-sexton@users.noreply.github.com>
@cursor
cursor Bot force-pushed the fix/skill-reference-verify-replace-all branch from d0be16b to 1bf70c4 Compare August 11, 2026 23:42
… resolution

Co-authored-by: Kyle Sexton <kyle-sexton@users.noreply.github.com>
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown

Warning

Automated security review did not complete — this is an infrastructure failure, not a review verdict.

Treat any Claude comment on this PR (including a placeholder like "I'll analyze this and get back to you") as incomplete, not "no findings."

The check is green on purpose, and it is not evidence. It certifies that a security pass ran, and this one did not complete — but the cause is outside this PR's control, so merging is deliberately left unblocked rather than locking every merge for the length of the outage. Nothing was reviewed at this head. Where this check is required, it is satisfied without that evidence; a human should review security-sensitive changes here before merging.

Re-run the job to retry the review; a new push also retries it only if the caller's pull_request triggers include synchronize (the canonical security caller keeps it). An automatic retry may already have run — it is skipped when a partial review could duplicate comments, or when the failure class needs an operator.

Re-running does NOT help for every class:

  • rate-limit that persists across re-runs, or auth — the credential or usage budget needs an operator; retrying will not clear it.
  • a run that exhausted its turn budget ("subtype":"error_max_turns" above) will exhaust it again. As the PR author, split the change into smaller PRs; raising --max-turns is a change to the caller workflow, not something you can set on this PR.

Co-authored-by: Kyle Sexton <kyle-sexton@users.noreply.github.com>
@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown

Warning

Automated review did not complete — this is an infrastructure failure, not a review verdict.

Treat any Claude comment on this PR (including a placeholder like "I'll analyze this and get back to you") as incomplete, not "no findings."

Re-run the job to retry the review. A new push re-triggers this lane only if the caller's pull_request triggers include synchronize (the canonical caller omits it).
An automatic retry may already have run — it is skipped when a partial review could duplicate comments, or when the failure class needs an operator (auth).

Co-authored-by: Kyle Sexton <kyle-sexton@users.noreply.github.com>
@kyle-sexton
kyle-sexton merged commit 56f5cd2 into main Aug 12, 2026
34 checks passed
@kyle-sexton
kyle-sexton deleted the fix/skill-reference-verify-replace-all branch August 12, 2026 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(guardrails): skill-reference-verify reports an untouched reference under replace_all: true

2 participants