ci(scripts): flag a bare & in a ${var//pat/repl} replacement - #2097
Conversation
Since bash 5.2 an UNQUOTED `&` in the replacement half of a pattern substitution expands to the text the pattern just matched — the `sed` rule, under the `patsub_replacement` option that is on by default. Before 5.2 the same character was an ordinary literal. macOS ships bash 3.2 and every runner here ships 5.2+, so this lands on the exact uncovered-platform axis the shell-portability gate exists for, reached through a bash version rather than through a utility dialect. It has already shipped a defect in this repo: #2008 fixed a sentinel restored to itself in plugins/guardrails/hooks/block-hook-bypass.sh, where the restore was a silent no-op on bash >=5.2 and produced a live false positive (`echo x >&2` blocked as a file write). The class is implemented in the script rather than as a token ERE because it cannot be one: matching runs on views in which every separator inside a masked run — `&` among them — has been neutralized, and a `${…}` body is masked whole, so a pattern has nothing to match. Activation stays data all the same: a token line beginning with `!` names a script-implemented class, an unrecognized one fails the run closed, and a class-scoped fixture enables exactly this class the way an ERE fixture enables a pattern. Grammar measured against bash 5.3.15, not recalled: - the pattern ends at the FIRST unquoted, unescaped `/`, so a `/` after it is replacement text (`v=aXbXc; "${v//X/Y/Z}"` -> aY/ZbY/Zc); - a quoted or backslash-escaped `/` in the pattern is NOT the separator, while a `[...]` bracket expression does not protect one; - `\&`, `"&"` and `'&'` are each a literal ampersand and none is flagged; `\&` is what the failure message recommends, because the quoted spellings carry their own pre-4.3 quote-removal divergence; - `${var//pat}` (deletion) has no replacement to flag, and an expansion whose operator is not `/` is not a substitution at all. Known limit, in the under-flag direction: the rule is applied after the replacement expands, so an `&` arriving through a variable or through `$(...)` output is a live match reference this static check cannot see — the same indirection class the gate already declares out of scope. Proof on real history: the rule flags the pre-fix file at 2b60bf0, line 439, and nothing else in it. An `--all` sweep of all 418 in-scope shell files finds ZERO occurrences of the class on current main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Claude finished @kyle-sexton's task in 5m 53s —— View job Review complete
SummaryThis adds a script-implemented detection class ( I manually traced the core parsing ( Finding posted inline ( Everything else — the fail-closed handling of an unrecognized |
|
Claude finished @kyle-sexton's task in 1m 32s —— View job Security review
No security issues found. This PR is scoped entirely to Notes from the review, none of which rose to a reportable finding:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8546355c4a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Claude has reviewed this PR 3 times. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
The new paragraph sat in the unconditional failure epilogue, so every `grep -P` or unsuffixed `sed -i` failure was followed by advice about an ampersand the developer never wrote. It is now printed only when a reported hit belongs to !subst-replacement-ampersand, pinned in both directions by a paired case — present on an & failure, absent from an unrelated one — the same present/absent pairing the inertness case uses. Also narrows the `\&` justification to what was actually measured. BASH_COMPAT is NOT a pre-5.2 oracle for this rule: a bare `&` still expanded at every level down to 32 on bash 5.3.15, so the option is not compat-gated. What the ladder does establish is that the backslash before an `&` is removed even under the pre-4.3 quote-removal regime; the manual supplies the rest. The earlier wording read as pre-5.2 coverage it never had. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 2m 3s —— View job Security review
No CRITICAL or IMPORTANT security issues found. One low-severity observation below. This PR is scoped entirely to
Everything else — the fail-closed handling of an unrecognized |
|
Claude finished @kyle-sexton's task in 4m 32s —— View job Review complete
SummaryThis is the second commit on this PR ( The previously-flagged footer bug is fixed. My earlier review on this PR flagged that the Two Codex findings from the prior commit are unaddressed — I traced both directly against the current
Both are in the over-flag direction (false positive), the opposite of the "Known limits" section's stated invariant that the class never produces a false pass — so if confirmed, either merits a fix before this ships as a hard-error gate, since there'd be no escape short of a per-site No other issues. The rest of the diff — |
Two review findings, both reproduced against real bash 5.3.15 before
being accepted.
OVER-flag: a process substitution in either half was walked into rather
than skipped, so `${v//X/<(cmd1 && cmd2)}` was reported. Its body is a
command list — measured, `v=aXb; "${v//X/<(a && b)}"` yields
`a/dev/fd/63b` with patsub_replacement BOTH on and off, so there is zero
version divergence and the hard error red-lined portable code. `<(` and
`>(` now open a frame; skip_frame already handled the shape, so the fix
is the opener test, factored into opens_frame() and shared by both
halves.
UNDER-flag: `#` straight after `${` was taken to be the length operator
unconditionally. It is the length operator only while what follows could
start a parameter name; a `/` cannot, so `${#//2/&}` is a substitution
on the positional-argument COUNT. Measured with two positional
parameters: `2` with patsub_replacement on, `&` with it off — the exact
divergence this class exists for, reported clean. This was a false PASS
on a literal ${var//pat/&}, which the limits section claimed could not
happen; the claim is true again now rather than being relaxed.
`${#}`, `${##}`, `${#v}` and `${#arr[@]}` keep a non-`/` successor and
stay length expansions.
Also rewrites a comment garbled by an earlier edit, and puts the
report_hit/next_hit doc block back over the function it describes.
Ten new cases, including both halves of the frame skip proving a real
hit AFTER a skipped process substitution is still found.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 1m 43s —— View job Security review complete
No security issues found. This is a re-review triggered by the third commit, I re-traced the fix itself for any new security-relevant surface:
No workflow files, permissions, or secret-handling code are touched by this commit (unchanged from prior rounds — still scoped to |
|
Claude finished @kyle-sexton's task in 5m 7s —— View job Review complete
SummaryThis PR has already been through several review rounds on the earlier commits ( 1. 2. Both fixes are precisely scoped to the two open findings, correctly reasoned against the bash grammar as I can verify it statically, and each has dedicated test coverage in both directions (hits and non-hits). No new issues introduced by this commit that I could find on inline trace. One thing I checked and ruled out: a local No |
No linked issue
Summary
Adds one class to the shell-portability gate: an unquoted
&in the replacementhalf of
${var/pat/repl}/${var//pat/repl}.Since bash 5.2 that
&expands to the text the pattern just matched — thesedrule — under the
patsub_replacementshell option, which is on by default.Before 5.2 the same character was an ordinary literal. The construct is accepted
on both sides and silently means something different on each, with no error
either way.
This is not a new axis for the gate. Its stated exposure is macOS — the one
platform no runner here covers — and macOS ships bash 3.2 while every runner in
this repo ships 5.2 or later. Same shape as the existing
mktemp -pclass, whosetoken comment already describes a silent precedence divergence rather than an
absent flag, and it is a hard error for the same reason that one is (there is
no warning channel; a hit is exit 1, escapable per site with
portability-ok: <reason>).Repro
Verified on bash 5.3.15:
Historical-detection proof
The class shipped a real defect in this repo, fixed by #2008. Run the new rule
against the pre-fix file:
32add0fais the pre-fix parent of #2008's fix onmain, so this reproduces foranyone. One hit, at the exact line #2008 fixed, and nothing else in that file. That
no-op restore produced a live guardrails false positive (
echo x >&2blocked asa file write) on bash >=5.2 only.
Repo sweep
--allaudit over every in-scope shell file, with only this class active:ZERO occurrences of the class on current
main(434 tracked*.sh, 418 inscope after the gate's existing
vendor/and cross-plugin-sync exclusions — nonew exclusion was added). Nothing to fix; no live bug found.
The full
--allrun with the shipped token list reports 14 hits, all from thepre-existing regex-escape classes (
\b/\s/\w/\S) in four files this changedoes not touch. Confirmed pre-existing by scanning those same four files with
origin/main's unmodified gate and token list in a throwaway tree — byte-identicaloutput.
--allis an audit mode; CI gates changed files only.The stale pre-fix copy under
.claude/worktrees/agent-ac8ee00680b8101e3/that asweep would legitimately hit does not exist in this worktree (it is untracked
in another session's worktree), so nothing was excluded for it, and nothing needed
to be: CI runs changed-file mode, where an untracked nested checkout can never
appear in a
git diff.Why it is script-implemented, not a token ERE
The gate keeps what is detected in
scripts/shell-portability-tokens.txt. Thisclass cannot live there as a pattern: matching runs on the
qline/clineviews,and
neutralize()replaces everySEPScharacter —&among them — inside amasked run, while a
${…}body is masked in its entirety. That is exactly rightfor every other class (a
;in an expansion body is data, not an operator) andleaves this one nothing to match on.
Activation stays data anyway: a token line beginning with
!names a class thescript implements in code, it runs only while that line is active, an
unrecognized
!namefails the run closed, and a class-scoped unit fixtureenables exactly this class the way
one_token_listdoes for an ERE. The extent ofeach
${…}comes frommask_quotes()— the one existing authority on quote andframe structure — rather than from a second tracker written beside it.
Grammar, measured not recalled
Every expectation below was probed against bash 5.3.15 before it was encoded.
/, not the last:v=aXbXc; "${v//X/Y/Z}"yieldsaY/ZbY/Zc, so the pattern isXand thereplacement is
Y/Z. (The task brief said last; that is measurably wrong, anda last-slash reading would miss the
&in${v//X/b&/c}— there is a test forexactly that line.)
/in the pattern is not the separator(
s=a/b; "${s//"/"/-}"and"${s//\//-}"both yielda-b), while a[...]bracket expression does not protect one (
"${p//[/]/-}"leavesa/buntouched).
\&,"&"and'&'are each a literal ampersand — the manual's "Quoting anypart of string inhibits replacement in the expansion of the quoted portion" —
and none of them is flagged. The failure message steers to
\&, and it isworth separating measurement from inference there:
BASH_COMPATis not apre-5.2 oracle for this rule — a bare
&still expanded at every level down to32 on 5.3.15, so the option is not compat-gated. What the ladder does establish
is that the backslash before an
&is removed even under the pre-4.3quote-removal regime (tested at 32/42/44/50/51 and the default); the manual
supplies the rest ("the backslash is removed in order to permit a literal
'&'"). The quoted spellings are the ones with a version quirk of their own
(compat42: "The replacement string in double-quoted pattern substitution does
not undergo quote removal, as it does in versions after bash-4.2"), which
leaves the quote characters in the output on the older regime — a reason to
prefer
\&, not a reason to flag them.${var//pat}(deletion) and an empty replacement have nothing to flag; anexpansion whose operator is not
/(${v:-a/b/&},${v#*/},${v%/*},${v:0:1},${#v}) is not a substitution at all.&outside any substitution —a && b,cmd &,2>&1,echo "a & b"— isnever flagged.
Sources: GNU Bash Reference Manual, Shell Parameter
Expansion
("Any unquoted instances of '&' in string are replaced with the matching portion
of pattern"; "Backslash escapes '&' in string; the backslash is removed in order
to permit a literal '&' in the replacement string") and Shell Compatibility
Mode;
bash NEWS records
patsub_replacementas new in bash-5.2.Known limits (stated, not overclaimed)
&that arrives by expansion is undetectable statically. The rule isapplied after the replacement expands, so an
&held in a variable, or in theoutput of a
$(...)inside the replacement, is a live match reference. Verified:for
v=aXb, a replacement of$(printf 'p&q')yieldsapXqb. Same indirectionclass the gate already declares out of scope (ci: harden shell-portability-lint detection precision (sed -i spellings, portability-scope precision, awk operand edge case) #1513).
&inside a nested$(...)/backquote in the replacement is skipped,because there it is ordinary command syntax (
&&, backgrounding). Its runtimeoutput is limit 1.
classes carry.
identifier, a digit run, or one of
@ * ? $ ! -) is skipped rather than guessed.All four are in the under-flag direction; none produces a false pass on a
literal
${var//pat/&}.Review round (both findings real, both reproduced before accepting)
Review caught that the limits list was incomplete in both directions. Both
are now fixed, not documented away, so the sentence above is true again.
Over-flag — process substitution.
<(/>(was not treated as a nestedframe, so
${v//X/<(cmd1 && cmd2)}was reported. Measured on 5.3.15:Identical — zero version divergence, so a hard error was red-lining portable
code.
<(/>(now open a frame (skip_framealready handled the shape; theopener test is factored into
opens_frame()and shared by both halves).Under-flag —
$#as the special parameter.#after${was taken to bethe length operator unconditionally. It is the length operator only while what
follows could start a parameter name;
/cannot, so${#//2/&}is asubstitution on the positional-argument count. Measured with two positional
parameters:
A genuine false pass on the literal shape. Fixed;
${#},${##},${#v}and${#arr[@]}keep a non-/successor and stay length expansions.Ten cases added, including both halves of the frame skip proving a real hit
after a skipped process substitution is still found.
Tests
New section in
scripts/check-shell-portability.test.shcovering: the #2008 defectshape (asserted with the exact
PORTABILITY: file:line:prefix, so a silentlyinert rule cannot pass), the first-slash grammar, escaped/quoted/nested-expansion
forms mixed with a bare one, quoted and escaped slashes in the pattern, array and
positional and anchored parameter spellings, the correct
\&form, both quotedforms, the deletion and empty-replacement forms, non-substitution operators,
&outside any substitution, all three excuse mechanisms (same-line
portability-ok:, the comment block above, whole-fileportability-scope:),per-physical-line attribution inside a quote-joined record, class inertness when
the
!nameline is absent, a directive-only token list not tripping thefail-closed empty-pattern check, an unrecognised
!namefailing closed, and theclass remediation paragraph appearing on an
&failure while staying off anunrelated one.
shell-portability-linton CI: PASS=312 FAIL=0 (268 before this change), andthe changed-file gate reports the two touched shell files clean. The job is
ubuntu-24.04only, so this class — like every other class in this gate — has noWindows CI coverage.
Related
proof runs against its pre-fix parent
2b60bf0b.${…}frame tracking this class reuses.