Skip to content

fix(core): match one sed character for . and bracket expressions - #15701

Open
2SinceJuly wants to merge 13 commits into
QwenLM:mainfrom
2SinceJuly:fix/15675-sed-regex-flags
Open

2SinceJuly wants to merge 13 commits into
QwenLM:mainfrom
2SinceJuly:fix/15675-sed-regex-flags

Conversation

@2SinceJuly

@2SinceJuly 2SinceJuly commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

The shell tool's in-process sed simulation now compiles substitution patterns with the s and u flags. This makes . match a carriage return and makes character classes and quantifiers operate on Unicode characters instead of UTF-16 code units. A literal ] outside a bracket expression is escaped during translation so the stricter Unicode mode still accepts patterns such as [\[\]].

Why it's needed

On CRLF files, s/=.*$/=new/ matched nothing and the tool reported that the edit made no changes. On files containing astral characters, s/^.// removed one half of an emoji and wrote the lone surrogate back as U+FFFD, silently corrupting the file. Real sed treats both cases as one character.

Reviewer Test Plan

How to verify

Run the sed parser unit tests. The new regression case covers the six rows from the issue: astral characters with ., a negated class, a zero-width x* match, and a CRLF substitution. Before the change it fails on the first astral row; after the change all cases pass. Also confirm that ordinary bracket patterns such as [abc], [^abc], and [\[\]] still take the simulation path rather than falling back.

Evidence (Before & After)

N/A for UI. Unit tests: 53 passed, including new pins for negated, mixed and equivalence-class POSIX forms ([^[:space:]], [[:alpha:]abc], [[=a=]]) and for the GNU/BSD multibyte-backreference split. The full core suite passed 39,351 tests; the single bun-pty.test.ts failure also reproduces on unmodified main. Typecheck, lint, and formatting pass.

Tested on

OS Status
🍏 macOS ✅
🪟 Windows ⚠️
🐧 Linux ✅

Linux was verified in a Docker node:22-bookworm arm64 container.

Environment (optional)

Node 22.23.4; unit tests only.

Risk & Scope

  • Main risk or tradeoff: the u flag can reject patterns that previously compiled, so those commands now fall back to real sed. That is the existing fail-safe path and never writes a wrong file.
  • Not validated / out of scope: Windows; GNU sed's zero-width x* iteration and multibyte backreferences are bytewise, while BSD sed and this simulation are characterwise. The POSIX-class guard now declines any pattern containing [:, [. or [=, so mixed and negated character classes also fall back to real sed instead of being simulated wrongly.
  • Breaking changes / migration notes: none.

Linked Issues

Closes #15675

中文说明

这个 PR 做了什么

shell 工具的进程内 sed 模拟现在会用 s 和 u 标志编译替换模式。这样 . 能匹配回车符,字符类和量词也按 Unicode 字符而不是 UTF-16 code unit 工作。翻译时会把字符类外的字面量 ] 转义,保证更严格的 Unicode 模式仍能接受 [\[\]] 这类模式。

为什么需要

在 CRLF 文件上,s/=.*$/=new/ 匹配不到任何内容,工具会报告没有产生修改。在包含星平面字符的文件上,s/^.// 只删掉 emoji 的一半,并把孤立代理项写成 U+FFFD,静默损坏文件。真实 sed 在两种情况下都按一个字符处理。

评审测试计划

如何验证

运行 sed 解析器单测。新增回归用例覆盖 issue 里的六行:星平面字符配合 .、否定字符类、零宽 x* 匹配,以及 CRLF 替换。修改前第一行星平面用例失败,修改后全部通过。同时确认 [abc]、[^abc]、[\[\]] 这类普通字符类仍然走模拟路径,而不是回退。

证据(修改前后)

UI 变化:N/A。单测 53 通过,新增取反/混合/等价类 POSIX 形态([^[:space:]]、[[:alpha:]abc]、[[=a=]])以及 GNU/BSD 多字节反向引用分歧的钉子。core 全量 39,351 通过;唯一失败的 bun-pty.test.ts 在未修改的 main 上也能复现。类型检查、lint、格式化均通过。

测试环境

系统 状态
🍏 macOS ✅
🪟 Windows ⚠️
🐧 Linux ✅

Linux 在 Docker node:22-bookworm arm64 容器中验证。

运行环境(可选)

Node 22.23.4;仅单测。

风险与范围

  • 主要风险/取舍:u 标志会拒绝一些以前能编译的模式,这些命令会回退到真实 sed。这是既有的 fail-safe 路径,不会写错文件。
  • 未验证/超出范围:Windows;GNU sed 的零宽 x* 与多字节反向引用按字节推进,而 BSD sed 和本模拟按字符推进。POSIX 字符类 guard 现在对含 [:、[.、[= 的模式一律回退,混合与取反写法也不再被错误模拟。
  • 破坏性变更/迁移说明:无。

关联 issue

Closes #15675

The simulated sed path compiled patterns without the s and u flags, so `.`
did not match `\r` in CRLF files and astral characters were matched as two
UTF-16 code units, leaving a lone surrogate that was written as U+FFFD.

Compile with su/gsu and escape a literal ] outside bracket expressions so
the stricter u mode still accepts patterns such as [\[\]].

Closes QwenLM#15675
@wenshao

wenshao commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Verdict: merge-ready — 77/77 scripted assertions passed at head 45edd5b72e405f90465c1b337146b2b9c388359e. Against real GNU sed 4.5, the base build simulates 16 of 59 comparable cells wrongly; the PR head simulates 0 wrongly. No regressions, no blocking findings. Three Suggestions below are non-blocking.

中文摘要
  • 结论:merge-ready(77/77 断言通过,head 45edd5b7)。
  • A/B(用真 GNU sed 4.5 做 oracle,59 格可比):base 有 16 格模拟结果与真 sed 不一致,head 0 格。CRLF 下 .、emoji(星芒字符)按一个字符、字面 ] 三类全部翻转为一致;base 在没有守恒的单元格上尖峰集中在 A/B/E 组(见下表)。
  • 变异矩阵:7 杀 2 活。新测试确实钉住核心改动(把 gsu 退回 g,新测试立即红),] 转义由既有 -E 括号测试钉住。两个 survivor 都不是缺陷:M1(gate 不带 su)是会真实抛错的覆盖缺口,M8(gate 只带 u)在 59 格上与 head 逐字节相同,说明 gate 里的 s 不可观测。
  • Findings(均为 Suggestion,不阻塞):S1 GNU/BSD 在「零宽全局替换 + emoji」上仍有分歧(head 选 BSD,GNU 给 -<byte>-…);S2 gate 与 apply 的 flag 一致性没有测试钉住;S3 gate 里的 s 标志是死代码。
  • 未覆盖:整包 core 套件、typecheck、lint 未跑(只跑了改动文件套件 + shell 的 sed 子集);没有跑完整 CLI 工具调用;本机无 BSD sed,BSD 那一格用的是 PR 自己测试声明的期望值;Windows/macOS 未测。

A/B against real GNU sed

Central claim

The simulated sed -i fast path (which writes files itself instead of running sed) must treat ., bracket expressions and astral characters the way sed does: . matches a carriage return, one astral character is one character, and a literal ] outside a bracket expression stays literal now that the translated pattern is compiled with the strict u flag.

The oracle here is not hand-written expectations — it is the real sed binary. For every cell the harness builds the same command, runs it through parseSedEditCommand and applySedSubstitution in each arm (the exact two calls ShellTool makes), and runs the identical expression through GNU sed -i on a real temp file. A cell is unsafe when the simulation accepts a pattern and writes bytes sed would not; a declined pattern is safe because production then runs the real sed.

A/B result

group cells base wrong head wrong
A — . must match \r (CRLF) 6 5 0
B — astral character is one character 10 8 0
C — literal ] outside a class 11 0 0
D — regression corpus (previously supported forms) 28 0 0
E — command spellings (-i -E, --in-place, double quotes) 4 3 0
total 59 16 0

The cells are byte comparisons, not visual checks. Examples from the run (base → real sed):

cell input base wrote real sed wrote
sed -i 's/=.*$/=new/' key=old\r\n unchanged (.* cannot cross \r) key=new\n
sed -i 's/^.//' 😀 Title\n lone low surrogate + Title (U+FFFD on disk) Title\n
sed -i 's/[^a-z]/_/g' a😀b\n a__b (two surrogates) a_b
sed -i -E 's/^.//' (also -i -E, --in-place) 😀x\n \uDE00x x

Both arms were bundled separately from their own worktree; the base bundle contains no gsu and the head bundle contains exactly one, so the arms really did load different code.

Mutation matrix

Mutation matrix

mutant change suite outcome
C1 (control) replaceAll forced false head 52 killed (15 failures)
M2 apply gsu → g (full revert of the central change) head 52 killed by the new test (matches one character for . and bracket expressions)
M3 apply gsu → gu (drop s) head 52 killed by the new test
M4 apply gsu → gs (drop u) head 52 killed by the new test
M5 remove the literal-] escaping head 52 killed by keeps backslashes literal inside bracket expressions under -E (existing test)
M6 POSIX-class check back on jsPattern head 52 killed by rejects POSIX bracket expressions before simulated edits
M7 M5 + M6 together head 52 killed by the same -E bracket test
M1 gate new RegExp(jsPattern) without flags head 52 survived → coverage gap (S2)
M8 gate new RegExp(jsPattern, 'u') (drop s only) head 52 survived → dead flag (S3)

The new test is not vacuous: reverting the whole apply-side change (M2) turns it red with the assertion it was written for, and each half of the bundled s/u change is pinned separately (M3, M4). M6 shows the hasPosixBracketExpression(sedInfo.pattern) change is load-bearing rather than cosmetic: that mutant compiles the check against the translated pattern, accepts s/[[:digit:]]/X/g, and writes a1b unchanged where real sed writes aXb — a silent wrong edit (the translated pattern no longer contains the literal [[:digit:]] once the trailing ] is escaped). The existing rejects POSIX bracket expressions before simulated edits test is what catches it.

Survivors were adjudicated by building the mutant, not by reading it:

  • M1 — its build accepts s/\q/X/g and s/a\-b/X/g (identity escapes that u mode rejects) and then throws Invalid escape inside applySedSubstitution. With auto-approve there is no preview pass, so that surfaces to the user as a tool error instead of falling back to real sed. Head is consistent (su gate / gsu apply) and declines both, so this is a missing test, not a defect.
  • M8 — byte-identical to head on all 59 cells. The s flag cannot change whether a pattern compiles, so only the u in that gate is observable.

Findings

S1 — a zero-width global match over an astral character follows BSD, not GNU. sed -i 's/x*/-/g' on 😀\n gives -😀- at head (the new test pins this and the comment documents it as BSD behaviour), while GNU sed 4.5 on this host gives -\xf0-\x9f-\x98-\x80-, i.e. - between each byte. This is not a regression — base wrote -<lone high surrogate>-<lone low surrogate>-, which lands on disk as U+FFFD — and the divergence is deliberate. It is worth an explicit acceptance note because on Linux the simulation still is not "what sed would do" for this shape. Reproduce: node harness.mjs base-tree/sed-bundle.mjs head-tree/sed-bundle.mjs, cell 16. No fix proposed; declining this shape so real sed runs would also be defensible, but that is a product decision.

S2 — nothing pins the gate's flags to the substitution's flags (M1). A one-line test would close it: expect(parseSedEditCommand("sed -i 's/\\q/X/' f.txt")).toBeNull(). Today the two are written independently, and a future edit to either can reintroduce the throw-with-no-fallback path M1 demonstrates.

S3 — the s in the gate's new RegExp(jsPattern, 'su') is unobservable (M8). Not harmful, and arguably kept for symmetry with gsu; noted only so a future reader does not attribute meaning to it.

Observation (not a finding): head declines two patterns base simulated — s/\q/X/g and s/a\-b/X/g — because those escapes are invalid under u. Both fall back to real sed, so they are correct but leave the fast path (51 accepted cells at head vs 52 at base). No output drift was measured on any cell that base simulated correctly.

Corrections

None. There was no earlier verification round or bot review on this PR describing the code, so there is nothing to correct; the two author claims I tested (the ] escaping and the CRLF/astral behaviour) both hold.

Not covered

  • Full packages/core suite, npm run typecheck, and lint were not run. Only the changed file's suite (head 52/52, base 51/51) and the 31 sed-related tests in shell.test.ts (31 passed | 343 skipped) ran.
  • No end-to-end CLI tool invocation. The harness calls the same two functions ShellTool.prepareSedEdit calls, but the preview/approval plumbing was read, not driven. The auto-approve consequence in S2 is inferred from shell.ts, while the throw itself is measured.
  • BSD sed is not installed here. The BSD cell uses the expected value the PR's own test asserts, not a live BSD sed; GNU 4.5 under en_SG.UTF-8 is the only oracle that ran.
  • Windows/macOS, and the interaction with -i.bak/combined flags, are untouched by the diff and were not exercised.
  • Real CI was still in flight (precheck passed, other jobs pending/skipping) when the evidence was captured.

Methodology

Two detached worktrees, one at baseRefOid 7b09c0a9 and one at headRefOid 45edd5b7. Each tree's sedEditParser.ts was bundled with esbuild into a self-contained ESM module, so no workspace symlink can leak one arm's code into the other; the base bundle has zero gsu occurrences and the head bundle has one, as a positive control that the arms differ. The harness (harness.mjs) runs 61 cases, of which 59 are comparable (two are excluded because real sed itself rejects the expression, e.g. the empty pattern). Mutation rows run the PR's own test file via the JSON reporter and attribute every kill to a named failing test; the tree is restored with git checkout after each row and the restore is verified by sha256. Tests were run through a minimal vitest config because a bare worktree lacks the built workspace dist/ the package's global setup requires. Environment: Node v24.22.1, GNU sed 4.5, locale en_SG.UTF-8. Raw logs, bundles, and harnesses live in tmp/pr15701-verify-20261027-153859/.

wenshao
wenshao previously approved these changes Oct 27, 2026
@wenshao

wenshao commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

@wenshao
wenshao enabled auto-merge (squash) October 27, 2026 09:22
@2SinceJuly

2SinceJuly commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

Head is f51590bb — ci: rerun the checks on the unchanged tree — same tree e00b2c97 as the previously reviewed 45edd5b7, so no code change. It was pushed only because the earlier runs on that head ended cancelled and the required checks never completed. The fresh Qwen Code CI run is action_required (GitHub is holding the fork's runs), so it needs a maintainer approval before it starts.

@qwen-code-review-bot qwen-code-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not explored to full depth (tool budget reached): "agent 6b": packages/core/src/tools/shell.test.ts sed-integration tests were not run — the file needs packages/core/dist/index.js , and this worktree is unbuilt; the uni…; "agent 6c": did not run the full packages/core suite (only the parser file plus targeted greps of shell.test.ts ), and did not run the mutation experiment by deleting th….

Test Plan (not a blocker): tests: 52 passed — this review observed 18646, 2035, 38776, 2278, 306, 13338 passed.

中文说明

未探索到全部深度(达到工具调用预算):"agent 6b":packages/core/src/tools/shell.test.ts sed-integration tests were not run — the file needs packages/core/dist/index.js , and this worktree is unbuilt; the uni…;"agent 6c":did not run the full packages/core suite (only the parser file plus targeted greps of shell.test.ts ), and did not run the mutation experiment by deleting th…。

Test Plan(非阻断):tests: 52 passed — this review observed 18646, 2035, 38776, 2278, 306, 13338 passed。

— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.2)

@wenshao

wenshao commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Verdict: blocked — 289/297 scripted assertions passed at head f51590bb9e1c95ebc7e76d754baf39ce1b4c096b (base 9df10f873b016f2db6b32cfb1a42823bcd8020d1). The 8 failures are one reproduced regression, R1-1: with -E, the new u flag turns sed-literal escapes into JavaScript Unicode metacharacters, and the simulation writes bytes that neither real GNU sed 4.10 nor real BSD sed writes (4 cells × 2 oracles). The central CRLF/astral fix itself is confirmed load-bearing: all six issue #15675 rows flip from base-wrong to head-match against both real seds. The author has acknowledged R1-1 and promised a fix commit; none has landed — the head tree is byte-identical to the one reviewed in round 1.

中文摘要
  • 结论:blocked(297 条脚本断言,289 通过,8 失败;head f51590bb)。8 条失败全部是同一个已复现的回归 R1-1(4 格 × GNU/BSD 双 oracle),归因 PR。
  • 中心修复成立:issue sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed #15675 的六行在 base 上全部与真 sed 不一致,head 上全部一致(GNU 4.10 与 macOS BSD sed 双 oracle,en_US.UTF-8)。A/B 见 01-sed-oracle-ab.png。
  • R1-1(Critical,阻塞):sed -E -i 's/\p{L}/X/g' 这类模式在 head 被模拟成 JS Unicode 属性转义,把每个字母都替换掉(ab1 p{L} A → XX1 X{X} X);真 GNU sed 直接报错退出、BSD sed 按字面 p{L} 替换。\P{L}、\u{41}、\p{Lu} 同类。base 对 BSD 全部正确,故为 PR 引入的回归。字节证据见 02-r11-regression-bytes.png。作者已确认并承诺修复,但截至本轮 head 未推送。
  • 建议修复已实测:把 hasSedJavascriptDivergentEscape 扩为 [<>dDwWsSpPu] 后,5 个 hostile 格全部回退真 sed,59 个非 F 格逐字节零附带损伤,套件 54/54 绿;新增断言在 head 上变红(M11),证明断言钉得住修复。
  • 变异矩阵:12 行全部符合预期(10 杀 2 活),r1 的 S2/S3 缺口复现;R1-2 的两条验收断言能杀掉 M1(M9)。见 03-mutation-matrix.png。
  • 后续轮分类:新 head 是空 CI 重跑提交,树哈希与 r1 完全相同(e00b2c97,含阳性对照)。r1 的 merge-ready 结论被 bot 的 Critical 推翻——r1 语料未覆盖 u-only 接受类。
  • 姊妹 PR:fix(core): match sed text semantics in unicode mode #15694 与本 PR 是同一 issue 的竞争修复,trial merge 直接冲突(需维护者二选一);fix(core): keep sed anchors inside groups and alternations #14428 干净合并且合并树套件 60/60 绿。
  • 未覆盖:core 全量/typecheck/lint(CI 仍 queued);shell.test.ts sed 子集本轮环境缺新依赖无法收集(A/A 证明 base 同样失败,承接 r1 的 31/31);Windows;完整 CLI 工具调用。

Follow-up round: previous-finding status

Round 1 (2026-10-27, head 45edd5b7, tree e00b2c97) reported merge-ready, 77/77. This round's head f51590bb is an empty CI-rerun commit: both heads resolve to tree e00b2c970a49b23d4a673e4960f6a29df9b7f10b, git diff 45edd5b7 f51590bb is empty, and the comparator's positive control (git diff 9df10f87 f51590bb --stat) reports the real 2-file diff. Every carried-forward measurement below was nevertheless re-run on this machine (Node 22.23.1, GNU sed 4.10 + BSD sed, en_US.UTF-8) because the round-1 rig differed (Node 24, GNU sed 4.5 only).

Correction to round 1 (my own earlier report): its merge-ready verdict is superseded. The r1 corpus (59 cells, groups A–E) did not include the u-only acceptance class — patterns that compile under u with a different meaning than sed's. The bot review of 2026-10-28 12:50 UTC found exactly that class (R1-1), and this round reproduces it with byte evidence. r1's "0 wrong of 59" remains true for those 59 cells; the verdict was wrong because the corpus was incomplete, not because any r1 cell was miscounted.

# finding severity status at f51590bb
R1-1 (bot, 2026-10-28) u flag simulates \p{L} / \P{L} / \u{41} as JS Unicode metacharacters under -E; wrong bytes written Critical stands — reproduced on both real seds (F1/F2/F3/F8); author acknowledged 2026-10-28 13:03, fix promised, not pushed
R1-2 (bot, 2026-10-28) nothing pins the gate's u-only rejection class Suggestion stands — M1 survives 52/52; the two proposed decline assertions kill it (M9)
S1 (r1) zero-width global + astral follows BSD, not GNU Suggestion stands — B5 re-measured: head -😀- = BSD; GNU 4.10 inserts - between each byte; deliberate, test-pinned
S2 (r1) gate/apply flag consistency untested Suggestion stands — same gap as R1-2, re-measured (M1)
S3 (r1) the s in the gate's 'su' is unobservable Suggestion stands — M8 survives; gate 'u'-only bundle byte-identical to head on all 68 cells

Central claim + A/B

The simulated sed -i fast path must treat ., bracket expressions and astral characters the way real sed does. The oracle is the real binary, not hand-written expectations: each cell runs parseSedEditCommand + applySedSubstitution (the exact two calls ShellTool.prepareSedEdit makes) in each arm, and the identical expression through GNU sed -i and BSD sed -i on a real temp file, comparing bytes. Arms are esbuild-bundled in isolation; positive control: the base bundle contains zero gsu occurrences, the head bundle exactly one.

group cells base wrong head wrong
A — . must match \r (CRLF) 6 5 0
B — astral character is one character 10 8 0
C — literal ] outside a class 11 0 0
D — regression corpus 28 0 0
E — command spellings (-i -E, --in-place, double quotes, -Ei) 4 4 0
F — u-flag acceptance class (R1-1 + siblings) 9 2 5
total 68 19 5

"Wrong" counts a cell against both oracles unless the cell is marked as deliberately following one (B5 follows BSD; A6/D4/D12–D14 follow GNU — all pre-existing choices, identical in both arms). Of the 5 head-wrong cells, F1/F2/F3/F8 are the PR regression and F5 is pre-existing (both arms wrong). All six issue rows flip base-wrong → head-match on both oracles. The corpus is a reconstruction: r1's harness no longer exists locally, so groups A–E were rebuilt from r1's group descriptions, the issue's six rows verbatim, and existing-test shapes (r1's A/B counts 5/6 and 8/10 reproduce exactly; E is 4/4 here vs r1's 3/4 because this round's fourth spelling is -Ei).

A/B against real GNU and BSD sed

R1-1 regression (Critical, blocking)

Under -E/-r, toJavascriptPattern copies \p{L}, \P{L}, \u{41} through verbatim; none of the four guards matches them (hasSedJavascriptDivergentEscape covers only \<>dDwWsS); the gate's new RegExp(jsPattern, 'su') compiles them as Unicode property / codepoint escapes; applySedSubstitution then writes the simulated text, which prepareSedEdit hands to writeTextFile. There is no fallback because nothing throws.

Measured bytes (input ab1 p{L} A\n = 61623120707b4c7d20410a):

cell base sim head sim GNU sed 4.10 BSD sed
F1 s/\p{L}/X/g ab1 X A (literal p{L}) XX1 X{X} X exit 1 Invalid content of \{\}, file unchanged ab1 X A
F2 s/\P{L}/X/g unchanged abXXpXLXXA exit 1, unchanged unchanged
F3 s/\u{41}/X/g unchanged (u{41} = 41 u's, as in sed) rewrites A→X exit 0, unchanged unchanged
F8 s/\p{Lu}/X/g on AB c unchanged XX c exit 1, unchanged unchanged

Against BSD (the issue's own platform) every row is a clean match→wrong regression. Against GNU, F1 was already wrong at base (base simulated a pattern GNU rejects — pre-existing), and head changes the wrong bytes to a worse shape. BRE is safe: the translator emits \p\{L\}, which u rejects, so the command declines and real sed runs (F4 measured). V8 probe confirms the mechanism: non-u \u{41} compiles as the quantifier u{41} (sed's reading), u-mode \u{41} is the codepoint A.

Blast radius: every -E/-r simulated sed -i whose pattern contains \p{…}, \P{…} or \u{…} — the simulation silently writes wrong content with no error and no fallback. \u0041-style 4-hex escapes (F5) are wrong in both arms (pre-existing; JS hex escape vs sed literal) and are covered by the same proposed fix.

Reproduce: node tmp/pr15701-verify-20261029-020933/harness.mjs <base-bundle> <head-bundle> out.json, cells F1–F3, F8; raw bytes in harness-results.json and the capture below.

R1-1 regression bytes

The proposed fix is measured, not eyeballed. Widening the guard to /\\[<>dDwWsSpPu]/u (the bot's suggestion, which the author accepted) in a scratch copy:

  1. hostile fixtures clean: F1/F2/F3/F5/F8 all decline on both oracles → real sed runs;
  2. benign fixtures byte-identical: 0 of 59 non-F cells change sim output or classification. One documented over-decline: F9 [\p{L}] (class-member \p, previously simulated correctly) now declines too — safe, costs only the fast path, consistent with the module's stated policy;
  3. suite: 54/54 green with the R1-1/R1-2 decline assertions added (M10); removing the widening while keeping the assertions reds exactly the R1-1 acceptance test (M11), so the assertions pin the fix.

The fix should ship with those assertions; the author's promised commit has not landed at this head.

Mutation matrix

12 rows, head's 52-test suite via JSON reporter, fresh scratch copy per row, every anchor asserted unique, kills attributed to named failing tests. Runner positive control: C1 (replaceAll forced false) kills with 15 failures; unmutated baseline 52/52.

row change outcome
C1 control: replaceAll forced false killed (15 failures)
M2 apply gsu→g (central revert) killed by the new test
M3 apply gsu→gu (drop s) killed by the new test (CRLF row)
M4 apply gsu→gs (drop u) killed by the new test (astral row)
M5 remove literal-] escaping killed by keeps backslashes literal inside bracket expressions under -E
M6 POSIX check back on jsPattern killed by rejects POSIX bracket expressions before simulated edits
M7 M5+M6 combined killed by the same -E test
M1 gate without flags survived → coverage gap (S2/R1-2)
M8 gate 'u' only survived → dead s flag (S3); byte-identical to head on all 68 cells (measured this round)
M9 M1 + R1-2's two decline assertions killed by the R1-2 acceptance test
M10 widened guard + R1-1/R1-2 assertions green 54/54 (the measured fix)
M11 head source + R1-1/R1-2 assertions killed by the R1-1 acceptance test (assertions are non-vacuous)

The new test is not vacuous: transplanted onto base source it fails exactly once, on its first astral assertion (expected ' Title\n' to be ' Title\n', U+FFFD vs space) — the author's "fails on the first astral row" claim holds verbatim.

Mutation matrix

Trial merge and sibling PRs

Findings

  1. R1-1 — Critical, blocking. Reproduced above; author acknowledged, fix promised but not pushed. The PR's own risk note ("the u flag can reject patterns that previously compiled, so those commands now fall back to real sed") covers only the reject direction; the accept-with-different-meaning direction is the regression. Suggested fix and its acceptance assertions are measured in the previous section.
  2. R1-2 / S2 — Suggestion. Gate/apply flag consistency is unpinned (M1 survives). The two toBeNull() assertions the author agreed to add kill M1 (M9 measured). Ship them with the R1-1 fix.
  3. S1 — Suggestion (carried, re-measured). Zero-width global + astral follows BSD (-😀-), GNU 4.10 gives - between each byte. Deliberate and test-pinned; on Linux the simulation still is not "what sed would do" for this shape. Acceptance note, not a blocker.
  4. S3 — Suggestion (carried, re-measured). The s in the gate's 'su' is unobservable (M8 byte-identical on 68 cells). Kept for symmetry; noted so no future reader attributes meaning to it.
  5. Pre-existing observations (not PR regressions, both arms identical). F5 \u0041 under -E: JS hex escape vs sed literal — wrong in both arms, covered by the proposed widened guard. D4 2g flag and D12–D14 BRE \+ \? \|: the simulation follows GNU semantics; macOS BSD sed rejects 2g outright and reads those BRE escapes literally. A6 \r: GNU extension, both arms match GNU. These are the module's existing GNU-leaning choices, unchanged by this PR.

Not covered

  • Full packages/core suite, typecheck, lint. The author claims 39,351 passed with one bun-pty.test.ts failure that also reproduces on main; the re-run CI (Qwen Code CI 40619669842) was still queued (Test/Lint jobs queued) at capture time, so the claim is unverified this round.
  • shell.test.ts sed-integration subset. Attempted on a head worktree with cloned node_modules; collection fails on @modelcontextprotocol/client — a dependency main added after this machine's install (head's packages/core/package.json requires @modelcontextprotocol/client@2.0.0; only sdk@1.29.0 is installed). A/A attribution: base's mcp-client.ts imports the same package and the PR touches no manifest, so both arms fail collection identically — environmental. Round 1's 31/31 on the identical tree is carried forward with that proof, not re-measured here.
  • No end-to-end CLI tool invocation. The harness drives the same two functions prepareSedEdit calls; the write path (writeTextFile) is read, not driven. R1-1's file-write consequence is inferred from that reading plus the measured sim bytes.
  • Windows (author marks it untested), BSD sed on Linux, -i.bak/backup-suffix interactions, and the CRLF line-ending restoration layer in shell.ts (the issue notes it restores \r\n on write; that layer is untouched by this diff and unexercised here).
  • Per-commit attribution beyond the aggregate: the PR has one production commit (45edd5b7) plus an empty CI-rerun commit (f51590bb, tree-identical, proven); the mutation matrix is per-guard over that one production commit's hunks.

Methodology

Local maintainer round on macOS (Node v22.23.1, GNU sed 4.10 at /opt/homebrew/bin/gsed, BSD sed at /usr/bin/sed, locale en_US.UTF-8 for both oracles). Two arm sources exported via git show at head f51590bb and merge-base 9df10f87 (the PR's parent; sedEditParser.ts/shell.ts are byte-identical across merge-base, baseRefOid 7b09c0a9, and current main, verified by empty git log over those paths), each bundled with esbuild into a self-contained ESM module so no workspace link can leak one arm into the other (gsu census: base 0, head 1). The 68-cell harness compares sim bytes against real sed -i bytes per oracle; divergent-oracle cells are scored against the oracle the simulation deliberately follows, and the delta is reported. Unit gates run through a minimal vitest config in scratch dirs (the package globalSetup requires built workspace dist); the mutation runner uses a fresh copy per row with unique-anchor assertions and JSON-reporter tallies that exclude pending tests. Trial merges run in a scratch worktree at main's tip, local-only, never pushed. Guard snapshots taken at round start and re-checked before posting. Raw logs, bundles, harnesses, ledgers: tmp/pr15701-verify-20261029-020933/ (harness-results.json, fix-results.json, mutations-results.json, assertion-ledger.json, test-*/results.json, tree-identity.txt, sibling-*.log).


Verification round 2 by @wenshao's local /verify-pr rig, 2026-10-29. Advisory evidence for human reviewers; this round does not approve, merge, or close anything.

@qwen-code-review-bot qwen-code-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test Plan (not a blocker): tests: 52 passed — this review observed 19060, 2067, 39336, 2340, 306, 13620 passed.

Deferred under the convergence posture (round 2, not a blocker) — recorded, not requested in this round:

  • packages/core/src/utils/sedEditParser.test.ts:264 (+1 locations) — [review] the new s/x*/-/g assertion pins a result GNU sed does not produce for zero-width global matches over an astral character
中文说明

Test Plan(非阻断):tests: 52 passed — this review observed 19060, 2067, 39336, 2340, 306, 13620 passed。

收敛姿态下延后(第 2 轮,非阻断)——已记录,本轮不要求修改:共 1 条(原文未翻译,列表见上方英文部分)。

— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.3)

@2SinceJuly

Copy link
Copy Markdown
Contributor Author

Two Criticals at 3be40510, both valid. Note the head moved only by a merge of main; the R1-1 fix I agreed to earlier has not landed, so R1-1 reproduces exactly as described.

R1-1 (still standing). Confirmed on 3be40510: hasSedJavascriptDivergentEscape is still /\\[<>dDwWsS]/u, so \p{…}, \P{…} and \u{…} pass every guard under -E, new RegExp(jsPattern, 'su') compiles them as JavaScript Unicode property / code-point escapes, and the simulated text is what writeTextFile receives. The u flag stays — your ^. / .$ astral cases depend on it. Fix is the narrow decline: widen the class to [<>dDwWsSpPu], and add toBeNull() cases beside rejects sed escapes that diverge in JavaScript regexes for -E 's/\p{L}/X/g', -E 's/\P{L}/X/g', -E 's/\u{41}/X/g'; deleting p/P/u from the class must red those three.

R1-2. Taking the decline assertions as written, a{ included, so removing u from the matcher reds the suite instead of flipping the pattern to "accepted, then Invalid escape at apply time".

R2-1 (new, accepted). Confirmed at shell.ts:2068: the gate compares the parser's string before writeTextFile normalises LF to CRLF for lineEnding: 'crlf'. abc\r\n + s/.$// therefore produces abc\n, passes the gate, is rewritten to abc\r\n, and reports sed edit applied with the bytes unchanged. I will compare what will actually be written — normalise the new content the way the write path does, and fall through to the existing made no changes branch when the result equals originalContent — without changing the CRLF write policy, plus the abc\r\n + _meta: { lineEnding: 'crlf' } case asserting sed edit made no changes that reds if the comparison is deleted.

Both fixes and their assertions go in one commit on top of 3be40510, no rebase.

肖洪进 added 2 commits October 29, 2026 09:24
`\p{…}`, `\P{…}` and `\u{…}` are literals to sed but Unicode property and
code-point escapes to a `u`-flagged regex, so the simulator accepted patterns
real sed rejects or leaves alone and wrote the simulated bytes to disk. Widen
the divergent-escape guard and pin the declines, plus the `u`-only rejection
class the flag introduced.
On a CRLF file, `s/.$//` consumes the trailing carriage return, but the write
path restores CRLF, so the edit reported success while the bytes on disk were
unchanged. Compare the prepared write content against the original instead of
the parser's intermediate string.
auto-merge was automatically disabled October 29, 2026 01:25

Head branch was pushed to by a user without write access

@2SinceJuly

Copy link
Copy Markdown
Contributor Author

All three are fixed.

R1-1 — ec8cb95ce. hasSedJavascriptDivergentEscape now covers \p, \P and \u (/\\[<>dDwWsSpPu]/u), so -E 's/\p{L}/X/g', -E 's/\P{L}/X/g' and -E 's/\u{41}/X/g' decline and real sed runs. The u flag stays; the decline is the fix. Three toBeNull() cases sit beside rejects sed escapes that diverge in JavaScript regexes, and removing p/P/u from the class reds them (verified).

R1-2 — same commit. Added declines u-only invalid escapes so the real sed runs with s/a\-b/X/g, s/a\ b/X/g and -E 's/a{/X/g'.

R2-1 — 28b5d7a37. prepareSedEdit now normalises the new content through the same prepareTextFileContentAsync path the write uses and compares that against the original, so a CRLF edit whose only effect is the trailing \r falls through to sed edit made no changes. The write policy is untouched: s/=.*$/=new/ still writes key=new\r\nk2=v\r\n. New case does not report a CRLF edit whose only effect is the carriage return (abc\r\n + _meta: { lineEnding: 'crlf' }, s/.$//) reds when the comparison is removed (verified).

Evidence: sedEditParser.test.ts 53/53, shell.test.ts 375/375, both on 28b5d7a37. Two commits on top of 3be40510, no rebase.

@qwen-code-review-bot qwen-code-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not explored to full depth (tool budget reached): "agent 6c": I did not run the two touched test files ( npx vitest run for shell.test.ts / sedEditParser.test.ts ), so the new assertions are reviewed by inspection only…; "agent 3a": none — no check was cut short..

Convergence: round 3 posted 4 inline comment(s), 4 of them reported for the first time; the previous round posted 3 (1 new). Findings keep coming back to the same files: packages/core/src/utils/sedEditParser.ts (findings in rounds 1, 2; 1 more now). The rate of new findings is not falling. A cluster that keeps producing siblings usually means the fixes are treating instances of a shared root cause — triaging that cause before the next round, or splitting an independent cluster into its own pull request, tends to end the loop faster than fixing them one at a time. Batching the remaining fixes and verifying them before the next push, or dropping this PR's reviews to --severity-floor critical, keeps the loop from re-deriving the same set. (Observation only — nothing was withheld from this review because of this observation.)

中文说明

未探索到全部深度(达到工具调用预算):"agent 6c":I did not run the two touched test files ( npx vitest run for shell.test.ts / sedEditParser.test.ts ), so the new assertions are reviewed by inspection only…;"agent 3a":none — no check was cut short.。

收敛情况:第 3 轮发布了 4 条行内评论,其中 4 条是首次提出;上一轮发布了 3 条(其中 1 条首次提出)。发现反复回到同一批文件:packages/core/src/utils/sedEditParser.ts(第 1、2 轮已出过发现,本轮又有 1 条)。新发现的产出速度没有下降。一个不断再生兄弟发现的簇,通常意味着逐条修复只在处理同一根因的实例——先定位并处理该根因,或把独立的簇拆成单独的 PR,通常比逐条修复更快结束循环。把剩余修复攒成一批、验证后再推送,或将本 PR 的评审降到 --severity-floor critical,可以避免循环反复推导同一组发现。(仅为观察——本轮评审未因此扣留任何内容。)

— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.3)

Comment thread packages/core/src/tools/shell.ts Resolved
Comment thread packages/core/src/tools/shell.ts Resolved
Comment thread packages/core/src/tools/shell.test.ts Resolved
@wenshao

wenshao commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Verdict: findings — 62/62 scripted assertions passed at head 28b5d7a374ffda1fb4a2384fb37aaee57ec6c9da (base 35fd1f44ff58b29d4e9eaf1b6f1e5472b6d3de20). R1-1, R2-1 and R1-2 are fixed and each is pinned by its own witness; the central CRLF/astral fix remains load-bearing. Two blocking problems stand at this head: on BOM files the new byte guard still certifies sed edit applied while the bytes on disk are unchanged (R3-1), and \0 is still simulated as a NUL where sed reads a literal 0 (R3-3). One further consequence of the guard is stated with its bound below: it now reports "made no changes" for CRLF→LF edits that neither build ever wrote (N-1).

中文摘要
  • 结论:findings(62/62 脚本断言通过;head 28b5d7a3)。阻塞项不是断言失败,而是本轮实测出的行为问题,见 Findings 一节。
  • A/B(真 GNU sed 4.5 为 oracle,三臂:base 35fd1f44 / 第二轮 head f51590bb / head 28b5d7a3):. 匹配 \r(issue sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed #15675)与星芒字符按一个字符在 base 全错、head 全对;R2-1 的 \p{L}/\u{41} 在第二轮 head 被改写成错误内容,head 已改为回退真 sed。见 01-ab-three-arms.png 与 Central claim 表。
  • 上一轮三条全部修复并钉住:R1-1(变 M2/M4/M5 全杀)、R2-1(M3/M7/M8 全杀)、R1-2(M6 杀掉——第二轮同一变异是存活)。
  • 本轮新发现:
    • R3-1(Critical,复现):带 BOM 的 CRLF 文件仍报 sed edit applied,磁盘字节未变。机器人建议改为比较 encodeTextFileContentAsync 的两侧字节,实测可修,且 10 个端点格零附带变化。
    • R3-3(Suggestion,复现,双向都错):\0 在 JS 里是 NUL、在 sed 里是字面 0;加进字符类后三个用例全部回退真 sed。
    • N-1(关于写入路径的边界说明):s/\r$// 在 CRLF 文件上,base 与 head 都没有真正改变磁盘字节(行尾还原层把模拟器产出的 LF 又变回 CRLF),但 head 现在如实说“没有变化”,base 之前会谎报“已应用”。我起初把它写成“head 丢弃了真实编辑”,实测后已改正。
    • R3-2 / R3-4(Suggestion):写入服务不还原行尾时(ACP/IDE)同样不写入;作者测试里没有“CRLF 正向写入”用例,套件对这个轴不设防。
  • 一处更正(我自己的第二轮报告):第二轮“补丁在 59 格零附带损伤”建立在 |sim| 与 |sed| 锁步的前提上,本轮实测该前提不成立,见 Correction 一节。
  • 未覆盖:Windows、BSD sed(本轮只有 GNU 4.5)、真实 ACP 会话(只用按原样写入的服务替身)、\0 之外的兄弟形状、core 全量套件(作者声称 39,351 通过,该 head 的 CI 为绿)。

A/B: base, round-2 head and head against GNU sed

Previous-finding status

Round 2 (2026-10-28, head f51590bb, tree e00b2c97) reported blocked. This head adds two production commits on top of that tree. Nothing was carried forward by assumption — every row was re-run in this round (Node 24.22.1, GNU sed 4.5, this checkout).

# finding severity status at 28b5d7a3
R1-1 -E turns sed's \p{…} / \P{…} / \u{…} into JS Unicode escapes and writes wrong bytes Critical fixed — F1/F2/F3/F8 now match sed; M2/M4/M5 killed
R2-1 CRLF edit whose only effect is the \r reports "applied" with unchanged bytes Critical fixed — M3/M7/M8 killed; the round-2 head is kept as the failing control
R1-2 gate's u-only rejection class unpinned Suggestion fixed — M6 killed (round 2 measured the same edit as a survivor)
S1 (r1) zero-width global over an astral character follows BSD, not GNU Suggestion stands — cell B5 identical at all three heads
S2/S3 (r1) gate/apply flag consistency; s in the gate unobservable Suggestion superseded — dropping u from the gate now reds the suite
R3-1 (bot, 2026-10-29) BOM / non-UTF-8 file: the guard decodes prepared bytes, never fires, false "applied" Critical stands — reproduced; the bot's own fix is measured below
R3-2 (bot) a write service that does not restore line endings (ACP/IDE) gets an edit skipped Suggestion stands — reproduced
R3-3 (bot) \0 is NUL to JS, a literal 0 to sed Suggestion stands — reproduced in both directions
R3-4 (bot) no positive-write CRLF test: the guard's writing direction is unpinned Suggestion stands — an added verbatim-writer test fails; candidate fixes leave the suite green

The bot's three corrected rulings (R1-1, R1-2, R2-1 "fixed by …") are accurate: each is confirmed by that fix's own witness going red when the fix is reverted.

Central claim + A/B

Claim unchanged: the simulated sed -i fast path must treat ., bracket expressions and astral characters the way sed does. Oracle: the real binary, never hand-written expectations. Each arm is its own bundle of sedEditParser.ts / tools/shell.ts, executed in a working copy holding only that arm's originals. The arm census is green with a negative control (arm-census.txt): per-bundle banner, exactly one parser and one shell copy, "no other arm's source present", and a runtime reach proof (\p{L}, \u{41}, ^.) that differs per arm.

cell base 35fd1f44 round-2 head f51590bb head 28b5d7a3 claim
A1 s/=.*$/=new/ on CRLF wrong (key=old… untouched) matches sed matches sed . spans the \r (issue #15675)
B1 s/^.// on 😀1 wrong (lone surrogate) matches sed matches sed one astral character is one sed character
F1 -E s/\p{L}/X/g wrong (ab1 X A) wrong (XX1 X{X} X) declines → real sed R2-1
F3 -E s/\u{41}/X/g matches wrong (XB) declines → real sed R2-1
L1 s/.$// on abc\r\n (pattern space) wrong matches sed matches sed dotAll + per-line processing
L3 s/\r$// on a\r\nb\r\n (pattern space) matches matches matches a greedy sticky tail must not cross \r
shell S1 s/.$// (verified R2-1) "no changes" "applied", bytes unchanged "no changes" the endpoint
shell S2 s/=.*$/=new/ on CRLF "no changes" applied key=new\nk2=new\n applied key=new\nk2=new\n issue #15675 end to end
shell S7 s/\r$// on CRLF "applied" (bytes unchanged) "applied" (bytes unchanged) "no changes" (bytes unchanged) N-1

Derived from those rows: head differs from base on four parser cells (A1, B1, F1, L1), all in the fixed direction; the shell matrix moves S2 into the fixed column and changes S7's report only.

Mutation matrix

Runner: the author's suites through a throwaway copy of the head tree, baseline green (53/53, 375/375) before any row, each anchor asserted unique, each revert replayed as that commit's production diff. The coverage claim is per-commit: git log --no-merges f51590bb..28b5d7a3 --not 4196cf81d1 over the two production files lists exactly ec8cb95c and 28b5d7a3, and both have rows (mutations.json).

row commit change outcome
M2 ec8cb95 whole commit reverted killed — rejects sed escapes that diverge in JavaScript regexes
M4 / M5 ec8cb95 drop u / drop p from the class killed — same witness
M6 ec8cb95 gate compiled 's' instead of 'su' killed — declines u-only invalid escapes so the real sed runs
M3 28b5d7a whole commit reverted killed — does not report a CRLF edit whose only effect is the carriage return
M7 / M8 28b5d7a gate disabled / gate compares pre-write text killed — same witness
B0 — baseline green
M1 — control: one assertion falsified red as expected (vitest names the describe block, not the it() title)

Both survivors were adjudicated by running their builds. Removing s from applySedSubstitution is caught by this round's corpus (A1, B5), not by the author's suite. The bracket-expression backslash escape (toJavascriptPattern) shows no behavioural difference: its variant scored 32/33, the same as head, declining on the backslash cells instead of simulating them — redundant defence today, pinned by nothing.

Mutation matrix

Correction to round 2 (my own earlier report)

Round 2 said the widened guard left "0 of 59 non-F cells" with collateral damage, reasoning from |sim| against |sed| treated as lockstep. That premise does not hold: this product restores LF→CRLF on write when the read reported lineEnding: 'crlf', while GNU sed on the same file drops the \r terminator (probe lockstep-noproduct: a plain rewrite round-trips abc\r\n). Round 2 was not miscounting its cells; it was measuring a model the write path does not implement. I also corrected one of my own claims mid-round: an earlier draft of this report said the guard "suppresses real CRLF edits". Measured on disk (probe/disk-crlf.mjs), s/\r$// never changed the bytes at base either — the write path turns the simulated abc\n back into abc\r\n. What the guard changes is the report: base claimed sed edit applied, head says made no changes, and both leave the file as it was. The pre-existing write-path behaviour is documented here, not blamed on this PR; only the BOM case below is a newly false success.

Findings

R3-1 — Critical, blocking, reproduced. On a UTF-8-BOM CRLF file the read returns BOM-stripped abc\r\n; prepareTextFileContent's BOM branch returns a Buffer with no encoding; the guard decodes it as UTF-8 ("\uFEFFabc\r\n"), the comparison never matches, and the tool reports sed edit applied while the bytes on disk are unchanged (probe r31-shell: head report="sed edit applied", bytesUnchanged=true). At the merge base . could not match \r, so the same input answered made no changes — the false success is new to this line of changes. The bot's suggested fix — compare encodeTextFileContentAsync(newContent) with encodeTextFileContentAsync(content) — was applied in a scratch copy and measured: the same probe reports sed edit made no changes, and all 10 endpoint cells are byte-identical to head (shell-fix_combo.json, shared library filesystem.test.ts 34/34). It should ship with a bom: true fixture beside the new CRLF test.

R3-3 — Suggestion, reproduced in both directions. sed -i 's/\0/X/' reads a literal 0 to sed (a0b → aXb); head's simulation treats \0 as U+0000 (a0b unchanged, a NUL byte rewritten). Widening the class to [<>0dDwWsSpPu] makes all three cells decline to real sed (probe r33). Same class as R1-1 one character over — the bot's point that the fixes have been treating instances of one root cause is fair.

R3-2 — Suggestion, reproduced. The guard assumes StandardFileSystemService's line-ending preparation; AcpFileSystemService.writeTextFile forwards content verbatim (packages/cli/src/acp-integration/service/filesystem.ts:214), so in an IDE session the guard's decision is about bytes the service never writes. Same root cause as R3-1: the decision belongs to the service that writes.

R3-4 — Suggestion. Every positive-write assertion in the simulated sed edit block uses an LF fixture; the only CRLF fixture asserts a skip. Candidate fixes (endpoint, BOM, \0) applied in scratch leave the suite at 428/428 with nothing changing colour — the axis has no fixture, and I would add the one above.

N-1 — note on the write path (pre-existing, not a regression of this PR). A CRLF→LF conversion simulated in-process never reaches the file: the write path restores CRLF (s/\r$// on abc\r\n leaves abc\r\n at base, at the round-2 head and at head — probe/disk-crlf.mjs). Head's report is now truthful ("made no changes"); base's was not ("applied"). This is the line-ending preservation issue #15675 explicitly leaves in place, so I record it as a known limitation rather than a defect to fix here — R3-4's positive fixture is the guard against making it worse.

Pre-existing, unchanged at every arm: B5 zero-width global over an astral character follows BSD (round 1's S1, still standing).

Not covered

  • Windows (author marks it untested) and BSD sed — this rig has GNU sed 4.5 only, so every "matches sed" verdict is GNU-only.
  • A real ACP session. R3-2 is proven by a verbatim-writing FileSystemService stand-in following the ACP service's own code contract; no ACP connection was driven.
  • \0 siblings outside the four probe cells (e.g. \cX, \xHH agreement) were not enumerated.
  • The full packages/core suite (author: 39,351 passed, one bun-pty.test.ts failure also on main). The PR's own CI at this head is green for Test (ubuntu), Lint & Static, integration and web-shell smoke; macOS/Windows lanes were skipped on this run.
  • -i.bak, backup suffixes, symlinks, non-UTF-8 encodings beyond the BOM case were not exercised.

Methodology

Local maintainer round: three arm sources exported with git show at 35fd1f44, f51590bb and 28b5d7a3, each bundled on its own (parser and shell separately, @qwen-code/qwen-code-core external and shimmed) and run inside a working copy holding only that arm's originals plus a hardlinked node_modules. Parser cells compare applySedSubstitution with real sed -i output; endpoint cells drive ShellTool.getConfirmationDetails → execute with the configuration surface lifted verbatim from the author's shell.test.ts; the disk probe uses the real StandardFileSystemService. Unit gates run the head tree's own vitest config with only the fresh-clone dist guard relaxed. Everything is scripted under probe/ (ab-witness.mjs, parser-matrix.mjs, shell-harness.mjs, disk-crlf.mjs, r31-shell.mjs, r32.mjs, r33.mjs, run-mutations.mjs, ledger.sh); results in assertion-ledger.json, assertions.json, mutations.json (mutations-raw.json holds the runner's own tally), parser-*.json, shell-*.json.


Verification round 3 by @wenshao's local /verify-pr rig, 2026-10-29, at head 28b5d7a3. Advisory evidence for human reviewers; this round does not approve, merge, or close anything.

The write-back prediction decoded the prepared bytes as UTF-8 and compared
them with the read content, so a file carrying a BOM or a non-UTF-8 encoding
never matched and the false `sed edit applied` this PR removes survived.
Compare content strings instead, and only predict the CRLF normalization when
the writer applies it: a client-side write stores the content verbatim, so
`s/\r$//` is a real edit there and must not be suppressed.
`\0` is NUL to a `u`-flagged regex but a literal `0` to sed, so the simulator
found no match and answered `sed edit made no changes` instead of falling back
to the real binary.
@2SinceJuly

Copy link
Copy Markdown
Contributor Author

Round 3 is addressed in two commits on top of 28b5d7a3 (no rebase): b951ab0c3 and 9fc09d69b.

R3-1 (Critical — BOM / non-UTF-8 made the comparison false). The guard no longer decodes the prepared bytes. It compares content strings against the content the read returned, and applies the CRLF normalization only when the active writer applies it, so a BOM or a non-UTF-8 encoding can no longer make an unchanged file look changed.

R3-2 (a client-side write never restores CRLF). Added restoresLineEndingsOnWrite to FileSystemService: StandardFileSystemService reports true, and AcpFileSystemService reports !capabilities.writeTextFile, i.e. only the local-fallback path normalizes. When the write goes to the client the predicted content is left unnormalized, so sed -i 's/\r$//' is applied instead of being suppressed — s/.$// on a CRLF file is still reported as no change under the local writer.

R3-3 (\0). 0 is now in hasSedJavascriptDivergentEscape, so s/\0/X/ falls back to the real binary instead of simulating a miss. Verified here that BSD sed turns a0b into aXb while a u-flagged regex only matches NUL.

R3-4 (only the skip branch was pinned). shell.test.ts now also covers a CRLF edit that does change bytes (foo\r\n → bar\r\n) and the verbatim-writer case; the BOM skip case is pinned next to the existing one.

packages/core shell and sedEditParser suites: 431 passed. packages/cli ACP filesystem suite: 34 passed. tsc --noEmit clean for core and cli.

Resolving the threads. @qwen-code /review

@wenshao

wenshao commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Verdict: findings — 36/39 scripted assertions passed at head 9fc09d69b5441fa8db0ac6808484db3503ea9d4d (base 35fd1f44ff58b29d4e9eaf1b6f1e5472b6d3de20). R3-1, R3-2 and R3-4 are fixed and pinned; R3-3 is fixed but unpinned. One blocking regression stands: the new restoresLineEndingsOnWrite declaration is false for the repository's own qwen serve client, whose write path re-applies CRLF, so sed -i 's/\r$//' on a CRLF file reports sed edit applied while the bytes on disk are unchanged (R4-1). The same runtime also keeps R3-1's BOM-shaped false success.

中文摘要
  • 结论:findings(36/39 脚本断言通过;head 9fc09d69b,base 35fd1f44)。
  • 上一轮四条:R3-1(BOM 假成功)、R3-2(逐字写入被跳过)、R3-4(CRLF 正向用例)在本地写入路径上均已修复并被测试钉住(M20/M21/M22 全部杀死)。R3-3(\0)行为已修复,但没有任何测试(M24 存活;修正后的对照变异 M25b 能杀死套件)。
  • 本轮新问题(R4-1,Critical,已复现):AcpFileSystemService.restoresLineEndingsOnWrite 取 !capabilities.writeTextFile,而 qwen serve 广告的是 writeTextFile: true,于是子进程按“逐字写入”预测;但真正的写入端——daemon 的 writeTextOverwrite——会用现有文件的 meta 重新做 CRLF 还原。用仓库真实的 ACP 服务 + 真实 daemon 写入类端到端驱动:s/\r$// 与 BOM 文件上的 s/.$// 都报 applied,磁盘字节不变。相对上一轮 head(r3 报 “no changes”)这是新引入的假成功;本地运行时不受影响。
  • A/B:真 GNU sed 4.10 为 oracle,base/r3/head 三臂;. 跨 \r、星芒单字符、\p/\u{41}/\0 回退均与 sed 一致;B5 零宽全局仍沿用 BSD 语义(既有 S1,未变)。
  • 未覆盖:Windows/BSD sed(只有 GNU 4.10)、真实 ACP 传输(进程内直调)、真正拉起的 serve 会话、-i.bak/符号链接/非 UTF-8(BOM 之外)、core 全量套件、head 的 ubuntu/lint/integration lane 当时仍在排队。

A/B: base, round-3 head and head against GNU sed 4.10

Previous-finding status

Round 3 (2026-10-29, head 28b5d7a3) reported findings. This head adds two production commits on top of that tree (b951ab0c, 9fc09d69). Every row was re-run in this round — nothing is carried forward by assumption.

# finding severity status at 9fc09d69
R1-1 -E \p{…} / \u{…} simulated as JS escapes Critical fixed — both decline; the guard's own tests and my F1/F3 cells confirm
R2-1 CRLF edit whose only effect is \r reports applied with unchanged bytes Critical fixed locally; re-opened in qwen serve — see R4-1
R1-2 gate's u-only rejection class unpinned Suggestion fixed — M25b kills the suite when p is dropped
R3-1 BOM / non-UTF-8 file: false "applied" Critical fixed locally (string comparison); stands in qwen serve (same root cause as R4-1)
R3-2 a writer that stores content verbatim gets a real edit skipped Suggestion fixed — the verbatim-writer cell applies and changes bytes
R3-3 \0 is NUL to JS, a literal 0 to sed Suggestion fixed behaviourally, unpinned — M24 survives; the commit ships no test
R3-4 no positive-write CRLF fixture: the writing direction is unpinned Suggestion fixed — M22 is killed by two tests, including the new positive-write one
S1 (r1) zero-width global over an astral character follows BSD Suggestion stands — B5 mismatch at all three arms
N-1 CRLF→LF never reaches disk under the local writer note unchanged — s/\r$// still leaves abc\r\n; the report is truthful

Central claim + A/B

Claim unchanged: the simulated sed -i fast path must treat ., bracket expressions and astral characters the way sed does. Oracle: the real binary (GNU sed 4.10), never hand-written expectations. Three arm worktrees at 35fd1f44 / 28b5d7a3 / 9fc09d69, each bundled from its own sources; parser cells call applySedSubstitution, endpoint cells drive the real ShellTool against real files.

cell base r3 head claim
A1 s/=.*$/=new/ on CRLF wrong matches sed matches sed . spans \r (issue #15675)
B1 s/^.// on 😀1 wrong (lone surrogate) matches matches one astral character is one sed character
F1 -E s/\p{L}/X/g wrong declines declines sed rejects it; fall back to the binary
ESC7 s/\0/X/g wrong (NUL) wrong (NUL) declines → aXb R3-3
L1 s/.$// on abc\r\n wrong matches matches dotAll + per-line processing
B5 zero-width global on 😀 mismatch mismatch mismatch S1, unchanged (BSD vs GNU)
E-R31 BOM CRLF s/.$// "no changes" applied, unchanged "no changes" R3-1 fixed locally
E-R32 verbatim s/\r$// applied, changed "no changes", skipped applied, changed R3-2 fixed
E-SERVE s/\r$// applied, unchanged "no changes" applied, unchanged R4-1
E-SERVE-BOM s/.$// "no changes" applied, unchanged applied, unchanged R3-1 stands in serve

The endpoint cells print in 01-ab-three-arms.png; the serve chain in 02-serve-chain.png

qwen serve chain: real ACP service + real daemon writer

Mutation matrix of the two new production commits.

Findings

R4-1 — Critical, blocking, reproduced. AcpFileSystemService declares restoresLineEndingsOnWrite = !capabilities.writeTextFile. qwen serve advertises { readTextFile: false, writeTextFile: true } (packages/acp-bridge/src/channel-handshake.ts), so in every served session the child predicts a verbatim write. The delegated write does not store content verbatim: the daemon's WorkspaceFileSystem.writeTextOverwrite re-reads the existing file's meta and encodes through encodeTextFileContentAsync, which re-applies CRLF (packages/cli/src/serve/fs/workspace-file-system.ts, mergeWriteMeta + atomicWriteTextResolvedFile). Measured on the real chain — real AcpFileSystemService, real writeTextOverwrite, only the ACP transport in-process: sed -i 's/\r$//' on abc\r\n → message sed edit applied, bytes 6162630d0a before and after. The BOM variant (EF BB BF abc\r\n, s/.$//) is the same: applied, bytes unchanged. A real edit (s/foo/bar/) still lands, so this is a false success, not a broken write. At the previous head the same two cells answered no changes (truthful); base answered applied for the non-BOM cell, so this is a re-introduced defect of R2-1's class rather than a pre-existing behaviour. The tool also records a file-history backup, commit attribution and a read-cache write for the edit, so the false success is not merely cosmetic.

Suggested fix: a capability flag says the client can write, not how it writes; the ACP service cannot distinguish a verbatim IDE from the in-repo daemon. Keep the conservative prediction (normalize) unless the writer proves verbatim — the r3 column of 02-serve-chain.png is that configuration, measured: both serve cells become truthful again, and the only cost is the hypothetical verbatim client's s/\r$// being skipped. A qwen serve fixture that drives the delegated write through writeTextOverwrite (as zz-pr15701-serve.test.ts does) should ship with the change; the author's unit tests use a mock whose property is set by hand, which is why M20/M21 stay green while the production client is mis-modelled.

R4-2 — Suggestion, measured. 9fc09d69 ("decline \0") changes only sedEditParser.ts; no test was added. Dropping 0 from the class (M24) leaves all 431 tests green, while the corrected control (dropping p, M25b) reds rejects sed escapes that diverge in JavaScript regexes in 5 ms — the suite is live and simply does not pin \0. Behaviour at head is correct (ESC7 declines; real sed writes aXb), so this is a coverage gap, not dead code. Add expect(parseSedEditCommand("sed -i 's/\\0/X/g' f.txt")).toBeNull() to the existing divergence test.

R4-3 — Suggestion, measured. Dropping needsCrlfLineEndings(filePath) from the predicate (M23) also leaves 431/431 green: the Windows .cmd axis has no fixture. This round cannot close it (macOS os.platform() makes the branch unreachable), so it is listed under Not covered as well.

Corrections

My first control mutation (M25) removed W from the class instead of p, so it survived and briefly looked like a dead suite. The corrected control (M25b) reds exactly the \p test; mutations.json carries only the corrected row. Also, round 3's oracle was GNU sed 4.5; this round's is 4.10, so \p{L} is rejected by the oracle with Invalid content of \{\} rather than being a no-op — the head's decline is correct under both.

Not covered

  • Windows and BSD sed — this rig has GNU sed 4.10 only (round 3 used 4.5); every "matches sed" verdict is GNU-specific. M23's Windows branch is unexercised.
  • A real ACP transport. The serve cells call the real AcpFileSystemService, real WorkspaceFileSystem and real encoder in-process; the AgentSideConnection object is a one-method stand-in. No daemon process was spawned.
  • Non-UTF-8 encodings beyond BOM, -i.bak, backup suffixes, symlinks.
  • The full packages/core suite (the author's two files: 431/431 at head; packages/cli ACP filesystem: 34/34). Head's Test (ubuntu), Lint & Static, integration and web-shell E2E lanes were still queued when this round captured (windows/macos lanes skipped), so no CI evidence is claimed beyond the plumbing lanes that had finished.

Methodology

Local maintainer round, three detached worktrees at base / round-3 head / head; each arm runs the same probe file (zz-pr15701-probe.test.ts) from its own packages/core/src/tools/, with parser cells comparing applySedSubstitution against /opt/homebrew/bin/gsed and endpoint cells driving the real ShellTool on real temp files (real StandardFileSystemService, plus declared stand-ins for the verbatim and daemon writers). A second probe (zz-pr15701-serve.test.ts) runs in the head arm's packages/cli with the real ACP service and the real daemon write class; core imports are aliased to packages/core/src through the package export map. Mutation rows mutate the head arm's production files, run the author's two core suites, and restore with git checkout HEAD --. Everything is scripted under probe/; results in assertions.json, mutations.json, probe/results-*.json, probe/mutations.json.


Verification round 4 by @wenshao's local /verify-pr rig, 2026-10-29, at head 9fc09d69b5. Advisory evidence for human reviewers; this round does not approve, merge, or close anything.

@2SinceJuly

Copy link
Copy Markdown
Contributor Author

Round 4 is addressed in 21d2c9dfb and 826c33325 on top of 9fc09d69b (no rebase).

R4-1 (Critical — a write capability cannot say how the client stores the content). I removed restoresLineEndingsOnWrite from FileSystemService rather than fixing its value in AcpFileSystemService: a capability bit says a client can write, not how it stores the bytes, and the in-repo qwen serve writer (WorkspaceFileSystem.writeTextOverwrite → mergeWriteMeta / encodeTextFileContentAsync) re-applies CRLF while a generic ACP host may not. prepareSedEdit now always predicts the normalization — the conservative direction you measured in 02-serve-chain.png: both serve cells become truthful again, and the cost is the hypothetical verbatim client's s/\r$// being skipped. That trade-off is recorded in the code comment, and the unit test that pinned the opposite model was deleted with the flag. R3-1's string comparison and R3-4's positive-write CRLF test are unchanged.

R4-2 (\0 fixed but unpinned). The divergence test now asserts parseSedEditCommand("sed -i 's/\\0/X/g' f.txt") is null. Mutation-checked: dropping 0 from the class (your M24) fails rejects sed escapes that diverge in JavaScript regexes; restoring it passes 53/53.

R4-3 (Windows .cmd axis). Deferred, not dismissed: needsCrlfLineEndings is unreachable where I can run tests (macOS os.platform()), and it stays listed as not covered in the PR. Per the repo's review-round guidance I would rather file this as a follow-up than grow this diff further — say the word and I will add a platform-mocked test here instead.

Verification at 826c33325: src/utils/sedEditParser.test.ts 53/53 and src/tools/shell.test.ts 377/377 in packages/core; ACP filesystem 34/34 in packages/cli; tsc --noEmit clean for both packages; eslint and prettier clean on the touched files.

中文说明

第四轮已在 21d2c9dfb、826c33325 上处理(未 rebase)。

  • R4-1(Critical):直接删掉了 FileSystemService.restoresLineEndingsOnWrite,而不是去修 AcpFileSystemService 的取值——写入能力位只说明客户端“能写”,无法说明它怎么存字节,而仓库内 qwen serve 的写入路径(writeTextOverwrite → mergeWriteMeta / encodeTextFileContentAsync)会重新套用 CRLF。现在预测一律按“会还原行尾”处理,即你 02-serve-chain.png 里测到的保守方向:serve 两条单元格恢复真实,代价是假设中“逐字写入”客户端的 s/\r$// 会被跳过,该取舍写在代码注释里。钉住相反模型的那条单测随字段一起删除;R3-1 的字符串比较与 R3-4 的正向 CRLF 用例保持不变。
  • R4-2:在 divergence 测试里补了 \0 断言,并做了变异验证:去掉 0(M24)会让 rejects sed escapes that diverge in JavaScript regexes 变红。
  • R4-3:暂缓而非忽略——本机(macOS)无法触达 needsCrlfLineEndings 的 Windows 分支,仍在 PR 的“未覆盖”里。按仓库“约 5 轮后只落 Critical 修复”的约定,我倾向另开 follow-up;如果你希望在本 PR 里补,我加一个按平台 mock 的测试。
  • 验证:core 53/53 + 377/377,cli ACP 34/34,两个包的 tsc --noEmit 干净,eslint / prettier 干净。

@qwen-code /review

@wenshao

wenshao commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator

Verdict: merge-ready — 215/215 scripted assertions passed at head df0288326fb0e1109561074b188e3ce7c9422559 (advanced mid-round from 826c333258abf992d5e7bf87119cf1cf84e26e81 by a mechanical main merge; base 35fd1f44ff58b29d4e9eaf1b6f1e5472b6d3de20). R4-1 is fixed and re-measured on the real qwen serve write chain: the two cells that reported sed edit applied while leaving the bytes unchanged at 9fc09d69 now report sed edit made no changes, and the r4 head is kept as the failing control. R4-2 is pinned by the new test; R4-3 stays deferred with its mutation surviving. The central fix still matches GNU sed on every corpus cell the base gets wrong, and the trial merge into current main (bc8ad53f95) is clean.

中文摘要
  • 结论:merge-ready(215/215 脚本断言通过;head df028832,base 35fd1f44)。
  • head 在本轮中途前进:从 826c3332 变为 df028832,内容是 merge main(bc8ad53f95)。其树 230e2949 与我本轮 trial merge 的树逐字节相同,且 PR 的六个文件在 826c3332/df028832 之间 git diff 为空——因此 A/B 与端点格的结果原样成立;合并树上的两套测试与端点探针又各重跑了一遍(430/430、7/7)。
  • R4-1 已修复并复测:在真实的 qwen serve 写入链上(真实 AcpFileSystemService + BridgeFileSystemAdapter + WorkspaceFileSystem.writeTextOverwrite),上一轮 head 9fc09d69 的 s/\r$// 与 BOM + s/.$// 两格仍是 “applied / 字节不变”;本轮 head 两格都变成 “made no changes”,本地写入与 serve 链两列重新一致。r4 head 作为失败对照保留在同一矩阵里。
  • R4-2 已钉住:去掉 \0(M3)会让新增断言变红;阳性对照 M4 同样被杀。
  • R4-3 仍暂缓:去掉 Windows .cmd 谓词(M7)430/430 全绿,说明该轴仍无测试;作者已在 PR 里声明延后,我同意按“第 5 轮后只落 Critical”处理。
  • 中心 A/B:59 格 × 3 臂,oracle 为容器内 GNU sed 4.9。base 15 格与 sed 不一致,r4/head 各 1 格(B5 零宽全局,GNU 按字节、PR 按 BSD 语义,既有 S1)。issue sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed #15675 的六行里五格 base 错→head 对;第六行(x*)head 与 BSD 一致,本容器无 BSD sed,未测。
  • 两个 Suggestion(不阻塞):①u 标志带来 16/25 个恒等转义写法(\-、\&、\%、\~、\{ 等)从“模拟”退化为“回退真 sed”;有 sed 的机器等价,PR 标注未验证的 Windows 上则会从可用变为依赖 sed 存在。②泛型 ACP 客户端若逐字写入,s/\r$// 会被跳过——这是作者 R4-1 修复的已知代价,我同意保守方向。
  • 未覆盖:Windows/BSD sed(只有 GNU 4.9)、真实 ACP 传输(进程内链)、非 UTF-8(BOM 之外)、core 全量套件、本 head 的 CI 实质 lane 在 2026-10-29 14:11Z 被整体取消(无测试结论)。

Previous-finding status

Round 4 (2026-10-29, head 9fc09d69) reported findings. Since it, the branch adds 21d2c9dfb (production), 826c33325 (test-only) and df028832 (a merge of main). The merge is mechanical: git merge-tree --write-tree 826c3332 bc8ad53f95 yields 230e2949, which equals df028832^{tree}, and git diff 826c3332 df028832 over the PR's six files is empty. Every row below was re-run — nothing is carried forward by assumption.

# finding severity status at df028832 (cells measured at 826c3332; files identical)
R1-1 -E \p{…} / \u{…} simulated as JS escapes Critical fixed — D1/D2/D3 decline at r4 and head; the real binary runs
R2-1 CRLF edit whose only effect is \r reports applied with unchanged bytes Critical fixed — LOCAL-NOOP and both serve cells are truthful; base control reproduces the false success
R1-2 gate's u-only rejection class unpinned Suggestion fixed — M10 (drop u from the gate) kills the suite
R3-1 BOM / non-UTF-8 file: false "applied" Critical fixed on both writers — LOCAL-BOM-NOOP and SERVE-BOM-NOOP report no changes; r4 serves it as the failing control
R3-2 a writer that stores content verbatim gets a real edit skipped Suggestion re-opened by design / declined with rationale — VERBATIM-CRLF now skips s/\r$//; this is the cost R4-1's fix chose. I agree with the direction: the in-repo qwen serve writer normalizes, and the r4 alternative was a false success in that same runtime
R3-3 \0 is NUL to JS, a literal 0 to sed Suggestion fixed and pinned — M3 kills the new assertion; behaviour cell D4 declines
R3-4 no positive-write CRLF fixture Suggestion fixed — LOCAL-POSITIVE and SERVE-POSITIVE change bytes; M5/M6 kill the suite
R4-1 served sessions predict a verbatim write while the daemon re-applies CRLF Critical fixed — SERVE-NOOP/SERVE-BOM-NOOP flip from applied/unchanged (r4) to no-changes/unchanged (head)
R4-2 \0 fixed but unpinned Suggestion fixed — the divergence test now asserts it; M3 kills it
R4-3 Windows .cmd axis unpinned Suggestion stands, deferred — M7 survives 430/430; author deferral is explicit, and I agree
S1 (r1) zero-width global over an astral character follows BSD, not GNU Suggestion stands — B5 differs from GNU at all three arms; head is BSD-consistent
S2/S3 (r1) gate/apply flag consistency; s in the gate unobservable Suggestion S3 stands — M2 (drop s from the gate) survives; S2 is covered by M10
N-1 CRLF→LF never reaches disk under the local writer note unchanged — s/\r$// still leaves abc\r\n; the report is now truthful

Central claim + A/B

Claim unchanged: the simulated sed -i fast path must treat ., bracket expressions and astral characters the way sed does. The oracle is the real binary (GNU sed 4.9, LC_ALL=C.UTF-8), never hand-written expectations. Three arms, each bundled from its own sedEditParser.ts; positive control: the base bundle contains zero gsu occurrences and the r4/head bundles exactly one. Cell counts are expectation-encoded, so a base cell that fails as predicted is a passing assertion.

arm match decline wrong note
base 35fd1f44 39 5 15 the control: every issue row the PR targets is wrong
r4 9fc09d69 46 12 1 the one is B5 (GNU-vs-BSD, pre-existing)
head 826c3332 → df028832 46 12 1 parser source is byte-identical to r4 (sha256 ee31865d…) and unchanged by the main merge

A/B: base, round-4 head and head against GNU sed 4.9

The six rows from issue #15675, verbatim:

issue row base head note
s/^.// on 😀 Title wrong — \ude00 Title (lone surrogate) match — Title B1
-E s/^.// wrong — same lone surrogate match B1E
s/.$// on abc😀 wrong — abc\ud83d match — abc B2
s/[^a-z]/_/g wrong — a__b match — a_b B4
s/x*/-/g on 😀 wrong — -\ud83d-\ude00- BSD-shaped -😀- B5; GNU 4.9 interleaves - at every byte, so this cell is excluded from the oracle count (S1)
s/=.*$/=new/ on CRLF wrong — unchanged match — key=new\nk2=new\n A1

The Reviewer Test Plan's three bracket patterns ([abc], [^abc], [\[\]]) all take the simulation path at head (C4/C5/C6 match), not the fallback. [abc]-style cells are asserted match, so a decline would fail the cell.

Endpoint matrix (the round-5 delta)

The changed predicate is judged at the endpoint, not by reading it. Real ShellTool → real AcpFileSystemService → real BridgeFileSystemAdapter → real WorkspaceFileSystem.writeTextOverwrite → real encoder; only the ACP transport object is a one-method stand-in. LOCAL-* cells use the real StandardFileSystemService; VERBATIM-CRLF is a generic ACP host whose writer stores content as given.

cell base r4 9fc09d69 head df028832 (re-run on the merged tree)
LOCAL-NOOP s/.$// on abc\r\n no-changes/unchanged no-changes/unchanged no-changes/unchanged
LOCAL-BOM-NOOP no-changes/unchanged no-changes/unchanged no-changes/unchanged
LOCAL-POSITIVE s/foo/bar/ applied/changed applied/changed applied/changed
SERVE-NOOP s/\r$// applied/unchanged applied/unchanged no-changes/unchanged
SERVE-BOM-NOOP no-changes/unchanged applied/unchanged no-changes/unchanged
SERVE-POSITIVE applied/changed applied/changed applied/changed
VERBATIM-CRLF s/\r$// applied/changed applied/changed no-changes/unchanged (accepted cost)

Endpoint matrix: local writer, qwen serve chain, verbatim client

Two things this fixes and one it costs. Fixed: the r4 head's applied/unchanged in the serve chain (a false success that also records a file-history backup, attribution and a read-cache write) and the same shape for the BOM file. Cost: a generic ACP host that writes verbatim would have applied s/\r$//; head now suppresses it because the prediction always normalizes. A positive edit on the same file still lands (SERVE-POSITIVE), so this is a narrow line-ending-only class, not a broken write path.

Mutation matrix

Runner: the author's two core suites through the head arm, baseline green before any row, each anchor asserted unique, each revert replayed and the tree confirmed clean. M4 is the positive control; M7 is the carried R4-3 gap.

ID mutation expected outcome tests verdict
M1 apply loses dotAll + unicode (gsu → g) kill killed 429/430 PASS
M2 gate loses dotAll (su → u) survive (S3) survived 430/430 PASS
M3 drop \0 from the divergent-escape class kill killed 429/430 PASS
M4 positive control: drop p/P/u from the class kill killed 429/430 PASS
M5 remove the CRLF no-op suppression kill killed 428/430 PASS
M6 prediction stops normalizing before comparing kill killed 428/430 PASS
M7 drop the Windows .cmd predicate (R4-3) survive survived 430/430 PASS
M8 stop escaping a literal ] outside a class kill killed 429/430 PASS
M9 never predict CRLF restoration kill killed 428/430 PASS
M10 gate loses the u flag (su → s) kill (R1-2) killed 429/430 PASS

Baseline before any row: 430/430 (53 sed parser + 377 shell). Every restore is byte-identical to the original; git status --porcelain --untracked-files=no is empty after the runner. The surviving rows are the two accepted coverage gaps, not missed kills.

Mutation matrix at head

Findings

F1 — Suggestion, measured: the u flag removes 16 of 25 literal-escape spellings from the simulation. The gate now rejects patterns JavaScript's u mode refuses, so s/a\-b/X/, s/a\&b/X/, s/a\%b/X/, s/a\~b/X/, s/\{/X/, s/\}/X/ and 10 more fall back to the real binary. On Linux and macOS that is equivalent or better (the binary runs, and it is the module's documented policy). On the Windows configuration the PR marks untested, the shell tool has no sed binary to fall back to, so those commands would now fail where the simulator once handled them. Not blocking — the class is narrow, the failure is loud rather than a wrong write, and the PR documents the fallback — but the risk note reads "never writes a wrong file", which is true and does not cover "may now require a sed binary". Reproduce: node sweep-declines.mjs (16 lost of 25).

F2 — Suggestion, carried from R4-3: the Windows .cmd predicate is still unpinned. Deleting needsCrlfLineEndings(filePath) from the restoration predicate (M7) leaves the two suites 430/430 green. The branch is unreachable on this rig (os.platform()), so this round cannot close it; a platform-mocked test would. Agreed with the author's deferral under the five-round rule.

F3 — Suggestion, accepted trade-off: a verbatim ACP writer loses line-ending-only edits. Measured in VERBATIM-CRLF; the author records the choice in shell.ts's comment and in the round-4 reply. I agree with it: the alternative (r4 head) purchases that hypothetical client at the price of a false success in the repository's own qwen serve runtime, and a capability bit cannot distinguish the two. The residual is that the skip is reported as "made no changes" — truthful about the bytes, silent about the dropped request.

No blocking finding. The false-success class that R2-1/R3-1/R4-1 chased is closed on every writer this round can reach.

Corrections

None to an earlier round. Two oracle notes for readers comparing rounds: this container's GNU sed is 4.9 (rounds 3–4 used 4.5/4.10), and on 4.9 -E 's/\p{L}/X/' is a no-op rather than an error, which is why base "matches" D1–D3 here while round 4 measured base as wrong under 4.10. The head's decline is correct under all three versions.

Not covered

  • Windows and BSD sed. Only GNU sed 4.9 is in the container. The issue's x* row is a BSD expectation and is reported as BSD-consistent (head), not oracle-matched; M7's Windows branch is unexercised.
  • A real ACP transport. The serve cells run the real service, adapter, writer and encoder in-process; the AgentSideConnection is a one-method stand-in. No daemon process was spawned.
  • Non-UTF-8 encodings beyond BOM, -i.bak, backup suffixes, symlinks.
  • The full packages/core suite and this head's CI. The author's two files are 430/430 at head; packages/cli ACP filesystem tests were not re-run. At 826c33325 (the last commit CI ran on), the substantive CI lanes (Test (ubuntu), Lint & Static, precheck, TUI parity, Managed Agent, web-shell E2E) were cancelled at 2026-10-29T14:11Z by the workflow, not failed — no test conclusion is available from CI.
  • The main merge's unrelated changes. The head tree is the trial-merge tree (230e2949), and the two suites plus the endpoint probe were re-run on it (430/430, 7/7); main's other changes are not otherwise exercised. Sibling PR fix(core): match sed text semantics in unicode mode #15694 still conflicts with this PR in sedEditParser.ts (competing fix for the same issue — a maintainer must choose one); fix(core): keep sed anchors inside groups and alternations #14428 merges cleanly.

Methodology

Local maintainer round in a credential-free node:22-bookworm container on a LAN host: four trees at base / r4 / head / trial-merge, each with its own git repo and hard-linked node_modules; every internal workspace link asserted to resolve inside its own tree (readlink -f node_modules/@qwen-code/qwen-code-core → that arm's packages/core). Parser arms are esbuild bundles of each tree's sedEditParser.ts; endpoint arms run the real ShellTool through packages/cli's vitest alias (core source, not dist). The head moved mid-round; it was classified as a merge-only advance by tree identity, re-measured on the merged tree, and the old SHA is labelled on the cells that were captured before it. Raw logs and all result files are in probe/; the runners are ab-parser.mjs, mutations.mjs, collect-serve.mjs, ledger.mjs. GNU sed 4.9 was the only oracle.


Verification round 5 by @wenshao's local /verify-pr rig, 2026-10-31, at head df028832 (cells captured at 826c3332, whose six PR files are byte-identical to the merged head). Advisory evidence for human reviewers; this round does not approve, merge, or close anything.

@2SinceJuly

Copy link
Copy Markdown
Contributor Author

CI 失败项是 Lint & Static 里的 Check lint gate freshness:main 在 2026-10-31 16:35 (CST) 合入 ca7d487(#15974,修改了 .github/workflows/ci.yml),本分支尚未包含该门禁改动。

已用普通 merge 把上游 main(fe30b5daa)同步进本分支(无冲突,未 rebase,未 force-push)。

  • 新 head:f244917f1
  • 测试:本次仅同步上游 main,无代码改动,未在本地跑测试;以 CI 复跑结果为准。

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Qwen Code review did not complete successfully. Qwen review exited with status 1. A transient error is retried automatically; if you are seeing this, retry with @qwen-code /review. See workflow logs.

@doudouOUC doudouOUC left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EN

Thanks — I checked the central claim against a real sed instead of the new expectations: the head source of
sedEditParser.ts was driven through parseSedEditCommand + applySedSubstitution and the same expression was
run through /usr/bin/sed -i on the same bytes, over 54 targeted cells (.,.*,.\{1\} on astral and CRLF
input, [^a-z], [😀], [😀-😁], [^😀-😁], a], [\\], zero-width x*, \(.\)\1) and 649 randomised
ones. Head matches the real binary in all 18 cells where the base build was wrong, and nothing in the diff
regressed a cell that base got right. The ] escaping, the hasPosixBracketExpression(sedInfo.pattern) input
change at packages/core/src/utils/sedEditParser.ts:182, and the CRLF write-side guard at
packages/core/src/tools/shell.ts:1951 all hold up. I also tried to build a script this path would accept that
writes outside the named file (s///w, s///e, w file;…, two -e, s///p, -i.bak, pipelines and
redirects): every one is declined, so the write surface looks intact.

Two non-blocking notes:

  1. packages/core/src/utils/sedEditParser.ts:204-206: feeding the raw pattern into this guard is the right fix,
    but the regex only matches a POSIX class that closes the bracket expression itself. Negated and mixed forms
    still pass and are then simulated wrongly rather than declined — s/[^[:space:]]/X/g on z]x writes Xx
    where sed writes XXX, s/[[=a=]]/X/g on a] writes X where sed writes X], and
    s/[[:alpha:]abc]/X/g on aabc] writes X where sed writes XXXX]. Both arms of the diff do this, so it
    is pre-existing rather than a regression, but it is the same "decline rather than guess" policy the comment at
    :215-224 states, and declining anything containing [:, [. or [= would close it.
  2. packages/core/src/utils/sedEditParser.ts:520: with u, a multibyte backreference now matches
    (s/\(.\)\1/X/g on 😀😀x writes Xx, while BSD sed leaves the file alone). The same split already
    existed for éé at base, so this is consistent with the GNU-oriented stance — just not covered by the
    GNU/BSD note at sedEditParser.test.ts:278-279, which documents only the x* case.

中文

感谢——我没有只依赖新增断言,而是用真实 sed 复核了核心结论:把 head 的 sedEditParser.ts 原样驱动
parseSedEditCommand + applySedSubstitution,再用同一条表达式通过 /usr/bin/sed -i 处理同样的字节,跑了 54 个
定向用例(星平面与 CRLF 输入上的 ./.*/.\{1\}、[^a-z]、[😀]、[😀-😁]、[^😀-😁]、a]、[\\]、零宽
x*、\(.\)\1)和 649 个随机用例。base 答错的 18 格 head 全部与真实 sed 一致,且没有任何 base 答对的用例被本次
diff 改坏。] 转义、packages/core/src/utils/sedEditParser.ts:182 改用原始 pattern 的改动,以及
packages/core/src/tools/shell.ts:1951 的 CRLF 写回判据,都站得住。我还尝试构造一条会被这条路径接受、却写到指定
文件之外的脚本(s///w、s///e、w file;…、两个 -e、s///p、-i.bak、管道与重定向):全部被拒绝,写入口
看起来是完整的。

两条非阻断意见:

  1. packages/core/src/utils/sedEditParser.ts:204-206:把原始 pattern 喂给这个 guard 是对的,但正则只匹配「自行闭合
    整个括号表达式」的 POSIX 字符类。取反和混合写法仍会漏过,然后被错误模拟而不是回退——s/[^[:space:]]/X/g 处理
    z]x 写出 Xx(sed 是 XXX),s/[[=a=]]/X/g 处理 a] 写出 X(sed 是 X]),s/[[:alpha:]abc]/X/g 处理
    aabc] 写出 X(sed 是 XXXX])。diff 的两侧行为一致,所以这是既有问题而非本次回归;但它正是函数上方注释里
    「宁可回退也不要猜」的那条策略,只要在 pattern 含 [:、[. 或 [= 时一律回退即可关闭。
  2. packages/core/src/utils/sedEditParser.ts:520:加 u 之后多字节反向引用会命中(s/\(.\)\1/X/g 处理 😀😀x 写出
    Xx,而 BSD sed 不动这个文件)。同样的分歧在 base 上对 éé 已经存在,所以这与「以 GNU 为准」的取向一致——只是
    sedEditParser.test.ts:278-279 的 GNU/BSD 说明目前只写了 x* 一种,没有覆盖它。

@2SinceJuly

Copy link
Copy Markdown
Contributor Author

Thanks for checking against the real binary rather than the new expectations — that is the evidence I could not produce from inside the simulation.

Both notes are addressed in b8ccdda4:

  1. hasPosixBracketExpression now matches the opening sequence [:, [. or [=, not only the self-contained [[:alpha:]] form. Your three shapes — s/[^[:space:]]/X/g on z]x, s/[[=a=]]/X/g on a], and s/[[:alpha:]abc]/X/g on aabc] — are now declined instead of simulated wrongly, and each has a toBeNull() pin. More patterns fall back to real sed; none are newly simulated.
  2. Added s/\(.\)\1/X/g on 😀😀x to the characterwise test, next to the x* case, and extended the GNU/BSD note to say the same split applies to multibyte backreferences (GNU writes Xx, BSD leaves the file). That is the GNU-oriented stance the PR already takes.

No other changes; 53 sed-parser tests pass.

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Qwen Code review did not complete successfully. Qwen review exited with status 1. A transient error is retried automatically; if you are seeing this, retry with @qwen-code /review. See workflow logs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed

5 participants