Skip to content

fix(core): match sed text semantics in unicode mode - #15694

Open
Sthreal wants to merge 1 commit into
QwenLM:mainfrom
Sthreal:fix/15675-sed-unicode-flags
Open

Sthreal wants to merge 1 commit into
QwenLM:mainfrom
Sthreal:fix/15675-sed-unicode-flags

Conversation

@Sthreal

@Sthreal Sthreal commented Oct 27, 2026

Copy link
Copy Markdown

What this PR does

The in-process sed edit simulation now treats text as Unicode characters instead of UTF-16 code units and lets . consume the carriage return retained from CRLF input. It keeps the parse-time and execution-time regular-expression flags consistent and escapes a literal closing bracket where Unicode mode requires it, so the existing supported sed patterns continue to use the simulation safely.

Why it's needed

Issue #15675 reports two user-visible divergences from real sed. In CRLF files, s/=.*$/=new/ can report that no changes were made because . does not consume \r. For non-BMP characters, patterns such as s/^.// can remove half of an emoji and write the orphaned surrogate as U+FFFD. Both behaviors make the simulated edit disagree with the shell command and can either block an edit or silently corrupt content.

Reviewer Test Plan

How to verify

Run the focused parser and shell regression suites:

corepack pnpm --dir packages/core exec vitest run src/utils/sedEditParser.test.ts
corepack pnpm --dir packages/core exec vitest run src/tools/shell.test.ts -t "simulated sed edit"

The first suite covers CRLF dot matching and non-BMP handling for ., [^...], and zero-width quantifiers. The second confirms that a CRLF substitution is written with lineEnding: crlf metadata and that all simulated-sed scenarios still pass.

Evidence (Before & After)

Before: a CRLF substitution could report no changes, and s/^.// on 😀 Title could leave U+FFFD in the file. After: both cases follow real sed semantics, with the substitution producing LF-normalized content and the shell write retaining the source file's CRLF metadata. This is an internal behavior change, so there is no UI screenshot.

Tested on

OS Status
🍏 macOS ⚠️
🪟 Windows ✅
🐧 Linux ⚠️

Local verification used Windows and Node.js 24.18.0.

Risk & Scope

  • Main risk or tradeoff: Unicode mode changes how translated regular expressions are compiled, so existing bracket and fallback behavior is covered by the focused regression suite. Open PR fix(core): keep sed anchors inside groups and alternations #14428 also changes sed pattern handling; a merge conflict is possible because the fixes are independent.
  • Not validated / out of scope: macOS and Linux execution were not available locally. The full shell test file reported one unrelated Windows-specific path-warning assertion; all 25 simulated-sed tests passed.
  • Breaking changes / migration notes: none.

Linked Issues

Fixes #15675

中文说明

本 PR 做了什么

进程内的 sed 编辑模拟现在按 Unicode 字符而不是 UTF-16 码元处理文本,并允许 . 匹配 CRLF 输入拆分后保留的回车符。它让解析阶段和执行阶段使用同一组正则标志,并在 Unicode 模式要求时转义字符类外的字面量 ],从而让现有受支持的 sed 模式继续安全地使用模拟路径。

为什么需要这个改动

Issue #15675 报告了两处与真实 sed 不一致的用户可见行为。在 CRLF 文件中,s/=.*$/=new/ 可能因为 . 不匹配 \r 而错误地报告没有发生修改。对于非 BMP 字符,s/^.// 这类模式可能只删除 emoji 的一半,并在写回时把孤立代理项变成 U+FFFD。这两种行为都会让模拟编辑与 shell 命令不一致,可能阻止编辑,也可能静默破坏内容。

Reviewer 验证计划

如何验证

运行聚焦的 parser 和 shell 回归测试:

corepack pnpm --dir packages/core exec vitest run src/utils/sedEditParser.test.ts
corepack pnpm --dir packages/core exec vitest run src/tools/shell.test.ts -t "simulated sed edit"

第一组测试覆盖 CRLF 下的点号匹配,以及 .、[^...] 和零宽量词对非 BMP 字符的处理。第二组测试确认 CRLF 替换会以 lineEnding: crlf 元数据写回,并确认所有模拟 sed 场景仍然通过。

证据(修改前与修改后)

修改前:CRLF 替换可能报告没有修改;s/^.// 作用于 😀 Title 时可能在文件中留下 U+FFFD。修改后:两种情况都符合真实 sed 语义,替换层输出 LF 归一化内容,shell 写回时保留源文件的 CRLF 元数据。这是内部行为修复,因此没有 UI 截图。

测试环境

本地使用 Windows 和 Node.js 24.18.0 验证;macOS 和 Linux 未在本地执行。

风险与范围

  • 主要风险或取舍:Unicode 模式会改变翻译后正则的编译方式,因此现有字符类和回退行为由聚焦回归测试覆盖。开放中的 PR fix(core): keep sed anchors inside groups and alternations #14428 也修改了 sed 模式处理;两个修复相互独立,但可能出现合并冲突。
  • 未验证或超出范围:本地没有 macOS 和 Linux 环境。完整 shell 测试文件有一个与本次改动无关的 Windows 路径告警断言失败;25 个模拟 sed 测试全部通过。
  • 破坏性变更或迁移说明:无。

关联 Issue

Fixes #15675

@qwen-code-review-bot

qwen-code-review-bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Qwen Triage finished — view run. See the stage comments in this thread for the result.

✅ Qwen Triage 已完成 —— 查看运行。结果见本线程中的各阶段评论。

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Thanks for the PR — and for the bug report behind it. #15675 is unusually well researched, which made this gate a lot easier to run.

Template looks good ✓

Problem: observed, not theoretical. The issue carries a six-row table comparing the simulation against BSD sed with UTF-8 hex output, a named root cause (the translated RegExp is built without s and u), and the commit it was reproduced from. Two of those rows are file corruption — a lone surrogate written back as U+FFFD — and one is a silent no-op on CRLF input. That is the strongest kind of evidence this gate asks for.

Direction: aligned. The in-process simulation only earns its place if it is indistinguishable from real sed; every divergence either blocks an edit the user asked for or rewrites a file wrongly. It also follows the philosophy this file already applies elsewhere — POSIX bracket expressions, leading ] literals and divergent escapes are all declined rather than approximated.

Size: core path (packages/core/src/utils/sedEditParser.ts). 17 production lines (+14/−3), 39 test lines, 0 generated/schema. Well under every threshold — no size escalation.

Approach: minimal, and it matches what I arrived at independently before reading the diff. One shared flag constant feeding both construction sites, one escape branch, one argument switch. I looked for a smaller version and don't think there is one: u without s leaves the CRLF row broken; stripping \r before matching and restoring it afterwards is more invasive and collides with the existing "preserves carriage returns in sed pattern space" contract; and adding the flags without the ] escape breaks an existing test.

One ask, because it currently reads like a drive-by and isn't: switching the POSIX-bracket check from the translated pattern to the raw sed pattern is load-bearing. Once ] gets escaped, [[:space:]]*$ translates to [[:space:]\]*$, the check stops firing, and that pattern compiles under u as a single-character class — a silent mis-edit. The existing rejects POSIX bracket expressions before simulated edits test does catch it, so you're covered, but a line in the PR body or a comment at the call site would stop the next reader "simplifying" it back.

Risk: no elevated risk signals — none of the high-revert-correlation paths are touched.

Two notes for later, neither a blocker: #14428 is still open and also edits this file, so the ordering is worth a deliberate choice; and your test evidence is Windows-only, which I come back to in the review comment.

Moving on to code review. 🔍

中文说明

感谢这个 PR,也感谢背后的 issue。#15675 的调研非常扎实,让这次准入检查轻松很多。

模板完整 ✓

问题: 是已观测到的 bug,不是理论性加固。issue 里有一张六行的对照表,用 UTF-8 十六进制列出了模拟结果与 BSD sed 的差异,指出了根因(翻译出的 RegExp 没有带 s 和 u 标志),还写明了复现所用的 commit。其中两行是文件损坏——孤立代理项被写成 U+FFFD——一行是 CRLF 输入下静默地什么都不改。这正是本关卡最希望看到的证据。

方向: 对齐。进程内模拟只有在与真实 sed 无法区分时才值得存在;任何偏差要么挡住用户想要的编辑,要么错误地改写文件。这也符合该文件已有的处理哲学——POSIX 字符类、开头的 ] 字面量、语义不一致的转义,全部选择拒绝而不是近似模拟。

规模: 触及核心路径(packages/core/src/utils/sedEditParser.ts)。生产代码 17 行(+14/−3),测试 39 行,生成/schema 0 行。远低于各项阈值,不需要因规模上升处理。

方案: 足够精简,和我在读 diff 之前独立想到的做法一致:一个共享的标志常量供两处构造点使用、一个转义分支、一个参数切换。我找过更小的写法,认为没有:只加 u 不加 s,CRLF 那一行仍然是坏的;改成匹配前剥离 \r、匹配后再还原,侵入性更大,还会与现有的 "preserves carriage returns in sed pattern space" 约定冲突;加了标志却不转义 ],会打破一个现有测试。

有一点想请你说明,因为它现在看起来像顺手改动,其实不是:把 POSIX 字符类检查的输入从翻译后的 pattern 换成原始 sed pattern 是关键改动。一旦 ] 被转义,[[:space:]]*$ 会翻译成 [[:space:]\]*$,该检查就不再命中,而这个 pattern 在 u 模式下会被编译成单字符字符类——即静默的错误编辑。现有的 rejects POSIX bracket expressions before simulated edits 测试确实能拦住它,所以覆盖是够的,但在 PR 描述里补一句、或在调用处加一行注释,可以避免后来的人把它"简化"回去。

风险: 无升级风险信号——未触及与回滚高度相关的路径。

两点后续说明,都不是阻塞项:#14428 仍处于开放状态且同样修改该文件,合并顺序值得有意识地安排;另外你的测试证据只在 Windows 上取得,这一点我会在代码审查评论里再提。

进入代码审查 🔍

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at 0ae72eba1e9a028ab37f3f3cec157015cb659231 · re-run with @qwen-code /triage

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Code review

Static review only — no PR code was built or executed. I read the diff against main and traced every path by hand.

No critical blockers found. The change is correct as far as I can establish by reading it, and the reasoning hangs together:

  • The ] escape is placed correctly. It fires only outside a character class — the in-class branch runs first and consumes the closing ] itself. It also can never double-escape an existing \], because both the BRE and ERE backslash branches append the pair and advance past it, so a ] that arrived escaped is never re-visited as a bare character.
  • s is safe here. applySedSubstitution splits on \n and only processes the non-newline parts, so . never sees a \n; the only new character it can consume is \r. That is the intended fix and matches both GNU and BSD sed, which keep \r in the pattern space.
  • $/^ anchoring is unchanged. No m flag was added, so the existing preserves carriage returns in sed pattern space assertions still hold — s/foo$/bar/ must not match foo\r, and it doesn't.
  • Every u-mode tightening fails safe. Patterns that u newly rejects (\,, a bare { in ERE, an inverted class range) make canCompileSedPattern return false, so parseSedEditCommand returns null and the command runs through real sed. There is no path where u turns into a wrong but accepted match — the file's established "decline rather than mis-edit" direction is preserved.
  • The shared constant strengthens an existing invariant. applySedSubstitution throws SedEditSimulationError, which surfaces to the user as EDIT_PREPARATION_FAILURE rather than falling back to the shell. That path is only reachable if parse-time and runtime flags diverge; SED_REGEX_FLAGS now makes that structurally impossible.
  • Downstream consumers, complete list: packages/core/src/tools/shell.ts is the only production consumer (parseSedEditCommand L1859, applySedSubstitution L1900), plus the two test files. Nothing else in packages/ imports this module.
  • The helper scans are unperturbed. countCapturingGroups, hasUnsafeQuantifiedGroup and skipCharacterClass all treat \] as an escaped pair, so the added backslash changes neither group counting nor quantifier-safety analysis.

I hand-traced all four new parser assertions and the new shell assertion against the translation and the engine semantics, and each expected value is right — including -😀- for s/x*/-/g, which works because u makes AdvanceStringIndex step a whole code point after an empty match, and key=new\nk2=new\n for the CRLF case, whose \r the shell layer then restores from _meta.lineEnding. I also traced the existing keeps backslashes literal inside bracket expressions under -E cases and confirmed they genuinely need the escape: [\[\]] translates to [\\[\\] plus a trailing bare ], which is a syntax error under u. That is exactly the one failure #15675 predicted, and the escape resolves it.

One thing I could not settle, and it is the gap that matters. The [\[\]] case above produces [\\[\\]\] — a bare [ inside a character class — and whether that compiles under u is load-bearing for an existing test. My reading of the grammar is that [ is only special inside a class under the v flag, so under u it is an ordinary literal and this is fine. I could not confirm that empirically: this environment blocks interpreter execution, and running the PR's own tests is out of scope for a static review either way. Note it fails safe regardless — a decline sends the command to real sed — and it would fail loudly in CI as an existing-test regression. But "I believe it compiles" is not evidence.

Non-blocking nit: the new shell test spells its write expectation inline instead of reusing sedWrite, because that helper hardcodes the lf meta. Fine as it stands.

Test evidence

There is none, and that is the substance of this comment. Every one of this PR's own CI workflows completed as action_required, which is GitHub parking a first-time fork contributor's runs until a maintainer authorizes them. The unit suite has never executed against 0ae72eba. The only checks carrying a real conclusion are bot orchestration jobs. There is no failing-job log to quote because no test job ran.

Check Conclusion
Qwen Code CI action_required — never ran
Managed Agent gate action_required — never ran
tui-parity action_required — never ran
precheck-pr / precheck success
authorize success
assign success
label success

For completeness, and clearly attributed: the author reports in the PR body that sedEditParser.test.ts passes and that 25 simulated-sed shell tests pass on Windows / Node 24.18.0, with one unrelated Windows-specific path-warning assertion failing in the full shell test file. That is the author's claim on a single platform. Not verified: this run is static, and no CI job has executed the suite. My trace above is consistent with those results — notably, the author would have hit the POSIX-bracket failure had they not made that switch — but consistency with a claim is not the same as the claim being checked.

Sandboxed verification would settle this, and two different gaps need two different lanes:

  • Authorizing CI settles whether the existing 51-test sedEditParser suite and the shell suite are still green — in particular the bare-[-inside-a-class compilation question above, which only actually running the engine answers.
  • @qwen-code /verify settles whether the new tests are load-bearing: that they fail on the base build, and that all six rows of the table in sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed #15675 now match real sed. The author has read permission only, so this is a sponsored run — a maintainer's @qwen-code /verify comment approves the head it was written against, and that run carries a pre-execution risk screen plus a full workspace wipe. Worth reading its report with the same skepticism as the fork's own CI logs: the code under verification is adversarial input, so a crafted PR can shape what the report says even though the sandbox bounds what it can do.

@qwen-code /tmux does not apply — the author lacks write access, and this is an internal-substitution change with no TUI surface of its own beyond the existing edit-preview diff.

Real-scenario testing (2c): N/A — unattended CI run, never driven locally.

中文说明

代码审查

仅做静态审查——没有构建或执行任何 PR 代码。我把 diff 与 main 对照阅读,并手工推演了每一条路径。

未发现阻塞性缺陷。 就我能通过阅读确认的范围而言,改动是正确的,推理也自洽:

  • ] 转义的位置是对的。 它只在字符类之外生效——类内分支先执行,会自己消费掉收尾的 ]。它也绝不会重复转义已有的 \],因为 BRE 和 ERE 的反斜杠分支都会把这一对字符一起输出并跳过,所以以转义形式出现的 ] 不会再被当作裸字符处理。
  • s 在这里是安全的。 applySedSubstitution 按 \n 切分,只处理非换行的部分,所以 . 永远看不到 \n;它唯一新增能消费的字符是 \r。这正是要修的行为,也与 GNU、BSD sed 一致——两者都把 \r 留在 pattern space 中。
  • $/^ 锚定语义未变。 没有加 m 标志,所以现有的 preserves carriage returns in sed pattern space 断言依然成立——s/foo$/bar/ 必须不匹配 foo\r,实际也确实不匹配。
  • u 模式带来的所有收紧都是失败即安全。 u 新拒绝的 pattern(\,、ERE 中裸露的 {、反向的字符类区间)会让 canCompileSedPattern 返回 false,于是 parseSedEditCommand 返回 null,命令交由真实 sed 执行。不存在"因 u 而产生错误匹配却仍被接受"的路径——该文件既有的"宁可不模拟也不误改"方向被保住了。
  • 共享常量还强化了一个既有不变量。 applySedSubstitution 抛出 SedEditSimulationError,对用户表现为 EDIT_PREPARATION_FAILURE,而不是回退到 shell。这条路径只有在解析期与运行期标志不一致时才可能触达;SED_REGEX_FLAGS 现在让这种不一致在结构上不可能发生。
  • 下游消费者(完整清单): packages/core/src/tools/shell.ts 是唯一的生产消费者(parseSedEditCommand L1859、applySedSubstitution L1900),加上两个测试文件。packages/ 下没有其他模块导入它。
  • 辅助扫描函数不受影响。 countCapturingGroups、hasUnsafeQuantifiedGroup、skipCharacterClass 都把 \] 当作转义对处理,因此新增的反斜杠既不改变捕获组计数,也不改变量词安全性分析。

我按翻译逻辑和引擎语义手工推演了四条新增 parser 断言和一条新增 shell 断言,每个期望值都是对的——包括 s/x*/-/g 得到 -😀-(原因是 u 让 AdvanceStringIndex 在空匹配后按整个码点前进),以及 CRLF 场景得到 key=new\nk2=new\n(其中的 \r 由 shell 层依据 _meta.lineEnding 还原)。我也推演了现有的 keeps backslashes literal inside bracket expressions under -E 各条用例,确认它们确实需要这个转义:[\[\]] 会翻译成 [\\[\\] 加一个结尾的裸 ],而这在 u 模式下是语法错误。这正是 #15675 预测到的唯一那条失败,转义解决了它。

有一件我无法确认的事,而它恰恰是关键缺口。 上面的 [\[\]] 用例产出 [\\[\\]\]——即字符类内部有一个裸 [——它在 u 模式下能否编译,直接决定一个现有测试的成败。我按语法推断:[ 只在 v 标志下才在字符类内有特殊含义,所以在 u 下它只是普通字面量,没有问题。但我无法实证确认:本环境禁止执行解释器,而静态审查本来也不应运行 PR 自带的测试。需要说明的是,无论如何它都是失败即安全——被拒绝就把命令交给真实 sed——而且在 CI 里会以现有测试回归的形式显式暴露。但"我认为它能编译"不等于证据。

非阻塞小项:新增的 shell 测试把写入期望直接内联写出,而没有复用 sedWrite,原因是那个辅助函数硬编码了 lf 元数据。保持现状即可。

测试证据

没有证据,这正是本条评论的实质。 本 PR 自身的每一个 CI 工作流都以 action_required 结束——这是 GitHub 在等待维护者授权首次贡献的 fork PR 运行。单元测试从未在 0ae72eba 上执行过。唯一带有真实结论的检查都是机器人编排任务。没有失败日志可引用,因为根本没有测试任务运行。

为完整起见并明确归属:作者在 PR 描述中称 sedEditParser.test.ts 通过,25 个模拟 sed 的 shell 测试在 Windows / Node 24.18.0 上通过,完整 shell 测试文件中有一个与此改动无关的 Windows 路径告警断言失败。这是作者在单一平台上的自述。未验证:本次为静态审查,且没有任何 CI 任务执行过该测试套件。 我上面的推演与这些结果一致——尤其是,如果作者没有做 POSIX 字符类那处切换,就会撞上那个失败——但"与自述一致"不等于"自述已被核实"。

沙箱验证可以厘清此事,而两个不同的缺口需要两条不同的通道:

  • 授权 CI 可以确认现有 51 条 sedEditParser 测试与 shell 测试是否仍然全绿——尤其是上面那个"字符类内裸 [ 能否编译"的问题,只有真正运行引擎才能回答。
  • @qwen-code /verify 可以确认新增测试是否真的承重:它们在 base 构建上是否失败,以及 sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed #15675 表中六行结果是否现在都与真实 sed 一致。作者只有 read 权限,因此这是一次代跑(sponsored run)——维护者发出 @qwen-code /verify 评论即批准其针对的 head,该运行额外带有执行前风险筛查和完整工作区清除。读它的报告时,建议保持与读 fork 自身 CI 日志同样的怀疑:被验证的代码是对抗性输入,精心构造的 PR 可以影响报告说什么,尽管沙箱限定了它能做什么。

@qwen-code /tmux 不适用——作者没有写权限,且这是内部替换逻辑的改动,除既有的编辑预览 diff 外没有自己的 TUI 界面。

真实场景测试(2c):不适用——本次为无人值守 CI 运行,未在本地驱动产品。

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at 0ae72eba1e9a028ab37f3f3cec157015cb659231 · re-run with @qwen-code /triage

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Confidence: 3/5 — I could not find a defect in this diff, but a core-path change with zero executed tests is not something I should sign off on alone.

Going back to what I proposed before reading the diff: the PR matches it almost exactly — one shared flag constant feeding both construction sites, s for the retained \r, u for code points, and the out-of-class ] escape that u forces. I did not find a simpler path it missed, and I did find one thing it does better than my sketch would have: switching the POSIX-bracket check to the raw sed pattern. I would have missed that, and missing it means [[:space:]]*$ compiles under u as a single-character class and quietly mis-edits a file. The author caught it, and the existing test suite pins it. That is a good sign about how carefully this was done.

What I like: the diff is 17 production lines that fix two distinct user-visible divergences, both of which are documented with measured hex output against real sed. Every new assertion I traced by hand is correct. Every way the change can go wrong that I could find goes wrong in the safe direction — canCompileSedPattern declines, the command falls through to real sed, and the file is edited by the thing the user actually asked for. Nothing here can silently produce a wrong but accepted match. If I had to maintain this in six months I would thank the author, not curse them: the shared constant removes a divergence that was already a latent hazard, and the comments say why rather than restating the code.

Why I am not approving:

  1. There is no test evidence at all. All three of this PR's CI workflows sit at action_required — the unit suite has never run against 0ae72eba. The 51-test sedEditParser suite is precisely the thing that would catch a u-mode compilation regression, and it has not executed. Approving now would attest to a result that does not exist.
  2. One load-bearing question is unresolved. Whether a bare [ inside a character class compiles under u decides an existing test's fate. My reading of the grammar says yes, but I could not confirm it empirically here. It fails safe either way, so I am not calling it a defect — I am calling it unverified.
  3. The mechanical deferral path does not cover this case. Pending-PR-CI computes to zero only because GitHub marks parked runs as completed, so there is no green-CI backstop to wait on and the finalize job would never revisit an approval I post now. This needs a human to authorize CI first.

Stage 0's bar for core paths is 100% confidence, and "the reasoning is sound but nothing has run" is not that. The fork-refactor guardrail is not the reason — this is a fix, so that check passes cleanly.

@yiliang114 — deferring to you, and you are already the assigned owner. Two things would unblock this: authorize the pending CI runs so the suite actually executes, and give the unicode-mode change a maintainer's eye, since it changes how every simulated sed pattern in the tool is compiled. Once CI is green on this commit, @qwen-code /triage will pick it back up. If you want the new tests proven load-bearing rather than merely green, @qwen-code /verify is available as a sponsored run.

Worth deciding deliberately: #14428 is still open and also edits sedEditParser.ts. The author flagged the possible conflict; the two changes are independent, but whichever lands second will need a rebase.

中文说明

信心度:3/5 —— 我在这份 diff 里找不到缺陷,但一个核心路径的改动在完全没有执行过测试的情况下,不该由我单独放行。

回到我在读 diff 之前自己写的方案:这个 PR 与它几乎完全一致——一个供两处构造点共用的标志常量、用 s 处理保留下来的 \r、用 u 处理码点,以及 u 所强制要求的类外 ] 转义。我没有找到它漏掉的更简路径;反而发现它有一处比我的设想做得更好:把 POSIX 字符类检查切换到原始 sed pattern。这一点我会漏掉,而漏掉的后果是 [[:space:]]*$ 在 u 模式下被编译成单字符字符类,静默地误改文件。作者发现了它,而且现有测试套件把它钉住了。这说明改动做得相当仔细。

我认可的地方:17 行生产代码修掉了两处用户可见的语义偏差,两处都有对照真实 sed 的实测十六进制输出作为依据。我手工推演过的每一条新增断言都是正确的。我能想到的所有出错方式,都朝着安全的方向出错——canCompileSedPattern 拒绝,命令落到真实 sed,文件由用户真正要求的那个工具来改。这里不存在任何能静默产生"错误但被接受的匹配"的路径。如果六个月后由我来维护,我会感谢作者而不是抱怨:共享常量消除了一处本就潜在的不一致隐患,注释写的是"为什么"而不是复述代码。

我不批准的原因:

  1. 完全没有测试证据。 本 PR 的三个 CI 工作流都停在 action_required——单元测试从未在 0ae72eba 上运行过。而 51 条测试的 sedEditParser 套件恰恰是能抓住 u 模式编译回归的那道防线,它没有执行。现在批准,等于为一个并不存在的结果背书。
  2. 有一个承重问题悬而未决。 字符类内部的裸 [ 在 u 模式下能否编译,决定了一个现有测试的命运。我按语法判断是可以的,但在这里无法实证确认。无论哪种结果都是失败即安全,所以我不把它称为缺陷——我称它为未验证。
  3. 机械化的延迟通道覆盖不了这种情况。 待处理的 PR CI 数量算出来是零,仅仅因为 GitHub 把被搁置的运行标记为已完成,所以没有"等 CI 变绿"的兜底,finalize 任务也永远不会回头处理我现在发出的批准。这件事需要先由人授权 CI。

Stage 0 对核心路径的要求是 100% 确信,而"推理成立但什么都没跑过"不是 100%。fork refactor 护栏不是原因——这是一个 fix,该检查干净通过。

@yiliang114 —— 转交给你,你也已经是本 PR 的负责人。有两件事可以解锁:授权待处理的 CI 运行,让测试套件真正执行;并以维护者视角看一下这个 unicode 模式改动,因为它改变了工具中每一个被模拟的 sed pattern 的编译方式。这个 commit 上 CI 变绿之后,@qwen-code /triage 会接着处理。如果你希望证明新增测试是承重的、而不只是绿的,@qwen-code /verify 可以作为代跑使用。

有一件事值得有意识地决定:#14428 仍然开放,且同样修改 sedEditParser.ts。作者已提示可能冲突;两个改动彼此独立,但后合并的那个需要 rebase。

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at 0ae72eba1e9a028ab37f3f3cec157015cb659231 · re-run with @qwen-code /triage

@wenshao

wenshao commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Verdict: findings — 87/91 scripted checks passed; verified head 0ae72eba1e9a (base 51c59e69bbfa, trial merge 46c1347b7f3c). The central CRLF + non-BMP fix is load-bearing and correct; one new silent-edit path on the ERE branch and one unpinned parse-time invariant are the open items.

中文摘要

结论:findings(87/91 条断言通过)。被验证 head 为 0ae72eba1e9a,base 为 51c59e69bbfa,另做了到当前 main 的 trial merge(46c1347b7f3c)。

  • 中心主张成立:sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed #15675 的六行规格(CRLF 的 s/=.*$/=new/、BRE/ERE 的 s/^.//、s/.$//、s/[^a-z]/_/g、s/x*/-/g)在 head 上都与 BSD sed 一致;base 上 3/3 报告行为全部偏离真实 sed,head 上 3/3 一致。更大语料 51 例中,base 有 21 例模拟结果与真实 sed 不符,head 在 41 个仍走模拟的格子里为 0 例不符。
  • F1(Medium)新分歧:ERE 路径不做 \{ 转义,u 把 \p{...}/\P{...}/\u{...} 变成真正的 JS Unicode 转义;head 会静默改写文件(s/\p{L}/X/g 把 abc 变成 XXX),而 BSD sed 退出 0 且不动文件、GNU sed 直接报错。这正是静态评审认为「不存在」的 wrong-but-accepted 路径。
  • F2(Suggestion):只把 parse 侧的 SED_REGEX_FLAGS 回退成 'g' 时,53/53 仍全绿;\q/\_ 会被接受后在运行期抛 Invalid escape。head 本身没这个问题,但共享常量在 parse 侧没有测试钉住。
  • CRLF 写回字节:6 个「. 吃掉 \r」的格子最终写回 CRLF,而真实 sed 写 LF;sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed #15675 已明确说明这是既有的行尾保留策略,不算本次 bug。
  • 未覆盖:Windows、Linux CI(三个 workflow 全是 action_required,没有 lane 真正跑过)、完整 shell.test.ts(只跑了 25 个 simulated-sed 用例)、真实模型驱动的 CLI 端到端、fix(core): keep sed anchors inside groups and alternations #14428 的交互。

修复建议:在 canCompileSedPattern 里加 hasUnicodeModeOnlyEscape(jsPattern)(/(?<!\\)\\[pPu]\{/u);已在一行修复臂上实测 51/51 语料格、53/53 parser、38 个良性格逐字节不变。

Central claim and A/B

The PR claims the in-process sed simulation now matches real sed for (a) CRLF input where . must consume the retained \r, and (b) non-BMP input where ./[^...]/quantifiers must count code points. The oracle is the real sed binary itself, run in a UTF-8 locale: BSD /usr/bin/sed and GNU gsed 4.10. A cell only enters the agreement count when both implementations agree; otherwise it is reported as an oracle split.

corpus A/B: base vs head against real sed

Arm Simulated cells Content differs from real sed Declined (real sed runs)
base 51c59e69 47 / 51 21 4
head 0ae72eba 41 / 51 0 of 46 oracle-agree cells 10

The three rows the PR is about (R1 CRLF s/=.*$/=new/, R2 BRE s/^.//, R3 ERE s/^.//) flip 3/3 from divergent on base to matching on head. The issue's other three rows (s/.$//, s/[^a-z]/_/g, s/x*/-/g) also match BSD sed on head; s/x*/-/g is an implementation split (GNU 4.10 inserts - between the bytes of the emoji), so it is excluded from the single-oracle count.

The broader corpus exercises the sibling shapes the fix's mechanism admits: CRLF anchors ($, ^), grouped captures, occurrence flags, mixed line endings, astral backreferences, astral character classes, counted quantifiers, and literal ] outside a class in both BRE and ERE. The head declines 10 cells, including five BRE patterns whose ES-only escapes (\p, \q, \_) u newly rejects — those fall back to real sed, which is the safe direction.

Byte-level note. 8 of 41 simulated head cells differ from real sed in the bytes written to disk: six are the CRLF cells where the match consumes \r (the substitution layer produces LF; writeTextFile re-applies _meta.lineEnding === 'crlf'), and two (C3 s/$/!/, C8 grouped replacement) are pre-existing on both arms. #15675 explicitly calls the CRLF difference "the existing line-ending preservation, not part of this bug", and the PR's new shell test pins that contract, so this is reported as a bounded divergence, not a finding.

Findings

F1 — Medium — ERE-only escapes are silently applied where both real seds leave the file untouched

toJavascriptPattern translates BRE patterns character by character, which turns a bare { into \{ and therefore makes \p{L} / \P{L} / \u{...} fail to compile under u — those decline and fall back to real sed. The ERE branch deliberately keeps the pattern closer to raw, so the same escapes survive and u gives them their ES2015+ meaning. parseSedEditCommand accepts them, applySedSubstitution applies them, and the shell writes the file without ever running sed:

Pattern (all -E) Input BSD sed GNU sed 4.10 head 0ae72eba
s/\p{L}/X/g abc\n exit 0, file untouched exit 1, RE error XXX\n
s/\P{L}/X/g ab1\n exit 0, file untouched exit 1, RE error abX\n
s/\u{1F600}/X/ 😀\n exit 1, RE error exit 1, RE error X\n
s/[\p{L}]/X/g (control) abc\n exit 0, file untouched exit 0, file untouched abc\n

The in-class control agrees because the ERE branch doubles a backslash inside a bracket expression (\\p), so the divergence is outside character classes only.

Reproduce directly against the head source:

const info = parseSedEditCommand("sed -E -i 's/\\p{L}/X/g' f.txt")!;
applySedSubstitution('abc\n', info); // "XXX\n"

Real sed -E 's/\p{L}/X/g' on the same input exits 0 and leaves abc (BSD) or rejects the command with Invalid content of \{\} (GNU). This is a new "wrong but accepted" path, which is the exact class the module's canCompileSedPattern exists to prevent.

Measured suggested fix (one guard, no collateral)
       hasSedJavascriptDivergentEscape(jsPattern) ||
+      hasUnicodeModeOnlyEscape(jsPattern) ||
       hasBracketCharacterEscape(sedInfo.pattern)
@@
+function hasUnicodeModeOnlyEscape(pattern: string): boolean {
+  return /(?<!\\)\\[pPu]\{/u.test(pattern);
+}

Applied to a third worktree at the same head: parser suite 53/53, corpus 51/51 cells as expected (the three hostile cells now decline and run real sed), and 38 benign cells byte-identical to head. The negative lookbehind is what keeps the in-class [\p{L}] control on the simulation path. No fixture currently pins this axis; the table above is the fixture that would go red.

ERE unicode-only escapes: head vs real sed vs fix

F2 — Suggestion — the parse-time half of SED_REGEX_FLAGS has no test

Mutating only canCompileSedPattern back to new RegExp(jsPattern, 'g') (runtime still 'gsu') leaves the parser suite at 53/53 green — a survivor. Adjudicating it through the real parse+apply path shows why the shared constant matters: \q and \_ are accepted at parse time and then throw Invalid regular expression: /\q/gsu: Invalid escape at apply time, where head would have declined and let real sed edit the file.

This is a coverage gap, not a defect in the current head. One parse-null assertion pins it:

expect(parseSedEditCommand("sed -i 's/\\q/X/g' f.txt")).toBeNull();

Correction to the static review

The bot review concluded there is "no path where u turns into a wrong but accepted match". That holds for BRE (the \{ translation makes the ES-only escapes fail to compile), but not for ERE, as F1 measures. Two of its other statements are confirmed by execution rather than by reading:

  • The unresolved bare-[-inside-a-class question is settled: the existing keeps backslashes literal inside bracket expressions under -E test passes on head and on the trial merge, so the translation compiles under u.
  • The POSIX-bracket switch is load-bearing and pinned: reverting hasPosixBracketExpression(sedInfo.pattern) to jsPattern turns rejects POSIX bracket expressions before simulated edits red (52/53), so [[:space:]]*$ cannot be quietly simulated as a single-character class.

Mutation matrix

mutation matrix: five guards and two controls

Baseline 53/53. Each row is a single exact-text mutation of the head parser, run through the same minimal vitest config, then restored with git checkout HEAD -- and verified by sha256 plus a clean worktree.

Mutant Result Witness
M1 runtime flags → g killed (51/53) the two new parser tests
M2 parse flags → g survived (53/53) see F2
M3 drop s killed (52/53) CRLF dot test
M4 remove ] escape killed (52/53) existing [\[\]] ERE test
M5 POSIX check on jsPattern killed (52/53) existing POSIX-decline test
M6 control: remove divergent-escape check killed (52/53) \d reject test
M7 control: neuter leading-bracket check killed (51/53) []a] / [^]] tests

The two controls prove the runner can see red on lines the suite already pins; without them the survivor would be indistinguishable from a dead harness.

Gates

Arm sedEditParser.test.ts shell.test.ts -t "simulated sed edit"
head 0ae72eba 53/53 25/25
base 51c59e69 51/51 24/24
trial merge 46c1347b 53/53 25/25

The trial merge is origin/main (5cb982da) plus the PR head; main touched none of the PR's files since the base, and the merge was conflict-free. The focused suites ran under a minimal vitest config because the repository's global setup refuses to run without a built packages/core/dist; these two suites use only relative imports, and the config omits the guard (the guard's own concern, workspace dist staleness, is unrelated to them).

Not covered

  • Windows — the author tested there; this round is macOS only.
  • Linux CI — all three of the PR's workflows are action_required (first-time fork contributor), so no CI lane has executed this commit. The trial merge is the closest substitute for "what lands".
  • The full shell.test.ts — only the 25 simulated-sed tests were run; the full 375-test file was not.
  • A real model-driven CLI turn — the harness drives parseSedEditCommand, applySedSubstitution, and the real prepareTextFileContent write-layer function directly, not through a model tool call.
  • PR fix(core): keep sed anchors inside groups and alternations #14428 — still open and also edits sedEditParser.ts; whichever lands second needs a rebase. This round did not evaluate the pair.
  • Replacement-side GNU escapes (\U, \L, \u in the replacement) — the parser's replacement filter already declines unsupported escapes; they were not swept here.

Methodology

macOS (darwin), Node 24.18.1, vitest 3.2.7, BSD /usr/bin/sed and GNU gsed 4.10 with LC_ALL=en_US.UTF-8. Four worktrees off the same repository: base 51c59e69, head 0ae72eba, trial merge 46c1347b (origin/main 5cb982da + head), and a fix arm at head plus the one-line guard. The corpus harness imports each arm's sedEditParser.ts through tsx, compares the substitution string to both real sed binaries, and computes the bytes the product would write through head's exported prepareTextFileContent (a file the PR does not touch). Mutation rows use exact-text replacement plus vitest's JSON reporter, with restore verified by sha256. Raw logs and harnesses live in tmp/pr15694-verify-20261027-192900/.

What I would do as a reviewer: the central fix is sound and I would merge it. F1 is narrow (ES-only escapes in ERE) but it is a silent edit where sed either no-ops or rejects, so I would either land the measured one-line guard in this PR or file it as a follow-up with the table above as the fixture; F2 is a one-assertion follow-up at most.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed

4 participants