Skip to content

test: rmrf exponential backoff + sweep remaining bare recursive rmSync (#1646 follow-up, #1910) - #1915

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-02_tmp-rm-backoff
Oct 2, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-02_tmp-rm-backoff

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

The ubuntu-22 release-gate leg of #1910 hit the #1646 ENOTEMPTY teardown race again: side-request-isolation #1129's finally-block rmrf exhausted the flat 10×50ms retry budget while a late debounced session flush (5ms debounce recreating <tmp>/<provider>/) kept the tree alive past every retry under full-suite load. All of the test's assertions had already passed — the gate was killed by its own cleanup, same signature as #1194 and #1646.

Changes

  • tests/tmp-rm.ts — escalate to a bounded exponential-backoff loop (25→1600ms, ~3.2s total, paid only when a race actually happens) around the same transient-error set Node's rimraf retries (ENOTEMPTY/EBUSY/EPERM/EACCES/EMFILE). Sync sleep via Atomics.wait — legal on any thread; teardown wants the loop blocked, not interleaved.
  • tests/tmp-rm.test.ts (new) — pins the contract: rmrf outlives a writer recreating the tree every 10ms for ~250ms (past the old 500ms budget under runner drift), force swallows ENOENT, non-transient errors still throw.
  • Sweep the 36 leftover bare rmSync(..., { recursive: true, force: true }) calls across 11 suites (port-zone, launcher, dsh-native, dsh-settings-ui, fix-decompress-tofile, compat-drop, log-mask, named-provider-bind, responses-seam-probes, responses-plugin-dev-position, codex-overlay-env) onto rmrf; codex-overlay-env's local rmrf wrapper replaced by the shared import.

Verification

  • tests/tmp-rm.test.ts 3/3; launcher + fix-decompress-tofile 261/261; side-request-isolation green; remaining swept suites green (96/101 aggregate — the 5 fails are dsh-settings-ui, which fails identically on clean master locally, stale dist; its CI leg is green)
  • npm run typecheck clean
  • No src/ changes — tests only

Closes nothing directly; unblocks release-gate reruns for the #1646/#1194/#1910 family.

…e rmSync (#1646 follow-up, #1910)

The ubuntu-22 release-gate leg of #1910 hit the #1646 ENOTEMPTY teardown
race again: side-request-isolation #1129's finally-block rmrf exhausted the
flat 10x50ms retry budget while a late debounced session flush (5ms debounce
recreating <tmp>/<provider>/) kept the tree alive past every retry under
full-suite load. All of the test's assertions had already passed — the gate
was killed by its own cleanup.

- tmp-rm.ts: escalate to a bounded exponential-backoff loop (25->1600ms,
  ~3.2s total, only paid on an actual race) around the same transient-error
  set Node's rimraf retries (ENOTEMPTY/EBUSY/EPERM/EACCES/EMFILE). Sync
  sleep via Atomics.wait — legal on any thread, and teardown wants the loop
  blocked, not interleaved.
- tests/tmp-rm.test.ts: pin the contract — rmrf must outlive a writer that
  recreates the tree every 10ms for ~250ms, force swallows ENOENT,
  non-transient errors still throw.
- sweep the 36 leftover bare rmSync(..., { recursive: true, force: true })
  calls across 11 suites onto rmrf; codex-overlay-env's local rmrf wrapper
  is replaced by the shared import.

Local: tmp-rm 3/3, launcher+fix-decompress-tofile 261/261, side-request-
isolation green, the other swept suites green; dsh-settings-ui (5) and
dsh-native fail identically on clean master locally (stale dist — CI legs
green); typecheck clean.

Refs #1646, #1194 (same signature on an earlier release gate).
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-02_tmp-rm-backoff (6470b2c)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1906-pr.1915.7d7f1f3

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1915.tgz
npm install -g package

Install instructions are refreshed on each push.

…s cannot interleave with sync rmSync (#1915 review follow-up)

The race test's setInterval writer runs on the same event loop as the
synchronous rmSync, so it can never recreate entries between readdir and
rmdir: the backoff loop was never exercised (the test body passes against
a bare one-shot rmSync too). Drive the loop through a _setRmImplForTest
seam instead — default impl unchanged — and pin recovery after transient
failures, budget exhaustion (8 attempts, full ~3.2s ladder), and
immediate throw on non-transient errors. Relabel the in-process writer
case as the smoke pass it is. Also fix the stale header note: the retry
lives in rmrf's own loop now, not in rmSync options.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review complete — one finding, fixed directly on the branch (commit 6470b2c6), new CI triggered.

方向关(审原 issue):问题真实存在且定位正确——#1910 的 ubuntu-22 release-gate 腿再次命中 #1646/#1194 同签名的 teardown 竞态,根因是 flat 10×50ms(~500ms)预算在 full-suite 负载下扛不过有界的 writer burst(5ms debounce 的延迟 session flush)。修法打在机制上(teardown 预算本身),不是症状。Sweep 完整性已验证:PR head 上 git grep 全仓仅剩 tests/tmp-rm.ts:26 一处 bare recursive rmSync(实现本身);11 个被扫文件逐行核对为纯机械替换;side-request-isolation 早已 import rmrf(line 16/859)。无 src/ 变更。

测试有效性关(发现并已修):原 tests/tmp-rm.test.ts 的竞态测试没有牙。in-process 的 setInterval writer 与同步 rmSync 共享同一个事件循环——同步调用(含 Atomics.wait 睡眠)期间 timer 根本不触发——所以 writer 永远无法在 readdir 与 rmdir 之间重建目录,backoff 循环从未被执行到。实证:把同一测试体对着一发即止、零重试的裸 rmSync(target,{recursive:true,force:true}) 跑,依然通过(比旧代码和新代码都弱)。真正的 #1910 对手是跨进程的(spawned lane 的 debounce persist flush),in-process 测试在结构上无法复现它。按「仅报告不是合法处置」,已直接在分支上修掉:

  • tests/tmp-rm.ts:加 _setRmImplForTest(impl?) seam(默认实现与 shipped 字节级一致 → 对所有既有调用方零行为变化),retry 循环改经 seam 调 delete;顺手修正过时的头注释("Do not simplify these options away" 指的是本 PR 自己删掉的 rmSync options,机制现在在外层循环里)。
  • tests/tmp-rm.test.ts:原竞态测试诚实降级标注为 smoke(coexistence only,注释写明单线程约束);新增三个确定性 loop-pin:① 3×ENOTEMPTY 后恢复(断言共 4 次调用 + 树被删);② 持续 ENOTEMPTY 风暴(断言 8 次调用 = 1 尝试 + 7 级 backoff,且耗时 ≥3.1s 钉死 25..1600ms 阶梯);③ 非瞬态错误 ELOOP 一次即抛(断言仅 1 次调用,无 backoff)。TypeError/force 用例保持原样。
  • 本地定向验证:tmp-rm 6/6 + port-zone 抽查 17/17(23/23,含风暴测试实付 ~3.2s);npm run typecheck clean。

结构化七问

  1. 是什么问题:teardown 清理与「延迟重建者」的 ENOTEMPTY 竞态下,旧 flat 预算(~500ms)耗尽后 finally-block 抛错,把已通过的测试判死——gate 被自己的清理杀死。
  2. 什么原因导致的:两层。表层:预算时长不足(writer burst 在 runner 漂移下可超过 500ms)。深层(本轮 review 发现):用来「钉住契约」的测试本身无效——单事件循环使 in-process writer 无法与同步删除交错,backoff 逻辑从未被测到。
  3. 怎么修复的:PR 本身用有界指数退避(25→1600ms,~3.2s,仅在竞态发生时付费)+ 统一 36 处 bare rmSync 到 rmrf;review 补充通过测试 seam 让 retry 循环可确定性驱动,把「恢复/预算耗尽/非瞬态即抛」三种行为真正钉住。
  4. 是否回归问题:无。逐项清点既有行为:(a) rmrf 对全部既有调用方的运行时行为不变(seam 默认实现与原 shipped 一致,本 commit 未动 backoff 逻辑);(b) 唯一有意变更——teardown 最坏时延 ~500ms → ~3.2s,仅当瞬态错误实际发生时才付费(PR 正文已披露,符合设计意图);(c) 错误语义未漂移:非瞬态错误仍然立即抛(新测试钉住),force/ENOENT no-op 保持;(d) 测试套件墙钟时间 +~3.2s/次(风暴测试按设计实付阶梯)。无 fail-fast 变吞错、无默认值/阈值无意漂移。
  5. 是否破坏缓存:不涉及(纯测试域,不触碰任何缓存/重试游标/去重键;前缀缓存路径零改动)。
  6. 是否改了设计方向:不涉及——tests-only,架构/边界/数据流无转向。
  7. 是否引入了新的配置:无新 env/config/CLI flag。

CI 状态:首查时全部 e2e 腿(real codex/dsh/pi/opencode、codex overlay、build-artifact、version-guard、hermetic registry)已绿;新 commit 重新触发的单测腿(含出事的 ubuntu-22)运行中。dsh-settings-ui 本地 5 败的声明(clean master 同样失败、stale dist)接受待其 CI 腿确认。

判定:代码可合——等 6470b2c6 的 CI 转绿即可合并(当前 mergeable: true;BEHIND master 不构成阻塞:自 PR base 以来 master 侧 tests/ 零改动,无热点文件冲突)。合并动作留给人工。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI 全绿,收工判定更新:

6470b2c6(review follow-up commit)上 13/13 check runs 全部通过,包括此前出事的 test (ubuntu-latest, 22) 腿、其余三平台单测腿、以及全部 e2e 腿(real codex / dsh / pi / opencode、codex overlay MCP、build-artifact、version-guard、hermetic registry ubuntu+windows)。

判定:可合并。 PR 现含两个 commit——原 backoff+sweep、review 补的测试 seam 与确定性 loop-pin。合并动作留给人工:#1915

@ranxianglei
ranxianglei merged commit 88bd34b into master Oct 2, 2026
13 checks passed
@github-actions github-actions Bot mentioned this pull request Oct 3, 2026
ranxianglei added a commit that referenced this pull request Oct 11, 2026
…EBUSY/EPERM teardown)

Every SessionStore created under BILI_STORAGE_UNIFIED holds the index.db
SQLite handle until GC; cancelAll() never closes the shadow, and Windows
cannot unlink an open file — the engproof teardown then pins the temp dir
in EBUSY (index.db) / EPERM (dir) on every run of the new fcac9e3 suite
(#1915 rmrf family; POSIX unlinks open files, hence linux-green). Switch
the last lifecycle call of each store to closeQuietly(); the corrupt-index
test closes only after its final writeNow (write-after-close would test a
different thing). Unblocks #2764 / #2774 windows legs.
@ranxianglei
ranxianglei deleted the 2026-10-02_tmp-rm-backoff branch October 15, 2026 05:32
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
… flake, #1646/#3720 family)

Incidental CI-red fix for #3930: the test(windows-latest, node 22, shard 1) lane of the rebased head failed in tests/issue3399-cross-route-load.test.ts teardown with ENOTEMPTY on the bare recursive rmSync - a leftover holdout from the tmp-rm sweep (#1646/#1915/#3721). Unrelated to the bootstrap regression test added by this PR, which passed on the same lane.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
… flake, #1646/#3720 family)

Incidental CI-red fix for #3930: the test(windows-latest, node 22, shard 1) lane of the rebased head failed in tests/issue3399-cross-route-load.test.ts teardown with ENOTEMPTY on the bare recursive rmSync - a leftover holdout from the tmp-rm sweep (#1646/#1915/#3721). Unrelated to the bootstrap regression test added by this PR, which passed on the same lane.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
… flake, #1646/#3720 family)

Incidental CI-red fix for #3930: the test(windows-latest, node 22, shard 1) lane of the rebased head failed in tests/issue3399-cross-route-load.test.ts teardown with ENOTEMPTY on the bare recursive rmSync - a leftover holdout from the tmp-rm sweep (#1646/#1915/#3721). Unrelated to the bootstrap regression test added by this PR, which passed on the same lane.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant