Skip to content

test: deterministic teardowns for the three observed CI flakes (#1208) - #1224

Merged
ranxianglei merged 1 commit into
masterfrom
2026-09-24_flaky-test-hardening
Sep 23, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-09-24_flaky-test-hardening

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Fixes #1208

What changed (test-only, zero src/ changes)

  1. tests/logger-rotation.test.ts — settle() now requires a stable non-zero size before returning and throws on timeout instead of silently returning. Previously last initialized to -1, which is exactly what a missing file reads as, so if the stream's async open had not completed within ~60ms (plausible under windows-latest load), settle returned early while streamFd was still undefined. The clobber then raced the in-flight open: isOrphaned() short-circuits to "not orphaned" while an open is in flight, both post-clobber logs reused the pending stream, and the expected single [warn] never fired (warns.length 0 ≠ 1 — the exact PR fix: register watcher pid for every native client attaching to a shared proxy (#1190) #1200 windows-24 failure). With the precondition guaranteed (open completed + line flushed), the clobber/reopen path is deterministic on every OS: libuv opens files with FILE_SHARE_DELETE on Windows, so the clobber succeeds either way, statSync(logPath) then throws ENOENT, orphan detection fires, the reopen fails with ENOTDIR, and exactly one warn is emitted.

  2. tests/session-gc.test.ts ("records rawInputTokens per turn and persists it ([Feature] Default state file cleanup strategy #1082)") — drains in-flight debounced saves with await store.flushAll([]) before the flushSync round-trip assertion (same pattern as Flaky: #1101 T7 cleanup races in-flight persist writes — ENOTEMPTY kills the release gate #1194). With debounceMs: 10, a turn-1 timer can fire between turns and start an async writeNow whose payload snapshot predates turn 2's rawInputTokens; its rename can land after the synchronous flushSync, leaving stale bytes on disk → recorded value survives persistence round-trip fails (the master@e8be19a1 windows-24 failure). After flushAll no competing writer remains.

  3. tests/e2e-export-handoff.test.ts — same one-line drain before the teardown rmSync: this test drives real proxy traffic (markDirty → scheduleSave, debounceMs 0) over an enabled store and deleted the dir without draining, the identical race class as Flaky: #1101 T7 cleanup races in-flight persist writes — ENOTEMPTY kills the release gate #1194/T7. Hardened proactively so it cannot become the next random red.

Not changed (verified safe)

Audited every other enabled-store test for the same race class (debounced save in flight at teardown rmSync):

  • fix-401-storage, persist-tail, cli-export, search-context-cross-session, store.test.ts envelope round-trip: use awaited writeNow or sync-only flushSync — no in-flight writer at teardown.
  • issue1086 T3/T8: foreign sessions never create local state → scheduleSave never called.
  • fix-config-session: sync eviction→flushSync only.
  • proxy-persist, host-usage-postfold: teardowns either drain (flushAll) or cancel before any write can be in flight; the residual fire-and-forget case is covered by the kernel-side unhandled-rejection fix from release v0.1.42 #158.

The third flake reported in #1208 (ubuntu-22 master@8daba7fd) was identified via the CI job log as issue1086-chain-self-exemption.test.ts T7 (ENOTEMPTY rmdir at teardown) — already fixed on master by 3f61cc1 (#1194); the failing run predated that commit. No action needed.

Behavior changes

None in production code — all three edits are inside tests/. No config schema, wire shape, persistence format, or default changes.

Pre-flight

  • npm run typecheck ✓
  • npm test: 2377 pass / 3 fail — the 3 failures are in tests/claude-native.test.ts (live ps-walk e2e tests) and fail identically on the untouched tree in this sandbox (verified via stash); they are environment-specific here, not regressions. All 3 touched files pass 5/5 standalone repeated runs.
  • npm run build ✓

Note: I did not add blanket CI retry-on-flake (issue suggestion 2) — with the root causes removed deterministically, retries would only mask future real regressions.

@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-24_flaky-test-hardening (5b58d20)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-1224

Each push to this PR publishes a new version under the pr-1224 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1224.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

一句话总结:审核了 PR #1224(#1208 三个 CI flake 的测试加固)——三个根因均在代码层面独立验证成立、对其他 enabled-store 测试的审计结论复核无误、CI 四平台全绿、纯测试改动零生产行为变更,可以合并。

① 原 issue 方向:#1208 建议「逐一定位 + 确定性等待」,本 PR 对三个观察到的 flake 全部照此实施;明确不做建议 2(CI retry-on-flake)——理由成立:根因已确定性消除,retry 只会掩盖未来真实回归,建议 3(标记 known-flaky)随之不需要。第三个 flake(ubuntu-22 master@8daba7fd)确认为 issue1086-chain-self-exemption.test.ts T7 的 ENOTEMPTY rmdir,已由 3f61cc1 (#1194) 修复 —— 已验证该 commit 是本 PR head 的祖先(git merge-base --is-ancestor),无需动作。方向关通过。

② 回归审查(既有行为清点):

  • src/ 零改动 → 生产行为零变化:config schema、wire shape、persistence format、默认值均未触碰。行为变更声明:无(按交付规则显式声明)。
  • settle() 超时语义从「静默继续」改为「throw」:严格改善的失败语义——文件 3s 内不达标就立刻失败并报明确错误,而不是继续跑出后续断言的误导性报错;不可能产生假绿。唯一理论风险是假红(窗口不足),CI 在两个 windows-latest runner 上实测全绿,说明负载下窗口足够。
  • 两处新增 drain(flushAll)只改 teardown/读取前时序,不动任何断言逻辑。
  • 无「修历史问题顺带改语义」类高危模式:改动全部限定在 test-local helper 与 teardown 块内。

③ 代码/机制审查(逐条读了相关代码路径,不只读 diff):

  • logger-rotation:旧 flake 机制确认——src/logger.ts:79 if (stream === s && streamFd === undefined) return false; // open still in flight。旧 settle 初值哨兵 -1 恰好等于「文件不存在」的读值,async open 未在 ~60ms 内完成时提前返回、streamFd 仍 undefined → isOrphaned() 短路为 not-orphaned → clobber 不被检测 → 0 个 warn,与 PR fix: register watcher pid for every native client attaching to a shared proxy (#1190) #1200 windows-24 失败签名(warns.length 0≠1)完全吻合。新前置条件(稳定非零大小 = open 完成 + 行已落盘)使后续路径确定:clobber → statSync(logPath) ENOENT → orphan 检测命中 → openStream() 的 mkdirSync(dirname) 因 dir 已被普通文件顶替抛 ENOTDIR → warnReopenFailed 恰好一次(第二次 log 被 reopenWarned flag 抑制)。机制成立。✓
  • session-gc round-trip:race class 确认——debounceMs: 10 时 turn-1 timer 可在两轮之间触发、启动一个快照早于 turn-2 rawInputTokens 的异步 writeNow,其 rename 可落在同步 flushSync 之后留下旧字节。await store.flushAll([])(src/persist.ts:636)保证 drain 后无竞争写者。与 Flaky: #1101 T7 cleanup races in-flight persist writes — ENOTEMPTY kills the release gate #1194 同模式。✓
  • e2e-export-handoff:new SessionStore({ dir, enabled: true, debounceMs: 0 })(line 38),真实代理流量 markDirty→scheduleSave,finally 原先未 drain 直接删目录——同一 race class,主动加固合理。✓
  • 独立复核了全部 enabled-store 测试(不止信 PR 清单):fix-401-storage / persist-tail / cli-export / search-context-cross-session / store.test.ts 均为 awaited writeNow 或 sync-only flushSync;issue404 / persist-encryption / persist-zstd / persist-rollback-guard / proxy-persist / host-usage-postfold / side-request-isolation 均在 rmSync 前 store.cancelAll();fix-config-session 的 eviction 路径是 getStore().flushSync(s)(src/session.ts:660)同步执行;absorb.test.ts 用同步 flushSync/loadSync 且无 rmSync;store-marker-gate 不删目录。「Not changed (verified safe)」一节成立。✓

④ 测试有效性:三处都是真测试、断言有牙——warns.length===1 + stderr-only 降级 + 文件不复活;rawInputTokens 持久化 round-trip 相等性;export handoff 标记断言。把此前非确定性的行为钉成了确定性回归防护。说明:原 flake 是 Windows 负载下的时序竞争,Linux 本地无法复现原始失败;证据链 = 机制代码级验证 + 本 head 在两个 windows-latest runner 的全量 CI 实跑。kernel 侧相关缺陷已有 acp-kernel#373 单独跟踪,本 PR 用 drain 规避,跨仓部分不在本 PR 范围。

⑤ diff 整洁度:+11/−2、3 个文件、单个线性 commit 直接坐在当前 master head(aae4de1c)上,每行都与目的相关,注释统一带 #1208 出处。无 version bump、无 lockfile/生成物 churn。干净。

验证证据:

  • GitHub API:PR open,mergeable_state: clean,base = 当前 master head。
  • CI check-runs @ 5b58d20:8/8 success(test windows-latest 22/24、ubuntu-latest 22/24、version-guard、build-artifact、codex e2e ×2)。
  • 本地(PR head):typecheck ✓、build ✓、全量套件 2377 pass / 3 fail——3 个失败均为 claude-native.test.ts 的 live ps-walk e2e,本沙箱环境特有问题(四个 CI 平台上均通过,非回归);三个被触碰文件连续 5 轮各 15/15 全过。

结论:可以合并。按惯例 Agent 不合并 PR,请人工操作:#1224

@ranxianglei
ranxianglei merged commit a10f872 into master Sep 23, 2026
8 checks passed
@ranxianglei
ranxianglei deleted the 2026-09-24_flaky-test-hardening branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI 测试套件存在间歇性单测 flake(不同 runner/不同测试随机失败),需排查并加固

1 participant