Skip to content

fix(tests): settle dsh-native attach chains before teardown (#1791) - #1798

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-01_dsh-test-flake-settle
Oct 1, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-01_dsh-test-flake-settle

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why

Pre-existing file-level flakiness in tests/dsh-native.test.ts, unrelated to #1785: intermittent reds rotating between #983 maybeRetry self-heals a base-less register after a failed respawn and apply() runtime-info (#956): a mid-resolve model switch discards the stale resolve, plus one windows/22 CI occurrence (check-run 109983788889) with the fast throw bili: proxy is down — recovery in progress, retry shortly at tests/dsh-native.test.ts:1028.

Root cause (reproduced locally with a timestamped console trace): the #1772 web-profile caveat test fires four attach chains through apply() and tears down (proxy.close()) without settling any of them. The deferred health probes then hit the closed mock → ECONNREFUSED → verifyAttachAndRecover runs its spawn fallback inside later tests' bodies (that test installs no spawn stub, so it reaches the real bootstrap() and spawns actual bili child processes, which exit code 1) and clobbers the module-shared register.base / register.toolsReady / state.attach / preset env mid-test. Whichever test's early window the clobber lands in goes red — hence the rotating failures. This is the same pollution class the #1117 test's house comment already documents ("a deferred probe hitting a closed proxy would run the fallback mid-way through a LATER apply() … sequence pollution exposed when #1130's suites follow this one"); #1772 was added later without that discipline.

#956 itself has no internal race — its assertion sequence is microtask-deterministic by static analysis (the stale flight A's discard chain is pure microtasks and drains before the 20 ms timer; flight B cannot start until a later headersFor poll). Its reds are the upstream clobber landing in its window. So: test-hygiene defect, not a production race — src/ is untouched.

Fix (test file only)

  1. dsh.bundle.patch.yml 禁用自动压缩在 web profile 下静默失效 #1772: hermetic fallback seam — _setSpawnForTest(async () => undefined) (cleared in finally) so a deferred probe can never reach a real bootstrap child process; and settle each of the four apply chains while the mock is still open: await _stateToolsReadyForTest() after each phase, with _resetRegisterForTest(proxy.origin) before phases 2/4 so each [Enhancement] E2E native lane: real dsh + bili profile vs deterministic fake upstream (from #1239) #1268 gate strictly waits its own chain (otherwise the previous chain's toolsReady=true lets the next gate resolve immediately and leaves it dangling again). All original assertions unchanged.
  2. Five latent instances of the identical pattern (single apply, mock closed before the chain settled) get the same one-line settle await before teardown: apply() /acp-cache (#1146), the three apply() /acp (#1677) variants, and apply() /acp pre-first-request (#955).
  3. #983c: the fixed sleep(50) becomes settle-based — a counting stub plus await waitFor(() => spawnCalls >= 1, "initial failed respawn settled"). The stub body is synchronous, so once the counter is observable by a macrotask the base-less state (landed.then(base=undefined)) has already drained as microtasks and the single-flight is cleared; the subsequent headersFor deterministically takes the maybeRetry respawn branch. This closes the secondary timeout variant where probe + 30 ms routed-evidence grace + jitter exceeded 50 ms under load.

No production behavior change: src/ untouched, every existing assertion byte-identical.

Evidence

  • Pre-fix, loaded loop (1 CPU spinner, 40 runs of the single file): 39 pass / 1 fail — the failure reproduced the exact CI signature: not ok 29 - #983 maybeRetry self-heals…, error bili: proxy is down — recovery in progress, retry shortly, stack Object.execute (src/agent/dsh-native.ts:348) ← tests/dsh-native.test.ts:1028, duration_ms 88.
  • Post-fix, same loop: 40/40 green.
  • The failing run's timestamped trace shows four attach target http://127.0.0.1:<port> is not healthy — falling back to a spawned proxy lines (one per dsh.bundle.patch.yml 禁用自动压缩在 web profile 下静默失效 #1772 apply) landing after the two following tests had already finished, followed by the proxy bootstrap failed — model traffic goes direct (uncompressed) line inside the #983c window.
  • npm run typecheck clean; full npm test: 3450 pass / 0 fail / 19 skipped (of 3469); npm run build: success.

Coordination

Open PR #1793 edits the same file but only disjoint regions (the waitFor deadline near L407–418 and the two #1365 loops near L1386/L1433). No textual overlap with this change; semantically additive (longer poll deadlines + settled chains). Either merge order is clean.

config surface

None — test-only change; no config fields, env vars, or CLI flags added, renamed, or re-semantics'd.

Fixes #1791

@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-01_dsh-test-flake-settle (77f6527)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1771-pr.1798.381c965

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1798.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR #1798 review 结论(结构化七问)

已拉取分支逐行核对代码、对照源码验证根因链、独立复跑负载测试。CI 12/12 全绿(含 ubuntu/windows × node 22/24 四个 test 矩阵 + codex/opencode/pi/dsh 四条 e2e lane + build-artifact)。

  1. 是什么问题:tests/dsh-native.test.ts 的既有 flake——#983 maybeRetry… 与 apply() runtime-info (#956) 间歇性轮流变红,windows/22 CI 一次出现 bili: proxy is down — recovery in progress, retry shortly 抛出(check-run 109983788889)。与 [tests] dsh-native attach-mode cluster (#1130/#1365) times out / stalls on dev host (suspected environmental) #1785 无关,是独立的测试卫生缺陷。
  2. 什么原因导致的(已对照源码逐层验证):dsh.bundle.patch.yml 禁用自动压缩在 web profile 下静默失效 #1772 web-profile 警告测试通过 apply() 触发四条 attach 链后未 settle 就 teardown。apply() 每次调用都会武装一条全新的 verifyAttachAndRecover 探测链 + 一个全新 [Enhancement] E2E native lane: real dsh + bili profile vs deterministic fake upstream (from #1239) #1268 toolsReady 门(src/agent/dsh-native.ts:583-617,无去重);mock 关闭后延迟探测拿到 ECONNREFUSED → 30ms evidence-grace(文件内 bug: dsh-native — attach 目标瞬时不可达 → 回退 spawn 第二实例:routed baseURL 下工具/模型通道必然脑裂(工具 404 → preflight 502 → 会话不可自愈) #1365 既有 pin)→ 走 spawn fallback,修复前该 fallback 调用真实 bootstrap(),在后续测试体内拉起真 bili 子进程,覆写模块共享的 register.base/toolsReady、state.attach、preset env → 后续测试命中 src/agent/dsh-native.ts:348 的 fast-throw(基线 register.base === undefined)或状态被污染,失败面随负载轮转。CI 栈帧与 :348 精确吻合,根因定位成立,不是表面现象。
  3. 怎么修复的(纯测试文件,+33/-3,src/ 零改动已验证):三管齐下——(a) settle-before-teardown:dsh.bundle.patch.yml 禁用自动压缩在 web profile 下静默失效 #1772 四阶段 + 五个潜伏同类(/acp-cache #1146、三个 /acp #1677 变体、/acp pre-first-request #955)在关 mock 前 await _stateToolsReadyForTest();我逐一核对了每个 settle 点之前都有 _resetRegisterForTest(...)(dsh.bundle.patch.yml 禁用自动压缩在 web profile 下静默失效 #1772 第三阶段的 reset 是 master 既有,非本 PR 新增)——这一点是正确性关键:门只有在 register.toolsReady=false 时才会真正等自己那条链,否则立即 resolve、白等;(b) hermetic 兜底:_setSpawnForTest(async () => undefined) 在 try 前装、finally 清,即使有探测溜过 teardown 也永远到不了真实子进程;(c) #983c 去 sleep 化:sleep(50) → 计数 stub + waitFor(spawnCalls >= 1)。确定性论证成立:stub 体同步自增,其后的失败路径(landed.then 置 base-less)是纯微任务链,而 waitFor 在宏任务上轮询——观察到计数时状态必已排空;30ms grace ≪ 5s 上限,无新竞态。
  4. 是否回归问题:无。逐项清点既有行为:文件内 39 个断言全部原样保留(只把时序从"碰运气"变确定,没有弱化任何断言);新增 spawn seam 在 finally 清理,不泄漏到其他测试;grace 30ms pin、waitFor 5s 上限均为既有值未动;无阈值/时序/输出格式漂移。本地独立复跑:无负载 39/39,8 CPU spinner 压载下 10/10 全绿(作者证据:修复前 1 CPU 压载 39/40 复现精确 CI 签名,修复后 40/40)。
  5. 是否破坏缓存:不涉及(纯测试改动,不触碰任何缓存/重试/游标/去重键策略,生产路径零变更)。
  6. 是否改设计方向:不涉及。遵循既有 「BUG」native-intercept 进程级 globalThis.fetch 补丁按 URL 形状接管所有模型流量,无关调用者(如 MemOS)被卷入宿主压缩流程 #1117/[Enhancement] E2E native lane: real dsh + bili profile vs deterministic fake upstream (from #1239) #1268 测试纪律(settle-before-teardown),无架构/边界/数据流转向。
  7. 是否引入新配置:不涉及(无新 env/config/默认值)。

协调性:#1793 与本 PR 同文件但区域不相交(waitFor 上限 L412-418 一带 + 两处 #1365 循环),两者语义正交(拉长轮询预算 vs teardown 前排空),任意合并顺序均干净。分支基于当前 master(仅领先一个提交),diff 每行都服务于本 PR 目的,整洁。

判定:可合并。 无需补 commit。

@ranxianglei
ranxianglei merged commit a3d7ddc into master Oct 1, 2026
12 checks passed
@ranxianglei
ranxianglei deleted the 2026-10-01_dsh-test-flake-settle branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[tests] dsh-native.test.ts 存在与 #1785 无关的间歇性 flaky 用例(#983 / #956)

1 participant