Skip to content

fix: contain dsh-native attach/recovery chains within their own test (#1797) - #1799

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-01_dsh-native-flake-isolation
Oct 1, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-01_dsh-native-flake-isolation

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Fixes #1797

Problem

test (windows-latest, 22/24) intermittently failed on exactly one test — #983 maybeRetry self-heals a base-less register after a failed respawn (tests/dsh-native.test.ts:1010) — in the two reported forms (proxy is down at src/agent/dsh-native.ts:348; timeout waiting for plugin headers stamped), reproducing on pristine master with ubuntu always green.

Root cause (traced + CI-log forensics)

One mechanism, two timing failures — fire-and-forget attach/recovery chains outliving the test that started them:

  1. Cross-test clobber (the issue's forms A/B). Several suites call apply() and tear down their mock proxy without letting the attach chain settle (worst case: the dsh.bundle.patch.yml 禁用自动压缩在 web profile 下静默失效 #1772 web-profile test calls apply() four times against one proxy). On loaded runners the in-flight probe loses the race with proxy.close() → probe DEAD → 30ms evidence grace → the fallback branch evaluates _spawnForTest ?? bootstrap with no stub installed → a REAL bootstrap() child. Its late tail then lands mid-window of a later test and unconditionally clobbers the shared module-global register.base/toolsReady. On CI the children exit instantly code 1 (ci.yml runs npm test BEFORE npm run build, so no dist/ exists) — accounting for every proxy bootstrap failed … (code 1) line in the windows logs.
  2. Fixed-sleep race (a third form, same family). The self-heal test assumed the whole initial chain (dead-port probe + 30ms grace + failed spawn) settles within a fixed 50ms sleep. If the probe is slower, the first headersFor takes the registration branch against the still-set dead preset, fetchManifest fails, and parks retryAt at +10s — beyond the 5s wait cap — with nothing re-triggering during the wait → timeout waiting for self-healed tool registration. Reproduced deterministically locally by injecting ≥30ms delay into the :1 probe.

Fix (at the decision point, not wider sleeps)

Behavior change

None user-facing. The product-file diff is additive only (a Set + one exported test hook); wire shapes, both compression modes, defaults, thresholds, and persistence are untouched. Test-side changes only alter when async chains are awaited between tests.

Config surface

N/A — no config fields, env vars, or CLI flags added, renamed, or re-semantics'd.

Verification

  • npm run typecheck, npm run build — green
  • npm test full suite — 3470 tests / 3451 pass / 0 fail / 19 skipped (e2e-gated)
  • slow-loopback sweep on tests/dsh-native.test.ts (SLOW_MS ∈ {30,50,100,150} + jitter, ×2 each): 8/8 green — pre-fix the suite failed from 30ms up
  • standalone repro (exact test sequence, injected :1 probe delay 0/30/60/100/200ms ×2): 10/10 pass — pre-fix failed from 30ms up
  • duplicate screening: open+closed issues/PRs searched for flaky/windows/maybeRetry/dsh-native variants — no prior report

Follow-up (disclosed in the #1797 thread)

The investigation exposed a latent product-side race of the same family — verifyAttachAndRecover's fallback landed.then/onGiveUp clobber register.base/toolsReady unconditionally, so a stale older chain can destroy a newer healthy base (reachable in prod only if a profile reload re-applies while a dead-preset bootstrap is still in flight; self-heals via maybeRetry but with a direct/uncompressed window). Suggested fix: generation counter — only the newest chain may clobber. Left to owner whether to file it separately (source chain already one level deep).

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-01_dsh-native-flake-isolation (2566b36)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1811-pr.1799.cda6f92

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1799.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR #1799 复核完成 —— 代码与 CI 已核对无误;但发现一处原始分析未覆盖的关键协调问题,需你拍板后再合。


⚠️ 协调警报(最高优先,先读这个)

本 PR(#1799,源出 #1797)与另外两个同窗口、同源缺陷的 open PR 高度重叠。三者都改 tests/dsh-native.test.ts;其中 #1787 与本 PR 同改 src/agent/dsh-native.ts:

PR 创建时间 来源 性质 改动面
#1787 09-30 16:38 #1783 死锁 + 本族竞态 产品侧:新增 landingOwnsRegister() 所有权守卫——过期的 verifyAttachAndRecover 落地链只在仍「拥有」register 时才写(base undefined 或 === 陈旧 origin);并把 retryAt 退避限定到实际失败的 base(修 windows-22 死锁) dsh-native.ts + 测试
#1798 09-30 19:03 #1791 纯测试:用既有 _stateToolsReadyForTest() gate settle + #1772 加 hermetic spawn-stub seam + #983c 固定 sleep(50) 换 spawn 计数 waitFor 仅 dsh-native.test.ts
#1799 09-30 19:58 #1797 产品+测试(本 PR):pendingChains+trackChain+_settleNativeForTest(),afterEach 统一兜底,5 个 suite 先 settle 再关 proxy,#983 换事件驱动,新增 80ms 慢 probe 回归测试 dsh-native.ts + 测试

核心判定:#1798 与 #1799 是同一根因(跨测试污染共享 register + #983 固定 sleep)的两种竞争修法,都编辑 dsh-native.test.ts 的同一片 finally 块和 #983 用例、且 settle 调用不同(_settleNativeForTest() vs _stateToolsReadyForTest())——二者不能同时干净合并。两个 session 各自做了查重却都报「无重复」,而实际上彼此都早于对方存在(#1798 19:03 < 本 PR 19:58)。这是「One issue = one scope」+ 热文件竞争的标准情形。

建议处置(请你拍板):

  1. fix(dsh): stale verifyAttachAndRecover landings must never clobber the self-healed register (#1783) #1787 独立先行——它才是产品正确性修复(在源头让过期落地不再覆盖新 base,符合 root-cause-not-symptom;同时修掉 release v0.1.176 #1783 死锁)。它与本 PR 文本不相交、语义互补,可先合。
  2. 测试侧二选一(fix(tests): settle dsh-native attach chains before teardown (#1791) #1798 / fix: contain dsh-native attach/recovery chains within their own test (#1797) #1799),别都留。 我倾向 fix: contain dsh-native attach/recovery chains within their own test (#1797) #1799:① afterEach 统一兜底比逐 suite 插桩更健壮(漏插桩的 suite 也能被安全边界接住);② 带针对形态 C(慢 probe 越过短窗口)的真回归测试(注入 80ms、行为断言、修复前 ≥30ms 即确定性失败);③ pendingChains 显式跟踪所有在飞链,而 state.toolsReady gate 每次 apply() 重建、只反映最新一条。其产物改动极小(Set + 一个 _ForTest 钩子,行为中性,符合既有约定)。
  3. 注意:fix(dsh): stale verifyAttachAndRecover landings must never clobber the self-healed register (#1783) #1787 落地后,跨测试 clobber(形态 A)在生产与测试里都变无害(过期落地只能在仍拥有 register 时写);此时测试侧 settle 的价值收敛为 (a) 防本地 dev(dist 存在时)孤儿 bili 子进程,(b) 修 [dsh] native proxyEnvOrigin() 把临时 origin 冻结进 BILLION_CONTEXT_PROXY,绕过 ensureProxyRunning() 的 instance 验活;工具调用全部 fetch failed,需重启宿主 #983 固定 sleep 竞态(形态 B——fix(dsh): stale verifyAttachAndRecover landings must never clobber the self-healed register (#1783) #1787 不碰)。所以无论选哪个,仍需一个测试侧修复。

结构化七问(针对本 PR #1799)

  1. 是什么问题:windows matrix 上恒为同一测试(#983 maybeRetry self-heals)间歇红,两形态(proxy is down @ dsh-native.ts:348 / timeout waiting for plugin headers stamped),pristine master 亦复现,常态红掩盖真实回归、卡合并判定。
  2. 什么原因导致的:fire-and-forget 的 attach/恢复链活过启动它的测试。(a) 跨测试污染:多 suite 调 apply() 后不等链落定就关 mock proxy,慢 runner 上飞行中 probe 输给 close → fallback 求值 _spawnForTest ?? bootstrap 时无 stub → 真 bootstrap 子进程(CI 上 dist 不存在故 code 1 退出,对应日志全部 proxy bootstrap failed (code 1)),其慢尾无条件覆盖模块级共享 register.base/toolsReady;(b) 固定 sleep 竞态:[dsh] native proxyEnvOrigin() 把临时 origin 冻结进 BILLION_CONTEXT_PROXY,绕过 ensureProxyRunning() 的 instance 验活;工具调用全部 fetch failed,需重启宿主 #983 假设初始链 50ms 内落定,probe 一慢首个 headersFor 走注册分支撞死 preset,retryAt 钉到 +10s > 5s 上限 → 超时。ci.yml 顺序已核实(test L43 先于 build L46),支撑 code-1 取证。
  3. 怎么修复的:决策点修而非放宽 sleep——产物侧 pendingChains+trackChain(apply 两处 state.ready=start() + maybeRetry respawn 分支)+ _settleNativeForTest()(等全链 + 一宏任务 flush);测试侧 afterEach 统一兜底、5 suite 先 settle 再关 proxy、[dsh] native proxyEnvOrigin() 把临时 origin 冻结进 BILLION_CONTEXT_PROXY,绕过 ensureProxyRunning() 的 instance 验活;工具调用全部 fetch failed,需重启宿主 #983 换事件驱动、新增 80ms 慢 probe 回归测试。
  4. 是否有回归问题:无。产物 diff 纯增量(trackChain 返回同一 promise,state.ready 语义不变;Set 每条链 settle 自删,有界不泄漏)。高危模式逐项过:无线形/双压缩模式变更、无 fail-fast 变吞错、无阈值/时序/输出格式漂移、持久化不动。CI 全绿(含 windows 22/24、e2e、build-artifact、version-guard)。
  5. 是否破坏缓存:不涉及——未触碰缓存/重试策略/游标/去重键;register.retryAt 语义原样(settle 只改变测试间 await 时机)。
  6. 是否改了设计方向:架构/边界/数据流未转向。但必须并入上面的三 PR 协调决定——单独合本 PR 会与未决的 fix(tests): settle dsh-native attach chains before teardown (#1791) #1798 冲突。
  7. 是否引入新配置:无新增 env/config/CLI flag/默认值。

判定:本 PR 代码健全、CI 全绿,可审;但因与 #1798 冲突,不应独立合并。请你先定 #1798/#1799 取舍(我倾向 #1799)、确认 #1787 先行;定下后我按结果收口(关冗余 PR 或 cherry-pick 增量)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

解决冲突后重新审查一次

@ranxianglei
ranxianglei force-pushed the 2026-10-01_dsh-native-flake-isolation branch from a87b698 to 9f5f6e2 Compare October 1, 2026 05:13
…1797)

The windows-matrix flake in "#983 maybeRetry self-heals a base-less
register after a failed respawn" had two timing failures sharing one
mechanism: fire-and-forget verify chains outliving the test that started
them.

- suites calling apply() tore down their mock proxy while the attach chain
  was still probing; on loaded runners the probe then sees a dead target,
  falls back to an unstubbed real bootstrap(), and the child's late tail
  clobbered the shared module-global register.base/toolsReady mid-window of
  a later test ("proxy is down" at execute / "plugin headers stamped"
  timeout).
- the #983 self-heal test raced its own initial chain with a fixed 50ms
  sleep; a slow dead-port probe left the base set when the first headersFor
  ran, parking retryAt 10s ahead of the 5s wait cap.

Fix at the decision point instead of widening sleeps:
- src/agent/dsh-native.ts: track in-flight attach/recovery flights
  (pendingChains + trackChain at the apply() state.ready sites and the
  maybeRetry respawn branch); expose _settleNativeForTest() awaiting them
  plus a macrotask flush. Behavior-neutral outside tests.
- tests/dsh-native.test.ts: afterEach drains pending chains (universal
  backstop); command-behavior suites settle before closing their mock
  proxies (removes the fallback-to-real-bootstrap window); the #983 test
  waits for initial-chain settlement event-driven instead of sleeping 50ms;
  new regression test pins the slow-dead-port-probe shape by delaying :1
  fetches 80ms in-test (failed deterministically pre-fix at >=30ms delay).

Verified: typecheck + build + full suite (3470 tests, 0 fail) + slow
loopback sweep SLOW_MS=30/50/100/150 x2 (green; pre-fix failed at 30+) +
standalone repro delay sweep 0..200ms x2 (pass; pre-fix failed at 30+).
@ranxianglei
ranxianglei force-pushed the 2026-10-01_dsh-native-flake-isolation branch from 9f5f6e2 to 2566b36 Compare October 1, 2026 05:14
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 冲突已解决,重新审查完成。

处理过程

#1799 现在只剩什么(相对 #1798+#1787)
一层全链边界的纵深防御:


结构化七问

  1. 是什么问题:不是新 bug,是协调问题——fix(tests): settle dsh-native attach chains before teardown (#1791) #1798/fix(dsh): stale verifyAttachAndRecover landings must never clobber the self-healed register (#1783) #1787 合入后 fix: contain dsh-native attach/recovery chains within their own test (#1797) #1799 与 master 文本冲突(mergeable=False),需把它收敛为「相对已合入工作仍独有」的最小增量再复审。
  2. 什么原因导致的:两个并行 session 对同一 flake 各出 PR;owner 先合了 fix(tests): settle dsh-native attach chains before teardown (#1791) #1798(纯测试)+fix(dsh): stale verifyAttachAndRecover landings must never clobber the self-healed register (#1783) #1787(产品守卫)。二者落地后,fix: contain dsh-native attach/recovery chains within their own test (#1797) #1799 的 per-suite settle 与慢 probe 测试已被覆盖,净增量只剩「全链 drain」一处。
  3. 怎么修复的:rebase→union-of-intent 解冲突,保留 fix(tests): settle dsh-native attach chains before teardown (#1791) #1798 既有机制、丢弃 fix: contain dsh-native attach/recovery chains within their own test (#1797) #1799 冗余项,仅保留独有增量(全局 afterEach 全链 drain + 行为中性钩子)。
  4. 是否有回归问题:无。产物 diff 纯增量(trackChain 返回同一 promise、Set 每条链 settle 自删有界、_settleNativeForTest 仅测试调用);afterEach 对已 settle 的 suite 是 no-op。高危模式逐项过:无线形/双压缩模式变更、无 fail-fast 变吞错、无阈值/时序/输出格式漂移、持久化不动。本地证据:typecheck✅ build✅ dsh-native 41/41×7(含负载循环)✅ 全量套件 3463 pass / 0 fail / 19 skip✅。
  5. 是否破坏缓存:不涉及——未触碰缓存/重试策略/游标/去重键;register.retryAt 语义原样。
  6. 是否改了设计方向:未转向。方向仍是「attach/恢复链不越过测试边界」;只是从 fix: contain dsh-native attach/recovery chains within their own test (#1797) #1799 的完整方案收敛为「在 fix(tests): settle dsh-native attach chains before teardown (#1791) #1798 逐 suite settle 之上补一层全链兜底」。
  7. 是否引入新配置:无新增 env/config/CLI flag/默认值。

判定 & 需要你拍板的一点

你选哪个我就照做(A 我等 CI,B 我关 PR 并在此说明)。

@ranxianglei
ranxianglei merged commit a2f2ffb into master Oct 1, 2026
13 checks passed
@ranxianglei
ranxianglei deleted the 2026-10-01_dsh-native-flake-isolation branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI] dsh-native #983 maybeRetry 测试在 Windows runner 上间歇性失败(pristine master 亦复现)

1 participant