Skip to content

autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044

Closed
#3055
Closed
autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits#3044
#3055

Description

@loalissm-cell
loalissm-cell
opened on Oct 13, 2026

Bug report

  • Package: billion-context
  • Version: 0.1.195 (old instance, holding the port) / 0.1.196 (new instances, repeatedly spawned)
  • Host: DSH (DeepSeek Harness), profile web, Windows 11
  • Config: autoRestartOnUpdate: true
  • Severity: medium-high on long-running hosts — silent resource leak accumulating per hour, no crash

Symptom

Disk had 0.1.196 while the running proxy was 0.1.195 (stale: true). autoRestartOnUpdate tried to self-upgrade by respawning. The old 0.1.195 process kept holding its listening port (18789, with 11 live connections from the host parent). Each newly spawned 0.1.196 process failed to bind and silently hopped to the next port (18790, 18791, … 18803) — it neither stopped the old process nor exited with a failure. Result: ~3.8 orphan idle processes leaked per hour over a 9-hour accumulation.

Evidence (single machine, one afternoon)

  • 15 bili processes total, all children of the DSH backend.
  • Only the old instance on 18789 had connections (11). The 14 new processes on 18790–18803 had 0 connections, inFlight=0, stale: true.
  • Combined footprint of the orphans: WorkingSet 3,471 MB / Private 8,374 MB / cumulative CPU 28,183 seconds.
  • Killing the 14 zero-connection processes restored a clean state (kept 18789); a 120 s observation window showed no re-leak (window too short to call it fixed — upgrade intent still pending).

Expected behavior

Either of these would close the leak:

  1. When the target port is occupied by a prior bili instance with the same config (the upgrade scenario — the old instance is supposed to yield), notify/handshake the old instance to exit (e.g. POST to its admin shutdown endpoint), then bind; or
  2. Fail fast with a clear exit code / log line ("port occupied by prior instance, upgrade pending") so the host can decide the retry policy.

Anti-pattern today

Silently hopping to +1 ports turns "upgrade failed" into "a second seemingly-successful instance nobody routes to". The host never learns the upgrade didn't land, stale never converges to false, and the leak accumulates.

Reproduce

  1. Run bili 0.1.195 on port P with live traffic (port occupied).
  2. Let 0.1.196 land on disk with autoRestartOnUpdate: true.
  3. Watch the process table / netstat: each respawn attempt adds a process on P+1, P+2, … while the old process on P never exits.

Suggested direction

In the upgrade path, add a bounded grace handshake: probe the old instance's status endpoint, request shutdown, wait for exit (with a timeout), then bind. This would also fix "stale never converges" without host-side cooperation.

Happy to attach bili.log excerpts (startup + port-hop sequence for each spawned instance) if useful.

Activity

  1. ranxianglei commented on Oct 13, 2026

    @ranxianglei
    ranxianglei
    on Oct 13, 2026
    Owner

    🤖 Powered by ework · qwen3.8-27b

    收到,正在按「先审核再动手」的流程查证这个泄漏。已确认三块相关机制的代码位置(self-restart 交接、EADDRINUSE 端口阶梯、DSH 插件的 respawn 回路),目前正在核对:孤儿进程到底是谁 spawn 的(自重启子进程 vs DSH 宿主 respawn)、为什么新实例在旧实例已关监听的情况下仍看到端口被占、以及为什么跑着 0.1.196 的孤儿仍报 stale:true。有结论会在这里回复(含根因分层和修复方案对比),修复走 PR 引用本 issue。

  2. ranxianglei commented on Oct 13, 2026

    @ranxianglei
    ranxianglei
    on Oct 13, 2026
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 排查完成，修复已实现。PR: #3055

    按八问逐项回答：

    1. 是什么问题：自动升级重启时，旧进程还占着端口，新进程不退出、而是静默换到下一个端口继续跑——成为没人路由到的「闲置孤儿」；旧进程永远不退出，孤儿按小时累积（你报告的 14 个孤儿 / ~8 GB 私存就是这个形态）。
    2. 什么原因导致的（根因分层）：自重启派生的替换进程继承了启动器子进程的「端口升级权限」（spawn 环境变量），目标端口 EADDRINUSE 时按既有规则升 +1 而不是失败——这是孤儿的直接生产者。叠加两个复合缺陷：父进程用对旧端口的裸 TCP 探测判定「重启成功」，分不清「我的子进程在听」和「别的进程先抢了端口」；失败后 kill 子进程不验证是否真死，且父进程恢复服务时走同一套升级逻辑，旧代码副本也能漂到新端口变成第二个僵尸。三个缺陷同属一个机制域（占用端口上的交接）。
    3. 影响范围与严重程度：开了 autoRestartOnUpdate 的长驻宿主（如你的 DSH profile 场景）；触发条件是升级重启窗口与端口占用重叠；后果是静默资源泄漏（CPU/内存持续累积）、stale 永不收敛、宿主无法感知升级未落地。无崩溃。中高危，与你评估一致。
    4. 怎么修复的：让替换进程「绑定即成功、否则大声死掉」。spawn 时标记它为升级替换（内部 env），保留对旧实例释放的有界等待（port-zone 重启竞态：sticky 端口单调漂移，升级依附与无界自增互相矛盾（#1660 后续） #1723），之后仍拿不到精确端口 → 立即 exit(1) + 清晰日志行（"must own port N exactly … Exiting instead of hopping to N+1"）——即你的期望行为 fix: address all code review findings (condense, routing, session-id, dead code) #2，在源头实现而不是过滤症状。同时父进程改按替换进程自己的注册表记录（pid + 实际绑定端口）验证交接：绑错端口 → 终止并验证死亡、恢复服务；已有另一个存活 bili 实例占着该端口 → 视为「升级已在别处落地」、父进程干净退出（不再让旧代码副本复活成僵尸）；kill 不生效 → 大声升级并给出手动清理提示。
    5. 是否有回归问题：非替换路径的 EADDRINUSE 决策抽成纯函数，矩阵测试逐条钉住既有行为（launcher 子进程升级、strict-port、手动启动均不变）；本地定向套件全绿（listen-policy + auto-restart 18/18；相邻 port-zone / instance-governance / wire-contract 51/51）；typecheck 干净；全量交给 CI。有意行为变更（旧→新）：(a) 替换进程静默跳端口 → 显式 fail-fast exit(1)；(b) 交接失败后父进程无条件恢复 → 若另一存活 bili 已占端口则退出；(c) kill 不验证 → 验证 + 大声升级；(d) 成功判定从裸 TCP → 子进程自己的注册记录。逐项理由见 PR 表格。
    6. 是否破坏缓存：不涉及——缓存/重试/增量游标/去重键策略均未触碰；仅 spawn 环境合并多一个标记位，继承内容不变。
    7. 是否改了设计方向：架构不变。一个有意的边界决策：外部非 bili 进程占端口时保留原有恢复行为（本修复只针对「同配置前任实例」的升级场景，不想改变一般端口冲突语义）。你期望行为 fix: keep system prompt byte-stable for prefix cache #1 的握手方案（bind 前经管理端点请旧实例退出）需要新增管理端点 + 跨进程 kill 权限——共享代理上可能还有其他客户端，属产品级决策，本次未实现，已在 PR 中标注为 owner 决策的后续项。
    8. 是否引入新配置：一个新内部 env BILI_UPGRADE_REPLACEMENT（派生通道类别，由自重启自动设置，用户/宿主无需配置），已在 CONFIGURATION.md 中英文档化。无配置文件字段、无 CLI flag。

    另有一处需要你的日志确认：「所有孤儿父进程都是 DSH backend」这一证据暗示可能还有宿主侧竞态一层——drain 窗口内 dsh 插件健康探测失败会触发第二次 spawn，与替换进程抢同一个端口。本 PR 已堵住代理侧的泄漏机制（无论谁触发，跳端口孤儿都不会再产生）；但要确认宿主侧触发链路，麻烦附上你提到的 bili.log 摘录（各 spawned 实例的 startup + port-hop 序列），我在这个 issue 里跟进确认是否还需要宿主侧处理。

    判定：PR #3055 可审，合并后代理侧泄漏机制即被堵住。

  3. ranxianglei
    added 8 commits that reference this issue on Oct 13, 2026

    fix: self-restart replacement binds-or-dies on the freed port instead…

    f6acfec

    fix: deny post-bind ladder rights to marked upgrade replacements (#3044…

    61cbc4a

    fix: readiness peer check must not read our own stale registry record (…

    2a86aca

    fix: drop unused EaddrinuseAction export (unused-exports gate) (#3044)

    5d63510

    fix: self-restart replacement binds-or-dies on the freed port instead…

    ac50fd2

    fix: deny post-bind ladder rights to marked upgrade replacements (#3044…

    83ea416

    fix: readiness peer check must not read our own stale registry record (…

    81ad157

    fix: drop unused EaddrinuseAction export (unused-exports gate) (#3044)

    b33ca09

  4. loalissm-cell commented on Oct 14, 2026

    @loalissm-cell
    loalissm-cell
    on Oct 14, 2026
    Author

    Follow-up: recurrence on a clean restart (0.1.197) + new process-level evidence of the silent port hop, plus a second live instance contending on the shared state dir

    Original report: this issue (#3044). Today's recurrence adds three new facts, all from a clean host restart (no residue from yesterday's manual cleanup):

    Timeline (today, single host)

    • 18:04:59 — dsh web backend restarted. The dsh plugin channel updated the profile underneath: profiles/web/node_modules/billion-context now 0.1.197.
    • 18:10:50 — bili spawned with cmdline ... start --host 127.0.0.1 --port 18811 → actually listening on 18812 (18811 was occupied). This became the working instance.
    • ~20:10 — host-side audit found 4 bili processes total (18810/18811 dead orphans, killed manually; 18812 working). ~1 orphan per hour, same as yesterday's rate.
    • 20:12:04 — the backend spawned a replacement (after the manual kills), cmdline --port 18812 → listening on 18813 (occupied by the working instance). Traffic re-pointed to it; the 18:10 instance (2.7 GB WS, 5,330 s CPU) became the next orphan with only stale connections left.

    So the leak is not a one-off: every respawn race leaks one full instance (0.5–2.7 GB WS each) and advances the port cursor permanently.

    New fact 1 — intended port ≠ bound port, visible in process cmdlines

    PID A: cmdline "--port 18811"  → LISTEN 18812
PID B: cmdline "--port 18812"  → LISTEN 18813

    Direct proof the new instance silently hops instead of failing or yielding.

    New fact 2 — the port-zone cursor persists and never recycles

    State file after today: port-zone.json = {"lanes":{"dsh":18813}}. The cursor is a high-water mark: hops consume ports globally and forever, even across restarts, even though the failed binds never actually served traffic.

    New fact 3 — two LIVE instances contend on the shared state dir

    The shared log is currently spamming:

    [persist] rejected stale snapshot for session <id>: in-memory copy is older than
the on-disk one (another bili instance holds newer state) — keep...

    That's the working instance and the newest replacement both writing session snapshots concurrently. Beyond the memory leak, the respawn race creates a dual-writer window on the session store — the stale-snapshot guard catches it, but each rejected write is a real save that didn't happen.

    Trigger path, refined

    • Update check runs every 180s ([update] auto-update enabled (checking every 180s)).
    • Self-update is correctly skipped for dsh-managed dirs ([update] install dir is managed by dsh ... update it via the dsh plugin channel).
    • But the dsh plugin channel updates the profile dir underneath a running instance; autoRestartOnUpdate sees diskVersion != runningVersion and respawns without first stopping the old instance or checking whether its target port is free. Each such respawn leaks one instance and bumps the zone cursor. Respawns triggered by external kills (watchdog/manual) race the same way.

    Suggested fix, sharpened from yesterday's proposal

    1. Version-aware yield on bind conflict: before binding, probe the occupant's status endpoint; if it's a same-config bili instance, the older process exits gracefully and the newer binds (or vice versa — one deterministic rule). Never hop.
    2. Never advance the zone cursor on a failed bind — only after a successful bind.
    3. Restart = stop-then-start: autoRestartOnUpdate should request the old instance's shutdown (it exposes status/admin endpoints) and wait for exit before spawning, closing the dual-writer window too.

    Happy to keep feeding measurements — the host is live and reproduces this on every plugin-channel update.

  5. ranxianglei
    added a commit that references this issue on Oct 14, 2026

    fix: duplicate spawn against a live same-build instance exits instead…

    d0ee613

  6. ranxianglei commented on Oct 14, 2026

    @ranxianglei
    ranxianglei
    on Oct 14, 2026
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Thanks for the measurements — today's recurrence is exactly the kind of evidence this needed. Three conclusions up front:

    1. The recurrence was expected. The fix lives in PR #3055, which is still open — v0.1.197 predates it entirely. So today's events do not falsify the diagnosis; they extend it.

    2. Your evidence exposed a SECOND trigger path that the first draft of the fix did not cover — now fixed in the same PR. Yesterday's shape was the self-restart replacement hopping. Today's 20:12 event (--port 18812 → LISTEN 18813) is different: a host respawn raced its own attach scan against a live same-build instance. Launcher-spawned children carry no replacement marker, and the predecessor-wait only matches different-build holders — so a same-build twin sent them straight down the port ladder. That's your fact 1, confirmed in code (src/launcher.ts spawn env, findSameLanePredecessor predicate).

    The new rule (commit d0ee61304 on the PR branch): if the port you're trying to bind is held by a live bili instance running your own build on the same lane, you exit 1 immediately with a loud log line — "port N is already served by a same-build bili instance (pid …) — exiting instead of hopping: the duplicate loses and the serving instance keeps serving; the next launch will attach to it." No waiting, no ladder, applies to marked and unmarked spawns alike. The deterministic rule is older/serving wins: tearing down a live-serving instance to install an identical fresh one buys nothing, and it needs no admin endpoint or cross-process kill authority. Convergence after the rule: redundant spawn dies loudly → the launcher surfaces the failure → the dsh retry's next attach scan finds the live twin and attaches. No orphan, no cursor advance (an exiting child never binds). A live end-to-end test replaying your exact 20:12 event now runs in the CI lane.

    On your sharpened proposals:

    • fix: keep system prompt byte-stable for prefix cache #1 version-aware yield — implemented for the same-build case as above (you don't need a handshake when the winner is obvious: the serving twin). For true upgrade overlap (different builds), the existing design already makes the older instance yield by draining first; the new part only removes its silent-hop escape hatch. The "newer binds via requested shutdown" variant needs a brand-new admin shutdown endpoint plus cross-process kill authority over a shared proxy other clients ride — that's a product decision, flagged in the PR as out-of-scope follow-up rather than built unilaterally.
    • fix: address all code review findings (condense, routing, session-id, dead code) #2 zone cursor — deliberately NOT changed here. Making port-zone.json stop ratcheting on drifted binds changes persistence semantics that were set intentionally ([Design] 定死端口 lane(claude 48787 / zcode 48789)端口被占时的 0 配置解法 #1660: sticky follows drift so repeat launches find the daemon fast). Options: (i) don't settle sticky on a drifted bind — repeat launches re-ladder while the squatter persists, and the cursor recovers on its own when the squatter leaves (my recommendation); (ii) keep the ratchet and add a recycle command. This one is yours/owner's to call — say the word and it's a small separate change.
    • Use kernel renderTags:"text-only" + forbid tag echoing in prompt #3 stop-then-start — same surface as fix: keep system prompt byte-stable for prefix cache #1's handshake (new admin endpoint); covered under that follow-up. On the dual-writer spam (fact 3): the root cause is simply two instances coexisting, which is the leak itself. I checked the guard you quoted (src/persist.ts): it converges the older writer's memory TO the newer disk state, so there's no corruption — just the orphan's stale view being dropped. Once coexistence is eliminated at the source, the spam disappears with it; the guard stays as defense-in-depth.

    Fleet transparency: while I was on this, a parallel review session pushed two complementary follow-ups onto the same PR branch — denying post-bind ladder rights to marked replacements (a second hop path found via #2947/#2941) and fixing the readiness check reading its own stale registry record (#2895) — plus absorbing the current master merge. I rebased my commit onto that state, resolved the conflicts as a union, and re-ran verification green on the rebased head (typecheck clean; targeted 21/21; adjacent suites 51/51). Full-suite verdict is CI's to give.

    Still open, and your logs would close it: why did the original attach scan miss the live 18812 instance? The fix makes the redundant spawn harmless regardless of the answer, but I'd like the trigger nailed. Specifically, from bili.log around 18:10 and 20:12: which component spawned each process (a [restart] line = self-restart, vs a dsh-side respawn line), and any attach-refusal or dual-instance warning lines near those timestamps. You offered excerpts earlier — these two windows would be the ones.

    结构化八问

    1. 是什么问题:自动重启/宿主重生时,新进程发现目标端口被占,不报错也不让位,悄悄换到下一个端口空转——旧进程不退、新进程没人用,每小时漏一个完整实例(今天复现还带来端口游标永久上爬和双写冲突日志)。
    2. 什么原因导致的:两条触发路径,同一机制族。①自重启替换进程继承了启动器的"端口忙就+1"梯级策略,没有任何标记要求它必须独占原端口;②宿主(dsh 插件)在旧实例还活着时重新拉起新实例,而它的"先附着已有实例"扫描没扫到活着的同版本实例(触发原因待日志确认),于是走了梯级跳端口。父进程的交接验证只探原始 TCP,分不清应答来自谁,kill 结果也不校验。
    3. 影响范围与严重程度:所有开了 autoRestartOnUpdate 的长驻宿主 + 插件通道会原地更新 profile 的场景(你的 DSH web 配置即典型)。严重:内存/CPU 持续泄漏(单实例 0.5–2.7 GB)、端口游标单调消耗、会话存储双写窗口;无崩溃但永不收敛。
    4. 怎么修复的:在决策源头加确定性规则,而不是过滤症状。(a)自重启替换进程打内部标记,必须绑中精确端口(保留有界的前辈等待),否则 exit(1)+清晰日志;(b)同版本同车道活实例占着端口时,后来者一律大声退出,服务中的实例胜出(今天的复发场景);(c)父进程改用替换进程自己的注册表记录(pid+实际绑定端口)验证交接,错端口就杀掉并验证死亡,端口已被别的 bili 实例接管则干净退出而非复活成僵尸;(d)kill 结果校验,杀不死就大声升级提示手动清理。
    5. 是否有回归问题:逐条审过既有行为——非替换、非双生的启动器子进程/strict-port/手动启动路径与原内联处理器逐字节一致(策略矩阵测试钉住);外部非 bili 进程占端口时保留原有 resume 行为(有意边界);port-zone 重启竞态：sticky 端口单调漂移，升级依附与无界自增互相矛盾（#1660 后续） #1723 前辈等待语义不变。行为变更 a–f 六项已全部列在 PR 描述表格里(旧值→新值+原因)。本地定向测试全绿(typecheck + 21/21 + 相邻套件 51/51),全量回归以 CI 为准。
    6. 是否破坏缓存:不涉及——改动只在 listen 失败决策与自重启交接路径,不触碰压缩缓存、前缀缓存锚点、增量游标或去重键。
    7. 是否改了设计方向:没有转向,是把"端口冲突"从隐式梯级改为显式决策表(纯函数可钉测)。握手式 shutdown/停旧起新(admin 端点+跨进程杀权)判定为新功能面,未擅自实现,留 owner 拍板。
    8. 是否引入了新的配置:一个内部 env BILI_UPGRADE_REPLACEMENT(仅自重启派生通道使用,用户/宿主不应设置),已按 en+zh 文档规范生成进 CONFIGURATION.md;无配置字段、无 CLI 旗标、无默认值变更。

    收工判定:PR #3055 已更新到覆盖两条触发路径(commit d0ee613,含并行评审补上的两个后续修复与 master 合并),等 CI 绿灯后即可合入;合并发版后你这台主机在下次插件通道更新时应观察到"大声退出+附着存活实例"而非新增孤儿进程。端口游标是否回收、握手式停旧起新做不做,等你和 owner 一句话。

  7. ranxianglei
    added a commit that references this issue on Oct 14, 2026

    Merge pull request #3055 from ranxianglei/2026-10-13_fix-3044-replace…

    bd2aa6e
  8. ranxianglei
    closed this as completedin #3055on Oct 14, 2026
  9. ranxianglei
    added 5 commits that reference this issue on Oct 14, 2026

    ci: per-file zone-port bands for live suites + deterministic teardown…

    1a320fb

    fix(web): serve /__bili lists from the unified index and bound boot p…

    c0d89e2

    fix: escalate replacement teardown SIGTERM -> SIGKILL before manual-c…

    36bb80a

    fix: escalate replacement teardown SIGTERM -> SIGKILL before manual-c…

    c49948e

    test: port-zone #3044 leg accepts the #3654 twin-adopt exit code 3

    4c9123f
  10. aolshevskiy
    added a commit that references this issue on Oct 19, 2026

    fix(ranxianglei#3801): ladder off config-drifted same-lane holders in…

    c368abb
  11. z0d0dOr
    added 2 commits that reference this issue on Oct 24, 2026

    feat: concurrent bili codex sessions on one CODEX_HOME — per-session …

    786c0fc

    feat: concurrent bili codex sessions on one CODEX_HOME — per-session …

    b7400e9
  12. ranxianglei
    added a commit that references this issue on Oct 28, 2026

    fix: stop draining proxies from claiming their freed port — stale-twi…

    350e833
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions