Skip to content

[PR] fix(storage): refuse gc full-VACUUM while another live process holds the db (#4612) - #4628

Merged
ranxianglei merged 5 commits into
masterfrom
2026-10-25_4612-gc-online-guard
Oct 30, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-10-25_4612-gc-online-guard

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why

Field report (bili 0.1.204, 14 GB index.db, #4612): running bili storage gc online while a proxy serves from the same sessions dir — the default --vacuum full rewrites the whole database under an exclusive write lock for O(db size) with no intermediate progress (SQLite VACUUM is atomic). The user saw 14m25s of silence, and meanwhile the live proxy's unified shadow layer got 3-striked out (database is locked ×3 → legacy-only for its whole process lifetime).

This PR implements the issue's proposals 1+2+3:

  1. Preflight guard — UnifiedStore.gc() refuses to start a full VACUUM when another live process holds the db, reusing the existing Storage Reliability Trio: open-time self-check (quick_check) + online backup (VACUUM INTO) + second-process probe on the | 存储可靠性三件套:开库自检 quick_check + 在线备份 VACUUM INTO + 同库第二进程探测 #4175 <db>.lock pid discipline: the holder captured at open (revalidated for liveness at gc time) ∪ a fresh lockfile read (catches an opener that started after us). The CLI exits 1 with an actionable message naming the pid; the panel endpoint returns HTTP 409. No new flags/env/config surface.
  2. Progress — the zero-ref delete sweep now commits in bounded batches (rowid subselect; SQLite DELETE takes no LIMIT clause) with per-batch progress lines (gc zero-ref sweep: 12000/82856 rows (14%)); the batch grows with the table so log volume stays bounded. Full-vacuum start/finish lines report main+WAL size and elapsed time.
  3. Docs — README en+zh: full vacuum on large dbs is an offline operation; use --vacuum incremental for online-safe shrink.

Behavior changes (old → new)

Surface Old New
storage gc --vacuum full with a live foreign writer silently contended: O(db) exclusive lock hold, no output, peer's storage layer degraded until restart refuses up front: CLI exit 1 + actionable message / panel HTTP 409, nothing executed
Zero-ref sweep transaction granularity one unbounded DELETE statement (one write-lock hold across the whole scan) ≤1000-row commit batches (floor), lock released between batches; crash leaves garbage that the next sweep collects (rows are refcount≤0 by definition)
gc log surface one line before the vacuum sweep progress per batch + vacuum sizing/elapsed lines

Unchanged by design: --vacuum incremental and no-vacuum sweeps stay online (short transactions, existing busy-timeout handling); dry-run stays read-only even with a writer present; sessions rm's graceful busy-skip of its shrink step; the lockfile protocol itself.

Known residual (pre-existing, out of scope): the lockfile is single-slot last-writer-wins, so if an unrelated opener clobbers the file between the peer's start and our open, the peer is undetectable via this signal — same limitation #4175's open-time detection already accepts.

Tests

New tests/issue4612-gc-online-guard.test.ts (a spawned node child stands in for the running proxy): open-time holder capture + refusal + incremental exemption + liveness re-validation on exit; later-opener detection via fresh lockfile read; batched sweep outcome + ≥2 progress lines ending at 100%; CLI code 1 / json fields / dry-run exemption / success after exit. Local: typecheck + build clean; targeted suites green (issue4612 + storage-maintenance-cli + storage-archive-gc + storage-backend-contract + storage-review-p2s = 29 pass, 1 env-gated skip). Full-suite verdict: CI.

Closes #4612

ci-check added 3 commits October 25, 2026 07:06
…the db (#4612)

Field report (bili 0.1.204, 14 GB index.db): an online 'storage gc'
(vacuum=full) held the exclusive write lock across the whole O(db-size)
VACUUM rewrite for 14 minutes with zero progress output, starving the
running proxy until its unified shadow layer degraded (3 consecutive
'database is locked' failures -> legacy-only for the process lifetime).

- UnifiedStore.activeForeignWriter(): union of the open-time captured
  #4175 lock holder (revalidated for liveness) and a fresh <db>.lock read
  (catches an opener that started after us); fail-open on unreadable file.
- UnifiedStore.gc(): vacuum:'full' refuses up front when a live foreign
  writer is present (actionable error naming the pid); incremental /
  no-vacuum sweeps are short transactions and stay allowed.
- Zero-ref sweep now commits in bounded batches (rowid subselect; SQLite
  DELETE takes no LIMIT) with per-batch progress lines; batch grows with
  the table so log volume stays bounded. Crash-safe: rows are garbage by
  definition, leftovers are collected by the next sweep.
- Full-vacuum sizing logs: main/WAL size before, elapsed + size delta
  after.
- CLI lane surfaces the refusal as exit 1 + actionable text; panel
  endpoint returns 409. Dry-run stays read-only.
Dummy live process (spawned node child) stands in for the running
proxy: open-time holder capture, later-opener lockfile detection, liveness
re-validation on exit, incremental/dry-run exemptions, batched zero-ref
sweep outcome + progress lines, and the CLI refusal surface (code 1,
actionable text, json fields).
README en+zh: 'storage gc --vacuum full' refuses to start while another
live process holds the database (CLI exit 1 / panel HTTP 409); use
--vacuum incremental for online-safe shrink.
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-25_4612-gc-online-guard (f5fd932)

Latest build — install this exact version

npm install -g billion-context@0.0.5626-pr.4628.36238b1

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
f5fd932 billion-context@0.0.5626-pr.4628.36238b1 build
d08d4e6 billion-context@0.0.5620-pr.4628.daec416 build
822df1f billion-context@0.0.5616-pr.4628.50d076b build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4628.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix(storage): refuse gc full-VACUUM while another live process holds the db (#4612) [PR] fix(storage): refuse gc full-VACUUM while another live process holds the db (#4612) Oct 24, 2026
The new gc online-guard test spawns an idle node child as its writer
fixture; the run-ci-tests SPAWN_RE tripwire requires every such suite in
LIVE_FILES or it aborts all shards before any test runs (#4604 class).
…ted red)

Unused-exports gate fails on every PR head rebased onto master tip
146b34f with the same line master itself fails on:
INTERNAL_ONLY src/agent/client-build-note.ts: ClientBuildCheck
(#4545 train left the type exported though nothing outside the file
imports it; missed by the 4d34abf de-export sweep). One-word fix,
identical to #4606 commit 1; drop this commit when rebasing after
#4606 merges.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——本次是 CI 自动修复车道(attempt 1/2),失败签名 INTERNAL_ONLY src/agent/client-build-note.ts: ClientBuildCheck 与 master 自身失败的 check-run 逐字相同、与 #4599 盘点第 1 条同一根因;按 CI 协调政策未开新 issue。

结论先行: 这条红是 master 继承性失败,不是本 PR 代码的问题。已在分支上以独立标注小修 commit 修掉(一词去 export),等 CI 重触发确认。

验证过程

  • 拉取失败 job 日志(run 40141525559):唯一报错行即上面那条,门禁提示「drop the leading export (INTERNAL_ONLY)」;
  • 本地在 PR head 复现同一输出;再核对 src/agent/client-build-note.ts 与 master tip(146b34f27)逐字节相同,且 master 自身的 check-run 也红在同一行 → 确认继承;
  • 全仓 grep:ClientBuildCheck 只在本文件内被引用(type-only 类型,零外部引用、零运行时影响)。本 PR 的 diff(9 个文件,gc VACUUM 守卫 + 测试 + 文档)不触及该文件及其任何引用方。

处置

结构化八问

  1. 是什么问题:PR 检查矩阵里 unused-exports 红条——master 自身带一个只在本文件使用却带 export 的类型 ClientBuildCheck,每个 rebase 到 master tip 的 PR 都继承这条红。
  2. 什么原因导致的:[PR] dsh lane: note in-session when running an unverified @deepseek-ai/dsh client build (soft, exact-match) #4545 train 引入该类型时带了 export,后续 4d34abfaf 的去导出清扫漏了它;门禁判 INTERNAL_ONLY 即红。与本 PR 改动无关(文件两树逐字节相同)。
  3. 影响范围与严重程度:所有 rebase 到 master tip 146b34f27 的 PR 的检查矩阵(信号级问题,无用户侧功能影响)。叠加 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 哨兵 attempt 3 报告的合并死锁(required checks 覆盖全部 14 个 test 分片 + born-red 集群 → 当前任何 PR 都走不了正常合并),整体解冻需 owner 拍板(哨兵建议选项 A:override + 合并 [PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606)。
  4. 怎么修复的:一词去 export(type-only、零运行时影响),独立标注 commit,与在途修复载体 [PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606 同法;重复修复可接受,[PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606 合入后 rebase 丢弃。
  5. 是否有回归问题:无。type-only 符号无外部引用(全仓 grep 核实),去 export 不改任何运行时行为、wire 形状、配置或默认值;本 PR 主修复路径(gc 在线守卫、批量零引用清理、admin 端点)此 commit 一行未动。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有。
  8. 是否引入了新的配置:没有。

收工判定:本车道交付完成——unused-exports 已在分支侧修复并推送,待重触发确认转绿。预期残留红(若有)= test 层撞盘点第 2 条 LIVE_FILES tripwire(tests/issue3890-parent-gone-confirm-drain.test.ts 未登记,11 分片跑测试前退出),属继承红,根治在 #4606;该 tripwire 不在本车道夹带范围(单独修会显形 born-red 集群,超出 auto-fix 边界)。本分支无需进一步动作。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重结论:不重复。 #4628 是修复 #4612 的唯一 PR。相关但不同的条目:#4565(纯文档,给 website Commands 段补 bili sessions / bili storage gc 行)、#4617(fold-substrate vacuum 上报,不同机制)、#4607(post-fold resave 的 block interval bounds)、#3337(web Context Trace)。症状与机制均不同,判非重复。

先给初步核查(完整八问结论随后补,正在收尾验证):

按 CI 协调纪律(#4599):继承性的 master 红不在本 PR 里顺手修、也不为此新开 issue——我会在 #4599 留一行记录,等 master 侧修好后 rebase 本 PR 即自动转绿。

接下来我会:① 复核 master 的 GitHub CI run 坐实「master 也红」;② 过完 persist / admin / shadow / storage-maintenance / 测试文件五个 diff 做回归审查;③ 本地定向跑 issue4612 + storage 相关套件拿直接证据。收完回一条完整八问结论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

结论先行:#4612 的修复本身正确、完整、非回归,代码可合。当前 CI 红全部来自 master 自带的两处「born-red」CI 缺陷,不是本 PR 引入。 其一(unused-exports)已被上一个 run 用独立标注 commit f5fd932ed 在分支内修好、本地门禁验证 clean;其二(LIVE_FILES tripwire)仍卡着,属 master 侧欠账,需你/#4606 拍板处置顺序(末尾给了 A/B 两个选项 + 我的推荐)。

查重:不重复(见上一条回复)。

我这轮做了什么:分支已从 issue 正文的 4-commit 版推进到 head f5fd932ed(多了一个去 export 的 commit),我把 worktree 切到 f5fd932ed 重新复核:逐文件过了 store/persist/shadow/admin/storage-maintenance + 测试 + live-files 的 diff,本地双树复现 unused-exports 报错、并在 head 上确认该 gate 已转绿,再拉 CI annotation 定位到唯一真实失败。

结构化八问

  1. 是什么问题:bili storage gc --vacuum full 在另一个活进程(如在线 proxy)写同一 index.db 时,会独占写锁做 O(db 尺寸) 的原子 VACUUM 重写、全程无进度,把对端存储层饿死降级(storage gc full VACUUM holds the write lock across the whole O(db) rewrite with no progress — online proxy shadow disabled mid-run | storage gc 全量 VACUUM 全程持写锁且无进度:在线代理影子层被降级 #4612 现场:14GB 库静默 14m25s,proxy shadow 三连 database is locked 后永久 legacy-only)。
  2. 什么原因导致:full VACUUM 是 SQLite 原子操作、无中间提交点;gc 启动前没有「是否另有活写入方」的预检;zero-ref sweep 又是单条无界 DELETE(整段扫描持一把写锁)。
  3. 影响范围与严重程度:仅当「online + --vacuum full + 存在活的外部落盘进程」三者同时成立才触发;触发后果严重(对端存储层降级到重启)。p1 级可用性缺陷,但触发面窄——离线 gc 或 --vacuum incremental 均不受影响。
  4. 怎么修的:① 预检守卫——复用 Storage Reliability Trio: open-time self-check (quick_check) + online backup (VACUUM INTO) + second-process probe on the | 存储可靠性三件套:开库自检 quick_check + 在线备份 VACUUM INTO + 同库第二进程探测 #4175 <db>.lock pid 纪律,open 时捕获的 holder 在 gc 时重验存活 ∪ gc 时 fresh 读一次 lockfile(抓「我们之后才开」的 opener),有活写入方即拒绝 full VACUUM(CLI exit 1 带 pid 的可操作提示 / panel HTTP 409);② sweep 改 ≤1000 行分批提交 + 每批进度日志,full VACUUM 首尾打 main+WAL 尺寸与耗时;③ README en/zh 说明大库 full vacuum 属离线索引。fail-open:拿不到活写入方的确凿证据就不拦,正常单实例 gc 行为不变。
  5. 是否有回归:无。改动全部 gated 在 vacuum==="full" + 「确有活写入方证据」双重条件;incremental / no-vacuum / dry-run 路径逐一核对保持原样(dry-run 在 CLI lane 于守卫之前就返回,恒只读);UnifiedStore 构造函数加 path 参数只有 open() 一处调用点、已同步;store.gc() 自身也抛错作兜底。f5fd932ed 只删一个 export 关键字,纯编译期、零运行时影响。
  6. 是否破坏缓存:不涉及。这是存储层维护路径,与请求路径的前缀缓存 / KV / 重试游标 / 去重键完全正交。
  7. 是否改设计方向:不改。沿用既有 Storage Reliability Trio: open-time self-check (quick_check) + online backup (VACUUM INTO) + second-process probe on the | 存储可靠性三件套:开库自检 quick_check + 在线备份 VACUUM INTO + 同库第二进程探测 #4175 锁纪律与既有分层(store→shadow→persist→cli/admin),只加预检 + 分批 DML + 日志。
  8. 是否引入新配置:无。没有新 env / flag / config 字段(测试里设置的都是既有变量)。

CI 现状(head f5fd932ed,run 40144692485)

给 owner 的决策请求(两选项对 #4612 的最终合入代码完全一致,只差 master born-red 在哪落地;差异在干净度 vs 速度)

⚠️ 请勿按现状合并 #4628:head 里带着 f5fd932ed 这个 throwaway commit,直接合会把无关的去 export 带进 master(虽无害但污染 diff,且与 #4606 撞同一行会冲突)。走 A 或 B 收敛后再合。

要执行 B(补那一行)说一声,我马上推。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——本次转发指向旧 head d08d4e635(run 40142343042),是乱序/陈旧通知;该红已由上一轮在分支上修复并推送(commit f5fd932ed),本轮为 attempt 2/2 的状态核实与收工判定。

当前 head f5fd932ed 的 CI 实况(live check-runs 核实):

结构化八问

  1. 是什么问题:PR 检查矩阵剩余红条全部来自 master 继承的 LIVE_FILES tripwire(一个会起子进程的测试文件未登记,杀掉所有测试分片的预检),非本 PR 代码问题。
  2. 什么原因导致的:master tip 146b34f27 的 train 落了该测试文件但漏登记 scripts/live-files.mjs;[PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606 已集中修复,尚未合入。
  3. 影响范围与严重程度:所有 rebase 到该 master 的 PR 的 test 层(信号级,无用户侧功能影响);叠加哨兵报告的合并死锁(required checks × born-red 集群),整体解冻等 owner 拍板(建议选项 A:override + 合并 [PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606)。
  4. 怎么修复的:本车道范围内已做的部分——unused-exports 一词去 export(f5fd932ed,独立标注,[PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606 合入后 rebase 丢弃)。tripwire 登记不在本分支夹带:单独修它会显形约 35 用例的 born-red 集群(kernel/storage/快照渲染敏感面),超出 auto-fix 边界,归 [PR] fix(ci): clear the two master-red gates in one branch — unused-exports (#4602) + the #3890 LIVE_FILES tripwire (#4604) #4606 统一处理。
  5. 是否有回归问题:无。本 PR diff(gc 在线守卫 + 批量零引用清理 + 文档)未触及任何失败路径;失败签名与不含本 PR 改动的干净 master 完全一致。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有。
  8. 是否引入了新的配置:没有。

收工判定:auto-fix 车道 2/2 用尽,交付完成——转发点名的 unused-exports 已在 CI 上确认转绿;剩余红为纯继承(有 #4606 在途修复 + owner 决策待办),本分支无需进一步动作,不再重复小修。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant