Repository navigation
[PR] storage: default unified mode flips to full + six #3135 cleanups (#3818) - #3819
Conversation
- resolveStorageMode: absent-everywhere default shadow -> full (#3818; the shadow dry-run reconciled cleanly since #2815 - JSON envelopes keep being written, legacy remains the one-line escape hatch) - writerOff saves: a mid-build 3-strike sidecar disable now lands one synchronous JSON envelope write in the same tick (#3135 1) - persistence disabled (BILI_PERSIST=0 / enabled:false) is recorded as an intentional opt-out - no fabricated storage-degraded banner (#3135 2) - the storageMode stamp follows the live instance: a dead/absent sidecar stamps legacy so records describe the copy that actually holds data (#3135 3) - verifyDrift + driftedSessions on the /__bili/status wire; drifted sessions serve from the JSON envelope until a clean sample (#3135 4) - self-bootstrap: an unloadable cached engine copy is removed and one pinned sha512 download retried instead of warn-forever-stuck (#3135 5) - bounded boot unions index-only sessions so jsonWriter=off sessions stay enumerable after restart (#3135 6)
ranxianglei
changed the title
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4551-pr.3819.fa5fc06This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3819.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 本 PR(#3819,分支 标签: bug + scope/storage + p3(六项均为存储子系统缺陷,#3135 原级 p3;翻默认是有回滚通道的产品变更,不升档)。 开始审核:已拉取分支逐文件读 diff(knobs/persist/shadow/runtime-status/engine-bootstrap + 三个测试文件 + 文档),接下来做既有行为回归清点、本地定向测试、核对推送头 CI。结论按八问体例在本线程给出。 |
…ded-byte budget (#3818 review) The #3135-6 union in bootBounded re-loaded every indexed id, including envelope-backed ones the file scan had already offered under the decoded-byte budget (#2803). With the full default (#3818) that bypassed maxBytes for every legacy file on disk (bounded-storage warm-budget test red: 2 sessions warmed where <=1 was allowed). Skip ids whose state file still exists; the union now only warms pure-index (jsonWriter=off era) ids, which is its stated purpose.
…3818 review) The flip changed resolveStorageMode's absent-everywhere result from shadow to full, but two suites still asserted the old shadow default: web-storage-banner (requested mode + blob text + healthy-sidecar block) and storage-mode-label (absent-eboth label row). Update expectations to full; explicit-shadow rows untouched.
…review) The #3135-5 retry path cleared the cooldown whenever the download itself succeeded, even if the freshly downloaded engine still would not load (AV quarantine, truncated extract, platform drift). In a persistently broken environment that re-downloads ~11 MB on every boot and reported status "downloaded" for an engine that never registered. Now that path mirrors the catch block: writeCooldown(reason) + status "failed" + a warn naming the reason. Behavior change (new path introduced by this PR): per-boot retry -> 6h cooldown bounded retries; panel/status wire shows bootstrap="failed" instead of "downloaded" in that case.
(ai: vllm-qwen/qwen3.8-27b) Review 结论(PR #3819,head 现为
|
| 变更 | 旧 | 新 | 原因 |
|---|---|---|---|
| knob 全缺省时的默认 unified 模式 | shadow | full | #3818 owner 决策 |
| 「下载 OK 但引擎加载不了」路径的 bootstrap 状态(本 PR 新增路径) | "downloaded" + clearCooldown(每 boot 重下 ~11MB) |
"failed" + 6h cooldown |
11e5bda66:避免谎报成功 + 无界重下 |
其他说明
- 热文件争用:[PR] fix: surface SQLite bootstrap cooldown expiry and force-retry lever in storage diagnostics #3790 同触 engine-bootstrap.ts/runtime-status.ts,[PR] fix: cold load after provider switch recovers the session fold ledger across routes (#3399) #3765 同触 persist.ts——合并顺序需协调(语义冲突风险),建议本 PR 先合或让对方 rebase。
- CI 状态:旧 head
ec56adc90的主 CI run 被取消(无有效裁决);3 个 commit 已推送,PR head 确认更新为11e5bda66,新 head 的远端 run 写作时尚在排队索引,以那次运行为准。本地定向:14 个相关套件 103 测试 → 100 过 / 2 挂(恰为 full mode: reloading a session with only archived originals silently drops those originals (blockContents lost through the unified view) #3831 钉死的两例)/ 1 skip;bootstrap+degraded+banner 37/37 过;typecheck 干净。 - 那两例红是正确的信号而非噪音:它们钉住「归档原文必须存活重载」,full 默认下代码违反之。
收工判定
暂不可合并 —— 等 #3831 拍板:A(kernel 侧给 blockContents 独立载体,根治,Kernel Boundary 人审)/ B(host 侧 loadSync 统一路径发现 view 丢了 envelope 里还有的 contents 时改 serve envelope,小改恢复 flip 后默认可用性,writerOff 不覆盖)/ C(hold flip,默认留 shadow)。我的推荐:B 先行 + A 跟踪根治。owner 拍板后对应修法落地(或按决策处理两例测试),其余部分本 PR 已具备合并条件。
…ow-ups) - shadow record/importRecord skip records whose blockContents carry ids no state.blocks entry claims: the ingest cannot represent them, so a viewFor rebuild silently dropped the orphans on first load. Real kernel sessions never trip this (state.blocks keeps consumed blocks); hand- built archives and web seeds do. JSON envelope stays the serving copy (orphanSkips counter on the stats surface). - bootBounded union now respects the decoded-byte budget as well as maxSessions: index-only (jsonWriter=off) candidates are measured by their serialized record — the same proxy readRawFileWithBytes gives the file path (#2803: budget is a RAM proxy). - loadSync organic import serves the freshly parsed envelope view (lossless incl. orphans) instead of a viewFor rebuild. - test expectations for the default flip: identity-registry restores the legacy baseline instead of deleting the env var (absent = full now), web-sessions stamp legs build per-leg stores (#3135-3: the stamp is the live instance's, not the knob at write time) and read the encrypted leg back with its key, storage-mode-label absent leg says full, stale 'defaults to shadow' comments updated.
|
ci失败 |
The per-leg stores open a sidecar each (shadow/full legs) and were never closed — on Windows the live index.db handles can outlive rmrf(dir) and flake the suite teardown. Close each leg after its write.
…3818 review) reapDestroyedSubstrate keeps blockContents while state.blocks loses the entry — real reaped sessions trip the orphan-contents guard, not only hand-built archives. Pin both legs: the save skips the mirror (orphanSkips), and same-process plus cross-restart reads serve the lossless envelope view with imports still 0.
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) CI 失败已查清:两处红,都不是本 PR 的代码回归。 当前 head ①
②
顺带确认: 结论:等 windows shard 重跑复绿后,本 PR CI 即为实质干净;合并前唯一剩余事项仍是 #3831 的状态收口(guard 已在分支内,属"已修复待验证关闭")。 |
…view follow-up) reapDestroyedSubstrate (#2695) keeps blockContents while state.blocks loses the entry — real reaped sessions trip the orphan-contents guard (owner-pinned in 21b70f4: never gutted into the index). Under BILI_STORAGE_JSON_WRITER=off that skip left the session on NO copy at all (index skipped the ingest, writerOff wrote no file). The valve lands one synchronous JSON envelope for exactly that shape; every later read stays envelope-served (imports remain 0).
|
#3831 缺陷已在当前分支修掉(时间线说明) review 头
#3835 与本修法重叠(serve 时比对 vs ingest 前拦截,同一止血目标)。建议:本 PR 合并后关闭 #3835 为 superseded,#3831 收敛到「选项 A(kernel 侧 blockContents 独立载体)+ 子缺陷 b(label 重复行)」两个后续项。 |
(ai: vllm-qwen/qwen3.8-27b) 同意时间线与收敛方案,已逐条核实: 收敛执行
CI 终态(头 结论:分支处于可合并状态(CI 全绿、mergeable=true;当前 blocked 仅为等待人工合并)。按你的方案,合并后我来关 #3835 + 收敛 #3831。 |
(ai: vllm-qwen/qwen3.8-27b) 补充一条对我上一条「blocked 仅为等待人工合并」的精确化——核实合并状态时发现 blocked 的真实构成: master 分支保护仍钉着旧版 12 个 test 分片,而 core scope(普通 PR push)只跑 6 个(ubuntu 22×3、ubuntu 24×1、windows 22×2)。所以本 PR 的 排查中顺带钉死两处 CI 管道问题,已分别立项(均带来源标记与查重声明):
本 PR 本身的内容结论不变:CI 全绿(6/6 core leg + static + guards + e2e)、#3831 子缺陷 a 已在分支内修掉并有测试钉住,可合并;合并后我按约定关 #3835、收敛 #3831。 |
(ai: vllm-qwen/qwen3.8-27b) 刚拉了当前头 实质 CI 全部绿:该 commit 上共 77 条 check-run——49 success + 28 skipped,零 failure。这个头上跑过两批完整批次(13:48 与 14:25),两批都是全绿:主 CI(6 个 core test leg 含 windows-22-1、static-gates、kernel-guard 等全部 guards)+ 全部 e2e 泳道(real codex / fake-codex / pi / opencode / dsh / local-registry / persona / ws)+ Windows codex overlay + PR Build Artifact。 页面上唯一的红是 ci-gate-plugins 幻影 run(run 39263296529,13:48:24 创建、秒败、零 job)——就是已立项的 #3852:dispatch-only workflow 在每次 push(含 master)上的平台侧幻影,不在必需检查里、不阻塞合并。你看到的「ci 还是失败」应该就是它。 如果你看到的是别的具体某项红(名字或链接发我一下),我马上查——按 API 数据现在不存在第二条红的 check-run。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.617-master.fc15b1e # this exact build
npm install -g billion-context@master # always the newest merged build
|
Fixes #3818
Fixes #3135
What
One PR, two layers:
resolveStorageModeabsent-everywhere default shadow → full. The shadow dry-run has been the default since Inbound markers: one declarative strip engine for all four wires (#2789, #2798, #2811 plan B) #2815 and digest reconciliation held everywhere it mattered; with full, the SQLite index becomes the load path (unified-first, organic import, legacy fallback). JSON envelopes keep being written;legacyremains the one-line escape hatch (unset-trumps-everything tier unchanged).stats.disabledafter the build and lands one synchronous JSON envelope write (store.writeNow+stampOwnWrite) in the same tickBILI_PERSIST=0/enabled:falsefabricated a "storage degraded - no SQLite engine" banner for a sidecar that was never supposed to openrecordRequestedModetakespersistDisabled; the banner reports intentional opt-out (same class as the encryption opt-out)storageModestamp described the REQUESTED mode: with full as default, every node<22.5 box (engine download fails) would stampfullonto records that only ever exist on the JSON envelopeliveStorageStampgetter on the store instance (disabled / no sidecar / sidecar self-disabled →legacy);guardedBuildstamps it; the module-leveleffectiveFileStorageMode()consults the live singleton (peek, never construct) and keeps pure knob resolution only pre-storeverifyDrift+driftedSessions(live count) on the/__bili/statuswire;loadSyncserves drifted sessions from the JSON envelope (fresher copy) until a clean sample re-admits them; organic import is skipped for drifted records so the watermark is not re-stamped as if verifiedbootBoundedwarmed only what the file scan saw — pure-index sessions (jsonWriter=off) were invisible after restartshadow.listSessionIds()(still budget-bounded, stop atmaxSessions)Stakes-first
Seven red tests were written on clean master first (
tests/storage-full-flip.test.ts6 +tests/storage-engine-bootstrap.test.ts1), each failing on the intended defect; all seven green after the fixes. One stake assertion was deliberately revised during implementation (module-level stamp when the singleton is disabled-by-config): the implemented semantic — disabled persistence reportslegacybecause nothing is unified — is the honest reading of #3135-3, documented in the test.Compatibility
legacy) + restart.storage-unified.test.tsfile-tier expectations updated ([undefined, "shadow"]→"full"); every other storage/status/golden family passes unchanged (storage-replicas4/4,golden-storage-render5/5,acp-statusfamilies,memory-diagnostics3/3).Docs
CONFIGURATION.md(+zh) and thewebsite/config-reference/process.yamlseed regenerated viatools/gen-config-docs.mjs(default column + EN/ZH descriptions state the #3818 flip and the escape hatch).Not in this PR
GC / index.db growth (#3463) — disclosed, separate; multi-lane posture doc (#3633); incremental-lane cold-resync gap (#3798); replicas 409-inheritance family (#3011/#3296/#3025).
🤖 Generated with pi