Repository navigation
Bound startup session loading and preserve historical recovery - #2803
ranxianglei merged 8 commits into
Conversation
3eb0c5b to
c3b212b
Compare
|
ework 接单 |
…anglei#2803 review) The existsSync gate in loadEnvelope can race the hourly GC sweep (ranxianglei#1082) or an operator archiving files into the -cold sibling between gate and read; the bare readFileSync then throws ENOENT and turns an ordinary absence into a 5xx. ENOENT now degrades to null like any other missed probe; every other error still fails closed.
…lei#2803 review) Bounded startup refuses corrupt originals instead of degrading silently, so the old 'never a crash' wording no longer matches loadContentStore.
🏷 Review 结论(PR #2803 / Fixes #2776) 判定先说:方向正确、实现质量高、代码层面可合;已顺手推了 2 个 additive 小修到分支(原作者 commit 未动),另有 4 处行为变更需明示确认 + 2 个设计决策点等 owner 拍板。均不阻塞。 验证证据(本地,基于当前 master
结构化八问:
测试有效性:9 个单测 + 规模回归断言有牙——精确字节相等、对 diff 整洁度:4 文件,每一行都在目的之内;无空白 churn、无锁文件/生成物;kernel-guard / version-guard / config-surface-guard 全绿。 收工判定:代码侧我已放行;待 owner 拍板 (a) 硬编码预算 vs 配置项,(b) legacy 迁移非破坏性立场。两个新增 commit 的 CI 转绿后,剩下的门在你手上。 |
|
需要配置 注意放到合理位置 json配置文件里面 默认值不能影响正常会话 另外我看回归问题比较多 |
…etric (ranxianglei#2803 owner ruling)
拍板两项都已落实。PR 分支新增提交 ① 是什么问题:你拍板的两件事——(a) warm 数量/字节预算硬编码不可调;(b) 我上轮 review 列的回归清单需要逐项处置,不能只披露。 ② 什么原因导致的:(a) 作者实现时选了常量,没走 config-surface 流程;(b) 回归项是 PR 与既有行为的语义差,每项要么修掉、要么确认有意并用测试钉住,必须二选一。 ③ 影响范围与严重程度:(a) 原来所有部署被迫接受 8/64MiB/512MiB,现在可调;默认值 = 原硬编码值 → 不配置的常规会话零影响(满足你"默认值不能影响正常会话")。(b) 回归清单里 F-H(zstd 归档绕过字节预算)是真实缺陷:高压缩比归档可让实际解码内存数倍于 64MiB 预算——已修;其余为有意设计(见⑤),无隐藏影响面。 ④ 怎么修复的:
⑤ 是否有回归问题(你点名的回归清单,逐项处置):
⑥ 是否破坏缓存:不涉及。warm 集只决定启动预加载哪些会话;消息内容、ref 编号、prefix cache 均不变。 ⑦ 是否改了设计方向:否。仍是"索引 + warm 集 + 懒加载"架构,只是三个常量入配置 + 修计量口径。 ⑧ 是否引入了新的配置:有(即你拍板项):上列 3 个 key,默认 8 / 67108864 / 536870912,各有 BILI_ env twin,无需迁移(新字段,旧配置自然回落默认)。 判定:拍板项全部落地并验证;等 CI 对新提交转绿 + 你对 F-G 处置(维持非破坏式 vs 要求清理)的确认。无异议即可合并。 |
…anglei#2803) CONFIGURATION.md/.zh-CN.md tables are generated from website/config-reference/*.yaml (tools/gen-config-docs.mjs); add the three ranxianglei#2803 budget keys to the process.yaml seed and regenerate instead of hand-editing the artifacts. Fixes config-surface-guard / docs-coverage / docs-sync on 740d258.
CI 跟进: 原因:CONFIGURATION.md / zh-CN.md 的表格是 修复:三个 key( 等 CI 对 |
…#2803) The ranxianglei#2816 tripwire requires every process-spawning suite to be listed in scripts/run-ci-tests.mjs before it may ride the c=4 pool (ranxianglei#1248 review). Concurrency assumptions reviewed: isolated mkdtemp dir, child-only env (BILI_PERSIST_ZSTD/BILI_SESSIONS_DIR), no port binding, one blocking spawnSync under a 128 MiB heap cap.
…tripwire gap on master) Pre-existing master gap, fixed in passing: c81be81 (ranxianglei#2752, 2026-10-11) added a process-spawning suite before the ranxianglei#2816 LIVE_FILES tripwire landed (e6ba45c, 2026-10-12), so every test job has died at the tripwire since then and the full suite never ran in them. Concurrency assumptions reviewed: listen(0) ephemeral ports, BILI_MCP_PROXY env save/restore, stdio transport to the spawned src/mcp.ts child.
CI 三个 ① 是什么问题:CI 测试调度器(#2816)有个 tripwire——所有会 spawn 进程的测试套件必须先登记进 ② 什么原因导致的:(a) scale test 是本 PR 新增文件,作者漏了登记;(b) issue2752 那个是 master 既有缺口——#2752(c81be81,昨天 20:05)先加了 spawn 套件,#2816 的 tripwire(e6ba45c,今天 11:15)后落地时没补上它。后果比看起来严重:tripwire 在测试阶段之前就 die,所以自 tripwire 合入起,master 的 test job 全都在门口挂掉,全量套件实际一天没跑过。 ③ 影响范围与严重程度:功能面零影响(纯 CI 调度元数据);但 master 三平台全量回归停摆是真问题,已顺手修掉。 ④ 怎么修复的:两个文件各一个独立标注 commit 登记进 LIVE_FILES(
本地验证:guard 逻辑独立复跑 GUARD-OK(34 条目、0 个未登记 spawner);两个受影响套件定向跑 23/23 pass(issue2752 自 tripwire 落地后 CI 里一次都没真跑过,这次确认它在本分支是绿的)。 ⑤ 是否有回归问题:无。仅列表常量加两行,测试内容/断言/调度参数(单阶段 c=4)未动。 等 CI 对 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.347-master.c55059c # this exact build
npm install -g billion-context@master # always the newest merged build
|
…#3016 review) The passthrough gate scans only the in-memory live-session map. Since #2803 (LRU session eviction) landed on master after this PR's base, a substrate-holding session can be flushed out of memory and become invisible to the scan. The previous wording ('disk-only sessions are not actively being compacted' / 'Never allows when the target could have substrate') documented a guarantee the merged code does not provide. Comments only; no behavior change.
…ded-byte budget (#3818 review) The #3135-6 union in bootBounded re-loaded every indexed id, including envelope-backed ones the file scan had already offered under the decoded-byte budget (#2803). With the full default (#3818) that bypassed maxBytes for every legacy file on disk (bounded-storage warm-budget test red: 2 sessions warmed where <=1 was allowed). Skip ids whose state file still exists; the union now only warms pure-index (jsonWriter=off era) ids, which is its stated purpose.
…ow-ups) - shadow record/importRecord skip records whose blockContents carry ids no state.blocks entry claims: the ingest cannot represent them, so a viewFor rebuild silently dropped the orphans on first load. Real kernel sessions never trip this (state.blocks keeps consumed blocks); hand- built archives and web seeds do. JSON envelope stays the serving copy (orphanSkips counter on the stats surface). - bootBounded union now respects the decoded-byte budget as well as maxSessions: index-only (jsonWriter=off) candidates are measured by their serialized record — the same proxy readRawFileWithBytes gives the file path (#2803: budget is a RAM proxy). - loadSync organic import serves the freshly parsed envelope view (lossless incl. orphans) instead of a viewFor rebuild. - test expectations for the default flip: identity-registry restores the legacy baseline instead of deleting the env var (absent = full now), web-sessions stamp legs build per-leg stores (#3135-3: the stamp is the live instance's, not the knob at write time) and read the encrypted leg back with its key, storage-mode-label absent leg says full, stale 'defaults to shadow' comments updated.
Startup currently hydrates every persisted session before applying the resident session count limit. An archive larger than the Node heap can therefore prevent the proxy from starting at all. A count limit also misses large idle sessions, and eviction can discard newer mutations after a failed flush if the session had previously been saved.
Fixes #2776.
Real native OpenCode end-to-end validation: 4/4 passed, including actual compression and persisted state. The external Windows runner allows persistence to flush before termination; tracked assertions are unchanged.
Index live and sibling cold archive files one record at a time, warm at most eight sessions within a 64 MiB disk-byte budget, and load older state and originals on demand. Preserve the existing unified SQLite read precedence and legacy migration identities. Bounded server reads fail closed on corrupt originals; legacy direct API callers retain their existing warned fallback. Original-store saves retain the existing unique atomic temporary filenames and prevent state advancement when they fail.
Bound the idle resident cache with an estimated 512 MiB target. Active requests stay pinned and may exceed that target; the existing session-count cap remains the hard admission limit. A regression specifically verifies that an oversized pinned request does not reject another conversation below that count limit. This is an idle-cache target, not a guarantee of total process RSS or a maximum size for one record.
Validation: typecheck, build and distribution import checks pass; the original 56 focused compatibility/recovery checks passed, and the final corruption compatibility/scale run passed 38 tests with one POSIX-only skip. A portable regression creates a 320 MiB synthetic archive, omits the migration marker, boots in a child with a 128 MiB heap, and restores an older exact original. Additional tests cover cold recovery, corrupt live precedence, failed original saves, failed-flush eviction, concurrent admission, and newer SQLite records winning over older JSON.
Final full patched Windows validation: 6,323 tests, 6,294 passed, 24 skipped, five failures, all reproduced in the unmodified baseline: name clearing (
issue2322-session-name.test.ts:223) and four Windows SQLite fixture cleanupEPERMerrors (storage-engine-proofs.test.ts:214,265,307,380). The baseline timing assertion did not recur. The direct corruption fallback compatibility failure found in the earlier run is corrected without changing the existing test. The pristine baseline had six failures: those same five and one timing assertion. The full suite is not entirely green; no new failures remain relative to that baseline.No kernel, configuration schema or version changes. Default direct
boot()behavior remains compatible; server startup opts into the bounded path. This draft needs storage-owner review for the newly merged unified engine and archive naming policy. It does not prune history, restore old snapshots, or change the current relay deployment.Config surface (#2803 owner ruling)
Per the owner ruling in the thread (需要配置 / 放到合理位置 json 配置文件里面 / 默认值不能影响正常会话), the three startup budgets are now config-file keys under the existing
sessionsobject (resolved env > file > default throughsrc/knobs.ts):sessions.warmMaxBILI_SESSION_WARM_MAX0warms none. Capped bysessions.max.sessions.warmMaxBytesBILI_SESSION_WARM_MAX_BYTESsessions.residentMaxBytesBILI_SESSION_RESIDENT_MAX_BYTESWhy the existing surface could not express these:
sessions.maxis count-only (#2776 facet F2) and no byte dimension existed anywhere in the schema. Defaults equal the previously hard-coded values, so unconfigured deployments behave exactly as before (no effect on normal sessions — per the owner constraint). Garbage/negative values fall back to defaults; explicit0is honored. Documented in CONFIGURATION.md + CONFIGURATION.zh-CN.md (key table, description, env-mapping table, long-form reference). Owner sign-off: the 决策点 1 ruling in this thread.