[ranxianglei]

What

One PR, five mechanism-level fixes — each removes a defect class instead of patching a symptom (per the issue's directive):

item	mechanism	before	after
#4192	COMPRESSION_STATE_FIELDS = Object.keys(mergeState(createInitialState())) in new src/state-hydration.ts	hand-copied field list in ingest.ts drifted (lost sweptOrphanCallIds, pendingOrphanSweepMinRef)	the list is derived from the hydration source itself; parity test fails the moment createInitialState() grows an unknown key
#3987	releaseContent deletes on zero (UPDATE … RETURNING + delete), sqlite and mysql	zero-ref rows lingered forever (12.6 GiB observed)	orphans cannot structurally accumulate; FK discipline made explicit — referencing rows now always deleted before release (5 call sites reordered)
#4216	orphan contents ride the ingest plan as sessions.orphans_json (merged ∪, LazyBlockContents-aware)	mirror skipped the shape entirely → post-reap saves froze at the pre-reap state, cold loads served hours-old data	the #2695 reap shape is representable; freeze repro pinned (requests keeps advancing); the mirror skip ×3, importRecord skip and the #3818 writerOff JSON valve are deleted
#4069	splitMediaContent also walks raw sidecars, swapping data-URL-form payloads (data:<mime>;base64,<payload> ≥2 KiB — the openai/responses carriers) for {$media:<hash>}; hydrate restores byte-identically (both backends)	every picture stored twice (top-level imageBase64 + raw-sidecar data URL) → dedup capped ~1.89×	one CAS row per payload, refcounted across both channels; small payloads (<2 KiB) stay inline. Known gap (review): anthropic rawAnthropicBlock.source.data and google inlineData.data carry bare base64 with no data-URL prefix and are NOT matched — widening the regex blindly would eat signed thinking bytes; those two carriers need slot-directed splitting (as #4075 does). Scope decision pending owner
#4004	legacy-envelope integrity tests pinned to storage.unified=legacy + new full-authority pin	four tests flaky-by-construction under the #3818 full default (index-vs-envelope race)	decided semantics pinned: index serves while it holds the record; reject only when both carriers are gone/tampered

Explicitly not here

• mysql twin of the orphan-journal merge behaves identically (same ingestSession input) but is exercised via the sqlite contract suite; a DSN-gated twin test can follow.
• Egress Judge false positive on Snapshot Render Lane I2: prefix comparison baseline needs to be aware of the render view | 出站判官对快照渲染车道误报 I2:前缀比对基准需要感知渲染视图 #4241 (invariant-judge exemption for the render lane) is a separate wire-scope fix.
• media splitting for the anthropic/google bare-base64 carriers (see [storage] #3762 media CAS split only halves the payload: the raw wire sidecar still embeds a second full copy — on real data the split yields 1.89× instead of 33.8× #4069 row) — slot-directed coverage as in [PR] fix: media CAS split covers all raw wire copies (#4069) #4075, pending owner scope call; the data-URL carriers this PR covers are a strict improvement over master either way.
• consolidation of [PR] fix: make full-mode storage tests deterministic (#4004) #4011/[PR] fix: media CAS split covers all raw wire copies (#4069) #4075/[PR] fix: keep unified session ingestion current with retained originals #4218 (overlapping earlier PRs for the same issue items) is an owner decision; [PR] fix: keep unified session ingestion current with retained originals #4218's author deserves credit in any close note.

Verification

• new: storage-state-fields 2/2, storage-orphan-representable 1/1, storage-media-sidecar-split 2/2, full-flip ⑦ authority 1/1; reworked ⑨/review tests to the new doctrine (9/9)
• families: unified 14, reap-journal 2, media-cas 3, archive-gc 5, bounded 11, scale 1, engine-proofs 12, backend-contract 6, replicas 4, json-writer-off 3, durability 5, ingest pointers 3, proxy-persist 23, issue2129 4 — all green; tsc clean; unused-exports clean
• storage-render goldens: an earlier head regenerated them after a local-only drift; CI computed the original master digests on every shard and the regenerated snapshot was restored (4b23f11) — final diff carries zero golden changes and full == legacy == goldens holds on every wire (engine-proofs). The local drift was a contaminated environment (stale kernel/dist), not a code property.

kernel-change: none

[github-actions]

📦 Built Package Artifact

Branch: 2026-10-22_storage-wrapup-3 (d9963eb)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.5056-pr.4249.01d25f9

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

1. Download the artifact from the Actions run
2. Extract the tarball and install:

tar xzf billion-context-pr4249.tgz
npm install -g package

---

Install instructions are refreshed on each push.

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:同家族但不算简单重复——与三个在途 PR 重叠,建议归并到本 PR,关闭动作留 owner 拍板(我不擅自关别人的 PR):

• [PR] fix: make full-mode storage tests deterministic (#4004) #4011(更早,Closes storage full mode: loadSync serves the sqlite index and bypasses the envelope id-mismatch guard (#2671-era tests fail locally) #4004):纯测试钉死,方向与本 PR 一致(legacy pinning),本 PR 另加了 ⑦ full-authority 测试,机制覆盖更全;
• [PR] fix: keep unified session ingestion current with retained originals #4218(更早,外部贡献者 0XCocoa,Fixes [Bug][storage] Retained orphan originals freeze unified session ingestion; cold loads restore hours-old state #4216):与本版同用 orphans_json 机制,唯一差异是它保留 writerOff 兼容 envelope(伞 issue Storage wrap-up wave 3: Single source of truth for status fields + Orphan representability + Authoritative "full" resolu | 存储收尾第三波:状态字段单一事实源 + 孤儿可表示化 + full 权威定案 + 媒体 CAS 全量拆分 + 归零即删(#4192/#4216/#4004/#4069/#3987) #4242 已决策删除);若归并,请在关闭说明里给作者署名;
• [PR] fix: media CAS split covers all raw wire copies (#4069) #4075(更早,Closes [storage] #3762 media CAS split only halves the payload: the raw wire sidecar still embeds a second full copy — on real data the split yields 1.89× instead of 33.8× #4069):不能直接关——它的媒体拆分覆盖全部四个载体(含 anthropic/google 的裸 base64),本 PR 只覆盖 data-URL 形态载体(openai/responses),见下文「缺口 ④」。归并前需先吸收那两个载体的覆盖,否则 [storage] #3762 media CAS split only halves the payload: the raw wire sidecar still embeds a second full copy — on real data the split yields 1.89× instead of 33.8× #4069 仍是半修。

标签:已打 bug / scope/storage / p1(p1 依据:#3987 属无界增长类、#4216 冷加载返回陈旧数据、#4192 水合静默丢字段,按 2026-10-19 口径一律 p1)。

审核方式(独立重审,未复读 issue 结论):全 diff 逐行通读 + FK 纪律逐点审计 + 本地定向运行——true master 与分支各跑 golden 5/5、新/改测试族全绿(state-fields 2、orphan-representable 1、media-sidecar-split 2、full-flip 9、archive-gc 5、issue2627-turn-sep 12、proxy-persist 23、issue2129 4)、tsc 干净;并做了 mysql-contract lane 的密闭复现(移除 kernel/dist 后 contract 套件 11/11 过,含 1 个无 DSN skip)。

逐项结论:
① state-hydration 字段列表派生 ✔ ——列表由 createInitialState() 自身派生,parity 守卫测试有牙(kernel 新增字段会立刻红);mergeState 逐字搬移无语义变化;kstate 序列化路径不受影响(JSON 解析顺序无关)→ 无持久化字节/digest 漂移,golden 实证。
② releaseContent 零引用即删 ✔ ——FK 审计全过:PRAGMA foreign_keys = ON,所有调用点满足不变量「refcount 归零 ⟺ 无任何存活引用行」,五处重排正确,undo-bump 位点结构性安全,gc sweep 保留为兜底。
③ orphans_json 可表示化 ✔ ——MERGE ∪ 语义 + 损坏容忍 + LazyBlockContents 裁剪场景有测试钉住;四处 skip 分支删除;writerOff JSON 阀删除属行为变更,CHANGELOG 与正文均已披露。
④ 媒体拆分 ⚠️ 机制正确但载体覆盖不全:只拆 data:<mime>;base64,<payload> ≥2 KiB 形态(openai/responses 两条车道);anthropic rawAnthropicBlock.source.data 与 google rawGoogleParts[].inlineData.data 是无前缀裸 base64,不匹配正则,这两条车道图片仍双存。正文「walks rawGoogleParts」对该载体的表述不准确。不能简单把正则放宽到任意长 base64——redacted_thinking 等 thinking 族字段是签名校验的字节,任何改写都会砖掉会话(KDD #10),扩展必须按槽位精确定向(#4075 的做法正是如此)。需 owner 决策:吸收 #4075 的两载体覆盖,或明确收窄 #4069 范围并登记残留。
⑤ #4004 钉死 ✔ ——四个 legacy 测试 pin + 新 ⑦ authority 测试正负双向齐备(tamper id→照常服务;index 行删+文件篡改→拒绝),符合 gate 纪律。

两件过程事项,独立核实过:

• golden 风波:原 head 的「重新生成 golden」在 CI 不复现(CI 在全部失败 shard 上算出的都是 master 原值)→ 后续 commit 恢复 master golden,最终 diff 净 golden 变化为零;我在 true master 与分支各自独立跑均 5/5,master 无既有红。PR 正文的 golden 段落现已过时,建议更新。
• mysql-contract 红:两跳根因(state-hydration 值导入 kernel → session.js 顶层也值导入 kernel),两个 commit 修复(相对 kernel 源码导入 + turn-separator 零依赖叶模块,session.ts 保留 re-export 兼容既有消费者),新 head 上该 lane 已绿。这四个修复 commit 是并行会话推的;我本轮没有推代码,只做独立验证。

非阻塞小观察(均不影响正确性;分支正被并行会话活跃迭代,留给收尾打磨,不在本轮推 commit):
a. sqlite hydrateMedia 缺 per-hash 缓存(mysql twin 有)——性能对齐,几行;
b. tests/storage-orphan-representable.test.ts:78 的 statsJson.includes("876") 是松断言(抓 freeze 回归仍有效,可收紧为解析后字段精确相等);
c. tests/storage-full-flip.test.ts ⑦ 结束时 env 残留 BILI_STORAGE_UNIFIED="full"(其后无测试,实际无害);
d. blocks-upsert redirect 路径重复了一段 15 列 INSERT 语句(DRY 味道,可抽 helper)。

八问:

1. 是什么问题:五个存储正确性缺陷类——水合字段列表漂移、零引用内容行永久堆积(真实库实测 92,004 行 / 11.32 GB,issue 标题记 12.6 GiB)、reap 形态会话保存冻结且冷加载返回陈旧数据、媒体双存(去重只到 ~1.89× 而非预期的 33.8×)、full 模式下 index 绕过 envelope id 守护致四个测试结构性 flaky。
2. 什么原因导致的:手维护的并行结构从真相源漂移——字段列表手工抄写、孤儿形状镜像可表示却被跳过、拆分做了一半、测试假设绑死旧默认模式。统一根因:缺单一真相源约束。
3. 影响范围与严重程度:p1 级。[storage] 0.1.202 existing install: 12.6 GiB of refcount=0 orphans (87% of content bytes), still growing on a long session — interim reclaim before the GC/CLI release? #3987 无界增长(unified 存储全体用户,随时间恶化);[Bug][storage] Retained orphan originals freeze unified session ingestion; cold loads restore hours-old state #4216 一切经历 reap 的会话(冷加载拿到数小时前数据);Unified Storage: MERGE_STATE_FIELDS and mergeState out of sync — full mode hydration drops sweptOrphanCallIds/pendingOrp | 统一存储 MERGE_STATE_FIELDS 与 mergeState 不同步:full 模式水合丢 sweptOrphanCallIds/pendingOrphanSweepMinRef(#2388 类) #4192 水合丢两个孤儿清扫簿记字段;媒体项属成本/性能面。
4. 怎么修复的:派生而非抄写(字段列表从 createInitialState() 本身导出)、可表示而非跳过(孤儿随 ingest journal 落库、删四处 skip)、事务内零引用即删(孤儿结构上无法累积)、字节恒等回放的 CAS 共享、已决语义钉死(index 服务、envelope 降为恢复源)。
5. 是否有回归问题:审过的既有行为——无新迁移(orphans_json 列 Reaped blocks resurrect after restart in full mode: stale index row outlives the #2695 reap and poisons the JSON envelope #3841 起已存在)、kstate 序列化字节/digest 不变、wire delta 为零(golden 实证)、FK 各调用点安全、gc 兜底保留;高危模式逐项过:无 fail-fast 变吞错、无阈值/时序漂移、无输出格式变化。唯一保留项:anthropic/google 两载体媒体残留(不比现状差,但修复不完整,见 ④)。
6. 是否破坏缓存:不涉及——前缀缓存/wire 体 sha 不变(golden 逐条相同),CAS refcount 内部自洽。
7. 是否改了设计方向:没有——执行 Storage wrap-up: 6 fixes + switch default mode to full (single PR) | 存储收尾六修 + 缺省模式翻 full(一个 PR 统一处理) #3818/Storage wrap-up wave 3: Single source of truth for status fields + Orphan representability + Authoritative "full" resolu | 存储收尾第三波:状态字段单一事实源 + 孤儿可表示化 + full 权威定案 + 媒体 CAS 全量拆分 + 归零即删(#4192/#4216/#4004/#4069/#3987) #4242 已决方向(journal 可表示、index 权威、envelope 恢复源)。
8. 是否引入新配置:没有(MIN_MEDIA_URL_SPLIT_BYTES 是内部常量非配置项;config-surface-guard 绿)。

判定:代码质量达标、测试有牙、本地定向验证全过;CI 现静态门禁与 mysql-contract 全绿,还有 8 个 lane(static-gates×2 + test shards×6)在跑,等全绿。不满足自动合并条件(多模块 feat 类 + 存储语义),人工合并。合并前需 owner 拍板两件事:① #4011/#4075/#4218 的归并/关闭处置(建议并入本 PR、吸收 #4075 的两载体覆盖、给 #4218 作者署名);② 上面四条小观察是否收进收尾。① 定案 + CI 全绿后即可合。

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 收尾（更正上一条「等全绿」）：head 175feab00 的完整结果已出——全部静态门禁、mysql-contract、e2e 各 lane 均绿，6 个 test shard 中 5 绿，唯一红的是 test (windows-latest, 22, 2)。

红的定性：与本 PR 无关的既有测试竞态。失败断言：

not ok 3681 - abort x proxy-responses-sse: client abort mid-stream destroys the upstream request
error: 'upstream socket(s) destroyed after client abort (closed=0)'   (tests/wire-exit-matrix.test.ts:257)

同一矩阵的兄弟用例 openai-sse / anthropic-sse / proxy-json 在同一 shard 全过，只有 responses-sse 变体在更长的耗时（1100ms）下读到 closed=0——典型的「abort 传播没赶上固定等待窗口」。根因在测试代码本身：tests/wire-exit-matrix.test.ts 里 json 单元格早已改成轮询（注释原话："Poll for the abort to propagate instead of a fixed 300ms sleep: slow runners (win/node24) destroy the upstream socket later"），但三个 SSE 单元格仍是固定 setTimeout(300)——同一个已知 flake 类只修了一半。本 PR 的 diff 只碰存储层（persist/ingest/store/shadow/mysql/state-hydration/turn-separator），与该文件零交集。

处置（已做）：按「CI 失败类小修在当前 PR 顺手修」的纪律，已在分支上补一个独立标注的 test-only commit d9963eb5b——把三个 SSE 单元格换成文件里现成的轮询惯用法（30×100ms），无 wire/源码影响；本地定向跑该文件 7/7 绿。新 CI 已在 d9963eb5b 上触发，等它绿完这个 PR 就是全绿。

另澄清一条避免误读：check 聚合视图里那条 lock-sync 红属于 PR #4255 的运行（它自己 kernel/package-lock.json 漂移 0.0.151→0.0.152），不是本 PR 的——本 PR 自己的 lock-sync 是绿的（kernel 版本两侧同步 0.0.151）。

结论不变：等 d9963eb5b 全绿 + owner 对两条决策点拍板（#4011/#4075/#4218 的并线处置、④媒体载体的 anthropic/google 覆盖缺口），即可人工合并。

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 已全绿：d9963eb5b 上主 CI（run 39718829085）22/22 job 通过，包括之前红的 test (windows-latest, 22, 2)——轮询化修复生效，该竞态类在 abort 矩阵四个单元格上全部收口。同 head 的 e2e 各 lane、Theorem Lane、PR Build Artifact 也全绿。

最终判定：代码与测试层面本 PR 可合（人工合并，非 auto-merge：多模块 feat 级 + 存储语义变更）。剩两条 owner 决策点，均已在首条评论里给出建议：

1. [PR] fix: make full-mode storage tests deterministic (#4004) #4011 / [PR] fix: media CAS split covers all raw wire copies (#4069) #4075 / [PR] fix: keep unified session ingestion current with retained originals #4218 并线处置——建议以本 PR 为准收敛；[PR] fix: media CAS split covers all raw wire copies (#4069) #4075 关闭前须吸收其 anthropic source.data / google inlineData.data 两个裸 base64 载体的覆盖（否则 [storage] #3762 media CAS split only halves the payload: the raw wire sidecar still embeds a second full copy — on real data the split yields 1.89× instead of 33.8× #4069 只修了一半），[PR] fix: keep unified session ingestion current with retained originals #4218 为外部贡献需保留署名归属；
2. ④媒体载体缺口是并入本 PR 补两个槽位定向扩展，还是显式收窄 [storage] #3762 media CAS split only halves the payload: the raw wire sidecar still embeds a second full copy — on real data the split yields 1.89× instead of 33.8× #4069 范围另立后续。

四项小 nit（sqlite hydrateMedia 无 per-hash 缓存、includes("876") 弱断言、⑦ env 残留、blocks-upsert 重复 INSERT SQL）不阻塞，随 owner 意愿顺手修或留档。

[github-actions]

📦 Merged Build Published

This merge to master was published as 0.0.737-master.c1d7eab (rolling master tag).

npm install -g billion-context@0.0.737-master.c1d7eab  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.