[PR] storage: seq ledger foundations — block interval bounds + ledgerSeq ref stamping (#4316 layer 3, phase 1) by ranxianglei · Pull Request #4346 · ranxianglei/billion-context

ranxianglei · 2026-10-23T00:50:26Z

First structural slice of the #4316 fix (design: #4316 (comment)). Pure foundations — no behavior flips yet; every consumer stays byte-identical unless the store serves it a stamped view.

kernel-change: refs.ts assignRefs ledgerSeq fast path (new, additive — the counter path is untouched), types.ts CoreMessage.ledgerSeq optional field, kernel 0.0.154.

What lands

CoreMessage.ledgerSeq? (kernel/src/types.ts) — a store-served view may stamp each message with its unified-transcript (session, seq); one allocation, never re-minted.
assignRefs honors the stamp (kernel/src/refs.ts) — when a message carries a valid ledgerSeq (integer within MIN_INDEX..MAX_INDEX, ref slot free), the ref is indexToRef(ledgerSeq) and the cursor advances past it. Collision or missing stamp falls back to the counter path; ids already in the kernel map keep their existing refs (the stamp only rescues cold rebuilds / post-churn views — stability is dual-channel).
Snapshot render stamps store rows (src/server/snapshot-render.ts) — parseRow sets msg.ledgerSeq = seq + 1 (store seq is 0-based, display refs are 1-based from MIN_INDEX=1).
Block interval bounds are now real (src/storage/ingest.ts) — planIngest fills start_seq/end_seq per block from the covered rows' seqs (-1/-1 = no row resolved, reads exactly like the old 0/0 placeholder). Phase 2 (fold bookkeeping keyed by seq intervals instead of h_id sets) builds on these columns.

Why phase 1 is safe

Nothing reads the stamp except assignRefs' new branch; counter-issued refs are unchanged.
Snapshot-render goldens stay green because the rendered refs they pin were already stable across salt flips (S2 asserts exactly that, now with the stamp active).

Tests (tests/seq-ledger.test.ts, 3/3)

S1 interval bounds: planIngest fills startSeq/endSeq from real lastMessages rows; ghost coverage → -1/-1.
S2 e2e ref stability: snapshot-render harness, salt flip mid-session — the leading ref sequence is deepEqual-stable across the flip (store-served prefix keeps its refs through an identity-function change).
S3 kernel seam: assignRefs direct — known id keeps its ref, fresh+stamped ledgerSeq:77 → m00077, fresh+counter → next counter, cursor advances.

Families: snapshot-render 12/12, storage-unified 14/14, golden 5/5, fold-reconcile 66/66, lane-matrix 58/58, reap 2/2, archive 5/5 (141 green); tsc clean; unused-exports clean.

Next (separate PRs, per the issue)

Phase 2: incremental lane stamps client appends with store seqs; fork clones the parent row set (keeping refs); one-shot remint window for existing sessions (the owner flagged the offset risk — the migration needs its own guard rails).

github-actions · 2026-10-23T00:51:10Z

📦 Built Package Artifact

Branch: 2026-10-22_seq-ledger (e444b1d)

Latest build — install this exact version

npm install -g billion-context@0.0.5399-pr.4346.33acf77

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit	version	build
`e444b1d`	`billion-context@0.0.5399-pr.4346.33acf77`	build
`a931892`	`billion-context@0.0.5300-pr.4346.e4e8322`	build
`04d5656`	`billion-context@0.0.5272-pr.4346.c089c4c`	build

Download artifact

Download the artifact from the Actions run
Extract the tarball and install:

tar xzf billion-context-pr4346.tgz
npm install -g package

Install instructions are refreshed on each push.

ranxianglei · 2026-10-23T00:57:25Z

mysql-contract red = pre-existing master breakage, not this PR. Evidence: master's own CI runs have been failing since 2026-10-22T15:30Z — run 39866373994 (head aa198fe6, 4 consecutive failure conclusions on the master branch) fails the same mysql-contract job plus 4 test shards. Reproduced locally on a clean checkout of origin/master: npm install --ignore-scripts leaves node_modules/acp-kernel/dist missing, so node --import tsx --test tests/storage-backend-*.test.ts cannot import acp-kernel from src/config.ts. This PR only ran the job because it touches kernel/** (scope gate widened); the failure predates it.

lock-sync (the one failure this PR genuinely owned — kernel/package-lock.json still stamped 0.0.153) is fixed in the follow-up commit.

ranxianglei · 2026-10-23T11:43:23Z

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——检索 open+closed 均无其他处理「seq ledger / ledgerSeq stamping / block interval bounds」的 issue 或 PR;本 PR 是 #4316 layer 3 设计(phase 1)的唯一实现切片。

标签: scope/storage + p3(结构地基期,无行为翻转;kernel 侧为纯增量接缝)。非缺陷类,不打 bug;无特定客户端车道。

开始按当前 master 重新审核(分支已 rebase 过一次但仍与 master 冲突:master 内核已到 0.0.160,本 PR 的内核版本号 0.0.154 与 master 上 #4289 用掉的同号冲突)。CI 现状:除 test (windows-latest, 22, 1) 红之外全绿(mysql-contract、lock-sync、kernel-guard 均已转绿)。正在:① 判定 windows shard 1 失败是否本 PR 引入(master 自身 10:22Z 的 ci.yml 也是红的,需先分清);② 解冲突并把内核版本推进到正确号;③ 逐文件深审 diff。结论稍后按八问补。

Review finding on #4346 phase 1: S2's e2e salt-flip scenario already held pre-change (store ids are stable, so the live ref map keeps serving the rendered prefix) — verified by running the suite against pre-change code (S2 green there). S2 therefore pins the user-visible property but cannot catch a regression of the stamp mechanism itself. S3 now covers the discriminating shape directly: empty existing map, stamps arriving out of order — each row must claim its ledger number (order-independent) and the cursor must advance past the highest stamp. Fails without the assignRefs fast path (verified against pre-change kernel).

ranxianglei · 2026-10-23T12:15:46Z

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

进度更新——冲突已解、分支已推到当前 master,CI 重跑中,测试有效性有一处发现并已顺手修。

1. Rebase 完成(新 head 7747503db,mergeable=True)
基线从 9e2d217 推进到 master tip 8d24bf9(master 在这 75 个提交里把内核从 0.0.153 推到了 0.0.160,核心矩阵改成三组合全 /3 分片 #4351,还加了 pretest dist 缓存 #4360)。冲突处置:

ci.yml:本 PR 的改动收敛为零。 本 PR 第三个提交(1b098b5,mysql-contract 前补 kernel build)与 master 的 master mysql-contract lane red: #4177's store→runtime-status chain pulled acp-kernel runtime imports into the storage co | master mysql-contract 车道红：#4177 的 store→runtime-status 链把 acp-kernel 运行时 import 带进了 storage 契约测试，job 没构建 kernel #4336(fd90437)是同一问题的平行修复——master 版已验证且足够(storage 两个测试文件不 import pi-subagents,无需连带构建)。取 master 版,本 PR 不再动 ci.yml。
内核版本号: 0.0.154 已被 master 的 Failed compression invocations are never folded back for reclamation, and the context keeps ballooning under the weight | 失败的压缩调用永远不会被折叠回收,上下文被压缩残渣越撑越大 #4289 用掉 → 按纪律推进到 0.0.161(master 现值 0.0.160 + 1),kernel/package.json、kernel/package-lock.json、root package-lock.json 三处一致。
CHANGELOG: 双方条目合并(本 PR 的 Added 条目置于 Unreleased 顶部)。
原两个 follow-up 提交(lock-sync、ci-fix)的目的已被上述解决吸收,不再单独存在。

2. 顺手修了 master 自身的一个漂移:#4292 把 kernel/package.json 提到 0.0.160 时没同步两个 lockfile(都停在 0.0.159)——现在在 master 上跑 npm install --package-lock-only 会改写 lockfile,lock-sync job 会红。本 PR rebase 后三处统一为 0.0.161,合入即修复 master 的这个漂移(未另开 issue:修复已在本 PR 内)。

3. 本地验证(rebased head):typecheck clean、build OK;定向套件全绿——seq-ledger / snapshot-render×3 / storage-unified / wire-contract(+golden)/ verif-subset-guard 共 69/69,kernel 全套 1460/1460。refs.ts 属 verif Tier-A 守护文件,子语言守卫已过。

4. 测试有效性发现(§④)+ 已修:S2(e2e salt-flip ref 稳定)对 stamp 机制本身没有牙——我把四个源文件+kernel 全部回退到改动前实测:S1、S3 红,S2 仍绿。原因:store 行 id 本身稳定,live ref map 一直能为渲染前缀供号,该场景的稳定性是本来就有的(OP 里「already stable across salt flips」一句也承认了这点)。S2 保留作用户可见属性的钉。真正区分性的形状(空 map 冷重建 + 乱序 stamp 各自认领自己的 ledger 号、cursor 跳过最大值)已作为独立标注提交补进 S3(7747503db 前一个提交),并验证无 fast path 时变红。

5. windows shard 1 失败归因(待 CI):该 lane 当时跑的是旧矩阵(/2 分片),且与 master 自身的 ci.yml 损坏窗口(duplicate concurrency block → 整个 workflow 零 job 失败,#4405/c90e6ea28 已修)重叠,不能排除环境因素。新 head 的 CI 在新矩阵(/3 分片)下全量重跑,windows 三条腿的结果会直接回答 flaky vs 代码问题。CI 绿/红后按八问给最终结论。

ranxianglei · 2026-10-23T12:18:03Z

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突唤醒核实:通知已过期,无需再次 rebase。 唤醒消息(11:43Z 转发)到达时,前一轮会话的 rebase 已经完成并推送(floor 17,head 7747503db)。我对照活体 master 复核过,分支现在完全干净:

分支 head 7747503db 与 master 的 merge-base = master 当前 tip 8d24bf9d4——分支完整包含 master 全部提交,不存在任何文本冲突面;
GitHub API 确认 mergeable: True,mergeable_state: blocked(等检查通过,不是冲突);
head 上的 25 个 check run 全部在排队/运行中——新矩阵(/3 分片)下的全量结果就是最终裁决,windows shard 1 是 flaky 还是代码问题会直接见分晓。

按纪律不重复 rebase(再跑一次只是对已线性化的分支做无操作改写)。

八问:

是什么问题:系统报「上游 PR 冲突(dirty)」要求 rebase;实际是过期通知——rebase 在通知生效前已完成。
什么原因导致的:原分支基线 9e2d21703 之后 master 推进了 75 个提交(内核 0.0.153→0.0.160、ci.yml 被 master mysql-contract lane red: #4177's store→runtime-status chain pulled acp-kernel runtime imports into the storage co | master mysql-contract 车道红：#4177 的 store→runtime-status 链把 acp-kernel 运行时 import 带进了 storage 契约测试，job 没构建 kernel #4336/[PR] fix(ci): drop the duplicate top-level concurrency block that invalidates ci.yml #4405 改过、版本号撞号),产生真实冲突;前一轮会话已解完,系统通知队列滞后送达。
影响范围与严重程度:仅本 PR 的合并状态,无用户面影响;解决后 PR 回到正常待审态,等待 CI。
怎么修复的:本轮只做核实不做动作——确认 merge-base==master tip + mergeable=True 后判定冲突已消解,不再推送。此前的解法(floor 17):ci.yml 取 master 版(平行修复收敛为零)、内核版本推进到 0.0.161(三处 lockfile 一致)、CHANGELOG 双方条目合并。
是否有回归问题:本轮零改动,无新增回归;内容 diff 自上一轮验证后未变(typecheck clean、build OK、定向套件 69/69、kernel 全套 1460/1460)。合入后顺带修复 master 自身的 lockfile 漂移([Bug][anthropic] fold summary rendered as user right after a mid-conversation system message → 400 "role 'system' must precede an 'assistant' message" (0.1.204, since #3992) #4292 提版本时漏同步两个 lockfile)。
是否破坏缓存:不涉及。
是否改了设计方向:不涉及。
是否引入了新的配置:不涉及。

收工判定:冲突已消解,分支干净,CI 在新 head 上重跑中;CI 绿即可合并,红了我再来定向排查。

ranxianglei · 2026-10-23T12:53:10Z

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

判定:无需改本 PR,没有可 push 的修复。 mysql-contract 红是 master 上的既有损坏,已在 master 修复,rebase 后的分支原样继承。本地已 A/B 验证。

证据链

原始失败(head f873248,job 120176514800,日志第 256 行):两个 storage-backend 测试都死在 import 阶段——Error [ERR_MODULE_NOT_FOUND]: Cannot find module '.../node_modules/acp-kernel/dist/index.js' imported from .../src/config.ts,尚未触到 MySQL。
根因:该 lane 用 npm install --ignore-scripts 安装后直接跑 suite,从不构建 in-repo kernel;自 acp-kernel 并入billion-context并保持相对独立 #2092 起 node_modules/acp-kernel/dist 只在 npm --prefix kernel run build 之后存在;[PR] storage: durability trio — open-time quick_check + online VACUUM INTO backup + second-process lockfile (#4175) #4177(aa198fe)新增 store.ts → runtime-status 导入链,把 acp-kernel 拉进这两个测试的 import graph(经 src/config.ts)→ master 自 2026-10-22T15:16Z(CI run 39803248052)起红,早于本 PR 存在。主 test job 不踩坑是因为 scripts/run-ci-tests.mjs 有 pretest build。
master 的 ci.yml 已在 mysql-contract job 的 install 之后加上 - run: npm --prefix kernel run build(注释引 master mysql-contract lane red: #4177's store→runtime-status chain pulled acp-kernel runtime imports into the storage co | master mysql-contract 车道红：#4177 的 store→runtime-status 链把 acp-kernel 运行时 import 带进了 storage 契约测试，job 没构建 kernel #4336/[PR] storage: durability trio — open-time quick_check + online VACUUM INTO backup + second-process lockfile (#4175) #4177);当前 head(7747503)rebase 在该 master 之上,diff 完全不碰 ci.yml,直接继承修复。(PR 分支上早前那个同类本地修复 commit 在 rebase 时被吸收——被 master 版本 supersede,丢弃正确。)
本地 A/B 验证(当前 head 7747503):npm install --ignore-scripts + 清掉 kernel/dist → 复现字节相同的 ERR_MODULE_NOT_FOUND;执行 ci.yml 同款步骤 npm --prefix kernel run build → import 通过。另确认 pi-subagents 不在这两个测试的运行时 import graph 里,所以 master 只构建 kernel 足够。
实况:旧 head 34112a5 的整轮 run 被新 push 并发取消(非失败);当前 head 7747503 的新 run(12:11:57Z 创建)写此回复时仍在排队等 runner。失败点 100% 在 import 阶段、已被本地验证覆盖,跑起来应为绿。

八问

是什么问题:mysql-contract CI job 红——两个 storage-backend 测试加载不了,import 阶段缺 acp-kernel dist 即死。
什么原因导致的:master 既有损坏,非本 PR 引入:--ignore-scripts 安装 + 该 lane 从不构建 in-repo kernel,[PR] storage: durability trio — open-time quick_check + online VACUUM INTO backup + second-process lockfile (#4175) #4177 又把 acp-kernel 拉进这两个测试的 import graph。master 自 10-22T15:16Z 起红,早于本 PR。
影响范围与严重程度:仅这一个 CI lane(以及未来任何 --ignore-scripts 安装又 import storage 链的 job);产品行为与用户零影响。本 PR 变红只因 scope gate 因触碰 kernel/** 而触发该 job。
怎么修复的:本 PR 无需改动——修复已在 master(ciyml 该 job 加一个 build 步骤),rebase 继承;本地 A/B 验证失败签名可被该步骤消除。
是否有回归问题:无。本 PR diff 不碰 ci.yml/import graph;master 侧修复只给该 job 增加一个 build 步骤。
是否破坏缓存:不涉及。
是否改了设计方向:不涉及。
是否引入了新的配置:不涉及。

收工判定:auto-fix lane 以无 push 关闭——PR 上没有要改的代码。新 run 跑完后 mysql-contract 应转绿;若 windows test shard(另一条单独转发的失败)仍红,走它自己的 lane 处理,不在本次范围。

…ef stamping (#4316 layer 3, phase 1) - planIngest fills blocks.start_seq/end_seq with the min/max unified-store seq of the covered messages (-1/-1 when no rows resolve). Coverage bounds become id-independent: byte churn can never invalidate them. - CoreMessage.ledgerSeq sidecar: store-served views (snapshot-render prefix rows) stamp the row's (session, seq); assignRefs issues the ref as the direct display of that number (seq+1, 1-based display space), advancing the cursor past the stamp so counter-allocated refs never collide. - Stakes first (tests/seq-ledger.test.ts): S1 interval bounds, S2 ref-set stability across a full id re-mint under snapshot render, S3 kernel seam.

Review finding on #4346 phase 1: S2's e2e salt-flip scenario already held pre-change (store ids are stable, so the live ref map keeps serving the rendered prefix) — verified by running the suite against pre-change code (S2 green there). S2 therefore pins the user-visible property but cannot catch a regression of the stamp mechanism itself. S3 now covers the discriminating shape directly: empty existing map, stamps arriving out of order — each row must claim its ledger number (order-independent) and the cursor must advance past the highest stamp. Fails without the assignRefs fast path (verified against pre-change kernel).

ranxianglei · 2026-10-23T14:19:08Z

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

进度更新(中途一次变基,分支已再次 force-push):

master 并入了 #4416(fix/ci-hygiene-locksync-unused-exports),恰好覆盖本次 review 中碰到的三个问题:① 两个 lockfile 的版本漂移同步;② tests/corrupt-index-child.ts 四个内部符号去掉多余 export(上一轮 unused-exports 红的根因,#4338 引入的文件);③ storage-engine-proofs.test.ts / web-storage-banner.test.ts 注册进 LIVE_FILES(再上一轮 全部 9 个 test shard 在跑任何测试前就死的根因——spawn 文件未过 tripwire)。所以之前我在 PR 分支上手工做的对应修复全部作废。

已重新变基到新 master(1491fcf):

PR 本体不变:9 个文件 +357/-7(kernel refs/types、snapshot-render、ingest、seq-ledger 测试等);
kernel 版本按新基线推进:0.0.160 → 0.0.161(本 PR 是唯一的 kernel 变更,lockfile delta 相对 master 只有这 4 行版本号);
我此前的 un-export / lock-sync 提交与 master 重复,丢弃;S3 冷重建回归用例保留为独立小提交(a93189232)。

新 head(a931892)本地定向验证:typecheck + build 干净;unused-exports gate 干净;LIVE_FILES tripwire 扫描干净(无漏注册的 spawn 测试文件);定向批次(seq-ledger + snapshot-render 族 + storage-unified + web-storage-banner/storage-engine-proofs + wire-contract+golden + verif-subset-guard)= 81 pass / 0 fail(1 skipped)。kernel 套件此前已在同内容上验证 1460/1460。

分支已 force-push(旧 04d565607 → 新 a93189232),PR 冲突已解除(mergeable),CI 正在跑。全绿后发完整八问结论;windows 三条 lane 的结果同时回答最初那个 windows shard 失败是环境问题还是代码问题的归属问题。

github-actions · 2026-10-24T05:54:46Z

📦 Merged Build Published

This merge to master was published as 0.0.794-master.02f544d (rolling master tag).

npm install -g billion-context@0.0.794-master.02f544d  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei changed the title ~~storage: seq ledger foundations — block interval bounds + ledgerSeq ref stamping (#4316 layer 3, phase 1)~~ [PR] storage: seq ledger foundations — block interval bounds + ledgerSeq ref stamping (#4316 layer 3, phase 1) Oct 23, 2026

ranxianglei force-pushed the 2026-10-22_seq-ledger branch from 8550618 to 1b098b5 Compare October 23, 2026 01:12

ranxianglei mentioned this pull request Oct 23, 2026

Accident re-anchor debris is unrecoverable: the overlay island lands mid-turn, and the turn-split guard means the orphan | 事故重锚碎屑无法回收:覆盖岛落在 turn 中段,turn-split 守卫让孤另一半永远折不了 #4348

Closed

ranxianglei added the 增量压缩 label Oct 23, 2026

ranxianglei force-pushed the 2026-10-22_seq-ledger branch from 1b098b5 to 34112a5 Compare October 23, 2026 12:02

ranxianglei and others added 2 commits October 23, 2026 22:09

ranxianglei force-pushed the 2026-10-22_seq-ledger branch from 04d5656 to a931892 Compare October 23, 2026 14:13

Merge branch 'master' into 2026-10-22_seq-ledger

e444b1d

ranxianglei merged commit 02f544d into master Oct 24, 2026
25 checks passed

ranxianglei mentioned this pull request Oct 24, 2026

Post-fold resaves erase resolved block interval bounds | 折叠后重存会把已解析的块区间边界抹回未知(-1/-1) #4608

Closed

Navigation Menu

Repository navigation

[PR] storage: seq ledger foundations — block interval bounds + ledgerSeq ref stamping (#4316 layer 3, phase 1) - #4346

[PR] storage: seq ledger foundations — block interval bounds + ledgerSeq ref stamping (#4316 layer 3, phase 1)#4346
ranxianglei merged 3 commits into
masterfrom
2026-10-22_seq-ledger

ranxianglei commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 24, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ranxianglei commented Oct 23, 2026

What lands

Why phase 1 is safe

Tests (tests/seq-ledger.test.ts, 3/3)

Next (separate PRs, per the issue)

Uh oh!

github-actions Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Built Package Artifact

Latest build — install this exact version

Recent builds (commit → version)

Download artifact

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

ranxianglei commented Oct 23, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 24, 2026

📦 Merged Build Published

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Oct 23, 2026 •

edited

Loading