[Bug] prune silently drops a new user message whose text matches an already-folded one (content-hash id collision)

Description

Symptom: a short message the user types in the chat box (e.g. 都按推荐来) never reaches the model. That turn instead sees a role:system [Compressed conversation section] — <topic> summary, and answers as if nothing had been said. The host records the message (its own session events and the body it sends to bili both contain it) — only the body forwarded upstream is missing it.

Trigger (all three must hold):

the session has an active compress block whose effectiveMessageIds covers some short text (in this report: 都按推荐来 → h_f68df23f93ebd113);
that text's older copy has already left the host's request (replaced by the host's compaction checkpoint / surface replace), so no second copy of it is on the wire;
the user sends the exact same text again.

The new message is then assigned the bare content-hash id (byte-identical to the old one) ⇒ isCovered() returns true ⇒ rebuildMessages() silently continues ⇒ the user message is dropped and its slot is replaced by the owning block's summary.

Ordered repro steps

Start bili (dsh-profile install) with ACP_DUMP_BODY=1 so every request lands as ~/.local/state/billion-context/dumps/req-<ts>-session-<sid>.json; the raw dir then also holds <ts>-session-<sid>-INCOMING.txt (host → plugin, raw HTTP) and -RES.txt.
Let the session accumulate several active blocks, one of which covers a short user message; confirm that text now appears at most once (or not at all) in the request body.
Send that short text again.
Compare the two dumps: the message is present in -INCOMING.txt (measured here at #253 and #255), absent from dumps/req-*.json, where #34 user len=1099 :: [Compressed conversation section] — … sits instead. bili.log's request: N msgs (which is the inbound count) goes up by one, while the forwarded body's message count does not change.

Expected: a fresh user turn should never be dropped because its text equals an already-folded older message. Summary substitution should only ever apply to messages that were actually folded.

Relation to #600: #600 documents that a ref number (mNNNNN) is never reused. This is a different layer — the semantics of the raw content-hash id: identical text ⇒ identical id ⇒ treated as the same message. The host treats the two sends as two independent messages (different turn, different event seq); the plugin treats them as one.

bili version

0.1.158 — already the latest ([update] dsh profile bundle up to date (current=0.1.158 latest=0.1.158 tag=latest)); the repro below is on this version. Install shape is a dsh-managed profile: <profile>/node_modules/billion-context, with [update] install dir is managed by dsh … skipping in-place self-update.

Usage mode

Launcher (child process spawned by the DSH host: node <profile>/node_modules/billion-context/dist/index.js start --host 127.0.0.1 --port <p>, parent = the dsh harness process).

Client and version

dsh 0.1.7-rc.2 (DeepSeek Harness web), node v22.23.2.

Operating system

Windows 11 Pro 10.0.26200 (64-bit, not WSL).

Upstream and model

Private OpenAI-compatible gateway (/zen/go/v1/chat/completions), model deepseek-flash; declared window 750k (usage=14% (101725/750000)), cache hit 100% ([acp-usage] input=101725 cached=101504 (cache hit 100%)).

Compression-related config (optional)

# ~/.config/billion-context/billion-context.json
{"providers": {}}

No compression switches were changed; the host-side DISABLE_AUTO_COMPACT=1 is set by bili itself.

Root cause (verbatim from `dist/index.js`, 0.1.158)

Id derivation is a pure content hash:

hashId2(s) = sha256(s, "utf8").hex.slice(0, 16)
deriveMessageId(role, contentType, text, options) =
  "h_" + hashId2(`${role}|${contentType}|${options.toolCallId ?? ""}|${options.toolName ?? ""}|${text}`)

When the n-th copy of identical text appears within one parse of a request body, ClusterCounter.next(baseId) appends _${n}.

Pruning:

function coveredMessageIds(state){const covered=new Set;for(const block of state.blocks){if(!block.active)continue;for(const id of block.effectiveMessageIds)covered.add(id)}return covered}
function baseIdOf(id){const hash=id.indexOf("#");return hash>0?id.substring(0,hash):id}
function isCovered(id,coveredBases){return coveredBases.has(baseIdOf(id))}

function prune(messages,state,options={}){
  const covered=coveredMessageIds(state);
  if(covered.size===0)return[...messages];
  const coveredBases=new Set;for(const id of covered)coveredBases.add(baseIdOf(id));
  const inject=options.injectSummaries??true;
  const firstUserIndex=messages.findIndex(message=>message.role==="user");   // ← only the FIRST user message is protected
  …
  return stripOrphanedReasoning(stripOrphanedToolResults(stripOrphanedToolCalls(
    rebuildMessages(messages,coveredBases,firstUserIndex,anchors))));
}

function rebuildMessages(messages,coveredBases,firstUserIndex,anchors){
  …
  if(index===firstUserIndex&&firstUserIndex>=0){result.push(messages[index]);continue}
  if(isCovered(messages[index].id,coveredBases))continue;   // ← the new user message disappears here
  …
}

Note one asymmetry: baseIdOf strips only # suffixes, not _N. So when two copies of the same text are on the wire at once, the second one gets h_…_1 and is not judged covered — which is exactly why the resend in incident 3 below arrived.

Evidence (three incidents, same signature)

1) The decisive one (this report, 2026-09-27T05:37Z)

host → plugin (258 messages on disk): #253 user len=5 :: 都按推荐来 and #255 user len=5 :: 都按推荐来, both with id h_f68df23f93ebd113, covered=true owner=b85.
plugin → model (37 messages on disk): #34 user len=1099 :: [Compressed conversation section] — 第二轮 grilling 答复（都按推荐来）+ 实现与 PR #67 全过程; the text 都按推荐来 occurs zero times.
Outcome: that model turn saw only the summary block; the user believed the message had been delivered.

2) 合并吧 (2026-09-27T04:01:05Z)

Host events: agent/inbox/spliced (id 18306ce6-fbde-4522-84cf-cf148420d7b5, text 合并吧) → turn/start → user/message (same id, surfaceOp: append).
bili.log inbound count: previous turn 310 → this turn 312 (+2 = the previous turn's closing assistant message + this user message) ⇒ the host did deliver it.
The model's reasoning for that turn opens verbatim with: The user hasn't said anything new — this is a checkpoint-style message (compressed conversation section)… — i.e. it saw block b45's summary, whose topic is literally 用户"合并吧" → PR #66 已合并.
Block b45 was created at 02:42:51Z ⇒ at 04:01:05 that bare hash was indeed in the covered set.

3) 都按推荐来, first send (03:13:09Z) vs resend (03:22:05Z)

The 03:13:09 send was swallowed (bare hash h_f68df23f93ebd113 covered by b7; b7 created 2026-09-26T13:47:40Z).
The 03:22:05 resend arrived, because the 03:13 copy was still in the body ⇒ the second copy got _1 ⇒ isCovered false. This also explains the only _N-suffixed record in the plugin's ref table (m01970: h_f68df23f93ebd113_1).

Relation to `[acp-drift]`

The same session's bili.log contains:

[acp-drift] fold coverage mismatch: 326/1033 covered message id(s) present in resent history — 707 covered id(s) missing from resent history — mutation (content edit invalidates content-hash refs, fold silently lost) or client-side deletion/truncation (benign, message no longer on the wire); observability complement to the #1328 overflow rescue (#1195)

That counter treats "a covered id that shows up in the request again" as present; it cannot distinguish "the old copy is still on the wire" from "the user just sent a new message with the same text" — the latter also counts as present (and this report's id is indeed among those 326). So the warning has no discriminating power for this case.

Suggested fix

Minimal: in rebuildMessages, protect the last user message in addition to firstUserIndex:

const lastUserIndex = messages.findLastIndex(m => m.role === "user");
…
if ((index === firstUserIndex || index === lastUserIndex) && messages[index].role === "user") {
  result.push(messages[index]);
  continue;
}

Rationale: a fresh user turn is new information by definition, and a content-hash collision must not outrank it; summary substitution should only apply to messages that were actually folded.

Optional (larger): give deriveMessageId an instance dimension (turn/step/ordinal) so identical text in different turns no longer collides. That touches ref stability and the kernel id contract (see #600) and needs to be designed together; the _N suffix shows the kernel tolerates suffixes, but baseIdOf does not strip _N, so the coverage set cannot rely on it.

Workaround (user side)

Do not resend identical text verbatim (changing one character changes the hash — a rewritten message was measured to arrive normally);
or clear that session's plugin state file to reset the covered set (at the cost of losing the compression memory and refs).

Repro tooling note

ACP_DUMP_BODY=1 (with optional ACP_DUMP_DIR) writes every body forwarded upstream to dumps/req-<ts>-session-<sid>.json; combined with the raw dir's -INCOMING.txt this gives a direct diff of "what the host sent" vs "what went upstream" — that diff is how this report was localized. Also note bili.log's request: N msgs is the inbound count (measured: 258 inbound vs 37 forwarded), so it must not be used to judge the forwarded size.

🤖 Powered by ework · qwen3.8-27b

🏷 Verified end-to-end against the shipped kernel — this is real, and the root cause sits one layer below where the repro points.

Duplicate screen: searched open + closed issues/PRs (content-hash collision / resend-dropped / prune-covered) — no existing item. It's distinct from #600 exactly as you frame it: #600 is ref-number (mNNNNN) reuse; this is raw content-hash id semantics. Agree.

What I checked (current master pins acp-kernel 0.0.95; every path confirmed present in the shipped source):

id derivation — deriveMessageId + ClusterCounter (acp-kernel/src/wire/message-id.ts): the counter is instantiated fresh per conversion pass, and its documented contract is "two passes over the same content produce the same cluster numbering (deterministic)". So a re-sent identical text gets the bare h_… again whenever the older copy is not co-present on that pass. That determinism is precisely what re-collides it with the folded id.
baseIdOf strips only #, never _N; isCovered(id) = coveredBases.has(baseIdOf(id)) (acp-kernel/src/prune.ts).
rebuildMessages protects only firstUserIndex and continues on every other covered id (your line refs match).
One link you didn't list, and it matters: assignRefs (acp-kernel/src/refs.ts) does if (map.byRaw[message.id]) continue; — first-wins — and syncBlocks never deletes byRaw entries (it only prunes tokenSnapshot). So the folded original's byRaw[h_…] → mNNN survives across turns.

So the actual chain for a resend where the old copy left the wire is three collapses, not one:

derivation hands the new message the bare h_… (collision);
assignRefs skips it because byRaw[h_…] is already claimed → it gets no ref (uncitable, undecompressable);
prune/rebuildMessages drops it because it's covered and isn't the first user message.

Your incidents 1 & 2 are exactly this. Incident 3 survived only by luck: the old copy was still on the wire, so the new one got h_…_1, and since baseIdOf doesn't strip _N, isCovered missed it.

Layer judgment: this is an acp-kernel identity-contract defect, not a billion-context bug. bili calls the kernel prune directly (src/persist.ts) and duplicates none of this logic — so the fix has to land in acp-kernel (kernel publish → pin bump → bili release).

Why your suggested lastUserIndex guard is a mask, not a fix:

It saves one slot, and only while the collided message is the newest user turn. Send one later message and the collided one is covered + not-first + not-last → dropped again (deferred silent loss).
It does nothing about step 2: even when it survives prune, the message is still ref-less, so the model sees the text but can never cite/compress/decompress it, and the shared-id ambiguity stays live for any range spanning it.
It's role-specific (user); the same collapse hits any content type (a repeated tool result / assistant text) — only user short-text is easy to notice.
Litmus test: same root cause, different shape tomorrow (non-last user message, tool result) → the guard doesn't hold.

Recommended fix (root cause): give the content-hash instance dimension durability. Seed the per-pass ClusterCounter from the session's already-claimed raw ids (e.g. messageRefs.byRaw occupancy, or a small persisted monotonic "used bases" set) so a re-sent identical text always resolves to h_…_1, h_…_2, … and never reuses a claimed bare id. This restores #600's invariant (one id ⇒ one message, forever) instead of routing around it, fixes the drop and the ref-collision in one stroke, is content-type-agnostic (covers the non-user cases too), and makes incident-3-style survival deterministic rather than accidental.

Why I've stopped at analysis rather than opening the PR yet: this is cross-repo (acp-kernel) and touches the #600 identity contract — both are owner-decision territory under our rules, and you flagged the larger fix as needing co-design. I want your pick on scope before I start the kernel change:

(recommended) Root fix in acp-kernel (durable instance counter above). I design + implement + add a regression test reproducing incidents 1/2, release the kernel, then bump the pin here.
(fallback, only if you need the bleeding stopped now) Ship the lastUserIndex guard as an explicitly-labeled interim mitigation (it does not fix the ref half), and track the root fix as a follow-up issue. I wouldn't leave it in long-term — it's a band-aid we'd have to remember to rip out.

My clear recommendation is the root fix alone. Say the word and I'll take it through the kernel PR + release ordering.

进行根本性修复

🤖 Powered by ework · qwen3.8-27b

🏷 收到,做根因修复。方案与顺序如下,开工。

根因修复设计(逻辑全在 acp-kernel;billion-context 只改 pin):

症结:deriveMessageId 的 ClusterCounter 是「每次转换 pass 内」计数。重发的相同文本在旧副本已折叠离开 wire 后,会被重新编号回裸 id h_…,与已折叠副本撞号;随后 assignRefs(first-wins,if (map.byRaw[id]) continue)让它拿不到 ref,rebuildMessages(只保护 firstUserIndex)把它丢掉。三个塌缩同源于这一个撞号。
判别器:某 base 是否被活跃 block 覆盖(coveredBases)。被覆盖 ⇒ 原始已在 block 里、不在 wire 上 ⇒ 任何带该 base 的 live 消息必然是新实例,必须换新 id;未被覆盖 ⇒ 保持原编号(incident 3:旧副本仍在 wire 上、base 未覆盖,in-pass 计数天然区分,不误伤)。
落点:在 wire→core 之后、assignRefs 之前加一步 re-mint——把「base 已被覆盖」的 live 消息重新编号为 h_…_1/_2/…(取未被覆盖且未被本 pass 占用的最小实例号,保留 # 投影尾)。一处集中修改:不动三个 converter 签名、不碰持久化格式(byRaw 只读)、非覆盖消息 id 不变(prefix cache 无影响)、两种压缩模式统一(改在入站转换方向)。
效果:撞号从源头消除,drop 与 ref-less 一并解决;同一 live 消息跨轮 id 稳定(每轮都被 re-mint 成同一个 _k)。

跨仓顺序(acp-kernel 必须先发布,billion-context 才能 bump pin):

acp-kernel:实现 + 回归测试(复现 incident 1/2)+ 本地绿(typecheck / test / build)→ 开 PR(需人工合并,CI 自动发布)。
确认 npm 上有新版本后,billion-context:bump pin + lockfile + rebuild/verify → 开 PR(Closes #1476)。

先做第 1 步,有进展随时同步。

🤖 Powered by ework · qwen3.8-27b

根本修复已实现,落在 acp-kernel(根因所在仓库)。PR: ranxianglei/acp-kernel#459

为什么是跨仓:这个 bug 的三处坍塌全在 acp-kernel,billion-context 只是直接调用它的 prune,没有复制任何相关逻辑。所以正确顺序只能是:修内核 → 发布 → 再在这里把精确 pin 顶上去。

根因:deriveMessageId(acp-kernel src/wire/message-id.ts)的簇计数器每趟转换都重置。一条文本在它上一份被折叠进活跃块之后重新发送时,会重新推导出同一个裸 h_… id。由这一处碰撞派生两个后果:

assignRefs 先到先得(src/refs.ts 的 if (map.byRaw[message.id]) continue),折叠原件已占着 byRaw[h_…],新实例拿不到 ref(不可引用/不可解压);
prune/rebuildMessages(src/prune.ts)丢弃任何非首条 user 的 covered id,于是这条新的 user turn 在上游静默消失。

你复现里它能活下来,纯粹是因为旧副本当时还在 wire 上(趟内计数给了 _1,而 baseIdOf 不剥 _N,所以没被判 covered)——旧副本一旦被折叠,重发就丢。这正是「症状 ≠ 机制」:表面是 prune 丢消息,病根是 wire 编解码给不出持久的实例维度。

修法:新增前置管道节点 reconcile-live-ids(逻辑在 src/instance-reid.ts,src/compress.ts 接入),跑在 assign-refs 和 prune 之前。对每个「有 live 实例携带被覆盖 base id」的内容哈希根,按到达顺序把这些实例重编号为 root_1、root_2…(跳过已被折叠副本占用的号)。每个重发实例因此拿到不同且持久的身份,既得到 ref,又不会被 prune 丢。判别条件刻意收窄:未被覆盖的 base 保持原编号不动——那是旧副本还在、未折叠的形状,趟内计数本就能区分,动它只会搅乱前缀缓存。映射对固定 (state, body) 确定性成立(常驻 live 消息每轮保持同一 _k,只有更新的重发加入才位移),前缀缓存稳定。

这恢复的是内核「id 永不复用」不变式,而不是绕开它。你提的最小改法(在 rebuildMessages 里额外保护 lastUserIndex)我评估后没用:只救一个槽位(再晚一条重复照样丢)、不修缺 ref 那一半、不覆盖非 user 内容类型,也过不了「同样根因换个形状明天再来」的试金石。

行为变更(需人工把关):

规范管道序列新增一个前置节点(§7.3 契约 feat: release CI (npm auto-publish on release branch) #8 节点排序);
wire 效果——某条消息在其旧副本被折叠后重发时,现在带不同的实例 id(h_…_1、h_…_2…),不再与折叠原件撞 id。之前这种 turn 要么被丢、要么没 ref。两种压缩模式的摘要载体不受影响(改动完全在入站 wire→core 调和层,位于 refs/prune 之前)。

预检:typecheck 干净;npm test 1167 通过 / 0 失败(含 8 个新回归测试 tests/issue1476-reid.test.ts,并更新了 tests/pipeline.test.ts 的管道顺序守卫以纳入新节点);build 成功;format:check 绿。

下一步(等你):这个内核 PR 需要你合并 → CI 自动发布到 npm。发布后我会回来把这里 acp-kernel 的精确 pin 顶到新版本、重建验证,再开 PR 关闭 #1476。合并发布之前我先挂起这一步——提前顶 pin 会让 CI 的 npm ci 装不到尚未发布的版本而变红。

🤖 Powered by ework · qwen3.8-27b

补充发布步骤(免得卡住):acp-kernel 与 billion-context 一样,合并 fix PR 本身不触发发布——发布走独立流程。所以是两步:

合并 fix: re-mint live ids that collide with a folded copy (billion-context#1476) acp-kernel#459 到 master;
触发一次发布:Actions →「Release (one-click)」(版本留空 = 在 npm latest 之上自动 next-patch),或走标准 *_release-v* 分支。

等 npm view acp-kernel version 出现新版本后说一声,我立刻回来把本仓的精确 pin 顶上去、重建验证,再开 PR 关闭 #1476。当前 npm 上是 0.0.95(与本仓 pin 一致),发完应为下一个 patch。

fix: re-mint live ids that collide with a folded copy

chore: pin acp-kernel 0.0.97

Navigation Menu

Repository navigation

[Bug] prune silently drops a new user message whose text matches an already-folded one (content-hash id collision) #1476

Description

bili version

Usage mode

Client and version

Operating system

Upstream and model

Compression-related config (optional)

Root cause (verbatim from `dist/index.js`, 0.1.158)

Evidence (three incidents, same signature)

Relation to `[acp-drift]`

Suggested fix

Workaround (user side)

Repro tooling note

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Bug] prune silently drops a new user message whose text matches an already-folded one (content-hash id collision) #1476

Description

Description

bili version

Usage mode

Client and version

Operating system

Upstream and model

Compression-related config (optional)

Root cause (verbatim from dist/index.js, 0.1.158)

Evidence (three incidents, same signature)

Relation to [acp-drift]

Suggested fix

Workaround (user side)

Repro tooling note

Activity

ranxianglei commented on Sep 27, 2026

ranxianglei commented on Sep 27, 2026

ranxianglei commented on Sep 27, 2026

ranxianglei commented on Sep 27, 2026

ranxianglei commented on Sep 27, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Root cause (verbatim from `dist/index.js`, 0.1.158)

Relation to `[acp-drift]`