Skip to content

fix: preserve non-text blocks in Anthropic tool_result via sidecar - #367

Merged
ranxianglei merged 2 commits into
masterfrom
2026-09-24_anthropic-tool-result-image-sidecar
Sep 25, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-09-24_anthropic-tool-result-image-sidecar

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #366

问题

anthropicToCore() 的 tool_result 分支对结构化 content 数组只提取文本(src/wire/anthropic.ts:101),image block 没有 .text → 像素在解析期蒸发;tool_result 消息不挂 sidecar,重建侧只从 m.text 还原 content。宿主经 Anthropic 线发送带图工具结果时每一轮往返都静默丢图(与 billion-context#1205 同类,但发生在 tool_result 内嵌场景,新会话第一条就丢)。

修复(与 #365 同一原则)

  1. 解析期挂 sidecar:content 数组含非 text block 时,把整个原始 tool_result block 存入既有 rawAnthropicBlock 字段(bili-message.ts:27-29 的文档早已预留该用途)。纯文本 tool_result 不挂 sidecar,保持旧形状字节稳定。
  2. 重建期原样吐回:coreToAnthropic 的 tool-result 分支优先使用 sidecar verbatim(与 user image 同路径),content 顺序、is_error、cache_control 全部保留。
  3. 折叠保护:hasMediaPayload(src/protected.ts)识别该载体——rawAnthropicBlock.type === "tool_result" 且 content 数组含非 text block 时为 true。由此自动接入 fix: preserve unknown OpenAI content parts and protect media payloads from folding #365 的三处机制:BLOCKED ref(assignRefsNode)、compressible-range gap(buildCompressibleRanges)、显式 range 的 media carve-out + warning(applySingleRange)。
  4. 配对原子性:无需新增逻辑——media result 被 carve 后,turn-integrity gate(INV2)自动把配对 call 一并 withdrawn,call/result 永不拆散(有测试钉住)。

行为变更披露(old → new)

项 旧 新
带非 text block 的 tool_result 经 proxy 往返 image/document block 静默丢弃,content 退化为 join 字符串 原 block 数组逐字节恢复,顺序不变
带图 tool_result 的可压缩性 可被折叠(= 数据销毁) BLOCKED ref,永不出现在 compressible range;显式 range 覆盖时被 carve out 并 warn
配对 tool-call(range 覆盖时) 可单独折叠留下孤儿 call integrity gate 自动 withdraw,与 result 一起保持可见
AnthropicToolResult.content 类型 string | AnthropicTextBlock[] string | (AnthropicTextBlock | AnthropicImage)[] —— 跨仓注意:adapter 若对 tool_result content 做 c.text 映射会收到编译错误,需处理 image 变体

不变的:text 提取逻辑与 id seed 逐字节一致(升级前后同一逻辑消息得到同一 id,ref 不漂移);纯文本 array / string content 的 tool_result 仍按旧路径重建为 join 字符串;OpenAI/Responses/Google 线不受影响。

验证

  • issue 给出的最小复现形状:修复前 rebuilt content 退化为 "done\n"、image 消失;修复后 [text, image] 原序逐字节恢复,hasMediaPayload = true。
  • 新增 7 个回归测试(4 codec round-trip + 3 保护/配对):全套件 1036 pass(typecheck/build green)。
  • format:check 在本仓库 master 上即有 119 个文件的既有失败(无 .prettierrc,代码风格与 prettier 默认不符);CI 实际只跑 typecheck+test+build,本 PR 未引入任何新的格式违规,也未顺带 reformat(避免 mass whitespace diff)。

依赖

基于 #365 的 head(90588d1)stacked——hasMediaPayload 由 #365 引入。请先合 #365;合并后本 PR 可直接合,或我 rebase 到 master。

范围外(issue 中列出的其他盲区)

issue 提到的 anthropic 未知 top-level block 无 default 分支、google 线远程 fileData 引用未挂 sidecar——按「一个 issue 一个 scope」未在本 PR 处理,是否立项请 owner 定。

ework-agent added 2 commits September 24, 2026 01:46
… from folding

OpenAI chat wire: user content parts that are neither text nor image_url (e.g. DeepSeek Files API {type:"file"} refs) were silently dropped by the parse/rebuild round-trip (ranxianglei/billion-context#1205). Non-text parts now ride rawOpenaiContentParts verbatim in wire order; coreToOpenai already re-emits them unchanged. A lone string-URL image keeps the legacy singular sidecar shape byte-stable (id seed untouched).

Compression: messages carrying media payloads (image/file sidecars) can no longer be folded into a summary — folding destroys bytes that live outside msg.text and hosts cannot resend (ranxianglei/billion-context#1188). Such messages get a BLOCKED ref, are excluded from recommended ranges, and are carved out of applied ranges with an operator warning. Escape hatch remains opt-in stripImages.

Behavior change: image/attachment messages were foldable before (pixels permanently lost); they are never folded now. Full suite: 1029/1029.
tool_result content arrays carrying image blocks lost them at parse time (c.text is undefined for non-text blocks) and coreToAnthropic rebuilt content from m.text alone, so every proxy round-trip silently dropped tool-produced images. Store the original block in rawAnthropicBlock when the content array holds non-text blocks and re-emit it verbatim; extend hasMediaPayload to recognize that carrier so such results are never folded. Text extraction and id seeds are byte-identical to before.
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-24_anthropic-tool-result-image-sidecar (035b489)

Option A — Install from npm PR tag (recommended)

In your adapter project:

npm install acp-kernel@pr-367

Each push to this PR publishes a new version under the pr-367 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf acp-kernel-pr367.tgz
npm install ./package

This comment is automatically updated on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

收到。这是 #366 会话(ses_7faa...)已完成的 PR,我不重做,按 PR 审查规范走:① 核对原 issue(#366)方向与根因定位;② 拉取分支核对代码(注意本 PR stacked 在 #365 的 90588d1 上);③ 本地最小验证(typecheck/test/build);④ 回归清点 + 测试有效性 + diff 整洁度。有结论回帖。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review complete on head 035b489 (stacked on 90588d1). Verdict per gate:

① Original issue — direction correct. Root cause verified on master b8b4d22 by code read: parse did b.content.map((c) => c.text).join("\n") (image blocks contribute undefined → "done\n"), attached no sidecar, and rebuild emitted content: m.text ?? "". Loss happens at parse time, every round-trip — #366's diagnosis is right, and this fix attacks the root cause (carrier + protection), not the symptom. The rawAnthropicBlock field doc (bili-message.ts:27-29) literally pre-reserved this use ("an image or a structured tool_result"); this PR implements the missing half.

② Regression inventory (existing behaviors on touched paths):

  • string-content tool_result: untouched (typeof check first) — pinned by test.
  • text-only array tool_result: no sidecar, still rebuilt as joined string — pinned by test.
  • id seeds: (c.type === "text" ? c.text : "") is byte-identical to c.text under join (String(undefined)===""); test pins res.text === "done\n" → same message, same id across upgrade. ✓
  • user-image path: early return before the new branch, unchanged.
  • cache_control: map still populated; raw-block path re-emits the identical value.
  • hasMediaPayload: behavior identical for all pre-existing carriers (bare/tool_result sidecars without non-text content fall through exactly as before); additive only.
  • OpenAI/Responses/Google wires: untouched.
    All four disclosed old→new rows match what the code actually does. No unintended regressions found.

③ Code. Minimal (4 files, +244/−5), mirrors the existing user-image sidecar pattern. Mechanism chain verified end-to-end: hasMediaPayload → BLOCKED ref (compress.ts:605-610) → recommendation gap (recommend.ts:202-208) → explicit-range carve via filterProtectedToolMessages (compress.ts:1272-1298, pass 1 also shields the paired call's id) → INV2 pair gate (turn-integrity.ts:188-201) withdraws any call whose result was carved, so call/result never split. CCR/absorb already skip BLOCKED refs (ccr.ts:297/396, absorb.ts:163), so media results consistently bypass placeholder offload — no conflict between the two mechanisms.

④ Tests. Real teeth: round-trip tests pin the exact bug shape ([text, image] byte-for-byte, original order) and the legacy shapes (string / text-only array stay sidecar-free); protection tests pin both sides — advertised ranges never span the media result nor its call, and an explicit range keeps both visible together with both warnings. A regression of either the codec or the pairing would fail these.

⑤ Diff. Clean — every line serves this PR; no whitespace churn, no unrelated files. CI green on both heads (typecheck+test+build on Node 22/24); confirmed ci.yml does not run format:check, so the master's 119 pre-existing prettier failures are correctly out of scope here.

Non-blocking nits (leave or file later, your call):

  1. The widened union string | (AnthropicTextBlock | AnthropicImage)[] omits document/unknown future types Anthropic allows inside tool_result content arrays. Runtime is fine — the sidecar path keys off any non-text type — but TS-typed hosts constructing such bodies would get compile errors against the declared type. Could widen later if it bites.
  2. applySingleRange computes mediaExcluded (compress.ts:1002-1010) solely for warning text while the actual removal happens in filterProtectedToolMessages. Same predicate, same input set, cannot diverge — just redundant; fine as-is.
  3. No round-trip test for a document-type block (pairs with nit 1).

Process notes: duplicate screening found no parallel work; the two out-of-scope blind spots (anthropic unknown top-level block, google remote fileData) have no issues yet — per loop guard they stay owner-decision items, as you noted. Per §7.4 this PR touches a §7.3 wire-artifact contract (rebuild output shape) plus a cross-repo type change, so it stays human-gated regardless.

Recommendation: mergeable once #365 lands (or rebase to master after that merge — either works, mergeable_state is clean). I can't merge — human-only operation: #367

中文摘要:审核了 PR #367(Anthropic tool_result 内嵌图片在解析期静默丢失的修复,sidecar 载体 + 折叠保护),已逐行核对根因、四处既有行为无回归、7 个新测试断言有牙且钉住精确 bug 形状,diff 干净、CI 全绿——可以合并(需先合 #365)。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Anthropic 线 tool_result 内嵌 image block 解析期静默丢弃(与 #1205 同类)

1 participant