Design discussion: should localInputEstimate count images by BILLING mode (compress triggers move) or by context-window pressure? #1118

Activity

1. ranxianglei commented on Sep 22, 2026

   

   ranxianglei

   on Sep 22, 2026

   OwnerAuthor

   More actions

   🤖 Powered by ework · qwen3.8-27b

   [bot] 🏷 Analysis done — recommendation: keep the hunk, document the semantics, no new knob. Every claim below verified against source on current master (9c01dbe, v0.1.134); the #1072 hunks re-read from the PR diff.

   First, the framing dissolves: there is only one master. The "two masters" premise assumes some consumer of the number exists for cost accounting. There isn't one — every imageBillingFor call site makes a window-fit decision:

   Consumer	Site	Decision
   Output clamp ×2	src/server.ts:2434, :2869	reserve headroom so input + max_output fits the window
   Side-request guard	src/server.ts:1550	reject side requests that would overflow the window
   Safe-forward payloadEstimate	src/server.ts:3456→:3476 (#767/#496/#300)	forward-once fit gate
   Nudge fallback ×3 (this hunk)	src/server.ts:2277/2462/2920 → effectiveTokenCount src/server.ts:2106-2119	numerator of decideNudge(tokenCount, limit) where limit = window

   Nothing in bili invoices anything — "billing" in imageBilling denotes the provenance of the formula (the provider's published per-image token figures, documented alongside its pricing), not a cost computation. And when the provider reports usage, the true count wins automatically: effectiveTokenCount returns lastInputTokens first (src/server.ts:2107). The billing-derived figure is strictly a blind-spot filler for upstreams that never report usage (the #728 regime). The codebase's own invariant says exactly this: "Every payload-size decision below consults this so one over-estimate cannot block all of them at once" (src/server.ts:635-639). This hunk brings the last consumer into line with the other five.

   Q1 — billing mode or dedicated context-accounting estimator? Keep the billing mode (keep the hunk). There is no independent observable for "context accounting": the proxy cannot see how many tokens the upstream counts an image except through usage reports, which take precedence whenever present. The published formulas are the best available proxy for the blind spot, and for every known provider they coincide with what the model actually counts (OpenAI/Anthropic bill what they count, by tiles). A second estimator would be a second number that must agree with five other consumers anyway, sourced from… the same published formulas. Calibration direction also matters: pixels mode deliberately overestimates both first-party ceilings for typical screenshots (src/image-tokens.ts:163-183 — 85 + 170×tiles, 765–2805), unparsable containers fall back to flat 16384 (PIXEL_IMAGE_FALLBACK_TOKENS, src/image-tokens.ts:31), remote URLs are flat 4096 in both modes (src/image-tokens.ts:25) — conservative for gates that fail closed.

   Q2 — separate knob? No. Its only reach is the narrow #728 fallback path (non-reporting upstreams), and its hypothetical benefit requires a provider that bills ≠ counts its context — none known. That would add config surface plus a consistency burden across six call sites for a hypothetical. Operator override already exists at three levels: per-route imageBilling, global BILI_IMAGE_BILLING (live-read), and BILI_IMAGE_TOKEN_CAP as a per-image ceiling (applyCap, src/image-tokens.ts:65-68). If a bills≠counts provider actually appears, set imageBilling to match the model's counting (documented); a dedicated knob earns its place then, with real data.

   Q3 — default-behavior delta: much smaller than the issue framing suggests, and never in the wrong direction. One structural fact dominates: effectiveTokenCount caps the recorded estimate by the current request's inbound bound, and that bound is text-only — estimateCoreMessagesUpper sums m.text chars, and the wire codecs move images out of CoreMessage.text into sidecars (src/preflight.ts:198-202, src/image-tokens.ts:1-9):

   const raw = estimateCoreMessagesUpper(msgs);
   return Math.min(est, raw);   // src/server.ts:2117-2118

   So whenever the previous turn's image contribution exceeds the current turn's text growth, the image term is capped away and the numerator is effectively the current request's text bound — regardless of billing mode. Per route class (reporting upstreams have zero delta everywhere — lastInputTokens takes precedence):

   Route	Delta from the hunk
   Unknown host (auto→bytes), explicit bytes	none — same mode, same term
   First-party pixel host (auto→pixels), explicit pixels	per-image term moves b64/4 → ≤2805 (16384 unparsable fallback). Where the term survived the cap on master (light image loads in chatty sessions), it was overstated up to ~15× vs tiles → trigger now moves later, toward truth. Where it was capped away (heavy loads), no delta — residual undercount tracked separately below. Net: toward the provider's published figure either way
   BILI_IMAGE_BILLING env set	now honored at these 3 sites too (previously ignored there) — consistent with the other five consumers

   Worst-case inversion: an unparsable container under ~64KB charges the flat 16384 in pixels mode vs its smaller b64/4 under bytes — bounded, conservative (overestimate), and dial-able via BILI_IMAGE_TOKEN_CAP. The feared case ("pixels billing on a provider counting fewer context tokens" → early compression) requires a host whose model counts less than the OpenAI tile formula while being classified/configured as pixel-tile — contradicting the classification's own premise — and is still bounded by the 2805/img ceiling.

   One pre-existing defect found during this analysis → filed as #1119 (来源-marked): the same text-only min() cap means that on byte-counting relays with non-reporting upstreams, large inline images make the true input exceed the bound while the numerator stays text-only → late compression / 400 risk. Identical before and after the #1072 hunk (orthogonal to it); suggested fix there is making the bound image-aware.

   Recommendation: merge the #1072 hunk as-is and add a one-line doc update so the semantic owner is explicit. Proposed wording for both imageBilling sections in CONFIGURATION.md (lines 130, 218) and the BILI_IMAGE_BILLING row (line 492):

   The resolved mode also feeds the silent-backend nudge fallback (localInputEstimate, #728): for upstreams that never report usage, compress-trigger timing follows this setting's per-image figure. For any upstream that reports usage, the reported count always takes precedence and the setting affects only fit gates (preflight, output clamp).

   No code change beyond docs. Happy to land that doc commit on the PR branch once you confirm keep-billing.

   中文摘要:核实全部消费方后建议保留 #1072 的计费模式改动——localInputEstimate 的所有消费方都是窗口压力估算(无任何成本计算),"billing" 只是公式来源的命名,且 usage 上报存在时真实计数永远优先;实际行为差异远小于 issue 预期(仅影响不上报 usage 的上游 + 像素分类路由,方向是向 provider 公布的 tile 计数收敛,且受 min() 上限约束后幅度很小);不加新旋钮(现有 imageBilling/BILI_IMAGE_BILLING/BILI_IMAGE_TOKEN_CAP 三级覆盖已足够),补一行文档说明语义归属即可;分析中发现的 min() 文本上限对图片盲区问题已另立 #1119。
2. ranxianglei commented on Sep 22, 2026

   

   ranxianglei

   on Sep 22, 2026

   OwnerAuthor

   More actions

   简述这个是啥问题
3. ranxianglei commented on Sep 22, 2026

   

   ranxianglei

   on Sep 22, 2026

   OwnerAuthor

   More actions

   🤖 Powered by ework · qwen3.8-27b

   [bot] 🏷 简述如下:

   背景:bili 在判断"要不要触发压缩"时,需要一个输入侧 token 估算值(localInputEstimate)。对会回报 usage 的上游,直接用真实计数;只有对从不回报 usage 的上游(如 ChatGPT 登录态后端,#728),才用这个本地估算值喂给压缩阈值引擎。其中图片部分的计量方式(#488/#767 引入)有两种模式:bytes(base64 长度/4,保守)和 pixels(解析图片头按 OpenAI tile 公式算,约 765–2805/张)。

   问题:PR #1072 把三个估算点从固定 bytes 默认值改成了跟随路由配置的计费模式(imageBillingFor)。但这里有个语义错位:压缩阈值守护的是上下文窗口压力(窗口快满了就压缩),而图片计量现在取自一个**计费(billing)**配置。如果某路由配的计费方式比 provider 实际的上下文计数更贵,估算就会高估窗口占用 → 压缩比必要的更早触发(多一轮摘要丢失);反过来则低估 → 压缩太晚、有 400 风险。也就是说:压缩节奏会不会跟着商业配置走? 这是设计取舍问题,不是机械 bug,所以单独拎出来讨论。

   我的结论(已在上条详述证据):保留 #1072 的改动,不加新旋钮,补一行文档说明语义归属。理由:① 这个数字的所有消费方都是窗口适配决策,没有任何成本计算,"billing" 只是公式来源命名;② proxy 无法独立观测 provider 给图片计多少 token,公布的计费公式就是唯一可观测的代理值,且像素模式刻意向上取整;③ 上游一旦回报 usage,真实计数永远优先,这个估算只在盲区生效;④ 实际行为差异远小于预期——未知主机/bytes 路由零变化,像素分类路由也只是把单图计量从 b64/4 收敛到 ≤2805,方向是向真实计数靠拢。

   另:分析中发现一个独立的既有缺陷(与 #1072 正交)——effectiveTokenCount 的 min() 上限只数文本、看不到图片,导致重图会话在静默后端上仍会低估压力、压缩偏晚,已立 #1119。
4. ranxianglei commented on Sep 22, 2026

   

   ranxianglei

   on Sep 22, 2026

   OwnerAuthor

   More actions

   🤖 Powered by ework · qwen3.8-27b

   [bot] 🏷 更正上一条里 #1119 的链接(域名少了个点):#1119
5. ranxianglei

   closed this as completedon Sep 26, 2026