fix: keep system prompt byte-stable for prefix cache by ranxianglei · Pull Request #1 · ranxianglei/billion-context

ranxianglei · 2026-08-06T15:02:22Z

Problem

Cache hit rate collapsed to ~0% across turns. Root cause: the nudge (which changes every turn — token count, growth %, dynamic compress example) was merged into the system prompt. The system prompt is the prefix-cache anchor; any byte change at its tail invalidates the entire cached prefix.

Reproduce

Before fix, consecutive requests saw system_len jump every turn:

system_len=17501 → 23432 → 17501 → ...

Fix

Mirror pai-acp's design: the system block carries ONLY the static compress prompt (byte-stable); the nudge is appended as a separate trailing user message.

system_len=16871 → 16871 → 16871   ✅ stable

Changes

injectSystem(): drop nudge param, only inject static compress prompt
prepareOpenai(): append nudge as trailing {role:user} message
prepareAnthropic(): append nudge as trailing {role:user} message

Verification

typecheck ✅ / 79 tests ✅ / build ✅
end-to-end verified: system_len constant across 3 consecutive turns

The nudge (which changes every turn — token count, growth %, dynamic compress example) was being merged into the system prompt. The system prompt is the prefix-cache anchor: any byte change at its tail invalidates the entire cached prefix, so cache hit rate collapsed to ~0% across turns. Reproduce before fix: consecutive identical requests saw system_len jump 16871 → 23432 → 16871 ... each turn. Fix mirrors pai-acp's design: the system block now carries ONLY the static compress prompt (byte-stable); the nudge is appended as a separate trailing user message instead. Verified system_len stays constant across turns (16871 → 16871 → 16871). - injectSystem(): drop nudge param, only inject static compress prompt - prepareOpenai(): append nudge as trailing {role:user} message - prepareAnthropic(): append nudge as trailing {role:user} message 79 tests pass. typecheck clean.

Review #2 findings (more comprehensive than #1): #2 (compress-loop*.ts): fetch-timeout timer leak across compress-loop rounds. activeClearTimer = clearTimer overwrote the previous round's timer without clearing it, so rounds 1..N-1 each leaked a 10-min timer that self-fired harmlessly but accumulated handles in the event loop over a long session. Clear the previous timer before overwriting (3 lines, both openai and responses variants). #5 (orphan-gc.ts): orphanStreak was monkey-patched onto the kernel's Block type via (block as unknown as {orphanStreak?:number}).orphanStreak — fragile, no type safety, breaks if the kernel Block shape changes. Replaced with a module-level WeakMap<object, number> keyed by the block object. TS now type-checks the access; counts are GC'd when a block is dropped. #7 (README.md): stale test count (79 → 136) and missing env-var docs (BILI_PERSIST, BILI_PERSIST_DEBOUNCE_MS, MAX_SESSIONS, BILI_SESSIONS_DIR). Both fixed. #8 (tests/*.test.ts): 7 type errors under tsc --noEmit (role: string not narrowing to the union literal; Session literals missing new fields after inFlight/persisted/blockContents were added). Fixed with 'as const' on roles and the missing fields. Full tsc --noEmit now 0 errors. #10 (persist.ts): comment referenced AGENTS.md which doesn't exist; replaced with an inline note on the known concurrency limitation.

…ianglei#2,ranxianglei#3,ranxianglei#4,ranxianglei#6) ranxianglei#1 (CRITICAL) protocol conversion was lossy: image->[image] placeholder, thinking signature dropped (Anthropic rejects unsigned thinking), tool_result .is_error dropped, OpenAI developer->system, image_url/input_image deleted. Fix: BiliMessage sidecar (src/bili-message.ts) extends CoreMessage with optional raw*/signature/isError/originalRole/image* fields. The kernel only does {...msg} spread copies, so sidecar survives the pipeline to coreToX, which replays the original block losslessly. 8 round-trip tests added. #2b (MAJOR) session-id extractKey() lowercased the API key before hashing — two distinct valid keys differing only in case collided into one hash bucket. Remove toLowerCase; keep raw (trim only). ranxianglei#3 (MAJOR) auto-update stale-lock takeover never deleted the stale lock file, so writeFile({flag:'wx'}) always failed EEXIST -> catch returned null -> a single crash during update permanently blocked all future auto-updates. Add unlink(LOCK_FILE) before the wx write, with ENOENT tolerated. ranxianglei#4 (MAJOR, security) /__bili/ management endpoints (config read/write, reload, session stats) had no auth. With --host 0.0.0.0 a remote caller could rewrite upstream routing to exfiltrate API keys. Gate all /__bili/ + /__acp/ paths to loopback connections only (127.x / ::1); model-traffic endpoints stay open. ranxianglei#6 (MINOR) build artifact was not self-contained: acp-kernel stayed as a runtime dependency (dist had 'from "acp-kernel"'). tsup now sets noExternal:[acp-kernel,node-forge] and acp-kernel moves to devDependencies, keeping the exact pin (0.0.17). dist verified: 0 'from acp-kernel' refs.

…eload Review feedback from ranxianglei (PR#73 comments ranxianglei#3-ranxianglei#4): 1. [ranxianglei#9 critical] Restore Responses session-identity fallback chain (responses.ts): header → body.session_id → metadata.session_id → previous_response_id → content-fingerprint(hashId). The previous generated-randomUUID() broke compression for headerless clients (Pi, plain OpenAI SDK) — every request got a new session so compression state never accumulated. 2. [ranxianglei#1+ranxianglei#2 critical] parseState robustness (codex-takeover.ts): corrupted state file now returns undefined (treat as stale/recoverable) instead of throwing, which would leave Codex CLI unusable. Added sectionPrefixLength type validation (non-negative number) to prevent NaN slicing that could wipe the entire config.toml. 3. [Comment#4 ranxianglei#2] PUT /__bili/config onChanged now also refreshes routes (server.ts): previously only proxy fields were refreshed, so pure API callers (non-Web-UI) had to also POST /reload for provider config to take effect. Now PUT guarantees save-then-apply in one call. 4. Fix misleading context-window hint on routing page (page.ts).

…mum (#3071) #3071: on a model with NO declared output ceiling, the proxy's starved-budget restore used the client's own last non-starved high-water as its only lower bound. That estimate decays unbounded from the client's raw-history size (..., 3071, 2061, 252, <=200), so once starved the session pins at a death rattle and every turn truncates at max-tokens (finish=length, 0 visible chars); a fresh fork child inherits the rattle and forwards max_tokens=1 verbatim. On v0.1.195 this blanked 9 turns across 3 DSH sessions in one day. Fix (minimal — matches the issue's suggestion #1 literally): when no source knows a ceiling, the STARVED / born-starved / fork-inherited restore target is floored at OUTPUT_RESTORE_ABSOLUTE_FLOOR (8192). A non-starved value is still trusted verbatim (client's explicit choice; far smaller blast radius), and a known-ceiling model is untouched. The no-ceiling WARN dedup moves from once-per-model (global) to once-per-session+model so each affected session logs its own WARN instead of staying silent behind another session's first hit. Paths fixed: 1 (no-ceiling restore pinned at poisoned high-water), 2a (killed session got zero warns), 2b (fork child forwarded the starved budget verbatim). Path 3 (headroom reserving the shrunken budget) needs no change: the restore runs before the headroom reservation reads max_tokens, so the floored value flows in automatically. Deferred to owner sign-off (product decisions / pinned fail-closed behavior, flagged in-thread): #4 invalidating the #2490 dsh planning floor after a refused replay (contradicts its sticky-by-design overflow guarantee), and #5 retrying a starved finish=length turn (reverses #2756's deliberate exclusion; the root-cause floor fix should remove most of those empty turns first). Closes #3071

…anxianglei#2) Review ranxianglei#2 findings (more comprehensive than ranxianglei#1): ranxianglei#2 (compress-loop*.ts): fetch-timeout timer leak across compress-loop rounds. activeClearTimer = clearTimer overwrote the previous round's timer without clearing it, so rounds 1..N-1 each leaked a 10-min timer that self-fired harmlessly but accumulated handles in the event loop over a long session. Clear the previous timer before overwriting (3 lines, both openai and responses variants). ranxianglei#5 (orphan-gc.ts): orphanStreak was monkey-patched onto the kernel's Block type via (block as unknown as {orphanStreak?:number}).orphanStreak — fragile, no type safety, breaks if the kernel Block shape changes. Replaced with a module-level WeakMap<object, number> keyed by the block object. TS now type-checks the access; counts are GC'd when a block is dropped. ranxianglei#7 (README.md): stale test count (79 → 136) and missing env-var docs (BILI_PERSIST, BILI_PERSIST_DEBOUNCE_MS, MAX_SESSIONS, BILI_SESSIONS_DIR). Both fixed. ranxianglei#8 (tests/*.test.ts): 7 type errors under tsc --noEmit (role: string not narrowing to the union literal; Session literals missing new fields after inFlight/persisted/blockContents were added). Fixed with 'as const' on roles and the missing fields. Full tsc --noEmit now 0 errors. ranxianglei#10 (persist.ts): comment referenced AGENTS.md which doesn't exist; replaced with an inline note on the known concurrency limitation.

ranxianglei changed the base branch from main to master August 6, 2026 15:04

ranxianglei merged commit ab06546 into master Aug 6, 2026
2 checks passed

This was referenced Aug 8, 2026

fix: post-merge review fixes + per-route context window + XDG config #6

Merged

fix: 6 audit findings (lossless protocol + security + Windows + packaging) #65

Merged

ranxianglei mentioned this pull request Aug 10, 2026

feat: codex takeover, history sync, and embedded web UI #73

Merged

ranxianglei mentioned this pull request Aug 11, 2026

fix: Codex textProtocol 输出重复 + SPEC 同步 #93

Merged

This was referenced Sep 9, 2026

fix(mcp): recover tool calls when the shim's conversation id goes stale after host resume #657

Merged

fix: drop hostUsageCredit — report provider-measured usage only (#660) #691

Merged

ranxianglei deleted the fix/cache-prefix-stability branch October 15, 2026 05:33

superman007-glitch mentioned this pull request Oct 16, 2026

web_search on DSH never completes: the side request (server-tool utility call) lane forwards upstream but writes 0 bytes back to the client #3396

Closed

xSAR-research mentioned this pull request Oct 22, 2026

Codex TUI settings are lost between MCP-enabled Bili launches #4318

Open

Navigation Menu

Repository navigation

fix: keep system prompt byte-stable for prefix cache - #1

fix: keep system prompt byte-stable for prefix cache#1
ranxianglei merged 1 commit into
masterfrom
fix/cache-prefix-stability

ranxianglei commented Aug 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ranxianglei commented Aug 6, 2026

Problem

Reproduce

Fix

Changes

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant