server: keep live KV reusable when clients strip transient metadata blocks - #727
Open
Flor1an-B wants to merge 1 commit into
Open
server: keep live KV reusable when clients strip transient metadata blocks#727Flor1an-B wants to merge 1 commit into
Flor1an-B wants to merge 1 commit into
Conversation
…locks Coding agents (OpenCode, Claude Code, Kilo Code/Roo Code) inject per-turn transient metadata into user message content (<system-reminder>, <environment_details>) and rebuild the next request's history without it. The live KV keeps the bytes that were actually rendered, so the replayed transcript no longer token-matches the sampled session and every user turn pays a full reprefill (issue antirez#364). This is the same shape hidden thinking already handles: the live state is richer than the visible replay, so the existing visible-key continuation mechanism applies. After a finished turn, remember the transcript the next request is expected to render -- the current one minus the transient spans -- keyed to the live token frontier. A client that replays the blocks verbatim simply never matches the key and still gets exact token-prefix matching; nothing changes for it. Original design and implementation by @adv0r in antirez#378. That PR now conflicts with main: native Metal/CUDA session batching (36cd0ca and follow-ups) added a server_slot parameter to the session/checkpoint functions it touches (thinking_live_remember/clear, remember_thinking_checkpoint, canonicalize_tool_checkpoint) after antirez#378 was opened. This threads the same strip_transient_blocks()/should_remember_transient_checkpoint()/ remember_transient_checkpoint() design through the slot-aware signatures; the diff is otherwise unchanged from antirez#378, same test names and assertions. Tested: full `make ds4_test` suite passes. Live A/B on the real quant (ds4-server, 2-turn conversation with a <system-reminder> block dropped on replay, as OpenCode does): - before: `live kv cache miss live=88 prompt=97 common=5 reason=token-mismatch` - after: `transient live checkpoint remembered ... stripped=128` then `thinking live continuation match=visible-prefix cached=30 prompt=54` Also re-verified a 2-turn tool-calling round trip with no transient blocks present still gets exact-DSML-replay continuation unchanged (18 new tokens prefilled instead of a full 406-token reprefill), confirming the canonicalize -> remember-or-clear restructuring is behavior-preserving when nothing is stripped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #364.
Coding agents (OpenCode, Claude Code, Kilo Code/Roo Code) inject per-turn transient metadata into user message content (
<system-reminder>,<environment_details>) and rebuild the next request's history without it. The live KV keeps the bytes that were actually rendered, so the replayed transcript no longer token-matches the sampled session and every user turn pays a full reprefill.This is the same shape hidden thinking already handles: the live state is richer than the visible replay, so the existing visible-key continuation mechanism applies. After a finished turn, remember the transcript the next request is expected to render — the current one minus the transient spans — keyed to the live token frontier. A client that replays the blocks verbatim simply never matches the key and still gets exact token-prefix matching; nothing changes for it.
Credit: original design and implementation by @adv0r in #378. That PR now has a real conflict with
main, not just a stale rebase: native Metal/CUDA session batching (36cd0caand follow-ups) added aserver_slotparameter to every session/checkpoint function it touches (thinking_live_remember/clear,remember_thinking_checkpoint,canonicalize_tool_checkpoint), added after #378 was opened. This PR threads the samestrip_transient_blocks()/should_remember_transient_checkpoint()/remember_transient_checkpoint()design through the slot-aware signatures — the diff is otherwise unchanged from #378, same test names and assertions. Opening as a new PR rather than pushing to #378's branch since I don't have write access to it.Testing:
make ds4_testsuite passes clean.ds4-server, 2-turn conversation with a<system-reminder>block dropped on replay between turns, exactly as OpenCode does):live kv cache miss live=88 prompt=97 common=5 reason=token-mismatch— full reprefill.transient live checkpoint remembered ... stripped=128then, on the next turn,thinking live continuation match=visible-prefix cached=30 prompt=54— live continuation, no reprefill.canonicalize_tool_checkpoint→ remember-or-clear restructuring is behavior-preserving when nothing is stripped.