Skip to content

server: resolve Responses response ids to KV prefixes #752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
perfloop-agent wants to merge 2 commits into
base: main
Choose a base branch
from
Open

server: resolve Responses response ids to KV prefixes #752

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f649d32
bench: exercise end-to-end Responses continuations
perfloop-agent Aug 8, 2026
79a2cc3
server: resolve Responses response ids to KV prefixes
perfloop-agent Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ DS4_LINK_LIBS ?= $(CUDA_LDLIBS)
METAL_LDLIBS := $(LDLIBS)
endif

.PHONY: all help clean test test-metal-session-batch test-mxfp4-cuda test-cuda-session-batch test-cuda-mixed-batch dspark-acceptance dspark-verify-depth mtp-verify-depth cpu cuda cuda-spark cuda-generic cuda-regression strix-halo rocm
.PHONY: all help clean test responses-replay-bench test-metal-session-batch test-mxfp4-cuda test-cuda-session-batch test-cuda-mixed-batch dspark-acceptance dspark-verify-depth mtp-verify-depth cpu cuda cuda-spark cuda-generic cuda-regression strix-halo rocm

ifeq ($(UNAME_S),Darwin)
.PHONY: metal-decode-schedule-bench metal-prefill-variant-bench check-mxfp4-half-lut
Expand Down Expand Up @@ -433,6 +433,15 @@ else
$(NVCC) $(NVCCFLAGS) -o $@ ds4_test.o ds4_help.o ds4_kvstore.o rax.o $(CORE_OBJS) $(CUDA_LDLIBS)
endif

tests/responses_replay_bench.o: tests/responses_replay_bench.c ds4_server.c ds4.h ds4_ssd.h ds4_distributed.h ds4_help.h ds4_kvstore.h rax.h
$(CC) $(CFLAGS) -DDS4_NO_GPU -DDS4_TEST_HOOKS -Wno-unused-function -c -o $@ $<

tests/responses_replay_bench: tests/responses_replay_bench.o ds4_help.o ds4_kvstore.o rax.o ds4_cpu_test_hooks.o $(filter-out ds4_cpu.o,$(CPU_CORE_OBJS))
$(CC) $(CFLAGS) -o $@ $^ $(LDLIBS)

responses-replay-bench: tests/responses_replay_bench
./tests/responses_replay_bench

ds4_agent_test: ds4_agent_test.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CORE_OBJS)
ifeq ($(UNAME_S),Darwin)
$(CC) $(CFLAGS) -o $@ ds4_agent_test.o ds4_help.o ds4_web.o ds4_kvstore.o linenoise.o $(CORE_OBJS) $(METAL_LDLIBS)
Expand Down
49 changes: 38 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1058,6 +1058,23 @@ and `reasoning`. It is the preferred endpoint for Codex CLI. The server keeps
Responses continuations bound to live state when possible, and can fall back to
the same DSML rendering and KV prefix reuse used by chat completions.

Each completed Responses turn has an opaque `resp_` id. To continue that exact
turn, send it as `previous_response_id` and put only the new user or tool-output
items in `input`. DS4 resolves the id to its saved model/KV frontier, validates
prompt-affecting settings and tool call IDs, and renders/tokenizes only the new
tail. The in-memory index is FIFO-bounded (4096 ids by default; tune it with
`--response-state-max-ids`). With disk KV caching enabled, DS4 writes an exact
post-response frontier under its opaque id only when that frontier crosses the
normal continued-cache cadence. This avoids a synchronous full KV copy after
every short turn; ids between retained boundaries are live-process-only and
cannot survive a restart or session displacement. The checkpoint persists
configuration and tool-call bindings, so a retained compatible id can be
rebuilt after a server restart or session displacement. Disk budget eviction,
an oversized checkpoint,
an incompatible model/context, changed instructions/tools/reasoning/model, or
an edited branch still requires full input replay. `conversation` objects are
not implemented.

`/v1/messages` is the Anthropic-compatible endpoint used by Claude Code style
clients. It accepts `system`, `messages`, `tools`, `tool_choice`, `max_tokens`,
`temperature`, `top_p`, `top_k`, `stream`, `stop_sequences`, and thinking
Expand Down Expand Up @@ -1343,7 +1360,10 @@ Tool calls also keep a bounded exact-DSML replay map keyed by unguessable tool
IDs, so client JSON history can be rendered back to the exact sampled text. The
RAM map keeps up to 100000 IDs by default; tune it with `--tool-memory-max-ids`.
Use `--disable-exact-dsml-tool-replay` to disable this and fall back to
canonical JSON-to-DSML rendering.
canonical JSON-to-DSML rendering. Responses response-id checkpoints use an
opaque id as their disk key rather than a visible prompt copy. Their trailer
persists the bounded continuation metadata needed to validate and resume that
checkpoint after restart.

On disk, a cache file is:

Expand All @@ -1352,7 +1372,7 @@ KVC fixed header, 48 bytes
u32 rendered_text_bytes
rendered_text_bytes of UTF-8-ish token text
DS4 session payload, payload_bytes from the KVC header
optional tool-id map section
optional response-id continuation section followed by an optional tool-id map
```

The fixed header is little-endian:
Expand All @@ -1362,7 +1382,9 @@ The fixed header is little-endian:
3 u8 version = 1
4 u8 routed expert quant bits, currently 2 or 4
5 u8 save reason: 0 unknown, 1 cold, 2 continued, 3 evict, 4 shutdown
6 u8 extension flags, bit 0 = appended tool-id map
6 u8 extension flags: bit 0 = appended tool-id map; bit 1 = Responses-visible key;
bit 2 = thinking-visible key; bit 3 = session-title metadata;
bit 4 = opaque response-id key
7 u8 reserved
8 u32 cached token count
12 u32 hit count
Expand All @@ -1373,12 +1395,15 @@ The fixed header is little-endian:
40 u64 DS4 session payload byte count
```

The rendered text is the tokenizer-decoded text for the cached token prefix.
It is both the human-inspectable prefix and the lookup identity: its SHA1 is
the filename, and a file is reusable only when those bytes are a prefix of the
incoming rendered prompt. After load, the exact checkpoint tokens from the DS4
payload remain authoritative, and only the incoming text suffix after the cached
bytes is tokenized.
For ordinary entries, the rendered text is the tokenizer-decoded text for the
cached token prefix. It is both the human-inspectable prefix and the lookup
identity: its SHA1 is the filename, and a file is reusable only when those bytes
are a prefix of the incoming rendered prompt. Response-id entries instead store
their opaque server-local key as the lookup text and carry their continuation
configuration and tool-call bindings in the trailer. They are accepted only by
the response-state continuation path. After load, the exact checkpoint tokens from
the DS4 payload remain authoritative, and only the incoming text suffix after
the cached bytes is tokenized.

The optional tool-id map is present only when header extension bit 0 is set.
Appended sections use fixed bit order, so future extension bits can add fields
Expand Down Expand Up @@ -1482,15 +1507,17 @@ tokens.
- `--kv-cache-boundary-trim-tokens`
- `--kv-cache-boundary-align-tokens`
- `--tool-memory-max-ids`
- `--response-state-max-ids`
- `--disable-exact-dsml-tool-replay`

By default, checkpoints may be reused across the 2-bit and 4-bit routed-expert
variants if the rendered prefix matches. Use `--kv-cache-reject-different-quant`
when you want strict same-quant reuse only.

The cache directory is disposable. If behavior looks suspicious, stop the
server and remove it. You can investigate what is cached with hexdump as
the kv cache files include the verbatim prompt cached.
server and remove it. You can investigate what is cached with hexdump: ordinary
entries include their rendered prompt, while response-id entries use an opaque
server-local checkpoint key.

## Backends

Expand Down
Loading