kvdoc

Find shared token prefixes, repeated prompt content, and the points where your LLM requests start to differ.

kvdoc reads a request trace and gives you ranked findings with matching token intervals and source lines. Use it to inspect prompt structure across an LLM or agent workload. Exact token matches do not establish cache hits or performance savings; those require evidence from your serving system.

1. Install kvdoc

You need Python 3.11+, Git, and Python virtual-environment support. Run these commands in a Linux, macOS, or WSL terminal:

git clone https://github.com/AdudodlaVarish/kvdoc.git kvdoc
cd kvdoc
python3 -m venv .venv
source .venv/bin/activate
python -m pip install .

The base installation needs no tokenizer libraries or GPU. Keep this terminal open for the next steps. When you return later, enter the kvdoc directory and run source .venv/bin/activate again.

2. Try the included example

Create your first report:

kvdoc diagnose examples/repetition.jsonl --source captured --repeat-window 3 \
  --format html --output reports/findings.html

Open reports/findings.html in your browser. You should see:

Measure	Example result
Requests	3
Input tokens	24
Historical prefix overlap	7 tokens
Repeated-content coverage	16 tokens
Repeated tokens outside prefixes	9 tokens
Selected findings	5

These counts overlap, so don't add them together. The example uses a three-token window because its inputs are small; the default is 32 tokens.

The report works offline with no server, scripts, or external assets. Output directories are created for you. To rerun it, choose a new output filename: kvdoc never overwrites an existing report.

3. Prepare your own trace

Save one request per line in a UTF-8 .jsonl file. Keep requests in the order you want them compared; kvdoc uses file order, not timestamps.

Choose the input you have:

• Recorded token IDs: use the exact input_ids captured by your serving system.
• Messages: reconstruct IDs with the tokenizer and chat template used to serve your requests.

Every request in a run must contain the selected input. kvdoc does not switch between sources automatically.

If you have recorded token IDs

Save records like these in trace.jsonl:

{"request_id":"r1","model":"example-model","input_ids":[1,2,3,4],"tokenization_metadata":{"tokenizer":"example-vocabulary-v1"}}
{"request_id":"r2","model":"example-model","input_ids":[1,2,9,4],"tokenization_metadata":{"tokenizer":"example-vocabulary-v1"}}

Use actual recorded IDs for your workload. These numbers are a synthetic example. Include accurate model and tokenizer declarations: the same number can mean different tokens in different vocabularies.

Captured mode copies IDs unchanged and needs no tokenizer download. It treats your capture as supplied evidence, without independently verifying how it was recorded. Rendered text and character offsets remain unavailable.

If you have messages

From the repository directory, with the environment active, install the extra:

python -m pip install '.[tokenization]'

Save records like these in messages.jsonl:

{"request_id":"r1","model":"Qwen/Qwen3-0.6B","messages":[{"role":"system","content":"Answer briefly."},{"role":"user","content":"What is two plus two?"}]}
{"request_id":"r2","model":"Qwen/Qwen3-0.6B","messages":[{"role":"system","content":"Answer briefly."},{"role":"user","content":"What is three plus three?"}]}

Use your workload's tokenizer for your own requests. The Qwen commands below demonstrate a supported tokenizer and pinned revision. --tokenizer accepts a Hugging Face repository ID or a local tokenizer directory.

Only tokenizer/configuration assets are loaded. You need no model weights, PyTorch, or GPU inference. Remote Python code is disabled.

Reconstructed messages may differ from serving input. Match its template, tool serialization, and generation settings under Match your serving settings.

Check the file before diagnosing it

For recorded IDs:

kvdoc validate trace.jsonl

For messages:

kvdoc validate messages.jsonl

Validation checks JSON syntax, messages, IDs, and optional identifiers. It stops at the first error and reports the file and physical line without printing prompt contents. Correct that line and run validation again.

The success summary counts requests, distinct supplied session IDs, and requests with nonempty tool definitions. It appears after the whole file validates. Rendering and comparison settings are checked during diagnosis; validation does not check every provider API field or tool parameter schema.

4. Diagnose your trace

For captured IDs, get findings in the terminal:

kvdoc diagnose trace.jsonl --source captured

For messages, select the tokenizer:

kvdoc diagnose messages.jsonl --tokenizer Qwen/Qwen3-0.6B \
  --revision c1899de289a04d12100db370d81485cdf75e47ca

Message mode is the default. A diagnosis shows up to 10 findings with a 32-token minimum repeated window. It examines the whole trace before printing success.

Save an HTML report

For captured IDs:

kvdoc diagnose trace.jsonl --source captured \
  --format html --output reports/trace-findings.html

For messages:

kvdoc diagnose messages.jsonl --tokenizer Qwen/Qwen3-0.6B \
  --revision c1899de289a04d12100db370d81485cdf75e47ca \
  --format html --output reports/message-findings.html

Open the resulting file in a browser. Start with the totals, then inspect the ranked findings. Each names source lines and token intervals; expand its provenance section to check assumptions and rendering settings.

Save a JSON report instead

kvdoc diagnose trace.jsonl --source captured \
  --output reports/trace-findings.json

JSON is the default. HTML requires --format html and --output; the filename extension alone does not select the format.

5. Read the findings

Shared prefixes: how much of the beginning matches?

A historical prefix is the longest initial sequence shared with any eligible earlier request. For example:

Earlier: [1, 2, 3, 4]
Current: [1, 2, 9, 4]
Shared prefix: 2 tokens

The current request shares two initial tokens, then diverges at token index 2: current ID 9, earlier ID 3. A divergence finding shows both boundary tokens and the shared prefix. Its ranking count is the current suffix length, including the boundary token; here, two tokens. It does not count how many tokens differ or how many were recomputed.

Repeated content outside prefixes: what matches later?

An early change can break a prefix while later content remains identical:

Earlier: [1, 2, 3, 4, 5, 6]
Current: [9, 1, 2, 3, 4, 5, 6]

With a three-token window, the current request has zero prefix overlap and six repeated tokens outside its prefix. The finding names the current interval and an earlier matching interval.

Overlapping matches count each current token once. Different positions and preceding context mean identical content does not automatically have reusable KV states.

Frequent exact windows: what appears across many requests?

A frequent window is an exact K-token sequence present in at least three distinct requests. Repetition within one request counts once toward its frequency. Duplicate request IDs still represent separate physical requests.

For a three-token window in four requests, the ranking count is 3 × (4 − 1) = 9. This is a structural frequency score, not nine saved tokens. The report selects representative windows rather than maximal repeated spans.

Follow the evidence

Token positions start at zero. [3, 6) means positions 3, 4, and 5; the end is excluded. Source lines are physical file lines, including blank lines. Use the line and interval to locate a finding in your trace.

A finding shows at most three examples and discloses the full evidence count: supporting spans for outside-prefix findings, distinct requests for frequent windows, or one pair for divergence. Sampling does not reduce measured coverage or frequency.

Prefix and outside-prefix evidence refers to earlier requests. Frequency looks across the complete trace. The first request has no historical overlap or repetition evidence. Identical inputs and ordinary input-ending boundaries do not create divergence findings.

Keep the interpretation precise

The report establishes	It does not establish
Exact prefix overlap with eligible earlier inputs	Actual cache hits or whether a KV state was resident
Exact sequences repeated outside a prefix	Safe reuse of shifted KV states
Where a historical prefix first differs	Eviction, expiry, or routing loss
Exact-window frequency	Recomputed GPU work, latency, dollar savings, or KV bytes saved
Token positions and source requests	Semantic labels such as system instructions or tool definitions

Ranking counts have different meanings and can overlap. Do not sum findings or coverage bars into savings. Each bar uses all input tokens as its denominator. observed_cached_tokens is unavailable (null), not zero cache hits. kvdoc does not ingest cache telemetry or integrate with a serving cache.

Adjust your analysis

Compare requests within the same session

By default, eligible requests can match across sessions. Restrict history to requests with the same supplied session_id:

kvdoc diagnose trace.jsonl --source captured --scope session

Requests without a session ID are each isolated in session scope.

In either scope, comparisons require equal supplied model, cache_namespace, and tokenization identity. Supplied and missing values form separate groups. Use cache_namespace for tenant, adapter, model-revision, or other serving boundaries that must not share history. A missing namespace assumes a shared one.

For captured IDs, the entire declared tokenization_metadata must match; object key order does not matter. Missing declarations compare with equally unspecified declarations, assuming a shared vocabulary. For messages, identity uses the tokenizer backend/special-token fingerprint. Template/generation changes affect IDs without hiding an otherwise matching prefix. These labels and fingerprints do not verify cache compatibility or model weights.

Change window size or the number of findings

kvdoc diagnose trace.jsonl --source captured --repeat-window 16 --top 5

Both values must be positive integers. --repeat-window sets the minimum exact match length; windows start at every token position. A shorter prefix can count toward prefix overlap without qualifying as repeated coverage.

Findings sort by descending ranking count. Ties prefer outside-prefix repetition, frequent windows, then divergence, followed by earliest evidence line and token position. A frequent candidate is skipped if its first interval overlaps an already selected frequent interval on that source line. Touching intervals can both appear. Coverage across other findings can still overlap.

Match your serving settings

Use these controls when reconstructing messages. Captured mode uses supplied IDs and rejects tokenizer/rendering options.

You need to…	Use
Select a tokenizer version	--revision REF; pin a commit for repeatable assets.
Avoid downloading missing assets	--local-files-only; local/cached assets are required.
Supply a serving chat template	--chat-template FILE, a UTF-8 Jinja template.
Parse string function arguments as JSON objects	--tool-arguments json (default); invalid/non-object arguments fail.
Keep function argument strings unchanged	--tool-arguments preserve; objects are accepted too.
Append an assistant generation header	--generation-mode append (default), according to the template.
Continue a final assistant message	--generation-mode continue; final role must be assistant.
Use neither header nor continuation	--generation-mode none.
Supply shared template variables	--template-kwargs FILE, containing a JSON object.

For example, reconstruct bundled messages with thinking disabled:

kvdoc tokenize examples/demo.jsonl --tokenizer Qwen/Qwen3-0.6B \
  --revision c1899de289a04d12100db370d81485cdf75e47ca \
  --template-kwargs examples/template_kwargs.json \
  --output reports/tokenized.jsonl

The kwargs file sets enable_thinking to false. The same controls work with analyze and diagnose. Per-request chat_template_kwargs override shared variables, but cannot override controlled options such as messages, tools, padding, truncation, generation mode, or special-token values.

You need a fast tokenizer with character offsets and an available template. Text parts concatenate without separators; absent/null assistant content becomes an empty string. Message/tool order and other fields remain in a copy. Only function tool calls with nonblank names are supported. The full prompt is rendered once and tokenized without adding special tokens again, padding, or truncation.

Check the verification label in provenance:

Label	Meaning
unverified_reconstruction	Reconstructed IDs, not verified against serving input.
matches_supplied_input_ids	Complete match against captured IDs supplied on the same record. A mismatch fails.
supplied_not_independently_verified	Captured IDs and identity metadata accepted as declarations.

Provenance records the resolved remote revision or recognized local asset hashes, tokenizer/backend fingerprint, template hash, settings, normalization, and library versions. The extra specifies Transformers 5.17.0 and Jinja2 3.1–3.x. Custom serving preprocessing or time-dependent templates can change input even with pinned assets.

Export requests for closer inspection

Use analyze for a JSONL record per request with prefix and repetition evidence:

kvdoc analyze examples/repetition.jsonl --source captured --repeat-window 3 \
  --output reports/repetition.jsonl

Use tokenize for IDs, rendered text, offsets, and provenance:

kvdoc tokenize examples/captured.jsonl --source captured \
  --output reports/captured.jsonl

For messages, replace --source captured with your tokenizer and serving settings. Each export includes the absolute source path, physical line, supplied identifiers, IDs/count, text, offsets, and provenance. analyze adds prefix_analysis and content_repetition.

Message exports include full rendered prompts and token-ID sequences. Offsets refer to each request's own rendered string; unavailable or zero-width intervals are null, and Unicode offsets can overlap. Captured exports have null text and offsets. Use these files to inspect the earlier input named in a finding; paired finding provenance describes the current request.

Prefix evidence records the best earlier request and whether the boundary is a token difference, identical input, either input ending, or no prior request. Repetition evidence contains merged coverage and current/prior matching spans. Each window chooses the latest matching earlier request and its latest matching position. Adjacent windows merge only when they follow the same earlier occurrence; not every occurrence is enumerated. Outside-prefix evidence clips at the prefix boundary and shifts the prior interval equally; fragments may be shorter than the detection window.

Share reports carefully

HTML and diagnosis JSON omit full prompts and full ID sequences, but retain paths, supplied identifiers, assumptions, and configuration metadata. Review these before sharing. HTML values are escaped as text; the page has no scripts or external assets.

Tokenization and analysis exports may include full prompts. Keep private traces and reports in traces/ and reports/; both are ignored by Git. Being ignored does not encrypt files or remove sensitive contents.

Files publish only after complete success. Existing files, symlinks, and competing writers' outputs are protected. Failure leaves no partial export or success summary. Success returns 0; input, configuration, dependency, tokenizer, template, mismatch, and output errors return 2.

Troubleshooting

What you see	What to check
kvdoc is not found	Enter the checkout and activate .venv. You can also use python -m kvdoc ....
Virtual-environment creation fails on Ubuntu	Install python3-venv, then retry installation.
A file/line error	Correct the first reported line and rerun. Check the checklist below.
Captured mode requires IDs	Every request needs input_ids; use message mode for a messages-only trace.
Message mode requires a tokenizer or dependencies	Install .[tokenization] from the checkout and supply --tokenizer.
Tokenizer assets cannot load	Check tokenizer ID/directory, revision, network, and cache. Local-only mode cannot fetch missing assets.
Missing template or offset support	Use a fast tokenizer with a usable template, or supply --chat-template.
Reconstructed IDs do not match	Check capture, tokenizer revision, template, tool serialization, generation mode, and template variables.
Output already exists	Choose a new filename.
No findings qualify.	Check input lengths, window size, groups, and scope. Short inputs, isolated requests, or input-ending boundaries may yield no findings.
No character offsets	Captured IDs cannot provide rendered character offsets. Message reconstruction provides available mappings.
A large trace uses substantial memory	History grows with distinct prefixes, windows, and findings. Streaming input does not bound memory.

Analysis is a historical baseline over recorded inputs. It does not simulate cache capacity, routing, expiry, eviction, block alignment, or generated output tokens. It does not assign semantic components or suggest prompt rewrites.

Trace checklist

• Use one JSON object per nonblank UTF-8 line, containing messages, input_ids, or both. Both are validated when supplied. A leading BOM is accepted; blank lines are ignored but still count toward physical line numbers.
• input_ids must be a nonempty array of nonnegative integers. Booleans, fractional numbers, strings, and negative IDs are invalid.
• messages must be nonempty. Each message needs a nonblank string role and string content or text parts such as {"type":"text","text":"Hello"}. Roles are not restricted to a fixed list. Empty strings and content arrays are accepted. Image, audio, and other content parts are unsupported.
• An assistant with nonempty tool_calls may omit content or use null. Supplied tool calls and tools must be arrays of objects. Tool results use text content and can retain tool_call_id. See examples/demo.jsonl.
• Optional request_id, session_id, and model must be nonblank strings. Missing identifiers remain absent; duplicates are preserved. Source lines distinguish requests without invented identifiers.
• Optional tokenization_metadata must be a JSON object in captured mode; chat_template_kwargs must be a JSON object in message mode. Analysis requires a nonblank string when cache_namespace is supplied.
• Nonstandard constants such as NaN and Infinity are rejected. Tool definitions, extra fields, and metadata remain in decoded source records; exports do not reproduce every original field.

For commands, run kvdoc --help. For options, use kvdoc diagnose --help, kvdoc analyze --help, kvdoc tokenize --help, or kvdoc validate --help.