Stopgap for #309: shard persisted BM25/vector snapshots to avoid oversized state values #762

Note: This issue description was drafted with AI assistance. I personally experienced the underlying issue. If any AI-slop-like shortcomings remain, they are due to my prompting, and I welcome suggestions on how to communicate my requirements more clearly to Codex GPT-5.5 xhigh.

Existing related discussion

This is a focused follow-up to #309, not a standalone rediscovery. #309 is a closed architecture issue about moving BM25 and graph search away from in-memory indexing. Its comments also report the same current IndexPersistence.save() bottleneck: large search snapshots are persisted through oversized state::set values, #204 made those timeouts nonfatal, and the data-loss / stale-snapshot problem remains. One production report in #309 specifically identifies the vector snapshot being written as one kv.set(KV.bm25Index, "vectors", vectorIndex.serialize()) value and proposes chunking as an interim workaround.

This issue narrows that broader closed architecture discussion to a current-repo stopgap: keep the existing BM25/vector index implementations, but persist their serialized snapshots as bounded shards so large corpora do not depend on one very large state value.

This is related to, but not the same as, #204 / #209. Those fixed unhandled state::set rejection crashes. They did not change the size of the persisted values or the publication protocol for rebuilt search snapshots.

This also should not be read as superseding #300. That PR is an open sqlite-vec vector-backend direction, opt-in through VECTOR_BACKEND=sqlite-vec; as checked immediately before filing on 2026-06-01 at 22:56 UTC, it is still unmerged and reports mergeable_state=dirty. If #300 lands, it may reduce vector pressure for that opt-in backend, but it still does not make the current BM25 snapshot persistence bounded, and it does not by itself remove the need for safe legacy/default snapshot publication semantics in IndexPersistence.

Current behavior

IndexPersistence.save() persists the complete BM25 snapshot and the complete vector snapshot as two monolithic strings in the mem:index:bm25 state scope:

• SearchIndex.serialize() returns one JSON string for all BM25 entries, inverted terms, document term counts, and total document length.
• VectorIndex.serialize() returns one JSON string for all vector entries, including base64-encoded embedding data.
• IndexPersistence.save() writes those strings with kv.set(KV.bm25Index, "data", ...) and kv.set(KV.bm25Index, "vectors", ...).

Current upstream source at fd9e3bd: src/state/index-persistence.ts lines 31-40, src/state/search-index.ts lines 194-210, src/state/vector-index.ts lines 124-136, and src/state/schema.ts lines 3-13.

For a large corpus, a rebuild can finish in memory but fail while saving either serialized snapshot through state persistence. The next process start then has no durable rebuilt snapshot, or it keeps using the last older snapshot that was saved successfully. The evidence is split deliberately: upstream source shows that both BM25 and vector snapshots are saved as single state values; #309 includes a broad large-corpus timeout report and a concrete vector snapshot report where the serialized vectors value reached about 27 MB and timed out.

Expected behavior

Large rebuilt BM25 and vector snapshots should persist without requiring one very large state value per index. A failed save attempt should not corrupt or delete the last successfully published snapshot for that index.

Steps to reproduce

The proposed regression test gives a deterministic reproduction without any private data or external service:

1. Wrap the StateKV used by IndexPersistence so set() rejects any string value over a small limit.
2. Build a BM25 index and vector index whose serialized forms are larger than that limit.
3. Call IndexPersistence.save(), then IndexPersistence.load().

On current upstream behavior, the save path sends the whole BM25 snapshot as one data string and the whole vector snapshot as one vectors string, so the guarded KV rejects the oversized write. With the proposed fix, each string is split into bounded shards, the save succeeds, and the loaded BM25/vector indexes round-trip.

The same size pattern appears from the current source shape: BM25 JSON grows with entries, terms, and postings, while vector JSON grows with vector count and embedding dimensions. The strongest detailed public size report is vector-specific; BM25 is included here because it has the same monolithic persistence boundary and the same failure mode once one data value exceeds what the state layer can reliably commit. The reproduction only needs to prove the storage-granularity bug, so it uses generated test observations and generated vectors.

Root cause

The current persistence boundary is the whole serialized index string. KV.bm25Index is the shared state scope for search snapshots, and IndexPersistence.save() writes the complete serialized BM25 and vector indexes as single values under that scope. That makes save reliability depend on the maximum practical size of one state value instead of on the logical index contents.

Suggested fix

Persist BM25 and vector snapshots as bounded string shards:

• write all shards for a new generation before publishing it;
• publish a compact manifest in mem:index:bm25 after shard writes succeed;
• keep loading legacy monolithic data and vectors values when no manifest exists;
• delete partial new shards if a save fails before manifest publication;
• leave new shards in place if manifest publication is attempted but reports failure, because the manifest may have committed;
• delete old shards and legacy keys only after the new manifest is published;
• fail closed for that index if a published manifest points to a missing shard or a shard with the wrong length.