Repository navigation
perf(index): store vectors in fixed buckets and rebuild BM25 from content - #1416
Merged
Merged
Conversation
Every observation, compression and memory save scheduled a full index save 5 s later, and each save re-serializes the whole BM25 and vector index and rewrites every shard. On a busy store that is a full index rewrite almost continuously. Saves now run at most once per AGENTMEMORY_INDEX_SAVE_INTERVAL_MS (default 10 minutes), measured from the start of the previous save. Explicit saves (deletes, shutdown) still run immediately. Only one save runs at a time; saves requested while one is running collapse into a single follow-up save. The BM25 and vector saves now fail independently. Before, a BM25 failure skipped the vector save in the same run and both were reported as one warning. Each leg records when it last saved, its last error, since when it has unsaved changes, and its serialized size. /agentmemory/status reports a failing leg as an error, unsaved changes older than twice the interval as a warning, and a vector index above 80% of the Node.js string limit (536,870,888 characters) as a warning, since past that limit the index can no longer be saved or loaded. The status page shows each leg's save state and the vector index size.
…tent The vector index was persisted by serializing every vector into one JSON string, cutting that string into shards and writing a new generation of all shards on every save. Node.js cannot hold a string longer than 536,870,888 characters, which a 2,048-dimension index reaches near 49,000 vectors; past it the index can no longer be saved or loaded, silently. Each save also rewrote the whole index. Vectors now live in a fixed set of state scopes (256 by default, AGENTMEMORY_VECTOR_BUCKETS), chosen by a stable hash of the id, with one key per vector holding its id, session and base64 embedding. The index tracks which ids were added or removed since the last save, and a save writes or deletes only those keys, then updates a small metadata record. Loading lists the bucket scopes one at a time. Nothing builds a string of the whole index, and scope names never change between saves. One key per vector, rather than one packed value per bucket: the iii file store rewrites a whole scope when any key in it changes, so both layouts cost the same on disk per flush (one bucket scope, 1/256 of the index). Per-key writes avoid re-encoding a whole bucket in the worker for every changed vector, a single added vector is exactly one state write, and a failed write retries only that vector. BM25 is no longer persisted. It is rebuilt at boot from stored observations and latest memories, the same documents the live paths index. A 3,000-document rebuild takes about 120 ms in the unit test; real boots add the engine round-trips for one list per session, and the boot log now reports the rebuild time. The rebuild also drops entries for content that no longer exists. Documents added after the last vector save that have no vector are embedded in the background; when no vector state was ever saved, every document without a vector is. Existing installs migrate on first boot: the old sharded or monolithic vector snapshot is read once, written into buckets, and only then are the old vector and BM25 shards and manifests removed. If the old snapshot cannot be read it is left untouched. If writing the buckets fails, the old snapshot stays and the next save finishes the job. No vector is lost and nothing is re-embedded. Changing the bucket count moves vectors to their new buckets on the next boot. The dimension-mismatch guard is unchanged; when AGENTMEMORY_DROP_STALE_INDEX discards persisted vectors, the next save now deletes them from storage too. The status page drops the string-size warning, which no longer applies, and shows the bucket count and unsaved vector changes.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (14)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
runSave() captured one dirtyEpoch before the BM25 leg's write and reused it for the vector leg's dirty check. When scheduleSave() ran while the BM25 write was in flight, the vector leg would serialize the current (already up to date) state but still compare against the stale epoch, leaving vector.dirtySince set even though the just-written snapshot already reflected the change. status() would report the vector leg as dirty until an unrelated follow-up save cleared it. saveLeg() now snapshots dirtyEpoch itself, immediately before invoking the leg's own serialize+write callback, so each leg judges its dirty state against the epoch as of its own write rather than one captured before an earlier leg ran. Added a regression test that gates the BM25 manifest write, triggers a scheduleSave() while it is held, and asserts the vector leg's dirtySince clears once its own write completes.
The stopgap now snapshots the dirty counter per save leg. Bucketed storage has a single vector leg that already snapshots before its own write, so its save path is kept as is and the BM25-leg test that came with the merge is dropped here.
Guard rebuildKeywordIndex's sessions listing with a retried helper instead of a bare kv.list call; a failure now reports an incomplete BM25 rebuild on status instead of throwing. mem::search's cold-start path now calls rebuildKeywordIndex instead of rebuildIndex, so a query against an empty index can never clear or re-embed the persisted vector index; only an explicit dimension mismatch at boot still drops vectors. Gate the vector backfill so a whole-store embed never starts silently: AGENTMEMORY_VECTOR_BACKFILL=all is required when there is no prior save to catch up from, backfills are capped per boot by AGENTMEMORY_VECTOR_BACKFILL_MAX (default 500), the pending count is reported on /agentmemory/status, and backfillVectors now saves progress every few batches so a restart does not repeat finished embedding work. On load, compare the vector count actually read against the count the last save recorded; a shortfall is reported on status and falls back to a bounded backfill for the missing ids instead of trusting the saved timestamp. Meta is still written only after every bucket write of a save succeeds. Replace hash-mod-N vector bucketing with fill-and-roll: every vector's bucket is tracked in memory, built by reading all known buckets at load; new vectors fill the current open bucket up to AGENTMEMORY_VECTOR_BUCKET_SIZE (default 500) before a new bucket opens, and updates/deletes go straight to a vector's existing bucket. A save under load now touches only the buckets it actually changed instead of rewriting the whole store, and a bucket-size change no longer requires moving existing data. Migration from the pre-PR sharded/monolithic format is unchanged; the hash-bucket layout from this PR's earlier, unreleased commit is dropped rather than migrated. Tests: sessions-listing failure keeps vectors untouched and marks the rebuild incomplete; whole-store backfill opt-in and per-boot cap; partial bucket set detected against the saved count; a save spread over several waves of adds touches only the open bucket; deletes touch only the owning bucket; migration from the old sharded format still passes.
…efined The engine SDK returns undefined for a missing key. The legacy cleanup treated undefined as a failed read and returned early, so after migration the old BM25 shards and manifest stayed in the store for good. Both legacy cleanups now read through readIndexValue, which only reports failure when the read throws.
amosodhiambo989-spec
approved these changes
Sep 26, 2026
This was referenced Oct 1, 2026
rohitg00
added a commit
that referenced
this pull request
Oct 6, 2026
The bucketed index rewrite (#1416) replaced test/index-persistence.test.ts and dropped the gating suite that #1182 added. Bring it back against the current storage: index_persist rows now come from the legacy migration and its cleanup deletes, and audit rows land in the month scope. The suite clears AGENTMEMORY_AUDIT_INDEX_PERSIST going in and restores it on the way out, then checks that auditing is off by default, that trimmed and case-insensitive "1" and "true" turn it on (with migrate and delete rows), that "0", "false", "yes" and blank values keep it off, and that the index still migrates and reloads from buckets when auditing is off. Co-authored-by: Dmitrii Zhukov <dmitry0983@gmail.com>
11 of 25 tasks
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the index save stopgap PR; merge that first, then retarget this to main.
Today the vector index is saved by serializing everything into one JSON string, sharding it and writing a new generation of every shard. That string hits V8's limit at about 48,800 vectors at 2,048 dimensions (about 65,000 at 1,536), after which saves fail silently and the index cannot be reloaded (discussion #1358). Every save also rewrites the whole index.
What changes
mem:index:bm25:vec:NNNN. New vectors go into the current open bucket until it holdsAGENTMEMORY_VECTOR_BUCKET_SIZEvectors (default 500), then the next bucket opens. Every vector's bucket is known in memory (all buckets load at boot), so updates and deletes touch only the owning bucket. The engine rewrites a whole scope when any key in it changes, so a save now rewrites only the few buckets that changed instead of the whole store. Hash-assigned buckets were tried first and dropped: under steady load every save touched nearly every bucket.AGENTMEMORY_VECTOR_BACKFILL_MAX(default 500). A whole-store backfill (embeddings newly enabled on an existing store) needsAGENTMEMORY_VECTOR_BACKFILL=all. Pending backfill is shown on/agentmemory/status, and progress is saved during the backfill.AGENTMEMORY_DROP_STALE_INDEX) is unchanged.Makes #1256 and #1282 unnecessary for new saves (no generations are written any more).
Verified on a real engine (iii 0.22.1)
Store built with current main (60 observations, 1,536-dim vectors from a local fake embedding server), then booted with this branch, then rebooted:
After migration the store holds only
vec:0000for vectors; the oldvectors:andbm25:shard scopes are gone. That run also caught a bug fixed here: the engine SDK returnsundefinedfor a missing key, which the legacy BM25 cleanup read as a failure, leaving the old BM25 shards behind.Tests
npm run build,npm run skills:check: passnpm test: 1812 passed, 1 skippedtsc: 29 errors (one fewer than baseline)Not yet measured: boot time on a large real store.
Credits
Co-authored with @MarvinFS, whose discussion #1358 traced the one-string serialization to the V8 limit and measured the rewrite volume, and @inix-x, whose #1256 mapped the generation lifecycle this replaces. Thanks also to @Chewji9875 for the orphaned-generation work in #1282.
Summary by CodeRabbit
New Features
Bug Fixes