Repository navigation
perf(index): store vectors in buckets and rebuild BM25 from content - #7
Merged
Merged
Conversation
Port of upstream rohitg00#1415 + rohitg00#1416 (cfc1934 + bcf4f0d, combined end state; bcf4f0d's parent IS cfc1934) to this line, replacing the ledger/generation persistence design (this line's rohitg00#1115, submitted upstream as rohitg00#1256, not merged; rohitg00#1416 credits its analysis) with upstream's bucket design: - saves run at most once per AGENTMEMORY_INDEX_SAVE_INTERVAL_MS (default 10 min); explicit saves still immediate; single-flight with one queued follow-up - vectors live in fixed bucket scopes, one key per vector; a save writes only the changed keys (AGENTMEMORY_VECTOR_BUCKET_SIZE=500, fill-and-roll) - BM25 is no longer persisted: rebuilt at boot from stored content; the boot log reports docs + duration; the sharded snapshot and the ledger/generation reclaim machinery are removed with the AGENTMEMORY_INDEX_RECLAIM_BOOT_MAX_SHARDS knob - whole-store vector backfill requires AGENTMEMORY_VECTOR_BACKFILL=all, capped per boot by AGENTMEMORY_VECTOR_BACKFILL_MAX=500, progress saved - /agentmemory/status reports per-leg save state, buckets, pending backfill; new problem codes bm25-rebuild-incomplete, index-save-failing, index-save-stale, index-vector-backfill-pending, index-vector-count-shortfall - first boot migrates existing stores: old sharded/monolithic snapshot read once, written into buckets, then old vector and BM25 shards and manifests removed; unreadable snapshots left untouched Line-required adaptations (engine 0.11.2): - ISdk instead of IIIClient (search.ts import + registerSearchFunction) - search-index.ts gains the upstream dedup-on-add line from rohitg00#1396 (d547883) as a prerequisite - index.ts ctor new IndexPersistence(kv, vectorIndex); fork's reclaimOnLoad/bootReclaimMaxDeletes options removed along with the knob and its evict.test.ts coverage - api.ts keeps this line's consolidation-status import beside the expanded search.js import Verification: npm test 1790 passed / 1 skipped (baseline 1804; net -14 = ledger tests removed with the machinery, upstream rewrite plus new throttle/keyword-rebuild tests in); npm run build; npm run skills:gen + skills:check (17 skills, REFERENCE.md 43 -> 47 vars); npx tsc --noEmit one error FEWER than baseline (upstream fixes this line's TS18046 in isValidShardDescriptor).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of upstream rohitg00#1415 + rohitg00#1416 (
cfc1934+bcf4f0d, combined end state —bcf4f0d^iscfc1934, so rohitg00#1416's tree contains rohitg00#1415 whole; ported as one PR).What changes
AGENTMEMORY_INDEX_SAVE_INTERVAL_MS, default 10 min, from the start of the previous save); explicit saves (deletes, shutdown) still immediate; single-flight with one queued follow-up.{id, s, e}), fill-and-roll (AGENTMEMORY_VECTOR_BUCKET_SIZE=500); a save writes/deletes only changed keys; load lists buckets at concurrency 8.rebuildKeywordIndex, retried sessions listing, incomplete flag on status); boot log reports docs + duration.AGENTMEMORY_INDEX_RECLAIM_BOOT_MAX_SHARDSknob and itstest/evict.test.tscoverage go with it.AGENTMEMORY_VECTOR_BACKFILL=all, capped per boot byAGENTMEMORY_VECTOR_BACKFILL_MAX=500, progress saved every 10 batches; load compares actual vs recorded count and queues a bounded backfill for shortfalls./agentmemory/statusgains per-leg save state, bucket count, unsaved changes, pending backfill; new problem codesbm25-rebuild-incomplete(error),index-save-failing(error),index-save-stale(warn),index-vector-backfill-pending(info),index-vector-count-shortfall(warn).:gcledger keys are unknown to that cleanup and remain as harmless orphans (deletable by hand).Line-required adaptations (engine 0.11.2)
ISdkinstead ofIIIClient—search.tsimport +registerSearchFunctionsignature.search-index.tsgains the upstream dedup-on-add line (if (this.entries.has(obs.id)) this.remove(obs.id)) from chore(engine): pin iii-sdk and engine at 0.19.7 rohitg00/agentmemory#1396 (d547883) as a prerequisite; upstream has no dedicated test for it (verified:test/search-index.test.tsunchanged v0.9.29→bcf4f0d).index.ts: ctornew IndexPersistence(kv, vectorIndex)— fork'sreclaimOnLoad/bootReclaimMaxDeletesoptions removed with the knob.api.ts: keeps this line's consolidation-status import beside the expandedsearch.jsimport (3-way conflict resolved to keep both).Verification
npm test: 1790 passed / 1 skipped (baseline 1804 — net −14: ledger tests removed with the machinery; upstream's rewritten persistence tests plus newindex-persistence-throttle/index-keyword-rebuildsuites in). Fork's owntest/index-add-save-scheduling.test.tsstill passes (mock hook shape survives).npm run build✓;npm run skills:gen+skills:check✓ (17 skills; REFERENCE.md 43→47 vars).npx tsc --noEmit: one error fewer than baseline (upstream'sisValidShardDescriptorfixes this line's pre-existing TS18046); zero new.