Skip to content

perf(index): store vectors in buckets and rebuild BM25 from content - #7

Merged
codeasier merged 1 commit into
v0.9.29-patchfrom
perf/index-bucket-persistence
Sep 28, 2026
Merged

codeasier merged 1 commit into
v0.9.29-patchfrom
perf/index-bucket-persistence

Conversation

@codeasier

Copy link
Copy Markdown
Owner

Port of upstream rohitg00#1415 + rohitg00#1416 (cfc1934 + bcf4f0d, combined end state — bcf4f0d^ is cfc1934, so rohitg00#1416's tree contains rohitg00#1415 whole; ported as one PR).

What changes

  • Saves are throttled (AGENTMEMORY_INDEX_SAVE_INTERVAL_MS, default 10 min, from the start of the previous save); explicit saves (deletes, shutdown) still immediate; single-flight with one queued follow-up.
  • Vectors in fixed buckets — one key per vector ({id, s, e}), fill-and-roll (AGENTMEMORY_VECTOR_BUCKET_SIZE=500); a save writes/deletes only changed keys; load lists buckets at concurrency 8.
  • BM25 no longer persisted — rebuilt at boot from stored memories + observations (rebuildKeywordIndex, retried sessions listing, incomplete flag on status); boot log reports docs + duration.
  • Removes this line's ledger/generation machinery (fork Orphaned index generations accumulate when the previous manifest read fails — store grows until the server pegs a core while idle rohitg00/agentmemory#1115, submitted upstream as fix(state): reclaim index generations the manifest can no longer name rohitg00/agentmemory#1256, not merged — perf(index): store vectors in fixed buckets and rebuild BM25 from content rohitg00/agentmemory#1416 explicitly credits its analysis while replacing the approach): with buckets never rewriting unchanged scopes and BM25 having no shards, there is nothing to strand. The AGENTMEMORY_INDEX_RECLAIM_BOOT_MAX_SHARDS knob and its test/evict.test.ts coverage go with it.
  • Backfill is gated — whole-store embed requires AGENTMEMORY_VECTOR_BACKFILL=all, capped per boot by AGENTMEMORY_VECTOR_BACKFILL_MAX=500, progress saved every 10 batches; load compares actual vs recorded count and queues a bounded backfill for shortfalls.
  • Status reporting — /agentmemory/status gains per-leg save state, bucket count, unsaved changes, pending backfill; new problem codes bm25-rebuild-incomplete (error), index-save-failing (error), index-save-stale (warn), index-vector-backfill-pending (info), index-vector-count-shortfall (warn).
  • Migration — first boot reads the old sharded/monolithic vector snapshot once, writes buckets, then removes old vector and BM25 shards + manifests; unreadable snapshots stay untouched. Note: this line's two legacy :gc ledger keys are unknown to that cleanup and remain as harmless orphans (deletable by hand).

Line-required adaptations (engine 0.11.2)

  1. ISdk instead of IIIClient — search.ts import + registerSearchFunction signature.
  2. search-index.ts gains the upstream dedup-on-add line (if (this.entries.has(obs.id)) this.remove(obs.id)) from chore(engine): pin iii-sdk and engine at 0.19.7 rohitg00/agentmemory#1396 (d547883) as a prerequisite; upstream has no dedicated test for it (verified: test/search-index.test.ts unchanged v0.9.29→bcf4f0d).
  3. index.ts: ctor new IndexPersistence(kv, vectorIndex) — fork's reclaimOnLoad/bootReclaimMaxDeletes options removed with the knob.
  4. api.ts: keeps this line's consolidation-status import beside the expanded search.js import (3-way conflict resolved to keep both).

Verification

  • npm test: 1790 passed / 1 skipped (baseline 1804 — net −14: ledger tests removed with the machinery; upstream's rewritten persistence tests plus new index-persistence-throttle/index-keyword-rebuild suites in). Fork's own test/index-add-save-scheduling.test.ts still passes (mock hook shape survives).
  • npm run build ✓; npm run skills:gen + skills:check ✓ (17 skills; REFERENCE.md 43→47 vars).
  • npx tsc --noEmit: one error fewer than baseline (upstream's isValidShardDescriptor fixes this line's pre-existing TS18046); zero new.
  • No endpoint count change (132), no version bump (upstream didn't bump either).

Port of upstream rohitg00#1415 + rohitg00#1416 (cfc1934 + bcf4f0d, combined end state;
bcf4f0d's parent IS cfc1934) to this line, replacing the ledger/generation
persistence design (this line's rohitg00#1115, submitted upstream as rohitg00#1256, not
merged; rohitg00#1416 credits its analysis) with upstream's bucket design:

- saves run at most once per AGENTMEMORY_INDEX_SAVE_INTERVAL_MS (default
  10 min); explicit saves still immediate; single-flight with one queued
  follow-up
- vectors live in fixed bucket scopes, one key per vector; a save writes
  only the changed keys (AGENTMEMORY_VECTOR_BUCKET_SIZE=500, fill-and-roll)
- BM25 is no longer persisted: rebuilt at boot from stored content; the
  boot log reports docs + duration; the sharded snapshot and the
  ledger/generation reclaim machinery are removed with the
  AGENTMEMORY_INDEX_RECLAIM_BOOT_MAX_SHARDS knob
- whole-store vector backfill requires AGENTMEMORY_VECTOR_BACKFILL=all,
  capped per boot by AGENTMEMORY_VECTOR_BACKFILL_MAX=500, progress saved
- /agentmemory/status reports per-leg save state, buckets, pending
  backfill; new problem codes bm25-rebuild-incomplete,
  index-save-failing, index-save-stale, index-vector-backfill-pending,
  index-vector-count-shortfall
- first boot migrates existing stores: old sharded/monolithic snapshot
  read once, written into buckets, then old vector and BM25 shards and
  manifests removed; unreadable snapshots left untouched

Line-required adaptations (engine 0.11.2):
- ISdk instead of IIIClient (search.ts import + registerSearchFunction)
- search-index.ts gains the upstream dedup-on-add line from rohitg00#1396
  (d547883) as a prerequisite
- index.ts ctor new IndexPersistence(kv, vectorIndex); fork's
  reclaimOnLoad/bootReclaimMaxDeletes options removed along with the
  knob and its evict.test.ts coverage
- api.ts keeps this line's consolidation-status import beside the
  expanded search.js import

Verification: npm test 1790 passed / 1 skipped (baseline 1804; net -14 =
ledger tests removed with the machinery, upstream rewrite plus new
throttle/keyword-rebuild tests in); npm run build; npm run skills:gen +
skills:check (17 skills, REFERENCE.md 43 -> 47 vars); npx tsc --noEmit
one error FEWER than baseline (upstream fixes this line's TS18046 in
isValidShardDescriptor).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant