Orphaned index generations accumulate when the previous manifest read fails — store grows until the server pegs a core while idle

Package: @agentmemory/agentmemory 0.9.28
iii runtime: 0.11.2
Host: macOS 26.5.2, arm64, Node v24.16.0, launchd-managed (not Docker)
Store: file-based KV, ~/data/state_store.db, ~3,000 observations

Summary

IndexPersistence.save() cleans up the previous generation's shards only when it successfully read the previous manifest. When that read fails — which happens exactly when the store is under load — the cleanup is skipped entirely and that generation's shards are orphaned forever: every later save() only ever looks at its own previous, so nothing ever revisits older orphans.

Orphans are monotonically accumulating, and they feed back into the failure that creates them.

In my install this reached 8 index generations / 140 MB of shards, of which only 2 generations (37 MB) were live. The server then sat at 110–150% CPU while completely idle with every HTTP route timing out.

Evidence

Storage breakdown before cleanup (338 MB total):

Key prefix	Size	Kind
`mem:index:*`	140 MB, 77 files, 8 generations	derived (only 2 gens live)
`mem:graph:*`	85 MB	derived
`mem:audit`	42 MB	log
`mem:obs:*` (2,888 observations)	66 MB	real data

So 79% of the store was derived data, most of it unreachable.

The manifest names the live generations explicitly:

{"data:manifest":    {"generation": "idx_mrzxkql0_d41f7623ee4f", "shards": [...]},
 "vectors:manifest": {"generation": "idx_mrzxkrwf_8fe2c54a3dc8", "shards": [...]}}

The other six idx_* generations were referenced by nothing.

Effect of deleting the orphans + graph + audit (observations untouched):

Metric	Before	After
Idle CPU (no requests in flight)	110–150%	0–3%
Wrapper RSS	1.4–2.7 GB, oscillating	93–250 MB
`GET /agentmemory/health`	8–25 s, often timeout	6 ms
`POST /agentmemory/search`	20 s timeout	14 ms
`POST /agentmemory/smart-search`	44.5 s	0.82 s
Store size	338 MB	109 MB

The server has now been up 45+ hours with no restarts and no degradation.

Root cause

dist/index.mjs, end of IndexPersistence.save():

await this.deleteKey(KV.bm25Index, legacyKey, "legacy_cleanup");
if (previous?.v === 1 && Array.isArray(previous.shards)) {
  const currentShardIds = new Set(shards.map((shard) => `${shard.scope}\0${shard.key}`));
  for (const shard of previous.shards) {
    if (currentShardIds.has(`${shard.scope}\0${shard.key}`)) continue;
    await this.deleteShards([shard], "previous_generation_cleanup");
  }
}

previous comes from reading the manifest at the start of save(). That read fails under load — from my logs:

[agentmemory] warn index persistence: BM25 manifest read failed
  {"scope":"mem:index:bm25","key":"data:manifest","message":"Invocation timeout after 180000ms: state::get"}
[agentmemory] warn index persistence: vector manifest read failed
  {"scope":"mem:index:bm25","key":"vectors:manifest","message":"Invocation timeout after 180000ms: state::get"}

When that happens previous is nullish, the if is skipped, and the shards written by the preceding generation are never deleted by anyone. There is no reconciliation pass that would find them later.

I found zero previous_generation_cleanup audit records against 3 logged manifest-read failures, which is consistent.

Why it compounds

Store grows → state::get gets slower
Manifest read hits the 180 s invocation timeout
Cleanup skipped → one more generation orphaned permanently
Store grows further → go to 1

Each failure is permanent, so the floor only ever rises. This also explains why the symptom appears suddenly after weeks of fine operation: it is not linear, it is ratcheting.

A restart makes it worse, because a fresh save() mints yet another generation — so debugging by restarting is actively counterproductive. During one incident I restarted eight times and ended up with eight generations.

Reproduction

Any deployment where a manifest read times out at least once. To force it:

Grow the store until state::get on the manifest key approaches the invocation timeout (~3,000 observations with vectors + graph did it here).
Trigger an index save.
Observe manifest read failed in the log.
ls ~/data/state_store.db | grep 'mem%3Aindex%3A' — the previous generation's shards are still present and are not referenced by mem%3Aindex%3Abm25.bin.

Suggested fixes

Reconcile instead of relying on previous. When the manifest read fails, don't silently skip. Enumerate the mem:index:bm25 scope and delete any idx_* shard whose generation is not named by the freshly published manifest. The manifest is the authority and it is written before this step, so a scan-based sweep is safe and self-correcting.
Add a startup reconciliation pass. One sweep at boot would bound the damage to a single generation regardless of how many past failures occurred.
Reconsider the 180 s invocation timeout for manifest reads. Three minutes of a blocked read on the single-threaded runtime is itself an availability problem; a short timeout plus explicit retry would fail faster and more visibly.
Surface the skip. manifest read failed is logged at warn, but the consequence (cleanup skipped, generation orphaned) isn't stated. Making it explicit would have saved a lot of diagnosis time.

Happy to open a PR for (1) + (2) if the approach sounds right.

Workaround for anyone hitting this

Stop the server, back the store up, then keep only the generations the manifest names:

# 1. read the live generations
python3 -c "
import re;print('\n'.join(sorted(set(re.findall(r'idx_[a-z0-9]+_[a-f0-9]+',
  open('~/data/state_store.db/mem%3Aindex%3Abm25.bin',encoding='utf8',errors='replace').read())))))"
# 2. delete every mem%3Aindex%3A* shard whose generation is not in that list

⚠️ Do not delete mem%3Aindex%3A* wholesale — the index is not rebuilt automatically. I tried; the server came up healthy and fast, and every query returned 0 results while all 2,888 observations sat there intact. Keeping the manifest plus the two live generations restored search completely.

mem:graph:* and mem:audit are regenerated and can be dropped safely (dropping the 85 MB graph is what took smart-search from 44.5 s to 0.82 s).

Architecture: Migrate BM25 & Graph Search from In-Memory to SQLite #309 (closed) — in-memory BM25/graph architecture; this is a consequence of the sharded-persistence design that came out of it
Stopgap for #309: shard persisted BM25/vector snapshots to avoid oversized state values #762 (closed) — introduced the sharded manifest/generation scheme this issue is about
Idle CPU leak: iii worker threads busy-wait, growing to a full core over ~40h uptime #1093 (open) — idle CPU growth over ~40 h. Looks similar from the outside but is a different layer: that one is iii worker threads busy-waiting and clears on restart, whereas this one is the Node wrapper and survives restarts because the orphans are on disk.

Hitting this on Linux at about 4x the size you describe, and I think the trigger is broader than a load timeout: a plain systemctl --user stop reproduces the divergence with nothing else running.

agentmemory 0.9.29, Node 24.19, systemd user unit, file-based KV at ~/.agentmemory/data/state_store.db. 257 sessions, 11165 observations, 4429 memories.

Storage before I touched anything (1.2 GB total):

mem:index:*   855 MB  677 files  14 generations   derived (2 live)
mem:graph:*   112 MB                              derived
mem:obs:*      79 MB   201 files                  real data
mem:memories   22.5 MB                            real data
mem:audit      18.8 MB                            log

So 79% derived here too, and the index alone is ~38x the size of the memories it indexes. Eight vector generations and six BM25 generations, oldest dated 2026-08-01.

The part I did not expect: I stopped the daemon cleanly to apply your workaround, and the shutdown itself made the store inconsistent.

23:25:53  manifest written, names idx_mt2154i9 (bm25, 46 shards) + idx_mt2155we (vectors, 125 shards)
23:58:14  shutdown flush writes idx_mt22aq3p (bm25, 22 shards) + idx_mt22ard5 (vectors, 49 shards)
          and deletes part of the previous generation
          manifest is NOT rewritten - mtime stays 23:25:53

After the stop, disk holds only 20 of the 46 shards the manifest names for idx_mt2154i9, and 69 of 125 for idx_mt2155we. The complete newer pair is named by nothing. Restarting did not fix it either: health and search come back fine (search returns correct results in ~2 s, so the in-memory index is good), but the manifest is still stamped 23:25:53 and still points at the incomplete generation. Presumably related to #1180, since nothing schedules a persist after boot.

Which means your workaround is not safe on this state, and that is worth a warning in the issue: keeping only the generations the manifest names would have deleted the one complete index and kept a half-deleted one. I stopped before running the delete.

No idle-CPU symptom yet - iii sits at 2.5% idle - but RSS was 5.0 GB before the restart and 3.1 GB after, which looks like the same ramp you described before things fell over.

Two things that would help from outside:

Is there any supported way to force a consistent manifest write, so orphans can be identified safely? rebuildIndex only runs on an empty index per No way to rebuild the search index without restarting: rebuildIndex only runs when the index is empty #1179, so I could not find one.
If the shutdown path writes shards without publishing a manifest, then your suggested fix (2), a startup reconciliation sweep, would not be enough on its own - boot would reconcile against a manifest that is already behind. A sweep after the shutdown flush, or publishing the manifest before the flush returns, looks like it needs to happen too.

Happy to run any diagnostic on this store, it is a decent worst case.

fix(state): reclaim index generations the manifest can no longer name

docs(state): drop the review marker from the gc ledger append

Three things in this comment, from a third deployment on 0.9.29:

Confirmation. This store leaked four half-written copies of its search index, 371 MB beside a live index of 182 MB. All four were left by a shutdown that interrupted a save, which is the route this issue already describes. No log line, audit row or health field reported them.
Two routes the issue and the open PRs (fix(state): reclaim index generations the manifest can no longer name #1256, perf(state): write only the vector buckets that changed #1258, fix(state): reclaim orphaned index shards and add index diagnostics/healing #1282) do not cover. A publish that times out while its rollback deletes also time out leaks a generation with one generic "failed to save" line and nothing else; a fault-injection harness below reproduces it. And the three separate v === 1 version checks are not consistent with each other, so a version skew or a failed manifest read can run the rollback against the shards the live manifest points at and delete the working index.
A defect in one of the pending fixes. fix(state): reclaim orphaned index shards and add index diagnostics/healing #1282's orphan sweep, when the manifest is behind, deletes the complete unpublished generation and keeps the half-deleted one. It needs a guard.

The rest of this comment is the evidence for each: how the four leaked generations date themselves, the harness and its results, and three review points for whichever PR lands.

Third deployment. @agentmemory/agentmemory 0.9.29, Node v22.23.2, Linux x64, systemd user unit, file-backed state_store.db. Four orphaned generations, 194 shard files, 371 MiB, beside a live index of 182 MiB, in a unit that was already at its 4 GiB MemoryHigh. vectors:manifest was not rewritten once between 2026-08-27 and 2026-09-14, and every boot in between logged Loaded persisted vector index (7234 vectors), which is #1180 and the shutdown observation in the comment above seen from a third store. All line numbers are dist/index.mjs of the published 0.9.29 tarball.

The four generations date themselves: createIndexGeneration (2901) is generateId("idx") (1535), whose middle field is Date.now() in base36:

generation	shards	written	matching journal line
`idx_mtuv7hh4`	61 vector	2026-09-09 21:45:00	`Shutting down...`, clean exit 3.4 s later
`idx_mtuw3ilw`	7 BM25	2026-09-09 22:09:54	`Shutting down...`, clean exit 3.5 s later
`idx_mtuxhn5z`	39 vector	2026-09-09 22:48:53	`Shutting down...`, clean exit 3.4 s later
`idx_mtyy8b9j`	87 vector	2026-09-12 18:20:42	`Shutting down...`, SIGKILL at 18:21:44, `Failed with result 'timeout'`

All four are the shutdown save (22963-22975). The partial counts (61, 39, 87 of about 80 and 135) show the process was gone during the shard-write phase, before the publish, so the previous manifest stayed live and correct and nothing ever looked beside it. Three of the four stops exited 0 a few seconds into a save that had not finished. Why is not determined: the bundle has two process.exit sites, 22974 and 22981, the SDK installs no signal handler, and nothing was logged. The audit scope has no index_persist row for any of the four, and there is no Failed to save index on shutdown line, which #1335 notes can never fire.

Two routes are named neither in the issue nor in the PR descriptions. Both live in saveShardedIndex (2978-3044), where the manifest kv.set at 3017 is the commit point and the rollback on a failed publish is deleteShards(shards, "manifest_publish_rollback") at 3034.

The publish fails and the rollback fails too, with the process alive. deleteShards (3069) is a loop over deleteKey (3052-3068), and deleteKey catches every delete error, writes result: "failed" to an audit row, and returns void. deleteShards checks nothing, so a rollback in which every delete failed is indistinguishable from one that succeeded. The realistic case is a KV that is timing out: the same fault that failed the set fails the deletes, and the audit row recording that goes through the same KV. save() rethrows, logFailure (2960) prints one failed to save BM25/vector index line, the in-memory index is intact, the next debounce retries with a new generation, and the abandoned one is never touched again. fix(state): reclaim index generations the manifest can no longer name #1256's ledger covers the reclaim side of this (its "ledger entry survives a failed rollback delete" test), as long as the ledger write itself lands. Making the failure loud is still needed: today the only record is an audit row written through the KV that was failing.
A version skew deletes live shards or skips a whole set. Three hard-coded v === 1 checks are not symmetric. isManifestPublished (3072-3080) returns false for published?.v !== 1, and also for a kv.get that throws, since the read is .catch(() => null). A false answer sends the catch down the rollback, and shards is the generation the manifest that just landed points at. The result is shard missing (3128) and a full rebuild on the next boot. The read-failure half of that can fire today: a publish whose set timed out after landing, followed by a verification read that also times out. The previous-generation GC at 3038 is gated on previous?.v === 1, so a manifest that a newer writer left at another v is skipped and the entire set it names is stranded. perf(state): write only the vector buckets that changed #1258 is an instance: it moves vectors:manifest to v: 2, so a 0.9.29 process started against a store perf(state): write only the vector buckets that changed #1258 has written strands every v2 bucket on its first save (the BM25 stream in perf(state): write only the vector buckets that changed #1258 still writes v: 1). Neither fix(state): reclaim index generations the manifest can no longer name #1256's nor fix(state): reclaim orphaned index shards and add index diagnostics/healing #1282's description mentions 3072 or 3038.

The harness is src/state/index-persistence.ts from the 0.9.29 tarball unmodified (md5 d426526fe4f5ae9c767fab9e78f17b60), driven by a file-backed StateKV with fault injection on set and delete, shardChars: 10, 40-character payloads so each stream is 4 shards. Each mode: save once clean, inject, save again, clear the injection, save a third time, count generations on disk that no manifest names.

mode	injection	leaked generations
shard write rejects	`set` throws on one shard key	0
publish rejects, KV otherwise healthy	`set` throws on `vectors:manifest`	0
process dies at the publish	`set` on `vectors:manifest` calls `process.exit`; a fresh process then loads and saves	1, survives the clean load + save
publish times out and every delete times out, process alive	`set` on `vectors:manifest` and all `delete`s throw `code: "TIMEOUT"`	1, with one `failed to save` log line and nothing else

The first two rows are the rollback working as designed. The route in the issue itself, the previous read at 2979 returning null, is not a row here. It reaches the same state through a skipped GC, with the rollback never involved.

Three review points for whichever PR lands:

Make deleteKey's failure observable. perf(state): write only the vector buckets that changed #1258 already changes it to return result === "deleted". deleteShards should aggregate that, and both rollback sites (3007, 3034) should act on it, logging at error with the generation id and the count of shards left behind. A stranded generation should produce a log line as well as the audit row.
The load-time sweep needs a guard for a manifest that is behind. The comment above shows a manifest 30 minutes stale, naming a half-deleted generation beside a complete unpublished pair. A sweep that deletes whatever the manifest does not name (fix(state): reclaim orphaned index shards and add index diagnostics/healing #1282's sweepOrphanShards, with a 60 s grace period) would delete the complete pair in that state and keep the half-deleted one. Treat a manifest that names a generation with shards missing on disk as a bad read and skip the sweep for that stream. Over-keeping is harmless and under-keeping costs a full re-embed. fix(state): reclaim index generations the manifest can no longer name #1256's ledger avoids this because it reclaims from a record it wrote, but it cannot see generations from before the ledger, as it says, and StateKV.list(scope) (1366) returns values within one exact scope, so neither PR can enumerate what it has no record of. The 371 MiB here and the 855 MB above need a one-time offline reclaim either way, and the warning in the comment above applies to it.
Handle the version skew. isManifestPublished should compare on generation, the field that is unique per save, and treat a read failure during the check as unknown. On unknown, leave the shards for the reclaim pass, because deleting is the irreversible branch. At 3038, a previous manifest of any version that names shards should have those shards reclaimed. loadManifestData (3111) returns null on an unknown v as well. Any format change needs all three together.

#1372 (the MAX_STRING_LENGTH ceiling in VectorIndex.serialize()) first noted the three asymmetric checks. The second route above follows those checks through to what they do to shards on disk.

Patch against v0.9.29 for the points above, with tests:

0002-index-persistence-pending-generation-ledger-observable-cleanup-failures-unknown-publish-verdict.patch

fix(state): reclaim orphaned index shards and add index diagnostics/h…

Navigation Menu

Repository navigation

Orphaned index generations accumulate when the previous manifest read fails — store grows until the server pegs a core while idle #1115

Summary

Evidence

Root cause

Why it compounds

Reproduction

Suggested fixes

Workaround for anyone hitting this

Related

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Orphaned index generations accumulate when the previous manifest read fails — store grows until the server pegs a core while idle #1115

Description

Summary

Evidence

Root cause

Why it compounds

Reproduction

Suggested fixes

Workaround for anyone hitting this

Related

Activity

sla-te commented on Aug 20, 2026

possiblyneal commented on Sep 14, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions