Skip to content

Bug: no path to rebuild or reset a >25K legacy graph — snapshot-rebuild + graph-reset both heartbeat-crash on state::list (follow-up to #816) #825

Open
Open
Bug: no path to rebuild or reset a >25K legacy graph — snapshot-rebuild + graph-reset both heartbeat-crash on state::list (follow-up to #816)#825

Description

@allandelmare
allandelmare
opened on Jun 4, 2026

The following was generated with the help of Claude Code, based on hands-on testing against a large corpus. Apologies for any AI-slop — please push back on anything that reads wrong.

Follow-up to #814 / #816. The merged #816 fix resolves the original #814 scope completely — /graph/query and /graph/stats now degrade gracefully (snapshot-only hot path, 200 + warning envelope, ~0.002s, zero worker reconnects across 10 calls verified on a ~75K-node corpus). Thank you — that was the crash that made the daemon unusable at scale, and it's gone.

This issue tracks the one piece that #816 surfaced but doesn't resolve: there is currently no working path to rebuild OR reset a graph on a >25K legacy (pre-#814) corpus — both write-side operations crash the worker.

Repro (against 680 sessions / ~75K graph nodes / 344 MB)

Build at b494f23 (the merged head), fresh boot, no viewer, worker confirmed healthy first:

POST /agentmemory/graph/snapshot-rebuild -> 500 {"error":"Invocation stopped"} in 0.357s ; worker reconnect +1
POST /agentmemory/graph/reset            -> 500 {"error":"Invocation stopped"} in 0.337s ; worker reconnect +1

Both die at ~0.35s (far under the 6s budget). The budget timer does eventually log enumeration: exceeded 6000ms budget, but only after the worker already reconnected — iii's heartbeat deadline (~0.35s observed) is much shorter than the budget, so a synchronous enumeration starves the heartbeat before Promise.race can resolve.

Root cause: one shared primitive

StateKV exposes only get / set / update / delete / list. There's no clear-scope, no cursor/pagination on state::list, and no key-only keys variant. So every operation that must touch all existing keys is forced through state::list, which returns the whole scope as one array and lands a multi-MB payload that JSON.parse chews synchronously, starving the worker heartbeat.

That makes these the same bug:

  • rebuild — needs every node to compute top-degree → state::list
  • reset — needs every key to delete it (no clear-scope) → state::list
  • seeding an incremental key-index for a legacy corpus — can't be done without one state::list to bootstrap it

So a pre-#814 corpus hits a chicken-and-egg: the incremental snapshot/index architecture (which is correct) can't be seeded without the one call that crashes. The hot path is safe precisely because it never seeds — it serves whatever snapshot exists. The write side can't avoid the seed. Net effect: a large legacy graph is stuck at empty-snapshot-forever — stable, but unrecoverable.

The REBUILD_SAFE_NODE_CEILING = 25000 guard doesn't catch this because it reads snapshot.totalNodes, and a legacy corpus has no snapshot yet — so totalNodes is 0/unknown, the guard passes, and the full kv.list runs anyway.

Same latent bug elsewhere

src/functions/migrate.ts does kv.list<Memory>(KV.memories). On a corpus with tens of thousands of memories, a migration would heartbeat-crash the same way. (We only have 131 memories, so it's fine for us — flagging it because it's the same primitive.)

Possible directions

  • Engine-level (clean fix): a paginated/streaming state::list, a key-only keys, or a clear-scope primitive — any of which lets rebuild/reset walk in yielding batches so the heartbeat survives. I realize this is upstream of this repo and tangled with the v0.11.2 iii pin, so flagging as context.
  • Graceful-refuse (band-aid): when size can't be determined cheaply (no snapshot, no counter), refuse rebuild/reset with a clean 400 instead of attempting the crash-prone enumeration — converts a daemon-killing footgun into an actionable error. Doesn't restore the graph, but stops the crash.
  • Off-worker migration: a standalone client-mode script that does the one expensive enumeration outside the worker-heartbeat path (unproven — no client-mode pattern exists in the repo today).

Standing offer

We have the >50K repro that's hard to synthesize (680 sessions / ~75K nodes / 344 MB), a backup, and a one-command rebuild-and-test loop pointed at it. Push any branch and we'll report rebuild/reset/hot-path behavior + worker-reconnect deltas against the real corpus, usually within the hour.

Environment

Activity

  1. rohitg00
    added 2 commits that reference this issue on Jun 7, 2026

    fix(graph,search): cr followups for #843/#825/#817 wave

    f1e7e96

    fix(release): v0.9.28 hotfix wave (#843, #825, #817, iii pin) (#849)

    a6e22fa

  2. allandelmare commented on Jun 9, 2026

    @allandelmare
    allandelmare
    on Jun 9, 2026
    Author

    Re-test on the released v0.9.27 against the same ~75K-node corpus, via Claude Code. One half of this fixed cleanly; the other half has a guard hole. Push back on anything that reads wrong.

    Upgraded to v0.9.27 (the released build, not a branch) and re-ran against the 680-session / ~75K-node / 344 MB corpus. Fresh backup first; data intact throughout (131/133 memories, 686 sessions).

    ✅ graph/reset — fixed, confirmed enumeration-free

    POST /agentmemory/graph/reset
-> {"cleared":{"mem:graph:snapshot":1},"success":true,"tookMs":0}  [0.017s, http 200]
-> worker reconnect delta: 0

    Fast, no kv.list, no worker death. Exactly as intended.

    Also confirmed while I was here: the #816 hot path holds under a real browser — left the viewer dashboard open and idle for 12s, 0 worker reconnects. The continuous churn that originally made the daemon unusable is gone. And #843 (data loss on stop/restart) verified fixed — clean stop showed [flushing state], restart came back with identical counts. Nice.

    ❌ snapshot-rebuild — graceful-refusal guard doesn't fire on a legacy corpus

    The release notes say rebuild "refuses pre-flight when no prior snapshot exists." On a real >25K legacy corpus it still hard-crashes the worker:

    POST /agentmemory/graph/snapshot-rebuild            -> {"error":"Invocation stopped"} [0.42s, http 500] ; reconnect +1
POST /agentmemory/graph/snapshot-rebuild {force:true}-> {"error":"Invocation stopped"} [0.36s, http 500] ; reconnect +1

    Root cause, from the installed dist/src-MOg2zKJs.mjs: the pre-flight refusal keys off the existing snapshot's recorded totalNodes — but a legacy corpus has an empty snapshot stub (graph/stats reports fromSnapshot: true, totalNodes: 0). So:

    • "no prior snapshot" → false (a stub exists), refusal skipped
    • totalNodes > REBUILD_SAFE_NODE_CEILING (25000) → 0 > 25000 → false, ceiling guard skipped

    …and it falls through to the real totalNodes = nodes.length full kv.list of the 75K on-disk rows → heartbeat starvation → Invocation stopped at ~0.35s (the 6s budget timer fires too late, after the worker has already reconnected). The guard protects a pristine corpus, but the legacy corpora it was written for carry exactly the 0-count stub that defeats it.

    Possible fix directions: gate on the actual on-disk row count rather than the snapshot's self-reported count (a cheap state::count-style probe, if one can be added), or treat a totalNodes: 0 snapshot on a corpus with resetAt unset as "unknown size → refuse unless force." Either way, calling rebuild shouldn't take the worker down — even a no-op refusal would be strictly better than the crash.

    Related: #828 (read path) — same race, confirmed still live on 0.9.27

    Flagging because it's the same mechanism: graph-query with a query string still hard-crashes ({"query":"stripe"} → Invocation stopped 0.35s on 0.9.27), and startNodeId BFS likewise. The withTimeout→snapshot fallback intended to make these degrade gracefully never fires — the heartbeat dies at ~0.35s, well before the 6s budget — so it's a worker crash, not a degrade. Tracked under #828 (per-node adjacency index); noting here only so the "graceful degradation" assumption isn't carried forward as true at scale.

    Standing offer

    Same as before — we've got the >50K repro (680 sessions / ~75K nodes / 344 MB), backups, and a one-command rebuild-and-test loop. Push a branch touching either the rebuild guard or the adjacency index and we'll report behavior + reconnect deltas against the real corpus, usually within the hour.

  3. healdigital
    added a commit that references this issue on Jul 27, 2026

    fix(release): v0.9.28 hotfix wave (rohitg00#843, rohitg00#825, rohitg…

    3bc4cc0

  4. allandelmare commented on Aug 14, 2026

    @allandelmare
    allandelmare
    on Aug 14, 2026
    Author

    Re-test on the released v0.9.28 against the same corpus, via Claude Code. Still reproduces — and the guard hole is now cleaner to describe than it was in June. Push back on anything that reads wrong.

    Re-tested on released v0.9.28 (npm global, not a branch) against the same corpus. Fresh backup first; data intact throughout (131/133 memories, 689 sessions, snapshot unchanged).

    Short version: snapshot-rebuild still hard-crashes the worker, with and without force. graph/reset remains fixed and the #816 hot path remains clean.

    Results

    POST /graph/query {}                          -> 200                             0.039s  reconnect +0  (hot path, #816 — healthy)
POST /graph/snapshot-rebuild                  -> 500 {"error":"Invocation stopped"}  0.641s  reconnect +1
POST /graph/snapshot-rebuild {"force":true}   -> 500 {"error":"Invocation stopped"}  0.627s  reconnect +1

    Note graph.ts is byte-identical between v0.9.27 and v0.9.28 (997 lines, clean diff), so this is the expected outcome — recording it so the issue has a datapoint against a shipped release rather than a branch.

    The guard hole is now sharper than in my June report

    In June I reported that the pre-flight keyed off a snapshot recording totalNodes: 0. That framing was incomplete. On this corpus today:

    GET /graph/stats -> {"fromSnapshot": true, "totalNodes": 108, "totalEdges": 73,
                     "updatedAt": "2026-06-09T19:55:05.801Z"}

    The snapshot is non-empty and perfectly valid — 108 nodes accumulated by incremental extraction after the June graph/reset. But the raw KV.graphNodes / KV.graphEdges scopes still hold the pre-reset rows (~75K; I can't re-count, since enumerating them is the thing that crashes).

    So the pre-flight at graph.ts:875:

    const existing = await readSnapshot(kv);
if (!existing && !forceRebuild) { /* legacyCorpus refusal */ }

    …sees a healthy snapshot, !existing is false, refusal skipped. Execution falls to graph.ts:900 which kv.lists both raw scopes, hits ~75K rows, starves the heartbeat, worker dies at ~0.6s. The REBUILD_SAFE_NODE_CEILING check at line 909 is downstream of the enumeration it exists to prevent, so it can never fire.

    The generalisable statement: the guard gates on the snapshot's self-reported size, while the crash is caused by the on-disk row count — and after a reset those two numbers are unrelated by design. A totalNodes: 0 stub was just one way to hit it; a legitimate 108-node snapshot sitting on 75K orphans hits it just as hard, and force: true is not an escape hatch since it skips straight to the same enumeration.

    Log evidence for the unwinnable race

    [agentmemory] error Graph snapshot rebuild failed
  {"error":"graph-snapshot-rebuild enumeration: exceeded 6000ms budget"}

    That line lands after the worker has already reconnected. The heartbeat deadline (~0.4–0.6s observed) is an order of magnitude shorter than LIVE_ENUMERATION_BUDGET_MS = 6000, and the synchronous JSON.parse blocks the timer as well — so withTimeout cannot convert this into a graceful degrade. Any fix that relies on racing the enumeration will have this same problem.

    Related

    #1189, filed independently today, is the other half of this: graph-reset writes an empty snapshot but deliberately leaves KV.graphNodes / KV.graphEdges on disk. That is precisely what manufactures the "small snapshot, huge disk" state above — so #1189's orphan-retention and this guard hole compound. Worth fixing together: if reset actually cleared the rows (or reads/guards filtered on resetAt), this corpus would stop being a landmine.

    I also built and tested PR #893 against this corpus — it inherits the same snapshot-self-report gating pattern at boot. Details on the PR.

    Possible directions (unchanged, plus one)

    • Gate on an on-disk row count rather than the snapshot's self-report — needs a cheap state::count-style probe, which may not exist.
    • Treat "snapshot exists but resetAt is set and orphan rows may remain" as unknown size → refuse unless force, and make force do a bounded/paginated walk rather than a single kv.list.
    • Failing both: refuse unconditionally with a clean 400 when size cannot be established cheaply. A no-op refusal is strictly better than a daemon-killing 500.

    Standing offer

    Still open, and I'll keep honouring it: >50K repro (691 sessions / ~75K pre-reset rows / 335 MB), backups, one-command build-and-test loop. Push any branch and I'll report rebuild/reset/hot-path behaviour plus worker-reconnect deltas, usually within the hour.

    Environment

    • agentmemory v0.9.28 (released, npm global)
    • macOS Darwin 25.6.0 arm64, Node 26 (Homebrew), iii-engine pinned v0.11.2
    • LLM/embeddings: OpenRouter (gemini-2.5-flash-lite + openai/text-embedding-3-small)
    • Corpus: 691 sessions, 131/133 memories, snapshot 108 nodes / 73 edges, 335 MB on disk
    • Data verified intact after every crash in this run

  5. cbsincool commented on Sep 21, 2026

    @cbsincool
    cbsincool
    on Sep 21, 2026

    Adding a data point on how corpora reach the no-path-to-rebuild state, and one thing the ceiling check itself does that isn't covered above.

    The state this issue describes is also reachable without a legacy corpus, via the write side.

    persistGraphDelta reads the snapshot through the fail-open readSnapshot (graph.ts:80-93, catch → null) and coalesces it (graph.ts:557):

    const snap = (await readSnapshot(kv)) ?? emptySnapshot();

    A transient state::get failure therefore reads as "no snapshot yet" and the next extraction writes back a snapshot derived from emptySnapshot() — counters at or near zero, while KV.graphNodes/KV.graphEdges still hold the full corpus. Reported separately as #1381.

    How the two guards then behave — they disagree, and neither offers a non-destructive exit.

    The rebuild preflight (graph.ts:990-1014) reads through the lenient readSnapshot, so:

    • If the read failed, existing is null → refuses with legacyCorpus, and the message is correct about the cause but wrong about the corpus (it is not legacy — it is the same corpus that was fine a moment ago).
    • If the read succeeded but returned the zeroed snapshot, it is a present, version-1 snapshot → the preflight passes → execution reaches the ceiling check → refuses with tooLarge instead. Two different errors for one underlying fault, decided by whether a read happened to time out.

    Both refusals name POST /agentmemory/graph/reset as the remedy (graph.ts:1003 and graph.ts:1038). So on a corpus with live data and a corrupted snapshot, the safe path refuses and the destructive path is the one both error messages recommend — and it is one call away. That asymmetry is the part I'd flag: graph/reset is the only branch either guard offers, and it discards the last good copy of the graph.

    One detail in the ceiling check itself: the comparison at graph.ts:1026 runs on the raw enumeration, and the !n.stale filter is at graph.ts:1049 — after it. So nodes.length counts stale rows too. A corpus that is mostly stale, and whose snapshot could be rebuilt cheaply from the small live remainder, still refuses on the raw count.

    A data point on the ceiling as a proxy. Measured on a docker-hosted iii state adapter, kv.list over both scopes with 22504 nodes / 51082 edges completed in ~800ms, well inside LIVE_ENUMERATION_BUDGET_MS (6000). The 25K guard is protecting against payload-size/JSON.parse starvation, which is real on the 344MB corpus you tested, but the node count is a proxy for that rather than the quantity itself — so the threshold is conservative on a fast local store and, as you note, still not safe at 75K. Not a suggested fix; the cursor/pagination on state::list you identify is. I raised it to 1e5 in a build-time patch locally purely to get one rebuild through, and that is explicitly not a fix — it moves the failure rather than removing it.

    Two things that would break the dead-end without needing pagination:

    1. Have the ceiling refusal not lead with graph/reset, and offer a snapshot-only rebuild that recomputes stats from the live enumeration without the index-rewrite loops (the badge only needs the snapshot write; the BATCH_SIZE=100 index backfill is ~N/100 sequential round-trips against an already-struggling store).
    2. Have the preflight distinguish "read failed" from "no snapshot", the same fix as persistGraphDelta can zero the graph snapshot on a transient read failure (readSnapshot fail-open) #1381 — so a transient failure reports as a transient failure instead of as a legacy corpus.
  6. possiblyneal
    added 3 commits that reference this issue on Oct 4, 2026

    docs(triage): keep five iii-symptom rows as candidates

    aabc313

    docs(triage): keep five iii-symptom rows as candidates

    7f92c18

    fix(graph): rebuild and reset graphs past 25,000 nodes in pages

    f854bfc

  7. rohitg00 commented on Oct 8, 2026

    @rohitg00
    rohitg00
    on Oct 8, 2026
    Owner

    @allandelmare v0.9.30 adds POST /agentmemory/graph/compact, which trims a legacy graph in slices without blocking the worker, and runs it once on first start. Please try it on your >25K graph; if snapshot-rebuild or graph-reset still crash afterwards, keep this open with the log. https://github.com/rohitg00/agentmemory/releases/tag/v0.9.30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions