All data lost on restart — stop order kills iii engine before worker can flush

Bug: Complete data loss on `agentmemory stop` → restart

Summary

All memories, observations, graph nodes, and indexes are lost after agentmemory stop followed by a restart. The "Memories persisted to disk" message shown at shutdown is misleading — nothing is actually persisted.

Steps to reproduce

Start agentmemory, save some memories via memory_save
Verify with agentmemory status — shows N memories, N nodes
Run agentmemory stop (or stop --force)
Restart with agentmemory
Run agentmemory status — shows 0 memories, 0 nodes

Before stop

Memories:     5
Sessions:     1
Observations: 2
Graph:        4 nodes, 3 edges

After restart

Memories:     0
Sessions:     0
Observations: 0
Graph:        0 nodes, 0 edges

Root cause analysis (source code investigation)

1. Wrong stop order in runStop() (cli.mjs ~line 2459)

The CLI kills the iii engine FIRST, then the worker:

Line 2528-2534: Stops iii engine (SIGTERM → 3s → SIGKILL)
Line 2535-2542: Stops worker second

But the worker's shutdown handler (index.mjs, bottom) needs iii alive to flush:

const shutdown = async () => {
    indexPersistence.stop();
    await indexPersistence.save();    // calls kv.set() -> iii state::set -> DEAD
    await sdk.shutdown();             // iii already killed
};

indexPersistence.save() -> saveBm25Index() / saveVectorIndex() -> saveShardedIndex() -> this.kv.set() -> sdk.trigger("state::set") -> iii engine. But iii is already dead.

2. iii file_based store doesn't actually persist to disk

The iii config uses store_method: file_based, file_path: ./data/state_store.db. In practice, only a mem%3Ahealth.bin (~12K) reaches disk. All observation, memory, graph, and index data set via kv.set() remains in the iii process's in-memory state. The 3-second SIGTERM->SIGKILL window is insufficient for iii to flush.

3. Relative path issue

The file_path: ./data/state_store.db resolves relative to iii's cwd, which is wherever agentmemory was launched (e.g., ~/Project/), NOT ~/.agentmemory/. Data files scatter based on the user's working directory.

Suggested fixes

Reverse stop order: stop worker first (let it flush through iii), then stop iii engine
Use absolute path: $HOME/.agentmemory/data/state_store.db in iii config
Increase worker SIGTERM grace period: from 3s to 5s+ to allow full flush

Environment

agentmemory v0.9.26
macOS, Node.js LTS
Storage: native iii engine (not Docker)
LLM: local Ollama (qwen2.5-coder:7b)

Workaround

Re-save critical memories manually after every restart.

fix(graph,search): cr followups for #843/#825/#817 wave

fix(release): v0.9.28 hotfix wave (#843, #825, #817, iii pin) (#849)

Windows: the #849 worker-first shutdown fix is a no-op — `SIGTERM` is a hard kill, the worker flush handler never runs

Confirming #843 still loses all runtime data on Windows, even with the #849 fix (worker-first SIGTERM, 5s grace) present in the installed build. The reorder is correct on POSIX but cannot work on Windows, because Node maps process.kill(pid, "SIGTERM") to a hard TerminateProcess — the worker's SIGTERM handler (which runs indexPersistence.save() → kv.set → state::set → iii) never executes. The process is terminated, not signalled.

Environment

Windows 11 (10.0.22631)
@agentmemory/agentmemory@0.9.27 (npm latest, already contains the fix(release): v0.9.28 hotfix wave (#843, #825, #817, iii pin) #849 worker-first shutdown reorder)
iii engine 0.11.2
Server launched as a hidden daemon (wscript.exe start-agentmemory-hidden.vbs → node dist/cli.mjs)

Repro (normally-captured observation, not a manual import)

Emit a unique marker so the PostToolUse hook captures it as an observation:
SENTINEL_MARKER_ZX9Q7_unique_probe
Confirm it is indexed before any restart:
iii trigger --function-id mem::search --payload '{"query":"SENTINEL_MARKER_ZX9Q7","limit":5}'
→ results: 1, marker present. ✅
Stop via the supported graceful path (the fix(release): v0.9.28 hotfix wave (#843, #825, #817, iii pin) #849 code path):
node dist/cli.mjs stop
→ prints Stopped worker pid … / Stopped pid … / Stopped. Memories persisted to disk
Start again, re-run the same search:
→ results: 0, marker gone. ❌

Evidence it never flushed

.bin file mtimes are unchanged across the graceful stop (e.g. mem:sessions.bin, mem:obs:<id>.bin keep their old mtimes). Nothing is written on shutdown.
A manually imported session (POST /agentmemory/import, verified live via state::list mem:obs:<sid> = 14) is gone (= 0) after the graceful stop + restart.
After any restart, state::list mem:sessions returns only the active session; state::get of older session keys returns null; raw mem::search of older terms returns 0 — even though the corresponding .bin files exist on disk with valid contents. So the engine also does not reconcile on-disk scopes back into its in-memory catalog at boot (the read side of the same problem).

Net effect on Windows: search/recall is effectively ephemeral — observations are searchable only within a single continuous server uptime. Every restart resets recall to (at most) the active session. (SessionStart context injection is unaffected, because that hook reads sessions.bin/summaries.bin from disk directly, bypassing the in-memory catalog.)

Secondary observations

signalAndWait() uses process.kill(pid, "SIGTERM"); on Windows this is TerminateProcess, so no catchable signal reaches the worker.
findEnginePidsByPort() early-returns [] on Windows, so stop depends entirely on ~/.agentmemory/{iii,worker}.pid.
The Stopped. Memories persisted to disk outro is printed unconditionally and is misleading on Windows — no flush actually occurred.

Suggested direction
A POSIX SIGTERM handler can't be the flush trigger on Windows. Options:

An explicit flush/checkpoint control call (HTTP endpoint or IPC/named-pipe message) that the CLI invokes and awaits before terminating the worker — i.e. flush-then-kill instead of signal-then-hope.
A Windows-native graceful shutdown (job-object/console-control-event, or a state::flush-style trigger exposed for the CLI to call).
Independently, fix the boot reconcile so existing on-disk scopes are loaded back into the catalog/index on startup — this would also recover data already written to .bin by earlier versions/imports.

Happy to provide full command transcripts or test against a patch.

fix(release): v0.9.28 hotfix wave (rohitg00#843, rohitg00#825, rohitg…

perf(graph): extract only observations added since the last pass [por…

@joyearnaud

@joyearnaud fixed in v0.9.30: agentmemory stop waits for the engine flush, state is saved every 2 s, and memories and vectors written just before a stop survive the restart. https://github.com/rohitg00/agentmemory/releases/tag/v0.9.30

Navigation Menu

Repository navigation

All data lost on restart — stop order kills iii engine before worker can flush #843

Bug: Complete data loss on `agentmemory stop` → restart

Summary

Steps to reproduce

Before stop

After restart

Root cause analysis (source code investigation)

Suggested fixes

Environment

Workaround

Windows: the #849 worker-first shutdown fix is a no-op — `SIGTERM` is a hard kill, the worker flush handler never runs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

All data lost on restart — stop order kills iii engine before worker can flush #843

Description

Bug: Complete data loss on agentmemory stop → restart

Summary

Steps to reproduce

Before stop

After restart

Root cause analysis (source code investigation)

Suggested fixes

Environment

Workaround

Activity

dmitryzlatous commented on Jun 8, 2026

Windows: the #849 worker-first shutdown fix is a no-op — SIGTERM is a hard kill, the worker flush handler never runs

rohitg00 commented on Oct 8, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Bug: Complete data loss on `agentmemory stop` → restart

Windows: the #849 worker-first shutdown fix is a no-op — `SIGTERM` is a hard kill, the worker flush handler never runs