Skip to content

Sessions never end on crash/SSH-drop/Ctrl-C — graph extraction + consolidation silently skipped, then evicted #308

Closed

Description

@flamerged

Summary

The current session lifecycle relies on Claude Code's SessionEnd hook firing reliably to trigger graph extraction and consolidation (event::session::stopped → mem::graph-extract, plus the consolidation pipeline in session-end.mjs). In practice, SessionEnd doesn't fire reliably across the common failure modes that real users hit: SSH disconnects, Ctrl+C in remote terminals, laptop sleep, network blips, /clear (per anthropics/claude-code#6428), and even when it does fire, hook async work is killed before completion (anthropics/claude-code#41577). Filing as a design discussion rather than a bug — would like the maintainer's take on intent.

What I see in the code

In dist/index.mjs (v0.9.9):

  • event::session::stopped is the only path that fires mem::graph-extract (line ~16415). It's only triggered when something POSTs /agentmemory/session/end (i.e. via the SessionEnd hook, or a manual call).
  • mem::evict (line ~5045-5077) sweeps sessions older than staleSessionDays and deletes any that lack a summary, citing reason: "stale_session_without_summary". No graph extraction or consolidation runs before the delete.
  • memory_diagnose flags abandoned sessions at the 24-hour mark with status: "warn" but only logs — doesn't trigger any recovery action.

Net behaviour for a user whose Claude Code crashes / disconnects / SSHs drop:

  1. Session stays status="active" forever.
  2. No event::session::stopped ever fires → no graph extraction, no consolidation.
  3. After staleSessionDays (default unclear — I couldn't find it in the env doc), the eviction sweep deletes the entire session including its observations. Graph would-have-extracted content is permanently lost.

Why this matters in practice

Per the Claude Code issues referenced above, SessionEnd reliability is far from perfect even for "graceful" cases:

For agentmemory's "remote/SSH-heavy" user base (this is precisely the audience for coding agents on cloud machines), these aren't edge cases — they're the modal experience. A user who runs Claude Code over SSH and accidentally Ctrl+C's, or whose laptop sleeps, or whose connection drops, has their session-end pipeline silently skipped and the data ultimately deleted by the stale-sweep.

Questions for the maintainer

  1. Is the current design intentional? I.e., is the assumption "users run /exit cleanly, and if they don't, the data was disposable anyway"? Or is this a known gap that just hasn't been prioritized?
  2. /resume semantics — since /resume keeps the same session_id, the existing design does support cross-disconnect continuity… if the user remembers to resume. If they don't, the session ages out and is deleted. Was this the intended trade?
  3. Should the stale-sweep run extraction + consolidation before deleting? That would convert "data loss" into "data preserved as graph/insights only", which feels like the right safety net even if the rest of the design stays as-is.

Proposed design (one option among several)

A backend-side idle-session sweep, since agentmemory's persistent process sees every observation's updatedAt and doesn't depend on any hook firing:

  • New scheduled function in agentmemory, runs every ~30 min: iterate kv.list(KV.sessions), find each status === "active" where now - new Date(updatedAt || startedAt).getTime() > AGENTMEMORY_IDLE_SESSION_HOURS * HOUR_MS, and for each:
    • Trigger event::session::stopped (graph extraction, slot reflection)
    • Trigger the consolidation pipeline equivalent to what session-end.mjs does
    • Set status="abandoned" (or "completed" — design call) and endedAt
  • New env var AGENTMEMORY_IDLE_SESSION_HOURS (default 24 or similar)
  • Idempotency is already handled — I verified empirically that running /agentmemory/graph/extract on the same observation set multiple times produces +0 nodes / +0 edges on the re-runs (dedup by extracted-entity content). Same is true for consolidation per the content-merge semantics.
  • The existing stale-sweep can still run AFTER this — and now it deletes sessions that were already extracted, so the data is preserved as graph nodes and semantic facts even if the raw observations are GC'd.

Variations worth considering:

  • Use a shorter threshold (4h? 12h?) for "session went idle without /exit" — most users would either /resume quickly or want extraction to fire soon after they walked away.
  • Distinguish "session updatedAt is fresh, session has just been idle for a while" from "session has been abandoned" — the first is just normal /resume lag, the second is a genuine crash.
  • Optional: emit a Pushover/webhook on abandoned-session detection (configurable) so users notice they have an orphaned session worth resuming.

What I'm doing on my deployment in the meantime

Building a small systemd timer that polls agentmemory's API every few hours, finds active sessions with updatedAt older than 24h, and POSTs /agentmemory/session/end to trigger the full clean-end pipeline before the stale-sweep can delete them. Stopgap until upstream decides direction. Happy to share if useful as a reference for what an in-tree implementation might look like.

Context

I deploy agentmemory on a small production server (@agentmemory/agentmemory@0.9.9, host-systemd + docker compose for iii-engine). My session today has ~360 observations and the graph would have stayed at 0/0 forever without manually backfilling — that surfaced this whole design question. Earlier issues from same deployment: #299 (viewer port), #301 (volume perms) — both now merged in #304. Thanks for the responsive turnaround on those.


Plugin: @agentmemory/agentmemory@0.9.9. Verified end-to-end on a Hetzner Cloud host.

Activity

  1. flamerged commented on May 12, 2026

    @flamerged
    ContributorAuthor

    More angles worth considering

    Thinking about this from a few more directions — wanted to lay out the full design space so the call you make has more options to compare against. Not advocating for any specific one; some of these compose with each other, some replace each other.

    A. Decouple extraction from session-end entirely

    The premise here: a session is just a logical grouping. The extraction work doesn't fundamentally need to wait for it to "end" — it could happen continuously as observations come in.

    • A1 — Incremental rolling extraction. Every Nth compressed observation (e.g. every 25), or every M minutes of activity per session, fire a partial mem::graph-extract over the new observations since the last extraction watermark. Pros: data is in the graph within minutes, crashes never lose more than ~one batch worth. Cons: more LLM calls overall (though dedup makes the cost-per-extraction approach asymptotic), watermark bookkeeping per session.
    • A2 — Per-observation contribution. Drop the "session" boundary from extraction. When a compressed observation has graph-worthy content (decision / discovery / error / pattern), run it through extraction immediately. Same dedup applies. Closest to "stream-processed" — but might produce less coherent triples without surrounding context.
    • A3 — Time-bucketed batch. Run extraction across all sessions once per N-minute bucket, processing observations whose timestamp ∈ [bucket_start, bucket_end]. Crashes lose at most one bucket. Closer to how Hindsight / GBrain handle batching.

    B. Additional trigger paths (multiple firing routes)

    Don't replace SessionEnd — add fallbacks so at least one fires.

    • B1 — SessionStart-as-recovery. When a new session starts in a given cwd, find any other active session for the same cwd with stale updatedAt, fire event::session::stopped on it. Cheap; assumes "you can't be in two active sessions of the same project at once", which is true in practice for the Claude Code workflow.
    • B2 — Per-project rotation. Stronger version of B1: only one session can be active per cwd at a time. When SessionStart fires for a cwd that already has an active session, auto-end the older one. Common in dev tools (Jest, esbuild dev server, etc.).
    • B3 — PreCompact as natural extraction point. When Claude Code's PreCompact hook fires (either auto or via /compact), treat that as a "session is about to lose context" signal and run extraction on the to-be-compacted observations. Won't help on crash but covers the long-session case where extraction would naturally lag.
    • B4 — Backend idle-sweep (the original proposal in this issue). Still relevant; pairs well with any of A or B above.

    C. Make the existing failure mode safe (defensive, minimal change)

    If the rest of the lifecycle stays as-is, the minimum-viable fix is making the stale-sweep non-destructive of insights.

    • C1 — mem::evict extracts before deleting. Before deleting a stale_session_without_summary, run graph extraction + consolidation on its observations. Then delete the raw observations only; the graph nodes / semantic facts / procedural memories survive. One-function change in src/functions/retention.ts (or wherever evict lives). Closest thing to a true bug-fix: prevents the worst outcome (data loss) without changing trigger semantics.
    • C2 — Configurable retention. AGENTMEMORY_STALE_SESSION_DAYS env var (if not already configurable; couldn't tell from quick code read). Lets users with valuable orphaned sessions extend the window.

    D. User-facing escape hatches

    • D1 — agentmemory reap CLI subcommand. On-demand sweep that finds abandoned sessions and fires their end pipeline. Users wire it into their own cron / systemd timer if they want it automated. Smallest surface change; punts policy to the user.
    • D2 — Viewer surfaces "abandoned sessions" as actionable. Right now the dashboard's "abandoned-session" warning is informational. Add a "Reap now" button next to each that POSTs /agentmemory/session/end. Manual but discoverable.

    Combinations I'd consider

    • Minimum-disruption pair (probably what I'd ship first): C1 + D1. C1 prevents data loss with a one-function change; D1 gives users control. Doesn't change trigger semantics anywhere. Almost no risk surface.
    • "Self-healing": A1 + B1 + B4. Continuous extraction (A1) keeps the graph fresh; SessionStart-as-recovery (B1) catches abandoned sessions when a new one starts in the same cwd; backend idle-sweep (B4) catches everything else. Significantly more code.
    • Just the safety net: C1 alone. Don't change any trigger semantics. Just make the existing eviction non-destructive of insights. One-function change, probably <50 lines.

    Sentinel-side stopgap update

    While the design discussion happens here, I'll build the equivalent of D1 on Sentinel as a systemd timer (cron-style). Happy to share the code once it's stable — could serve as a reference implementation if D1 or B4 is the direction you'd pick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions