Idle CPU leak: iii worker threads busy-wait, growing to a full core over ~40h uptime #1093

Package: @agentmemory/agentmemory 0.9.28 (current latest on npm)
iii runtime: 0.11.2
Deployment: Docker, image agentmemory-combined, network_mode: host
Host: Synology DSM 7.3.2, 4-core x86_64, 31 GiB RAM
Env set: CONSOLIDATION_ENABLED=true (interval vars unset — defaults apply)

Summary

After ~40 hours of container uptime, the iii process sits at 94–99% CPU continuously while completely idle — no log output, no network I/O, no block I/O. CPU is consumed by a busy-wait spread across the runtime's worker threads, not by any agentmemory workload.

Restarting the container clears it completely (98.74% → 0.36%), and resident memory drops at the same time (1.769 GiB → 1.366 GiB), so this looks like an accumulating leak rather than a fixed cost.

On a 4-core host this is a permanent 25% CPU loss.

Evidence

docker stats while completely idle:

agentmemory-agentmemory-1   CPU=94.83%   MEM=1.77GiB / 31.32GiB   NET=0B / 0B   BLK=0B / 0B

Note NET=0B and BLK=0B — nothing is actually being processed.

Zero log activity during that CPU burn:

$ docker logs -t --since 10m agentmemory-agentmemory-1 | wc -l
0

Per-thread breakdown (top -b -H) — four workers sharing the burn roughly equally:

  PID USER  %CPU  S  TIME+      COMMAND
30300 root  95.33 R  380:58.24  /usr/local/bin/iii
30302 root   0.33 R  428:46.06  /usr/local/bin/iii
30301 root   0.00 S  371:58.30  /usr/local/bin/iii
30303 root   0.00 S  449:06.90  /usr/local/bin/iii
30299 root   0.00 S    0:03.89  /usr/local/bin/iii   <- main thread, idle

Roughly 27 CPU-hours accumulated across ~40 hours of wall time. The near-equal cumulative totals across the four workers, with the main thread idle, suggest a polling loop without backoff being serviced round-robin by the worker pool.

Restart clears it:

before   CPU=98.74%   MEM=1.769GiB
t+50s    CPU=0.45%    MEM=1.366GiB
t+65s    CPU=0.36%    MEM=1.366GiB

This is NOT the consolidation/auto-forget schedulers

I checked this first, and they are healthy. Timestamped logs show both firing exactly on their default intervals:

auto-forget:    01:02, 02:02, 03:02, 04:02, 05:02, 06:02, 07:02, 08:02   (hourly)
consolidation:  01:02, 03:02, 05:03, 07:03                              (2-hourly)

Matching the code defaults AUTO_FORGET_INTERVAL_MS || 3600000 and CONSOLIDATION_INTERVAL_MS || 7200000. Each consolidation run completes in ~30–45s and then logs nothing further — yet CPU stays pegged in between. memory_diagnose reports the store fully consistent (832 sessions, 16,822 semantic memories, 3,083 lessons, 552 summaries, 825 crystals, 1,083 insights).

Suspected area

The runtime workers configured in dist/iii-config.yaml:

- name: iii-queue    { adapter: { name: builtin } }
- name: iii-pubsub   { adapter: { name: local } }
- name: iii-cron     { adapter: { name: kv } }
- name: iii-state    { adapter: { name: kv, config: { store_method: file_based } } }

A poll loop in one of these (queue builtin, pubsub local, or cron kv polling the file-based state store) that tightens or fails to back off as state grows would match the observed behaviour: idle-but-spinning, worsening with uptime, cleared by restart.

Deployment (docker-compose.yml)

Secrets and host identifiers redacted; otherwise verbatim.

version: "3.8"
services:
  agentmemory:
    image: agentmemory-combined
    restart: unless-stopped
    network_mode: host
    labels:
      - "wud.watch=false"
    volumes:
      - iii-data:/data
      - /volume1/docker/agentmemory/imports:/imports
      - /volume1/docker/agentmemory/agentmemory-config:/root/.agentmemory
      - /volume1/docker/agentmemory/npm-cache:/root/.npm
    environment:
      - AGENTMEMORY_TOOLS=all
      - TOKEN_BUDGET=8000
      - EMBEDDING_PROVIDER=local
      - ANTHROPIC_MODEL=claude-haiku-4-5-20251001
      - VIEWER_ALLOWED_HOSTS=<nas-ip>,<nas-ip>:3114,<nas-ip>:3113,<nas-host>,<nas-host>:3114,localhost
      - HOST=0.0.0.0
      - ANTHROPIC_API_KEY=<redacted>
      - GRAPH_EXTRACTION_ENABLED=true
      - AGENTMEMORY_AUTO_COMPRESS=true
      - AGENTMEMORY_INJECT_CONTEXT=true
      - CONSOLIDATION_ENABLED=true

volumes:
  iii-data:

Note EMBEDDING_PROVIDER=local — embeddings are computed in-process rather than via an API, which may be relevant to where the idle CPU is going.

Impact / workaround

• Workaround: periodic docker restart clears it fully.
• Impact: on small hosts this is significant — 25% of a 4-core NAS consumed permanently for no work. Since 0.9.28 is the current release, any long-running deployment is affected.

Happy to help

I can gather more detail on request (per-thread stacks, strace, longer-window growth curve). Let me know what would be most useful for narrowing it down.