octop内存溢出根因报告_20261016.html

What happened
In octop 1.0.1 (bundling deepagents 0.7.9), an agent-triggered filesystem search can consume unbounded memory and freeze the entire host machine (not just the container).
An agent session invoked the filesystem grep tool with no path (path=None, i.e. search from the backend root /). Because ripgrep is not on PATH in the container (shutil.which('rg') → None), the search fell back to the pure-Python implementation in deepagents/backends/filesystem.py (_python_search). That implementation walks the whole filesystem (root.rglob("*")) and reads every file in text mode, line by line. When the walk reaches /proc//pagemap — a pseudo-file that

• looks like a regular file,
• reports st_size == 0 (so it passes the 10 MB max_file_size_bytes check),
• yields effectively unlimited content (the page-table mapping of the entire address space — we measured 200 MB read in 0.1 s with no EOF in sight),
• contains very long stretches without a newline,
  Python's readline() starts accumulating one single "line" with no bound. All timeout checks are per-file / per-2048-lines, so none of them ever fires while the loop is stuck in that one read. The async wrapper (deepagents/backends/protocol.py, agrep) uses asyncio.wait_for(asyncio.to_thread(grep_call), timeout=35). After 35 s it logs “agrep timed out”, but the worker thread is not stopped — its own docstring says: “The timeout bounds how long the caller waits; it does not stop the worker thread created by asyncio.to_thread.” The thread keeps reading, and the process keeps growing.
  Impact observed in production (two separate incidents, ~1 hour apart, same morning):
• The octop process reached 19.3 GB anon-rss + ~4.96 GB swap on a 21 GB host.
• The whole machine became unresponsive (all services, SSH, Docker). The first incident required a hard reset; in the second the kernel OOM killer terminated the octop process, which recovered after an automatic container restart.
  Both freezes were preceded, 1–2 minutes earlier, by an unscoped agent search:
• 08:32:19 search starts → 08:33:52 host freeze (incident 1)
• 09:25:58 search starts → 09:28:02 host freeze (incident 2) → 09:32:20 kernel OOM kill
  In other words: any agent user can trigger a host-wide denial of service just by asking a question that leads the agent to run an unscoped search, on a machine where ripgrep is unavailable.
  (Local mitigations applied: installed a static ripgrep binary into the container so the Python fallback is no longer used, and set a container memory limit. The underlying defects remain and can be re-triggered on any deployment that hits the fallback path.)
  Expected behavior
• A filesystem search should never consume unbounded memory. The Python fallback should at minimum:
  • skip pseudo/virtual filesystems (/proc, /sys, /dev) and any file whose content cannot be safely bounded despite st_size == 0;
  • enforce a per-line and per-file byte cap while reading (a single readline() must never be able to grow without bound).
• The 35 s tool timeout should actually stop the underlying work — e.g. run the fallback in a separately killable process, or at least check the deadline far more often than every 2048 lines.
• On a machine without ripgrep, a search should degrade gracefully (partial results or a clear error), not take down the host.
• Additionally, LocalShellBackend defaults to root_dir='/' (the startup warning itself says: “the agent has shell access to the entire host. Restrict backend.root_dir in production.”). A workspace-scoped default would eliminate this whole class of accident.
  Steps to reproduce

1. Run octop 1.0.1 (Docker) on a machine where rg is not on PATH inside the container (the case with the stock image in our deployment).
2. Trigger an agent filesystem search with no explicit path — either ask the agent a question that makes it call grep with path=None (a full-filesystem search from /), or call the backend directly:
   from deepagents.backends.filesystem import FilesystemBackend

b = FilesystemBackend(root_dir='/', virtual_mode=False)
b.grep('some-unlikely-string') # path=None → searches from the backend root ('/' in this deployment)

1. Watch the process memory: stuck on /proc//pagemap, RSS grows continuously until RAM/swap are exhausted — on an uncapped machine the host freezes and the process gets OOM-killed. (The tool call reports “grep timed out after 35s” while the worker thread keeps running and growing.)
   Safe observation: cap the process memory first (e.g. ulimit -v 3000000, or Python resource.setrlimit(RLIMIT_AS, ...) at 3 GB). The read then fails with MemoryError within ~12 seconds, with a VmHWM (peak RSS) jump from ~13 MB straight to the cap.
   Environment

• octop version: 1.0.1 (image ghcr.io/tencentcloud/octop:1.0.1); bundled deepagents 0.7.9
• Python version: 3.12.14
• OS: CentOS 7 host (kernel 3.10.0-1160) + Debian-based container, Docker
• Install method: pip / Docker / source → Docker (docker compose)
• LLM provider (if relevant): OpenAI-compatible endpoint (self-hosted gateway) — not related to this bug
  Logs / screenshots

1) ripgrep missing → Python fallback is used (logged once per process)

deepagents.backends.filesystem — ripgrep ('rg') not found on PATH; using Python grep fallback.

2) The "timeout" that doesn't stop the worker thread (fired 35 s after the search started)

deepagents.backends.protocol — agrep timed out after 35s (pattern='...', path=None, glob=None)

3) Kernel OOM kill after the machine ran out of memory (21 GB host)

Out of memory: Kill process 2524 (octop) score 800 or sacrifice child
Killed process 2524 (octop), UID 0, total-vm:26921268kB, anon-rss:19351876kB, file-rss:0kB

4) Controlled reproduction (memory capped at 3 GB) — peak RSS (VmHWM) within 12 s:

12,940 KB → 2,990,220 KB (hit the 3 GB cap)
ERR read /proc/1/pagemap: MemoryError()

5) Direct measurement of the pseudo-file (why the read never ends)

/proc/1/pagemap : 200 MB read in 0.1 s (hit the test cap; no EOF)
/proc/1/smaps : 481 KB (finite)
/proc/1/numa_maps : 104 KB (finite)
/proc/1/maps : 111 KB (finite)
Happy to provide the full reproduction script or additional logs if useful.