rtk grep uses ~31× the memory and ~4.9× the wall-clock of grep for the same search #3392

Activity

1. pszymkowiak

   added

   area:performance

   bugSomething isn't workingSomething isn't working

   help wantedExtra attention is neededExtra attention is needed

   priority:high

   on Aug 4, 2026
2. pszymkowiak commented on Aug 4, 2026

   

   pszymkowiak

   on Aug 4, 2026

   Collaborator

   More actions

   wshm · Automated triage by AI

   🔍 Automated Triage

   🐛 Category	bug
   🟠 Priority	high
   🎯 Confidence	90%

   Summary

   rtk grep buffers and re-copies grep's entire output multiple times (raw_output clone, HashMap of matches, plain re-render, skipped_block), causing ~31x memory and ~4.9x wall-clock overhead vs grep, with OOM risk scaling linearly on large repos; needs streaming/intake-capped rewrite of src/cmds/system/search.rs.

   📁 Relevant files

   • src/cmds/system/search.rs
   • src/core/stream.rs

   ---

   Triaged automatically by wshm · This is an automated analysis, not a human review.
3. jazelly commented on Sep 3, 2026

   

   jazelly

   on Sep 3, 2026

   Last edited by jazelly

   More actions

   I encountered a similar instance of this problem with rtk rg.

   Environment:

   • macOS on Apple Silicon
   • 96 GB RAM
   • RTK 0.43.0

   Several concurrent, high-output rtk rg searches were running. Two RTK parent processes grew to approximately 83 GiB combined, while their corresponding rg child processes used less than 0.4 GiB combined.

   The machine exhausted memory, entered severe swap pressure, became unresponsive, and was eventually rebooted by the macOS watchdog.

   After rebooting, memory and swap usage returned to normal and no oversized RTK processes remained.

   This appears different from #897: in this case, the memory was held by the RTK parent processes rather than the search subprocesses. It seems consistent with the full-output capture and re-copy behavior described in this issue and #3397.
4. HetCreep commented on Sep 3, 2026

   

   HetCreep

   on Sep 3, 2026

   Author

   More actions

   @jazelly's report above lines up exactly with the mechanism this issue describes, so I re-checked it against current develop (0de9610) to confirm before adding this.

   Provenance: AI-assisted source audit, human-directed, at develop @ 0de9610. Source-read and grep-verified; the machine that hit this is jazelly's, not mine — I have not reproduced the 83 GiB figure myself.

   rtk rg takes the identical path as rtk grep

   Engine::Rg and Engine::Grep are two variants of the same enum (search.rs:270-293) and both flow through the same engine_capture → stream::capture call. There is no separate rg-specific pipeline — the report's choice of rtk rg over rtk grep doesn't matter to the mechanism, which is:

   • stream.rs:590-592: cmd.output() buffers the entire child stdout with no cap. RAW_CAP (stream.rs:286, 10 MiB) only guards run_streaming; capture() never references it.
   • search.rs:568: raw_output.clone()
   • search.rs:595/621/641-645/681: several more full copies built while grouping and re-rendering matches (by_file, plain, skipped_block, full_block).

   So a search with a large match set exists as several complete copies simultaneously, in a process that captures rather than streams, with no size limit anywhere on the path. That is consistent with a parent process growing far larger than its child — the child's own output was never the thing holding the memory.

   What this report adds

   Everything I had before this was a code-level finding: unbounded intake, multiple full copies, no cap. That's a real defect but it describes a shape, not a measured consequence. This is the first report of it actually taking a machine down — 96 GB RAM, swap exhaustion, unresponsive, watchdog reboot. It doesn't change the severity tag (still (C): a local dev tool run by its own operator, no untrusted input), but it's evidence the ceiling on this isn't theoretical.

   The distinction from #897 in the report is correct — that one is subprocess memory, this is parent-process memory, and they're different bugs with different mechanisms. The connection to #3397 is also right; that's the same .clone()-heavy pattern in the streaming path's CaptureOnly arm, a sibling of the capture path this one exercises.

   Not checked

   I have not built a reproduction of the 83 GiB figure — that would need a genuinely large match set and a peak-RSS monitor, which is more than I can responsibly run against my own machine on a hunch. Flagging that the field numbers are unverified by me, distinct from the code path, which I did verify.
5. albatrossflyon-coder commented on Sep 13, 2026

   

   albatrossflyon-coder

   on Sep 13, 2026

   Contributor

   More actions

   We hit this same class of bug independently and already have a fix up: #3134 (routes rtk grep/rtk rg through the already-bounded capture path), which was closed in favor of #3849 continuing that work — but #3849 is also still unmerged. Just flagging so this doesn't stall waiting on a PR nobody's tracking anymore.
6. HetCreep commented on Sep 16, 2026

   

   HetCreep

   on Sep 16, 2026

   Author

   More actions

   Re-verified this at develop @ 5e0f92c. Short version: the bug's still there, unchanged — only the line numbers rotted, and the worst one now actively misleads. The original cite for raw_output = result.stdout.clone(), search.rs:568, lands today inside the passthrough helper's exit-code branch, nowhere near a clone. Reading that line now looks like the bug was removed; it wasn't, it just moved ~100-113 lines down the file after an unrelated arg-tokenizer/flag-classification rewrite landed in between.

   Provenance: AI-assisted source audit, human-directed, at develop @ 5e0f92c. Verified by reading the code at this commit — not built or run.

   Citations, refreshed

   in the report (700bdde)	now (5e0f92c)
   search.rs:568 — raw_output = result.stdout.clone()	search.rs:671,674
   search.rs:595-605 — grouping HashMap	search.rs:701
   search.rs:621-630 — full re-render into plain	search.rs:734-743
   search.rs:682 — match_block	search.rs:172-179
   search.rs:641-645 — skipped_block accumulation	search.rs:755-760, specifically :758
   stream.rs:245 — RAW_CAP	stream.rs:423

   The plain re-render at search.rs:734-743 is still unconditional — it runs over the full raw_output regardless of max_results, same as originally reported. And :758 still does exactly what it did before: every file past the display cap gets its full text rendered through match_block and appended to skipped_block anyway, so the cap adds a copy instead of avoiding one.

   One level below where the original report looked

   engine_capture (search.rs:416-424) calls exec_capture_stdin, never run_streaming — so this path was never in RAW_CAP's reach, exactly as the original report said. Underneath that, capture_raw (stream.rs:700-735) is where the whole child stdout actually lands in memory: stream.rs:745 — let output = cmd.output().context("Failed to execute command")?; — one uncapped Vec<u8>, before any of the four search.rs copies get taken from it.

   No fix landed in between

   git log --oneline 0de9610..5e0f92c -- src/cmds/system/search.rs is 29 commits, all one arg-tokenizer/flag-classification rewrite. The same range on src/core/stream.rs is 10 commits, all git-blob decoding and signal-relay work. Neither touches capture or buffering, and grep -n "MiB|_CAP|memory|RSS" src/cmds/system/search.rs still comes back empty — no cap exists anywhere in the file.

   The measured numbers, the scaling table, and everything else in the original report are unaffected — this is only the line citations catching up.
7. lkraider commented on Oct 2, 2026

   

   lkraider

   on Oct 2, 2026

   Last edited by lkraider

   More actions

   Adding a minimal, self-contained synthetic reproduction script and scaling measurements on macOS Apple Silicon (arm64, Darwin 24.3.0, RTK 0.51.0, also verified present on current develop @ 259d165).

   Synthetic Reproduction Script

   This generates a deterministic matching corpus of a given size and measures peak RSS with /usr/bin/time -l (on Linux, use /usr/bin/time -v):

   #!/usr/bin/env bash
   set -euo pipefail
   
   # Generate synthetic matching corpus (150 MB here; adjust range for other sizes)
   python3 -c '
   with open("repro_matches.txt", "w") as f:
       chunk = ("MATCH: " + ("x" * 90) + "\n") * 1000  # 100 KB chunks
       for _ in range(1500):                           # 150 MB total
           f.write(chunk)
   '
   
   echo "==> Native grep:"
   /usr/bin/time -l grep "MATCH" repro_matches.txt > /dev/null
   
   echo "==> rtk grep:"
   /usr/bin/time -l rtk grep "MATCH" repro_matches.txt > /dev/null
   
   rm -f repro_matches.txt

   Measured Scaling (macOS Apple Silicon)

   Match Output	Tool	Peak RSS	Wall Clock	Memory Ratio
   5 MB	grep rtk grep	1.68 MiB 50.1 MiB	0.01s 0.16s	1.0× ~30×
   20 MB	grep rtk grep	1.75 MiB 167.7 MiB	0.04s 0.57s	1.0× ~95×
   120 MB	grep rtk grep	1.82 MiB 830.5 MiB	0.27s 3.51s	1.0× ~456×
   150 MB	grep rtk grep	1.85 MiB > 1,020 MiB (>1 GiB)	0.33s 4.48s	1.0× > 550×

   Observations

   • Native grep streams its results with a flat resident set under 2 MiB regardless of output volume.
   • In rtk grep, peak RSS crosses 1 GiB on just ~150 MB of matching output due to the uncapped Command::output() in capture_raw (stream.rs:745) and the subsequent raw_output.clone(), by_file HashMap, plain string re-render, and skipped_block accumulation in search.rs.
   • This confirms the memory ceiling is not theoretical even for single-file, moderately sized search results, and reinforces the need for bounded stream capture as proposed in fix(search): bound grep and rg output memory #3849.