Skip to content

fix(search): route rtk grep/rg through the already-bounded capture path - #3134

Closed
albatrossflyon-coder wants to merge 1 commit into
rtk-ai:developfrom
albatrossflyon-coder:fix/rg-stream-memory-bound
Closed

albatrossflyon-coder wants to merge 1 commit into
rtk-ai:developfrom
albatrossflyon-coder:fix/rg-stream-memory-bound

Conversation

@albatrossflyon-coder

Copy link
Copy Markdown
Contributor

Fixes #3106.

The bug

engine_capture() and passthrough() in search.rs called exec_capture_stdin, which wraps Command::output() and buffers the entire child process's stdout/stderr in memory with no limit — on a large search corpus this could consume 50+ GB before the 200-result cap in search.rs ever gets a chance to apply.

The fix

rtk already has a memory-safe capture path for exactly this: run_streaming with FilterMode::CaptureOnly reads line-by-line through the existing RAW_CAP (10 MiB), already covered by test_run_streaming_raw_cap_at_10mb in stream.rs. It just wasn't being used by grep/rg.

Added exec_capture_stdin_bounded as a thin CaptureResult-shaped wrapper around run_streaming, so both call sites in search.rs needed a one-line swap — no changes to search.rs's grouping/capping logic at all. Removed the now-unused exec_capture_stdin (the unbounded function this replaces).

Testing

  • Full suite: 2457 passed, 8 ignored (unchanged — the bounded-capture behavior itself is already covered by test_run_streaming_raw_cap_at_10mb, so this PR doesn't need new unit tests to prove that mechanism works; it only needed to prove search.rs is actually using it).
  • Manual verification: rtk grep against a real 36MB file where every line matches. Completes cleanly (exit 0) and stderr shows [rtk] warning: output exceeds 10 MiB — filter input truncated — proof the cap is genuinely active now (this warning could not have fired through the old unbounded path).
  • cargo fmt --check and cargo clippy --bin rtk -- -D warnings: both clean.

Fixes rtk-ai#3106.

engine_capture() and passthrough() in search.rs called exec_capture_stdin,
which wraps Command::output() and buffers the entire child process's
stdout/stderr in memory with no limit — on a large search corpus this
could consume 50+ GB before the 200-result cap in search.rs ever got a
chance to apply.

rtk already has a memory-safe capture path for exactly this: run_streaming
with FilterMode::CaptureOnly reads line-by-line through the existing
RAW_CAP (10 MiB), already covered by test_run_streaming_raw_cap_at_10mb
in stream.rs. It just wasn't being used by grep/rg. Added
exec_capture_stdin_bounded as a thin CaptureResult-shaped wrapper around
run_streaming so both call sites in search.rs needed a one-line swap; no
changes to search.rs's grouping/capping logic. Removed the now-unused
exec_capture_stdin (the unbounded function my change replaced).

Testing:
- Full suite: 2457 passed, 8 ignored (unchanged from before this fix —
  the bounded-capture behavior itself is already covered by
  test_run_streaming_raw_cap_at_10mb).
- Manual verification: `rtk grep` against a real 36MB file where every
  line matches. Completes cleanly (exit 0) and stderr shows
  "[rtk] warning: output exceeds 10 MiB — filter input truncated" —
  proof the cap is genuinely active now (this warning could not have
  fired through the old unbounded path).
- cargo fmt --check and clippy (-D warnings): both clean.
@KuSh

KuSh commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Closing in favour of #3849, which bounds the same unbounded search capture and goes further: it caps per line as well as per stream, reaps the child on a broken pipe, and adds regression tests for both cases. Please take any further discussion of the capture path to that PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

duplicate This issue or pull request already exists

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rtk rg can consume over 50 GB by buffering and duplicating large search output

2 participants