fix(ast-grep): account for every match line and stop capturing other subcommands

[KuSh]

Fixes three regressions from the release review on #3979, all introduced by #3857, which added src/cmds/system/ast_grep_cmd.rs.

Part of a set of six, one per originating PR: #3681, #3552, #3265, #3772, #3857 (this), #3941.

1 — 87 % of matches dropped with no way to read them

The caps counted the lines they held back inside a shown file, but a file the total cap skipped outright was reported as a file and nothing else. On the module's own fixture (387 match lines across 32 files):

50 lines shown
82 lines accounted for by per-file notes
255 lines mentioned nowhere at all
… 20 more file(s) with matches not shown (use --json or narrow the pattern)

Now those lines are counted too, and the full output goes behind the [full output: …] handle that src/cmds/README.md requires of any capping filter:

… 255 more match line(s) in 20 more file(s) not shown
[full output: ~/.local/share/rtk/tee/…_ast-grep.log]

50 + 82 + 255 = 387. filter_ast_grep returns a hidden_lines count so the I/O stays in run().

The decision to store is taken on the text the hint is already part of. Deciding before appending it left a band where the hint tipped never_worse back to raw: the archive was written, raw was printed, and the file sat there with nothing pointing at it — having evicted an archive another caller still needed.

2 — scan --report-style short was captured despite the docs

That style emits path:line:col: message, which matches the module's path:line:content regex, so a 60-finding lint report was grouped and capped to 18 lines — while the module header and FEATURES.md both said scan passes through.

3 — lsp ran with stdin closed

exec_capture hands the child a null stdin, so ast-grep lsp served an editor an empty document:

$ printf 'EDITOR-BYTES' | ast-grep lsp        # native
lsp-read:[EDITOR-BYTES]
$ printf 'EDITOR-BYTES' | rtk ast-grep lsp    # develop
lsp-read:[]

The routing fix

Only run produces the shape this module parses. Every other subcommand, and a run --stdin, now goes through runner::run_passthrough, which inherits stdin.

Telling a subcommand from a flag's value needs a grammar, read in two passes, because which options take a value depends on the subcommand that has not been identified yet:

1. the options that may precede a subcommand (--config, --color, -c) — the first free positional this finds is the subcommand
2. once that positional is run, run's own grammar, re-read

Without the second pass, run -p scan src/ took its pattern for the scan subcommand and gave up the filter entirely.

A line this module cannot identify is not filtered: if the first positional is neither run nor a known subcommand, an option whose arity is unknown may have claimed the subcommand's place, so any of the other names anywhere on the line disqualifies it. --color always lsp reading as a run is what closed that stdin. The cost is a pattern or path spelled exactly like a subcommand going unfiltered — compression, not correctness.

--json is now read from tokens too, so a path named --json past -- no longer disables the filter.

Verification

15 unit tests, including: every line printed or counted on the real fixture; the trailer's line and file counts; a subcommand behind --color/--heading/--no-ignore/--globs/--threads; a run flag's value not read as a subcommand; paths past --; --json from tokens; and a capped output always carrying a route to the rest.

Verified end to end against a stub: lsp gets the editor's bytes with and without preceding flags, scan --report-style short passes 60 lines through untouched, run still filters and still emits the handle, and a small search writes no archive at all.

cargo fmt --all, cargo clippy --all-targets and cargo test --all are green, rebased on current develop.

🤖 Generated with Claude Code

[rtk-wshm-sync-bot]

wshm · Automated triage by AI

📊 Automated PR Analysis

🐛 Type	bug-fix
🟡 Risk	medium

Summary

Fixes three regressions in the ast-grep command filter introduced by a prior PR: match lines dropped by the total cap were no longer counted or pointed to via a recovery hint, scan --report-style short output was incorrectly captured/capped, and ast-grep lsp was run with stdin closed (breaking editor integration). It introduces a two-pass argument tokenizer to correctly identify the run subcommand versus other subcommands, routing all non-run invocations (and run --stdin) through a st…

Review Checklist

• Tests present
• Breaking change
• Docs updated

Linked issues: #3979, #3857

---

Analyzed automatically by wshm · This is an automated analysis, not a human review.

[pszymkowiak]

Reviewed by building the branch and running both binaries against the module's fixture (387 match lines, 32 files) through a stub ast-grep, plus real ast-grep 0.45.3 on the worktree.

Verified

• Accounting: develop shows 50 lines + per-file notes for 82, trailer names 20 files and zero lines, no archive: 255 lines exist nowhere. This PR: same 50 + 82, trailer … 255 more match line(s) in 20 more file(s) not shown, [full output: …] whose archive is byte-identical to the raw (cmp). 50 + 82 + 255 = 387, 12 + 20 = 32, from the actual output. Real ast-grep: 50 + 30 + 118 = 198 over 34 files, rtk recall content matches a fresh run.
• scan --report-style short: 60 lines capped to 51 on develop, 60 through untouched on this PR.
• stdin: develop runs everything through exec_capture (Stdio::null()); this PR routes non-run shapes to run_passthrough with inherited stdin. lsp, -c cfg lsp, -p X --stdin, outline --stdin all receive the piped bytes now.
• Near-band case (6 lines, compaction cannot pay for its hint) prints raw and writes no archive; 60-line case writes exactly one. Exit 3 and stderr propagate on both paths. Fixture savings 84 % with the hint included.

Follow-ups, none blocking

1. OTHER_SUBCOMMANDS (ast_grep_cmd.rs:38) was not checked against the binary: docs is not an ast-grep subcommand (0.45.3: "unrecognized subcommand 'docs'"), while outline and help are missing. Cost is compression only: rtk ast-grep -p X docs (a directory named docs) goes unfiltered, 387 lines instead of 60. outline without --stdin is captured and rescued by unparsed_signal. FEATURES.md documents the phantom entry too.
2. global_takes_value lists --color as a pre-subcommand global option; per ast-grep --help only -c/--config, -h, -V are. Harmless (moves toward passthrough).
3. The description says the store decision is "taken on the text the hint is already part of"; the code reserves a fixed HINT_RESERVE = 256 B before the hint exists. Fine in practice (hints are 40–70 B), but a custom tee_directory over ~230 chars plus a search within 256 B of the never_worse band could still orphan an archive. Either check the rendered hint length or make the description match.
4. ast_grep_cmd.rs:266-282: the same comment paragraph appears twice.

Approving. CI green on all three OSes.