feat(core): add arg_tokenizer and migrate git/grep/dotnet/golangci arg parsing onto it

[KuSh]

What this adds

src/core/arg_tokenizer.rs — one place where RTK turns an argv slice into typed tokens, replacing the ad hoc starts_with('-') scans and find("--") walks that each command module had grown its own copy of.

A Token carries its kind (Long, Short, Positional, DashDash), its flag name, its attached value (--flag=v), a link to its separate-token value (--flag v), and its index in the original argv, so a caller can always rebuild the exact arg a token came from.

The tokenizer never guesses which flags take a value. The caller supplies the grammar as a takes_value(kind, name) -> Option<ValueSpec> predicate scoped to that tool and that subcommand, and everything else follows from it: a flag's value is linked to the flag instead of counted as a positional, -- is a real boundary, and a positional that looks like a flag stays a positional.

One table answers every question the tokenizer has about a flag, rather than one predicate per question — that shape is how two lists start drifting apart, which is a bug class this PR fixes three times elsewhere.

• ValueSpec::value() — --flag=v or --flag v, and a literal -- is the boundary. The common case.
• ValueSpec::attached_only() — --flag=v only; the next argument is never the value (git's -M/-U/-C/-B take an optional attached number and nothing else).
• ValueSpec::solo_only() — a Short flag takes a separate value only when it is the whole argument: git log -n 2 does, git log -pn 2 does not.
• .claiming_dash_dash() — lets a literal -- be this flag's value. A per-tool split, not per-flag: grep and rg let any value-taking flag swallow it, git and cargo reject it whichever flag is asking.

Two entry points, split by whether the caller has a grammar at all: tokenize(args) for the callers asking only which arguments are flags and where -- is, and tokenize_grammar(args, takes_value, dialect) for the rest. Two dialects — Posix (-abc clusters, --flag) and Msbuild (/flag:value, for dotnet).

Helpers on top: before_dashdash, dashdash_index, injection_point, is_digit_run, Token::value, Token::is_free_positional.

What it changes

git.rs, search.rs (grep/rg), dotnet_cmd.rs and golangci_cmd.rs are migrated onto it. Every fix below has a regression test that fails without it, and every claim about a tool's own behaviour was checked against that tool (git 2.53, GNU grep 3.12, ripgrep 15.2, .NET SDK 9 and golangci-lint 2.13.1 via Docker) rather than assumed.

A -- boundary is a boundary. restore_double_dash now runs once at git's dispatcher instead of per-subcommand, and every flag lookup that should stop at -- does. A pathspec, worktree path or project file spelled like a flag is no longer parsed as one.

A flag's value is not a positional. dotnet test --filter X no longer reads X as a project path; git branch -u origin/main links its upstream; rg -r X TODO f no longer loses the pattern to --replace; grep -f patterns.txt reads its pattern file instead of searching for it.

One grammar per tool, and per subcommand. grep and rg get separate value-taking tables transcribed from each engine's own --help, because the overlap is smaller than it looks: -r is --recursive in grep and --replace in rg, -h is --no-filename in grep and --help in rg, -I is the reverse, and -N/-T/-z/-L/-E all diverge. Within git, log and diff likewise get their own: git log -cl 2 is an error while git diff -wl 100 is valid, and -u means -p in git log but --include-untracked in git stash show.

Output-shape detection reads tokens, not reconstructed strings. git log -U<n>, --cc, -c and --remerge-diff request a patch and now pass through raw instead of being compacted into [+N lines omitted]; -W does not request one and no longer dumps the unfiltered history. --check, --exit-code and --quiet keep their reports instead of being thrown away by RTK's own stat probe. git worktree repair reaches git instead of being answered with a worktree listing, and worktree list --porcelain passes through instead of being replaced.

Injection checks recognise every spelling of what they look for. dotnet --report-trx:true, /l:trx and -l trx all count as "already present", so RTK stops double-injecting a logger; /flag:value MSBuild switches can no longer swallow the next token as a value; golangci's output-flag check covers the full set of sink flags and is scoped to run-level ones.

Design rule

The rule the migration kept running into is documented in src/core/README.md: one grammar per tool and per subcommand — never reuse a sibling's predicate because it looks close enough. .claude/rules/rust-patterns.md gains a matching rule 6, so new commands, new flags and fixes to arg handling go through the tokenizer rather than adding another bespoke scan.

What is not in this PR

• Modules still on raw string scans: src/cmds/system/ls.rs, src/cmds/rust/cargo_cmd.rs, src/cmds/go/go_cmd.rs, src/cmds/python/ruff_cmd.rs, src/discover/registry.rs. Same bug class, pre-existing, each its own follow-up. go_cmd.rs needs an OsString conversion first, since the tokenizer is deliberately AsRef<str>-only.
• ValueSpec carries no semantic tag for what a value means — only how it attaches. A kind field (Path, or a MaybePath that asks for a filesystem check) drops in as one more field when a consumer appears, which is the headroom the struct was shaped for. Nothing reads such a tag today: every path-ish use in these four modules is keyed on a role — golangci's output sinks, search's pattern file, dotnet's results directory — and a generic Path tag would be strictly less precise than the name-keyed checks it would replace.
• Dialect::Msbuild folds ASCII case for every flag name, but its only consumer mixes two grammars: MSBuild switches really are case-insensitive, dotnet's own options are not. A wrong-case spelling is rejected by dotnet either way, so the only cost is RTK suppressing an injection for a command that will fail regardless.
• Token exposes the prefix as two bools (double_dash, slash) rather than the three-state enum it holds internally, so consumers spell "single dash" as a double negative.
• Some modules tokenize twice per invocation. Known waste, not a regression — startup is 3.8 ms release.
• MSBuild single-segment absolute paths (/app, /tmp) are structurally indistinguishable from a switch name. Real MSBuild resolves this with a filesystem stat(); this tokenizer is a pure function and does not.
• dotnet_takes_value's allowlist is subcommand-unaware and is what explicit_projects relies on: a value-taking flag missing from it whose value ends in .csproj would be read as an explicit project path.
• --diff-merges' routing is static. log sends it to the raw passthrough on the flag alone, because a predicate cannot know whether the revision walk contains a merge; on a merge-free walk that costs compaction for a flag which provably changes git's output by zero bytes. The alternative — compacting a combined diff whose two marker columns compact_diff misreads — is silent corruption rather than verbosity, so this is the deliberate side to err on.
• git show repeats its summary line under --oneline, and once per parent under --diff-merges' single-column formats, because show_cmd puts RTK's flags ahead of the user's and git repeats the header per parent even under --no-patch. Both are pre-existing and byte-identical on the merge-base; both are cosmetic against a compacted body.

Issues

• Fixes git worktree repair is routed to list mode: prints the worktree list, exits 0, repairs nothing #3710 — git worktree repair is routed to list mode. Verified: the stale gitdir pointer is now actually repaired.
• Refs "git log -p" omits the patch requested by the -p flag #2275 — bare git log -p already passed through before this branch, but -U<n>, --cc, -c and --remerge-diff were still being compacted into [+N lines omitted]. They now pass through too.
• Refs rtk grep -h prints help instead of searching — clap intercepts GNU grep's -h (--no-filename), exit 0, no native fallback #2391 / rtk grep claims -h, -l, -m and -t, which belong to grep/rg #3663 — the display side of -h/--no-filename (and rg's -I) is implemented here and works for every spelling clap does not intercept. The bare -h still never reaches the module; that needs disable_help_flag on the Grep/Rg variants, which is a separate UX decision.

Test plan

• cargo fmt --all --check, cargo clippy --all-targets (zero warnings), cargo test --all (3074 unit tests plus every integration suite)
• Every fix has a regression test verified to fail before it and pass after
• Differential testing against a binary built at the merge base, repeatedly, across ~90–430 git/grep/rg invocations per round, including fuzzed flag combinations compared byte-for-byte against the real engines
• Flag semantics verified against the real tools listed above, not from documentation or memory
• Focused fuzz campaign on the two ecosystems that needed a container to test: 1,890 argument vectors compared against a merge-base binary for what RTK forwards (every divergence is one of this PR's intended fixes), then real-tool matrices in Docker under LC_ALL=C — golangci-lint 2.13.1 on clean and issue-producing modules, .NET SDK 9 on passing and failing test projects, and .NET SDK 10 for MTP-native. Exit codes match the real tool in every case, including failures; no regressions found

[aeppling]

Hey @KuSh

Thanks for working on this which is a pain for commands rewirtting.
Some regressions found:

repo with a stash that includes an untracked file:

rtk git stash show -u
develop: f.txt 1 + / u.txt 1 + / 2 changed 2 + exit 0
PR: (empty) exit 0
git: 2-file stat

file f containing a -- b:

rtk grep -e -- f </dev/null
develop / grep: a -- b exit 0
PR: (empty) exit 1

directory with 4 files containing foo:

rtk rg -z foo </dev/null
develop / rg: f:foo one … (filenames)
PR: foo one … (filenames dropped)

[aeppling]

Also we may want to document this system in respective README.md

[KuSh]

@aeppling took all of this into account