refactor(tokenizer): make Dialect a struct of independent grammar axes

[KuSh]

arg_tokenizer::Dialect welded four independent grammar axes into two variants. Measured against the real binaries, five tools want five different combinations:

tool	single-dash multi-char	attach separator	-- semantics	case	/flag
git, cargo, rg, golangci-lint	clusters char by char	=	ends option parsing	sensitive	no
dotnet	atomic	= or :	forwards the tail	folded	yes
maven	atomic	=	ends option parsing	sensitive	no
gradle	clusters	=	ends global option parsing; tasks and their options keep parsing	sensitive	no
go	atomic	=	ends option parsing	sensitive	no (and --flag ≡ -flag)

Because the axes were fused, three callers compensated in their own filter instead of declaring their grammar.

The axes

Dialect is now a Copy struct of five fields. Each earned its place from a row above that no existing variant could express:

axis	values	evidence
single_dash	Cluster / Atomic / AtomicAliasingLong	mvn -Bo validate errors while mvn -B -o validate succeeds, and -pl could not exist if single-dash clustered; gradle -qi tasks does cluster; go's -run is also --run
attach	Equals / EqualsOrColon	--logger:trx and --logger=trx are both valid dotnet syntax; nobody else takes :
dash_dash	EndsOptions / Forwards / EndsGlobalOptions	git and commons-cli stop; dotnet forwards to the test runner; gradle ends only the global region
name_case	Sensitive / Folded	MSBuild folds (/nologo == /NoLogo); commons-cli distinguishes -b from -B
slash_flags	bool	/bl:x.binlog is a switch for MSBuild and a path everywhere else

find's -exec cmd ; is deliberately out of scope: a variadic value ended by a sentinel is an Attachment variant, not a Dialect axis.

The presets

A preset names a grammar family several tools can share — a parser library, or a real convention — so the name answers "can my tool reuse this?". A single tool's bespoke parser gets no preset; its caller composes the axes at its own call site. That rule is what stops this list growing one variant per tool, and it is written down in src/core/README.md.

Dialect::Posix and Dialect::Msbuild stay, as associated consts, so every existing call site compiles and behaves unchanged.

• Dialect::Posix — the GNU/POSIX convention: Cluster, =, EndsOptions, sensitive, no /flag. git, cargo, rg, golangci-lint.
• Dialect::Msbuild — Atomic, =/:, Forwards, folded, /flag. dotnet.
• Dialect::CommonsCli — Apache commons-cli: Posix with Atomic. maven:3-eclipse-temurin-21 ships commons-cli-1.11.0.jar, and the whole-word short options (-pl, -am, -gs, -emp) are the library's property, not Maven's — which is why mvn -Bo validate errors while -B -o succeeds. Maven is the first consumer, not the definition.
• Dialect::GoFlag — Go's stdlib flag package: Posix with AtomicAliasingLong.

Gradle gets no preset. Its parser is its own org.gradle.cli (gradle:jdk21 ships gradle-cli-*.jar and zero commons-cli), used by nothing else. It is Posix with dash_dash: EndsGlobalOptions and nothing more — gradle clusters (-qi tasks works) and its value-taking shorts are the ValueSpec::solo_only() shape git's -n already uses (-qp /w fails, -p /w works) — so gradlew_cmd.rs composes that const at its own call site. The EndsGlobalOptions axis value stays: it is shared infrastructure, and only the one-tool preset was unjustified.

The presets with no in-tree caller yet carry #[allow(dead_code)] until mvn and go migrate, as does EndsGlobalOptions until gradlew does.

Two contracts a migrating caller would otherwise trip over are documented and pinned by tests: SingleDash::Atomic tags its flags TokenKind::Long (a predicate keyed on Short — the word mvn's own docs use for -pl — never fires), and Dialect::CommonsCli does not model commons-cli's Java-property options, so -DskipTests=true is the flag DskipTests, not D with a value. It stays a non-positional either way, which is all goal detection needs.

Differential evidence

src/core/arg_tokenizer/frozen.rs is the pre-axes implementation, kept verbatim as an oracle. differential.rs generates 88,740 arg vectors — every combination of length 1..=4 over a 17-token alphabet covering clusters, attached values, =/: forms, -- in every position, /flag against /path, digit runs (-20), a bare -, empty strings and non-ASCII in both a cluster and a long name — crosses each with 3 value grammars (nothing takes a value, a mixed table exercising every Attachment and claims_dash_dash, everything takes a value), and asserts the new Posix and Msbuild presets are token-for-token identical to the frozen implementation on all seven Token fields, plus has_flag / has_double_dash_flag / double_dash_flag_value over 8 lookup names (the only way the case axis is observable). 266,220 comparisons per preset; all pass.

Mutation-checked: flipping any one of the five axes on either preset fails the test.

Runtime check: rtk git log -10, log --oneline -20, log -n 2 --stat, log --grep -p, log -20 -- src/core, branch -a and diff HEAD~3 -- src produce byte-identical output from binaries built before and after.

cargo test --all is green with zero changes to git.rs, search.rs, dotnet_cmd.rs or golangci_cmd.rs — the diff touches only arg_tokenizer and src/core/README.md. The documented -20 digit-run asymmetry (guarded under clustering, absent under atomic) is preserved exactly; it now falls out of the single_dash axis rather than being a dialect special case.

hyperfine --warmup 5 -N 'rtk git log -10': 6.4 ms ± 0.4 before, 6.2 ms ± 0.3 after.

Also fixes Dialect's doc comment, which linked tokenize_dialect — a function folded into tokenize_grammar before merge.

Axis independence, enforced

The /flag-vs-path guard used to split on a hard-coded ['=', ':']. That was correct only while slash_flags and attach were welded together in one variant; with them independent, a slash_flags dialect attaching on = alone would hide the second / in /opt:a/b and emit Long("opt:a/b") instead of a positional. The guard now derives its separator from dialect.attach via split_attached, and a test covers it (it fails against the hard-coded form).

[rtk-wshm-sync-bot]

wshm · Automated triage by AI

📊 Automated PR Analysis

♻️ Type	refactor
🟢 Risk	low

Summary

Refactors arg_tokenizer's Dialect from a two-variant enum into a Copy struct of five independent grammar axes (single-dash behavior, attach separator, dash-dash semantics, name case, slash flags), preserving Posix and Msbuild as presets and adding unused Maven/Gradle/GoFlag presets for future callers. Validated with an 88,740-vector differential test against a frozen pre-refactor oracle, mutation testing, runtime output comparisons, and benchmarks showing no perf regression.

Review Checklist

• Tests present
• Breaking change
• Docs updated

---

Analyzed automatically by wshm · This is an automated analysis, not a human review.

[aeppling]

lgtm, Land after #4460 , develop is red right now and this branch inherits the failure on merge.