feat(bench): measure against clap at mise's scale

[jdx]

Stacked on #824. This is the gate — the measurement the experiment was supposed to be judged by.

The result

At mise's full scale (211 commands, 711 flags, 128 arguments, four levels), parsing mise use -g node@20:

	usage	clap	ratio
instructions, cold parse	50,885	5,265,233	103×
instructions, warm parse	47,223	5,351,127	113×
wall clock	2.1µs	440µs	214×

clap's 440µs decomposes into 283µs building its command tree, ~135µs validating it, and ~23µs actually parsing — so even against clap's parse alone, with the tree already built and paid for, this is 11× faster. The tree is the thing this project exists to delete, and it is 64% of clap's cost.

How it is measured, and why that way

gen-shadow … clap emits the same CLI in clap's vocabulary from the same spec and the same traversal as the usage shadow, so this compares two parsers rather than two transcriptions. Both sides drop the same properties (aliases, mounts, default_subcommand, second long forms), and the generator counts them.

Three release binaries — one per parser, plus one that does everything except parse. The third is what makes the numbers mean anything: roughly two thirds of a small binary's instructions are the dynamic loader and libc starting up, and neither parser is responsible for those. Whole processes rather than a criterion loop with the tree hoisted out, because clap builds its command tree on the way to parsing and that is precisely the cost in question.

tak gates all three in CI, so a regression on either side shows up as a diff in the series.

Methodology, and a correction

The first version of this PR reported 48–58× and ~110µs. Both were wrong in the same way: they subtracted a baseline measured in a separate no-op binary. Two binaries do measurably different amounts of setup before main, and that difference was landing in whatever I attributed to parsing — it cost usage a factor of two and a half.

parse-n and parse-n-clap take a repeat count from the environment, so N=1 minus N=0 is a cold parse in a fresh process and N=2 minus N=1 is a warm one, with the binary held fixed. time-parse times the parse in-process, which is what the 2.1µs figure is.

I also checked whether the instruction count was really relocation work, since 211 commands' worth of statics hold thousands of pointers between them: linking the tables costs nothing measurable. A binary that reads one static and never parses matches one with no tables at all.

Both gating targets are met — under 100k instructions, under 50µs — and not narrowly. The one number still owed is allocations on the derive path: usage-argv's own are asserted at zero, but the derive allocates a String per value and nothing counts them yet.

One fidelity note, since it cuts against this parser rather than for it: neither shadow carries mise's mounts or default_subcommand, so both are slightly cheaper than the real thing. And clap keeps one default on a collecting flag that the usage derive refuses, so clap answers a hair more grammar there.

AI-assisted — Tool: Claude Code; model: anthropic/claude-opus-5; version: unavailable.

---

Note

Low Risk
Benchmark harness and plan updates only; no production parser, auth, or data-path changes.

Overview
Completes the performance gate by adding a clap-equivalent shadow of mise and binaries that measure both parsers the same way.

gen-shadow … clap already emits shadow-mise-clap from the same spec/traversal as the usage shadow. This wires it into gate and adds parse-clap, parse-n / parse-n-clap, and time-parse so cold/warm instruction counts and wall time can be differenced within one binary (avoiding the earlier separate no-op baseline error).

PLAN.md records the result: ~51k vs ~6M instructions (~117×) and ~2µs vs ~500µs wall clock, meeting both gating targets.

^{Reviewed by Cursor Bugbot for commit 4564106. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 4b7c9131-e6bc-4621-9509-66d6f7812e89

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

The PR adds a generated clap shadow of mise’s CLI and benchmark binaries for comparing clap with the compiled usage parser.

• Extends xtask gen-shadow with usage and clap output dialects.
• Adds instruction-count and in-process timing harnesses.
• Updates benchmark configuration, generated shadows, and the performance report in PLAN.md.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
xtask/src/shadow.rs	Adds shared traversal and dialect-specific generation for equivalent usage and clap shadow declarations.
benches/gate/src/bin/parse-n.rs	Adds an environment-controlled repeated-parse binary for same-process-image instruction differencing.
benches/gate/src/bin/parse-n-clap.rs	Adds the corresponding repeated-parse binary for the generated clap shadow.
benches/gate/src/bin/time-parse.rs	Adds in-process timing for usage parsing, clap tree construction, and clap parsing with a reused tree.
tak.toml	Adds recorded benchmark entries for the baseline, usage shadow, and clap shadow processes.
PLAN.md	Documents the benchmark methodology, corrected measurements, and remaining allocation measurement.

_{Reviews (5): Last reviewed commit: "chore(bench): regenerate the clap shadow..." | Re-trigger Greptile}

[jdx]

Correcting my own numbers, since they moved in my favour and that is exactly when to show the work.

The table first published here reported 48–58× and ~110µs of wall clock. Both were measured by subtracting a separate no-op binary from the parsing one. That looked equivalent and was not: two binaries do measurably different amounts of setup before main, and the difference had been landing in whatever I attributed to parsing.

Differencing two runs of the same binary over how many parses it performs holds everything else fixed:

	usage	clap	ratio
instructions, cold parse	50,885	5,265,233	103×
instructions, warm parse	47,223	5,351,127	113×
wall clock, in-process	2.1µs	440µs	214×

And clap's 440µs decomposes: 283µs building the command tree, ~135µs validating it, ~23µs parsing. Against clap's parse alone — tree already built — this is still 11× faster.

Two things I checked rather than assumed:

• Is the instruction count relocation work? 211 commands' worth of statics hold thousands of pointers between them, so a PIE binary could be paying at load time. It is not: a binary that reads one static and never parses matches one with no tables at all.
• Do 47k instructions and 2.1µs agree? They imply about 5 instructions per cycle, which is high but within what a wide modern core sustains on a tight, L1-resident, perfectly-predicted loop. The cold figure (50.9k, one parse in a fresh process) is the one a CLI actually pays.

AI-assisted — Tool: Claude Code; model: anthropic/claude-opus-5; version: unavailable.

[github-actions]

Instruction counts

benchmark	trend	instructions	Δ	wall (min)	Δ
markdown	▁▁▁▁▁▁▁█████████	148,242,482 → 148,265,798	+0.02%	13.93 → 13.71ms	-1.61%
startup	▄▄▄▄▄▄▄█████▁▁▁▅	1,199,527 → 1,201,965	+0.20%	0.95 → 0.94ms	-0.53%

No instruction-count regression above 1%.

New, nothing to compare against: gate-baseline on bamboo-v2-ubuntu24.04-x64-30vcpu-24gb-rust1.97.1, gate-clap on bamboo-v2-ubuntu24.04-x64-30vcpu-24gb-rust1.97.1, gate-usage on bamboo-v2-ubuntu24.04-x64-30vcpu-24gb-rust1.97.1

_{Only instruction counts gate. Wall clock is shown for context — on identical hardware it moves 4-20% run to run.}

_{Measured by tak — instruction-counted CLI benchmarks, stored in this repository's git notes.}

_{4564106901bb vs 7047ae1232b2 · measured on the runner, not pushed to the history.}