feat(bench): generate a shadow CLI from a spec, and compile mise's by jdx · Pull Request #824 · jdx/usage

jdx · 2026-08-12T02:28:00Z

Stacked on #823. This is the first half of the gate — the thing the whole experiment is supposed to be judged by.

xtask gen-shadow <spec.kdl> <dir> reads a usage spec and writes a crate of derived types declaring the same CLI. mise's committed 5,592-line spec compiles: 211 commands, 711 flags, 128 arguments, four levels deep, in 2.6 seconds.

What it drops, and says so

dropped, because the derive cannot express it yet:
  `double_dash = "automatic"` on an argument: 3
  a default on a flag that collects values: 1
  a flag's second long or short form: 13

Seventeen out of roughly a thousand. The generator names each class and counts it, because a silent cap reads as "all of it was expressible".

A first measurement

The gate parses one command line and exits, beside a null binary that does everything except parse. Subtracting the second from the first is what turns a process measurement into a parser measurement — about two thirds of a small binary's instructions are the dynamic loader and libc starting up, and neither parser is responsible for those.

invocation	instructions to parse
`mise`	99,281
`mise use -g node@20`	104,720
`mise settings set experimental true`	104,650
`mise -vv ls --installed`	105,754
`mise tasks run build -- --verbose`	104,650

Near constant regardless of depth, which is what static tables should look like: there is no tree to build. The clap side is the next PR — until it lands, this number stands on its own rather than as a ratio, and the ~3.1M figure from the original analysis is not something I have reproduced here.

Two things worth review

The generated crate is excluded from the workspace. A shadow of mise trips large_enum_variant on the commands with thirty flags, which the real mise answers by boxing its variants — something the derive cannot express yet, so it's on the roadmap rather than papered over with an allow. The gate depends on the shadow by path so it is still built, and the smoke tests live in the gate crate, where they get linted like anything else.
The shadow is checked in, not built by a build.rs. The compile-time comparison wants a fixed input, and a generated file in the tree is a diff a reviewer can read when the derive's vocabulary changes. CI runs mise run gen-shadow and fails on a diff, the same way it does for render.

Along the way, a note on fidelity: mise's spec declares no positionals on the top-level run, and that is deliberate — src/cli/usage.rs clears them (run.args = vec![]) and adds a mount of mise tasks --usage plus restart_token = ":::", so the project's real tasks are what complete after run rather than a static [TASK]. The shadow can carry neither the mount nor the restart token yet, so its run is slightly cheaper to parse than real mise's would be. The smoke tests use tasks run for the separator shape and the root's own [TASK] for a bare task.

(I first suspected the clap bridge was dropping them, since Run(Box<run::Run>) is the only boxed variant of the commands involved and the same struct registered unboxed under tasks run kept all three positionals. I tested that against clap 4 directly: boxing loses nothing, in clap or in the bridge. Recording it here because it is a tempting wrong answer.)

AI-assisted — Tool: Claude Code; model: anthropic/claude-opus-5; version: unavailable.

Note

Medium Risk
Large generated shadow and new workspace/CI wiring for the gate; not security-critical, but the shadow exercises the derive at full mise scale and CI will fail on generator drift.

Overview
Adds the first half of the performance gate: generate a compileable shadow of a real CLI from its usage spec, then measure parsing at mise's full scale.

xtask gen-shadow turns any .usage.kdl into a crate of derived types. mise's checked-in 5,592-line spec compiles (211 commands, 711 flags, 128 args). Features the derive cannot express yet are counted and reported rather than silently dropped. The generated shadow-mise crate is committed and excluded from the workspace (it trips large_enum_variant); the gate depends on it by path.

The new gate crate provides one-shot parse binaries (parse-usage vs a null parse-none baseline) plus smoke tests for real mise invocations — including the [ARGS]… / [-- ARGS_LAST]… separator shape and nested/global-flag cases.

CI now runs mise r gen-shadow and fails if the checked-in shadow drifts, same as render.

^{Reviewed by Cursor Bugbot for commit 4470be0. Bugbot is set up for automated code reviews on this repo. Configure here.}

coderabbitai · 2026-08-12T02:28:11Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: ff311f68-7b72-4143-aa15-5fd333e5c6d2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

greptile-apps · 2026-08-12T02:30:22Z

Greptile Summary

The PR adds an xtask generator that converts a usage specification into a checked-in shadow CLI crate and introduces a gate harness for measuring its parser.

Generates derived command, flag, and argument types from mise’s committed specification.
Explicitly reports unsupported grammar properties rather than silently omitting them.
Adds parsing smoke tests and a CI freshness check for the generated shadow.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the three previously reported fidelity issues are now fixed or explicitly accounted for in the generated-shadow report.

Important Files Changed

Filename	Overview
xtask/src/shadow.rs	Implements shadow-crate generation, unsupported-property accounting, required-subcommand preservation, identifier sanitization, and focused generator tests without a remaining blocking issue.
benches/gate/tests/parse.rs	Exercises representative mise invocations, including separator binding, globals, nested routing, counting flags, and required-subcommand rejection.
benches/shadows/mise/src/lib.rs	Contains the checked-in generated shadow for mise and reflects the corrected required-subcommand field shapes.
.github/workflows/test.yml	Regenerates the shadow in CI and rejects uncommitted generated-file changes alongside existing rendered artifacts.
Cargo.toml	Adds the xtask and gate crates to the workspace while intentionally excluding the generated shadows directory.
PLAN.md	Documents the shadow-generation milestone, measured parser cost, and explicitly known fidelity gaps.

_{Reviews (6): Last reviewed commit: "fix(bench): strip only the short form, n..." | Re-trigger Greptile}

github-actions · 2026-08-12T02:34:56Z

Instruction counts

benchmark	trend	instructions	Δ	wall (min)	Δ
markdown	`▁▁▁▁▁▁▁█████████`	148,242,482 → 148,255,805	+0.01%	13.93 → 14.23ms	+2.18%
startup	`▄▄▄▄▄▄▄█████▁▁▁▅`	1,199,527 → 1,201,983	+0.20%	0.95 → 0.95ms	+0.50%

No instruction-count regression above 1%.

_{Only instruction counts gate. Wall clock is shown for context — on identical hardware it moves 4-20% run to run.}

_{Measured by tak — instruction-counted CLI benchmarks, stored in this repository's git notes.}

_{4470be0e264b vs 7047ae1232b2 · measured on the runner, not pushed to the history.}

jdx · 2026-08-12T10:31:47Z

Both findings were right, and the alias one matters more than its severity suggests.

Aliases dropped without a count. The filter itself is correct — an alias and the name it points at share a map entry, so generating both would declare the same command twice — but not counting it made the report say 17 losses where there were 108: 67 visible aliases, 24 hidden ones, plus 2 mounts and 2 restart tokens. The generator's whole job is to say what could not be expressed, so that was the one bug it could not afford. Aliases are now the largest known gap in the derive rather than an invisible one, and PLAN.md records them as the next thing mise needs.

subcommand_required ignored. Every command got an Option<Commands>, so the 27 of mise's commands that require a subcommand accepted invocations the real CLI refuses. Fixed, with a test on bootstrap accounts — which requires one while bootstrap itself does not, so reading the spec is the only way to get it right.

I also corrected the note at the end of the description. I had said mise's missing positionals on the top-level run "may be a loss in mise's own clap→spec dump rather than intent" — it is intent: src/cli/usage.rs clears them and adds a mount of mise tasks --usage so real task names complete after run. I had a plausible wrong theory (Run(Box<run::Run>) is the only boxed variant, and the same struct registered unboxed under tasks run kept its positionals) and tested it against clap 4 rather than reporting it: boxing loses nothing.

AI-assisted — Tool: Claude Code; model: anthropic/claude-opus-5; version: unavailable.

`xtask gen-shadow <spec.kdl> <dir>` reads a usage spec and writes a crate whose derived types declare the same commands, flags and arguments. Comparing this parser against clap at a real CLI's scale needs the same CLI expressed both ways, and mise's 210 commands are not something to transcribe by hand. mise's committed 5,592-line spec compiles: 211 commands, 711 flags, 128 arguments, four levels deep, in 2.6 seconds. Seventeen things are dropped and the generator names each — 13 secondary flag aliases, 3 `double_dash="automatic"`, and 1 default on a collecting flag. A silent cap would read as "all of it was expressible". The gate crate parses one command line and exits, beside a null binary that does everything except parse. Subtracting the second from the first is what turns a process measurement into a parser measurement: about two thirds of a small binary's instructions are the loader and libc starting up, which neither parser is responsible for. At mise's full scale the parse costs 100k–106k instructions, near constant across invocation shapes — which is what static tables should look like, with no tree to build. The clap side of the comparison is the next PR; until it lands this number stands on its own. The generated crate is excluded from the workspace: a shadow of mise trips `large_enum_variant` on the commands with thirty flags, which the real mise answers by boxing its variants — something the derive cannot express yet. The gate depends on it by path, so it is still built, and its smoke tests live in the gate, where they are linted like anything else. The generator runs rustfmt on its own output, since the file is checked in and `cargo fmt --check` sees it.

Two ways the shadow was answering a different grammar from the spec it came from. Command aliases were filtered out of the subcommand map — correctly, since an alias and the name it points at share an entry and generating both would declare the same command twice — but never counted. That is 67 visible and 24 hidden aliases, along with 2 mounts and 2 restart tokens, silently absent from a report whose whole job is to say what could not be expressed. It claimed 17 losses where there were 108, and aliases are now the largest known gap rather than an invisible one. And `subcommand_required` was ignored: every command got an `Option<Commands>`, so the 27 of mise's commands that require a subcommand accepted an invocation the real CLI refuses. The field now follows the spec, with a test on `bootstrap accounts` — which requires one, while `bootstrap` itself does not, so reading the spec is the only way to get this right.

The spec-level properties were never looked at, so the one that changes the root's own grammar went unrecorded: mise sets `default_subcommand run`, which routes `mise build` through `run` there while the shadow answers it at the root's `[TASK]`. Counted now, along with a root mount and restart token, and the test that leans on the root's `[TASK]` says why it is the root.

Two smoke tests were weaker than they read. The one covering `[ARGS]…` / `[-- ARGS_LAST]…` — the shape that made the derive's validation wrong — only checked that `tasks run` had been selected, so a regression merging or dropping the words around the `--` would have passed. It now names each word on each side, and a second positional so a merge is visible. The global-flag test only checked that `ls` was selected. That one could not fail: unknown flag-like words are values by default and `ls` has a variadic positional, so a global that stopped being recognized after the command would still parse, with `-C` and `/tmp` landing quietly in the variadic. It asserts the bound value now. Arguments also carry `env` and `help_heading`, both of which reach `ArgMeta`, and the generator was writing neither. mise's spec puts `env` only on flags, so nothing in the generated shadow changes and nothing would have caught it — which is why `render` is now separate from `generate` and has tests of its own, on a spec written for the purpose.

jdx · 2026-08-12T11:07:10Z

Three of the four were real, and two were tests of mine that could not fail.

The separator test only checked that tasks run had been selected, while claiming to cover the [ARGS]… / [-- ARGS_LAST]… shape that made the derive's validation wrong. It now names each word on each side of the --, plus a second positional so a merge would be visible.

The global-flag test was worse: it asserted only that ls was selected. That could not fail — unknown flag-like words are values by default and ls has a variadic positional, so a global that stopped being recognized after the command would still parse, with -C and /tmp landing quietly in the variadic. It asserts the bound value now.

Arg-level env and help_heading were being dropped with no count, and both reach ArgMeta. Fixed — and since mise's spec puts env only on flags, the regenerated shadow doesn't change and nothing in it would have caught this. So render is now separate from generate and has tests of its own on specs written for the purpose, including one that checks both dialects make the same claims about the same flag.

default_subcommand was already fixed in the commit after the one reviewed.

AI-assisted — Tool: Claude Code; model: anthropic/claude-opus-5; version: unavailable.

Two more ways the shadow said something other than the spec did. A spec's long help usually opens with its short help, and the derive reads a doc comment the same way — first paragraph short, whole comment long. Writing both in full repeated that opening paragraph on nearly every command mise has: 467 lines of it in the generated file. Only the remainder is written now. And a repeatable flag can be bounded on the flag as well as on its argument, both of which usage-lib enforces against the values collected. Reading only the argument's dropped occurrence limits with no count. The flag's own now win, and a second differing pair is counted. mise's spec declares neither a flag-level bound nor an argument `env`, so nothing in the generated file would have caught either — both have tests on specs written for the purpose.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 061ac25. Configure here.}

Taking the long help's remainder with a blunt `trim` broke two shapes mise's help really has, both of them visible in the regenerated shadow. A long form whose first sentence ends in a period the short form leaves off stranded that period on a doc line of its own. And where the remainder opened with an indented example, `trim` took the indentation with the blank line. Only the orphaned period and the separating newlines go now. An indented example keeps its indentation, because spaces are not in the set. A long form that adds nothing but a period says nothing at all. Both have tests, and I checked both fail with the blunt version restored — the first attempt at the indentation test did not, since it put a line between the short form and the example, which is exactly where `trim` cannot reach.

jdx · 2026-08-12T12:26:03Z

Good catch, and it was a regression from my own fix in the previous round — both shapes were visible in the regenerated shadow, exactly as described.

--security's long help ends its first sentence with a period the short form leaves off, so stripping the short form left . behind and trim could not remove it from the middle: it became /// . on a line of its own. And where the remainder opened with an indented example, trim took the indentation along with the blank line — mise activate --shims's PATH="$HOME/…" lost its four spaces.

Now only the orphaned period and the separating newlines go, since spaces are not in the set, and a long form that adds nothing but a period says nothing at all.

Both have tests, and the useful part: my first attempt at the indentation test did not fail when I restored the blunt version, because it put a line between the short form and the example — which is the one place trim cannot reach. Moved the example to the front of the remainder, and now both assertions fail without the fix.

AI-assisted — Tool: Claude Code; model: anthropic/claude-opus-5; version: unavailable.

Stacked on #824. This is the gate — the measurement the experiment was supposed to be judged by. ## The result At mise's full scale (211 commands, 711 flags, 128 arguments, four levels), parsing `mise use -g node@20`: | | usage | clap | ratio | | --- | --- | --- | --- | | instructions, cold parse | **50,885** | **5,265,233** | **103×** | | instructions, warm parse | 47,223 | 5,351,127 | 113× | | wall clock | **2.1µs** | **440µs** | **214×** | clap's 440µs decomposes into **283µs building its command tree**, ~135µs validating it, and **~23µs actually parsing** — so even against clap's parse alone, with the tree already built and paid for, this is 11× faster. The tree is the thing this project exists to delete, and it is 64% of clap's cost. ## How it is measured, and why that way `gen-shadow … clap` emits the same CLI in clap's vocabulary from the *same spec and the same traversal* as the usage shadow, so this compares two parsers rather than two transcriptions. Both sides drop the same properties (aliases, mounts, `default_subcommand`, second long forms), and the generator counts them. Three release binaries — one per parser, plus one that does everything **except** parse. The third is what makes the numbers mean anything: roughly two thirds of a small binary's instructions are the dynamic loader and libc starting up, and neither parser is responsible for those. Whole processes rather than a criterion loop with the tree hoisted out, because clap builds its command tree on the way to parsing and that is precisely the cost in question. `tak` gates all three in CI, so a regression on either side shows up as a diff in the series. ## Methodology, and a correction The first version of this PR reported 48–58× and ~110µs. Both were wrong in the same way: they subtracted a baseline measured in a **separate** no-op binary. Two binaries do measurably different amounts of setup before `main`, and that difference was landing in whatever I attributed to parsing — it cost usage a factor of two and a half. `parse-n` and `parse-n-clap` take a repeat count from the environment, so N=1 minus N=0 is a cold parse in a fresh process and N=2 minus N=1 is a warm one, with the binary held fixed. `time-parse` times the parse in-process, which is what the 2.1µs figure is. I also checked whether the instruction count was really relocation work, since 211 commands' worth of statics hold thousands of pointers between them: **linking the tables costs nothing measurable**. A binary that reads one static and never parses matches one with no tables at all. Both gating targets are met — under 100k instructions, under 50µs — and not narrowly. The one number still owed is **allocations on the derive path**: usage-argv's own are asserted at zero, but the derive allocates a `String` per value and nothing counts them yet. One fidelity note, since it cuts *against* this parser rather than for it: neither shadow carries mise's mounts or `default_subcommand`, so both are slightly cheaper than the real thing. And clap keeps one default on a collecting flag that the usage derive refuses, so clap answers a hair more grammar there. *AI-assisted — Tool: Claude Code; model: anthropic/claude-opus-5; version: unavailable.*  --- > [!NOTE] > **Low Risk** > Benchmark harness and plan updates only; no production parser, auth, or data-path changes. > > **Overview** > Completes the performance gate by adding a **clap-equivalent shadow** of mise and binaries that measure both parsers the same way. > > `gen-shadow … clap` already emits `shadow-mise-clap` from the same spec/traversal as the usage shadow. This wires it into `gate` and adds `parse-clap`, `parse-n` / `parse-n-clap`, and `time-parse` so cold/warm instruction counts and wall time can be differenced within one binary (avoiding the earlier separate no-op baseline error). > > **PLAN.md** records the result: ~51k vs ~6M instructions (~117×) and ~2µs vs ~500µs wall clock, meeting both gating targets. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 4564106. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>

greptile-apps Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread xtask/src/shadow.rs Outdated

Comment thread xtask/src/shadow.rs