perf(go): generate cobra's shadow from the spec, so its row is a measurement

benches/go/cobra/go.mod

@@ -0,0 +1,15 @@
+// A module of its own, so that cobra is not a dependency of `github.com/jdx/usage/go`.
+//
+// That module has none, deliberately: an adopter's binary carries the tables and nothing
+// else. A benchmark that put cobra in its go.mod would be measuring the thing it is
+// comparing against while claiming to have no dependencies.
+module github.com/jdx/usage/benches/go/cobra
+
+go 1.24
+
+require github.com/spf13/cobra v1.10.1
+
+require (
+	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/spf13/pflag v1.0.9 // indirect
+)

benches/go/cobra/go.sum

@@ -0,0 +1,10 @@
+github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
+github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
+github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
+github.com/spf13/cobra v1.10.1 h1:lJeBwCfmrnXthfAupyUTzJ/J4Nc1RsHC/mSRU2dll/s=
+github.com/spf13/cobra v1.10.1/go.mod h1:7SmJGaTHFVBY0jW4NXGluQoLvhqFQM+6XSKD+P4XaB0=
+github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
+github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

go/README.md

@@ -23,16 +23,22 @@ Measured against a shadow of [mise](https://mise.jdx.dev)'s spec — 211 command
 | ----------------------- | -----------: | ------------------: | ------: |
 | a do-nothing Go process |            — |             0.95 ms | 2.31 MB |
 | **usage-go**, amortized |   **~1,600** |          **1.1 ms** | 2.37 MB |
-| cobra, one cold parse   |    2,008,880 |              1.8 ms | 3.87 MB |
+| cobra, amortized        |    3,249,052 |              2.0 ms | 3.41 MB |
 | urfave/cli v3, likewise |    5,591,321 |              1.7 ms | 5.74 MB |
 | kong, likewise          |   57,889,084 |              6.1 ms | 5.34 MB |
 
 Instruction counts are cachegrind, against mise's committed spec, on the argv the
 Rust shadows use so the two tables describe the same work. The column is labelled
-per row rather than once at the top, because the two kinds of figure are not the
-same measurement: the three frameworks are one cold parse, and usage-go's is
-amortized over a thousand binds — for the reason below, which is that a single one
-of ours cannot be measured at all.
+per row rather than once at the top, because the rows are not all the same
+measurement:
+
+- **usage-go**, amortized over 1,000 binds, because a single one of ours is below
+  the Go runtime's own startup jitter and cannot be measured at all — see below.
+- **cobra**, amortized over 20 resolves, each including the command tree it builds
+  on every process start. Twenty rather than a thousand because one of them is
+  three orders of magnitude dearer, and a thousand under cachegrind's 50x
+  slowdown would take minutes.
+- **urfave/cli v3 and kong**, one cold parse each, taken by hand.
 
 **usage-go's row is reproducible: `mise run perf:go`.** That harness
 ([`tasks/perf-go.sh`](../tasks/perf-go.sh)) reports the bind amortized over 1,000
@@ -42,11 +48,26 @@ jitter_, ±50,000 instructions run to run, which is thirty times the whole bind.
 Differencing `PARSE_N=1` against `PARSE_N=0` the way the Rust harness does gives a
 number here that changes sign between runs.
 
-The three framework rows are not reproducible yet: they were measured by hand
-against programs that are not in the repository. Generating those shadows from the
-same spec — as `xtask shadow` does for clap, argh and bpaf — is the next piece of
-this, and until it lands those numbers should be read as an order of magnitude
-rather than a measurement.
+cobra's row is reproducible too, and by the same command. `xtask gen-shadow
+benches/mise.usage.kdl benches/go/cobra cobra` writes mise's CLI out as a cobra
+program — 211 commands, each with its own flag set — which is checked in under
+[`benches/go/cobra`](../benches/go/cobra) and measured beside usage-go. So the two
+rows describe the same CLI rather than two people's transcriptions of it.
+
+Its figure includes building the command tree, because that is what cobra does on
+every process start. Hoisting that out of the loop would measure its parser
+against a program that had already paid for its model, which no CLI gets to do.
+
+That measurement replaced a hand-taken one of 2,008,880, which was lower because
+the program it was taken against was written by hand and smaller than mise: the
+generated one declares every command and flag the spec has. What cobra cannot
+express is printed when the shadow is generated rather than passed over — 128
+positionals, since cobra validates a count and not a name, 17 hidden aliases, 13
+second long forms, and one short-only flag.
+
+urfave/cli v3's and kong's rows are still hand-measured against programs that are
+not in the repository, and until they are generated the same way those two numbers
+should be read as an order of magnitude rather than as a measurement.
 
 Two things are worth reading off that table honestly. The win against cobra is
 real — about 40% of process startup — but it is bounded: 0.95 ms of usage-go's
@@ -274,7 +295,7 @@ claim is measured at real scale rather than against a fixture with four flags:
 - **A typed front door.** The conversions exist; what is missing is generated
   code that calls them, so a CLI author gets a struct rather than events.
 - **Shadow programs for the other frameworks.** usage-go's own numbers are
-  reproducible with `mise run perf:go`; cobra's, urfave's and kong's are still
+  reproducible with `mise run perf:go`; urfave's and kong's are still
   hand-measured, because generating mise-sized programs for them from the spec is
   its own piece of work.
 - **Running a spec's `complete` scripts.** A `run=` block shells out, which this

mise.toml

@@ -102,19 +102,30 @@ fi
 cargo semver-checks check-release -p usage-lib
 '''
 [tasks."lint:go"]
-dir = 'go'
+dir = "{{config_root}}"
 # `gofmt -l` prints what it would change and exits 0 either way, so the output is
 # the failure condition.
+#
+# Both modules. `benches/go/cobra` is generated, and a generated file that gofmt would
+# reformat is one the generator is not finished with.
+#
+# The cobra module is skipped only when its dependency cannot be *fetched* — asked as its own
+# question, because `go vet ./... || true` would swallow a generator regression as well: a
+# shadow that no longer compiles would pass the lint and then report itself unmeasured.
 run = [
-    'test -z "$(gofmt -l .)" || { gofmt -l .; echo "gofmt would change these; run mise run lint-fix"; exit 1; }',
-    'go vet ./...',
+    # The status as well as the output: a file gofmt cannot *parse* is reported on stderr with
+    # nothing on stdout, so `test -z "$(gofmt -l …)"` called that a pass — the one case where
+    # the answer matters most.
+    'list=$(gofmt -l go benches/go) || { echo "gofmt could not parse a file; see above"; exit 1; }; test -z "$list" || { echo "$list"; echo "gofmt would change these; run mise run lint-fix"; exit 1; }',
+    'cd go && go vet ./...',
+    'cd benches/go/cobra && if go mod download >/dev/null 2>&1; then go vet ./...; else echo "skipped: cobra could not be fetched"; fi',
 ]
 
 [tasks.lint-fix]
 run = [
     'cargo clippy --all --all-features --fix --allow-dirty --allow-staged -- -D warnings',
     'cargo fmt --all',
-    'gofmt -w go',
+    'gofmt -w go benches/go',
     'prettier -w .',
 ]
 
@@ -181,6 +192,9 @@ run = [
     "cargo run -q -p xtask -- gen-shadow benches/mise.usage.kdl benches/shadows/mise-clap clap",
     "cargo run -q -p xtask -- gen-shadow benches/mise.usage.kdl benches/shadows/mise-argh argh",
     "cargo run -q -p xtask -- gen-shadow benches/mise.usage.kdl benches/shadows/mise-bpaf bpaf",
+    # Go, and a module of its own: cobra is what usage-go is compared against, so it must not
+    # be a dependency of the module being measured.
+    "cargo run -q -p xtask -- gen-shadow benches/mise.usage.kdl benches/go/cobra cobra",
     "cargo run -q -p xtask -- gen-shadow benches/fleet/hk.usage.kdl benches/shadows/hk usage",
     "cargo run -q -p xtask -- gen-shadow benches/fleet/fnox.usage.kdl benches/shadows/fnox usage",
     "cargo run -q -p xtask -- gen-shadow benches/fleet/pitchfork.usage.kdl benches/shadows/pitchfork usage",

tasks/perf-go.sh

@@ -24,6 +24,11 @@ ARGV="use -g node@20"
 # 50x slowdown stays under a second.
 BINDS=1000
 
+# cobra builds its whole command tree on every iteration, which is the cost being compared and
+# about a thousand times usage-go's. Fewer iterations, so cachegrind's 50x slowdown still
+# finishes: 20 of them is 40M instructions, where 1000 would be two billion.
+COBRA_BINDS=20
+
 root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
 work=$(mktemp -d)
 bin=$work/parse-n
@@ -35,6 +40,26 @@ trap 'rm -rf "$work"' EXIT
 
 (cd "$root/go" && go build -o "$bin" ./internal/bench/parse-n)
 
+# cobra, from the same spec — `xtask gen-shadow … cobra`, checked in under benches/go/cobra.
+#
+# Its own module, so cobra is not a dependency of `github.com/jdx/usage/go`, and its build is
+# allowed to fail: it needs the dependency fetched, and a machine without a proxy should still
+# get the row this harness is mainly about. What it must not do is silently report nothing, so
+# the reason lands in the table.
+cobra_bin=$work/cobra
+cobra_why=""
+if [ -d "$root/benches/go/cobra" ]; then
+  if ! (cd "$root/benches/go/cobra" && go build -o "$cobra_bin" . 2>"$work/cobra.log"); then
+    cobra_why="build failed: $(tr -d '\n' <"$work/cobra.log" | cut -c1-120)"
+  fi
+else
+  cobra_why="benches/go/cobra is missing; run \`mise run gen-shadow\`"
+fi
+# shellcheck disable=SC2086
+if [ -z "$cobra_why" ] && [ "$(PARSE_N=1 "$cobra_bin" $ARGV)" != "1" ]; then
+  cobra_why="the shadow did not reach a subcommand"
+fi
+
 # Each harness prints 1 when the bind reached a subcommand. Anything else means the numbers
 # below would be describing a rejected command line, which is cheap for the wrong reason.
 # shellcheck disable=SC2086 # the argv is several words on purpose
@@ -44,9 +69,10 @@ if [ "$(PARSE_N=1 "$bin" $ARGV)" != "1" ]; then
 fi
 
 instructions() {
+  local binary=$1 n=$2
   # shellcheck disable=SC2086
-  PARSE_N="$1" valgrind --tool=cachegrind --cache-sim=no --branch-sim=no \
-    --cachegrind-out-file="$work/cachegrind.out.%p" "$bin" $ARGV 2>&1 |
+  PARSE_N="$n" valgrind --tool=cachegrind --cache-sim=no --branch-sim=no \
+    --cachegrind-out-file="$work/cachegrind.out.%p" "$binary" $ARGV 2>&1 |
     sed -n 's/.*I *refs: *//p' | tr -d ','
 }
 
@@ -85,14 +111,14 @@ runs=10
 # Ten whole processes, timed from outside: a timer inside the program cannot see the runtime
 # starting up, and that is most of what is being reported here.
 wall_ms() {
-  local n=$1
+  local binary=$1 n=$2
   case $clock in
   gnu)
     local start end
     start=$(date +%s%N 2>/dev/null) || start=
     for _ in $(seq "$runs"); do
       # shellcheck disable=SC2086
-      PARSE_N="$n" "$bin" $ARGV >/dev/null
+      PARSE_N="$n" "$binary" $ARGV >/dev/null
     done
     end=$(date +%s%N 2>/dev/null) || end=
     # Validated rather than trusted: a `date` that answers with anything else would
@@ -112,7 +138,7 @@ wall_ms() {
     # ten runs of about one millisecond each, so the fallback reported python rather than
     # the program it was pointed at.
     # shellcheck disable=SC2086
-    python3 - "$bin" "$n" "$runs" $ARGV <<'PYTHON' 2>/dev/null || echo "unavailable"
+    python3 - "$binary" "$n" "$runs" $ARGV <<'PYTHON' 2>/dev/null || echo "unavailable"
 import os, subprocess, sys, time
 
 binary, parse_n, runs, *argv = sys.argv[1:]
@@ -138,9 +164,14 @@ wall_cell() {
   fi
 }
 
-size=$(stat -c %s "$bin" 2>/dev/null || stat -f %z "$bin")
-size_mb=$(awk -v b="$size" 'BEGIN { printf "%.2f", b / 1048576 }')
-one_wall=$(wall_ms 1)
+size_mb() {
+  local bytes
+  bytes=$(stat -c %s "$1" 2>/dev/null || stat -f %z "$1")
+  awk -v b="$bytes" 'BEGIN { printf "%.2f", b / 1048576 }'
+}
+
+size_mb=$(size_mb "$bin")
+one_wall=$(wall_ms "$bin" 1)
 
 if ! command -v valgrind >/dev/null 2>&1; then
   {
@@ -153,8 +184,8 @@ if ! command -v valgrind >/dev/null 2>&1; then
   exit 0
 fi
 
-floor=$(instructions 0)
-many=$(instructions "$BINDS")
+floor=$(instructions "$bin" 0)
+many=$(instructions "$bin" "$BINDS")
 per=$(( (many - floor) / BINDS ))
 
 {
@@ -176,4 +207,38 @@ per=$(( (many - floor) / BINDS ))
   printf 'pays before `main`, it is three orders of magnitude larger than the bind, and it\n'
   printf 'varies between runs by more than the bind costs. Any single-bind measurement here is\n'
   printf 'a measurement of the runtime.\n'
+
+  printf '\n#### Against cobra\n\n'
+  if [ -n "$cobra_why" ]; then
+    printf 'Not measured this run — %s.\n' "$cobra_why"
+  else
+    cobra_floor=$(instructions "$cobra_bin" 0)
+    cobra_many=$(instructions "$cobra_bin" "$COBRA_BINDS")
+    cobra_per=$(( (cobra_many - cobra_floor) / COBRA_BINDS ))
+    cobra_wall=$(wall_ms "$cobra_bin" 1)
+    ratio=$(awk -v a="$cobra_per" -v b="$per" 'BEGIN { printf "%.0f", a / b }')
+
+    # shellcheck disable=SC2016 # the backticks are markdown, not command substitution
+    printf 'The same spec, declared in cobra by `xtask gen-shadow … cobra` and checked in under\n'
+    # shellcheck disable=SC2016
+    printf '`benches/go/cobra`, so the two rows describe the same CLI rather than two people'"'"'s\n'
+    printf 'transcriptions of mise.\n\n'
+    printf '| | one resolve | whole process | binary |\n|---|---:|---:|---:|\n'
+    printf '| usage-go | %s | %s | %s MB |\n' \
+      "$(printf "%'d" "$per")" "$(wall_cell "$one_wall")" "$size_mb"
+    printf '| cobra | %s | %s | %s MB |\n' \
+      "$(printf "%'d" "$cobra_per")" "$(wall_cell "$cobra_wall")" "$(size_mb "$cobra_bin")"
+    printf '| ratio | %sx | | |\n' "$ratio"
+    printf '\n'
+    printf 'cobra'"'"'s figure includes building its command tree, because that is what it does on\n'
+    # shellcheck disable=SC2016
+    printf 'every process start: a `cobra.Command` per subcommand, each with its own flag set.\n'
+    printf 'Hoisting it out of the loop would measure its parser against a program that had\n'
+    printf 'already paid for its model, which no CLI gets to do. usage-go has no such step —\n'
+    printf 'the tables are laid out by the linker — so the two figures are what each framework\n'
+    printf 'costs to answer one command line in a fresh process.\n'
+    printf '\nAmortized over %s iterations for usage-go and %s for cobra: one cobra resolve is\n' \
+      "$BINDS" "$COBRA_BINDS"
+    printf 'dear enough that a thousand of them under cachegrind would take minutes.\n'
+  fi
 } >"$out"