perfratio

A multi-dimensional command-line benchmarking tool with Hardware Performance Counters & Energy (RAPL) and Deep Statistics (Criterion-rs).

perfratio (formerly joulex) builds upon the heritage of sharkdp/hyperfine and incorporates statistical rigor inspired by Gabriella439/bench alongside hardware telemetry concepts from perf stat and poop. It is engineered for hardware-aware performance ratios, energy efficiency, and statistically robust systems benchmarking.

Key Features

• ⚡ Performance per Watt & Energy Benchmarking (-E, --energy): Hardware RAPL microjoule counters (Linux sysfs powercap) to measure energy consumption in Joules (mean ± σ) and average active Power in Watts.
• 🔬 Deep Statistical Rigor (--deep-stats): Direct integration with criterion-stats (Criterion.rs) computing 95% bootstrapped confidence intervals for mean and median, plus two-sample hypothesis testing ($p$-value significance) against reference commands.
• 🧠 Peak Memory Usage Display: Real-time process peak memory (RSS) formatted in B, KB, MB, GB and displayed in terminal output.
• 🛡️ Hardened Robustness & Security: Protection against CSV formula injection (CWE-1236), safe integer range iteration avoiding overflows near i32::MAX, and input validation preventing zero-run crashes.
• 🔄 Environment Iteration Forwarding: $PERFRATIO_ITERATION, $JOULEX_ITERATION, and $HYPERFINE_ITERATION forwarded to --prepare and --conclude commands.
• ⚙️ Modern CLI & Completions: Unified -e, --export <FILE> auto-detecting formats by file extension, --filter-failed for parameter scans, and built-in --generate-completions <SHELL>.
• Statistical analysis across multiple runs: Mean, median, stddev, min, max, user, and kernel CPU time.
• Support for arbitrary shell commands: Raw or custom shells.
• Outlier detection: Modified Z-score outlier detection to identify caching or interference.
• Warmup runs & cache-clearing commands: Set up before timing runs.
• Export results to various formats: CSV, JSON, Markdown, AsciiDoc, Org-mode.
• Parameterized benchmarks: Vary arguments or matrices of parameters.
• Cross-platform: Linux, macOS, Windows.

Usage

Basic benchmarks

To run a benchmark, you can simply call perfratio <command>... (or hyperfine <command>...). The argument(s) can be any shell command. For example:

perfratio 'sleep 0.3'

perfratio will automatically determine the number of runs to perform for each command. By default, it will perform at least 10 benchmarking runs and measure for at least 3 seconds. To change this, you can use the -r/--runs option:

perfratio --runs 5 'sleep 0.3'

If you want to compare the runtimes of different programs, you can pass multiple commands:

perfratio 'hexdump file' 'xxd file'

Understanding the output

What do mean ± σ, User/System, CPU %, Peak Memory, the energy line and the relative-speed summary mean exactly, and how do you get stable numbers? See doc/understanding-output.md. It also covers benchmarking across git branches and a checklist for reducing noise.

Warmup runs and preparation commands

For programs that perform a lot of disk I/O, the benchmarking results can be heavily influenced by disk caches and whether they are cold or warm.

If you want to run the benchmark on a warm cache, you can use the -w/--warmup option to perform a certain number of program executions before the actual benchmark:

hyperfine --warmup 3 'grep -R TODO *'

Conversely, if you want to run the benchmark for a cold cache, you can use the -p/--prepare option to run a special command before each timing run. For example, to clear harddisk caches on Linux, you can run

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

To use this specific command with hyperfine, call sudo -v to temporarily gain sudo permissions and then call:

hyperfine --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' 'grep -R TODO *'

Parameterized benchmarks

If you want to run a series of benchmarks where a single parameter is varied (say, the number of threads), you can use the -P/--parameter-scan option and call:

hyperfine --prepare 'make clean' --parameter-scan num_threads 1 12 'make -j {num_threads}'

This also works with decimal numbers. The -D/--parameter-step-size option can be used to control the step size:

hyperfine --parameter-scan delay 0.3 0.7 -D 0.2 'sleep {delay}'

This runs sleep 0.3, sleep 0.5 and sleep 0.7.

For non-numeric parameters, you can also supply a list of values with the -L/--parameter-list option:

hyperfine -L compiler gcc,clang '{compiler} -O2 main.cpp'

Warning

Parameter values (from -P, -L and -F/--parameter-file) are inserted into the command verbatim. They are not shell-escaped. With the default shell, a value such as a; rm -rf ~ or $(curl …) is executed as shell code. This is by design for values you control, but be careful when the values come from somewhere else (file names, branch names, PR titles, CI inputs, the lines of a --parameter-file):

• Prefer -N/--shell=none. The command is then split into arguments without a shell, so a value can at most become extra arguments, never shell code.
• Or quote the placeholder in the command, e.g. perfratio -L f "$files" "wc -l '{f}'". This only helps if the values cannot contain a single quote themselves.
• Validate untrusted values (e.g. allow only [A-Za-z0-9._/-]) before passing them to perfratio.

Intermediate shell

By default, commands are executed using a predefined shell (/bin/sh on Unix, cmd.exe on Windows). If you want to use a different shell, you can use the -S, --shell <SHELL> option:

hyperfine --shell zsh 'for i in {1..10000}; do echo test; done'

Note that hyperfine always corrects for the shell spawning time. To do this, it performs a calibration procedure where it runs the shell with an empty command (multiple times), to measure the startup time of the shell. It will then subtract this time from the total to show the actual time used by the command in question.

If you want to run a benchmark without an intermediate shell, you can use the -N or --shell=none option. This is helpful for very fast commands (< 5 ms) where the shell startup overhead correction would produce a significant amount of noise. Note that you cannot use shell syntax like * or ~ in this case.

hyperfine -N 'grep TODO /home/user'

Shell functions and aliases

If you are using bash, you can export shell functions to directly benchmark them with hyperfine:

my_function() { sleep 1; }
export -f my_function
hyperfine --shell=bash my_function

Otherwise, inline them into or source them from the benchmarked program:

hyperfine 'my_function() { sleep 1; }; my_function'

echo 'alias my_alias="sleep 1"' > /tmp/my_alias.sh
hyperfine '. /tmp/my_alias.sh; my_alias'

Exporting results

Hyperfine has multiple options for exporting benchmark results to CSV, JSON, Markdown and other formats (see --help text for details).

Markdown

You can use the --export-markdown <file> option to create tables like the following:

Command	Mean [s]	Min [s]	Max [s]	Relative
find . -iregex '.*[0-9]\.jpg$'	2.275 ± 0.046	2.243	2.397	9.79 ± 0.22
find . -iname '*[0-9].jpg'	1.427 ± 0.026	1.405	1.468	6.14 ± 0.13
fd -HI '.*[0-9]\.jpg$'	0.232 ± 0.002	0.230	0.236	1.00

JSON

The JSON output is useful if you want to analyze the benchmark results in more detail. The scripts/ folder includes a lot of helpful Python programs to further analyze benchmark results and create helpful visualizations, like a histogram of runtimes or a whisker plot to compare multiple benchmarks:

Detailed benchmark flowchart

The following chart explains the execution order of various timing runs when using options like --warmup, --prepare <cmd>, --setup <cmd> or --cleanup <cmd>:

Reducing noise

Most "noisy" or irreproducible results come from the environment, not from the benchmarked command. perfratio --check-system reports the usual suspects before benchmarking, with a hint for each problem (--check-system=strict aborts with exit code 4, e.g. on a CI benchmark runner):

System check:
  ✖ CPU governor     powersave on 8 CPUs  → sudo cpupower frequency-set -g performance
  ✖ Turbo boost      enabled  → echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
  ✔ Load average     0.21 (8 CPUs)
  ! Power source     battery  → connect the power adapter (on battery, CPUs are often clocked down)
  ✔ Thermal          max 48 °C

Other things that help:

• Fixed CPU frequency: use the performance governor, and disable turbo boost (see above).
• Pinning and priority: --affinity 2 keeps the command on one core, avoiding migrations and P-/E-core mixes; --priority high or realtime reduces preemption.
• Caches: --warmup N (or --warmup auto) fills them. --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' empties them before every run, for cold-cache benchmarks. --first-run=separate reports the cold first run on its own.
• Drift: --schedule round-robin interleaves the commands, so slow drifts (thermal throttling, background jobs) affect all of them equally. perfratio warns about trends, multimodal distributions and outlier-inflated variance.
• Shell overhead: for very fast commands, -N/--shell=none avoids the intermediate shell.

Installation

On Ubuntu

On Ubuntu, hyperfine can be installed from the official repositories:

apt install hyperfine

Alternatively, for the latest version, you can download the appropriate .deb package from the Release page and install it via dpkg:

wget https://github.com/sharkdp/hyperfine/releases/download/v1.20.0/hyperfine_1.20.0_amd64.deb
sudo dpkg -i hyperfine_1.20.0_amd64.deb

On Fedora

On Fedora, hyperfine can be installed from the official repositories:

dnf install hyperfine

On Alpine Linux

On Alpine Linux, hyperfine can be installed from the official repositories:

apk add hyperfine

On Arch Linux

On Arch Linux, hyperfine can be installed from the official repositories:

pacman -S hyperfine

On Debian Linux

On Debian Linux, hyperfine can be installed from the official repositories:

apt install hyperfine

On Exherbo Linux

On Exherbo Linux, hyperfine can be installed from the rust repositories:

cave resolve -x repository/rust
cave resolve -x hyperfine

On Funtoo Linux

On Funtoo Linux, hyperfine can be installed from core-kit:

emerge app-benchmarks/hyperfine

On NixOS

On NixOS, hyperfine can be installed from the official repositories:

nix-env -i hyperfine

On Flox

On Flox, hyperfine can be installed as follows.

flox install hyperfine

Hyperfine's version in Flox follows that of Nix.

On openSUSE

On openSUSE, hyperfine can be installed from the official repositories:

zypper install hyperfine

On Void Linux

Hyperfine can be installed via xbps

xbps-install -S hyperfine

On macOS

Hyperfine can be installed via Homebrew:

brew install hyperfine

Or you can install using MacPorts:

sudo port selfupdate
sudo port install hyperfine

On FreeBSD

Hyperfine can be installed via pkg:

pkg install hyperfine

On OpenBSD

doas pkg_add hyperfine

On Windows

Hyperfine can be installed via Chocolatey, Scoop, or Winget:

choco install hyperfine

scoop install hyperfine

winget install hyperfine

With conda

Hyperfine can be installed via conda from the conda-forge channel:

conda install -c conda-forge hyperfine

With cargo (Linux, macOS, Windows)

perfratio can be installed from source via cargo:

cargo install --locked perfratio

Make sure that you use Rust 1.88 or newer.

From binaries (Linux, macOS, Windows)

Download the corresponding archive from the Release page.

Alternative tools

Hyperfine is inspired by bench.

Integration with other tools

Chronologer is a tool that uses hyperfine to visualize changes in benchmark timings across your Git history.

Bencher is a continuous benchmarking tool that supports hyperfine to track benchmarks and catch performance regressions in CI.

Drop hyperfine JSON outputs onto the Venz chart to visualize the results, and manage hyperfine configurations.

Make sure to check out the scripts folder in this repository for a set of tools to work with hyperfine benchmark results.

Origin of the name

The name hyperfine was chosen in reference to the hyperfine levels of caesium 133 which play a crucial role in the definition of our base unit of time — the second.

Citing hyperfine

Thank you for considering to cite hyperfine in your research work. Please see the information in the sidebar on how to properly cite hyperfine.

License

hyperfine is dual-licensed under the terms of the MIT License and the Apache License 2.0.

See the LICENSE-APACHE and LICENSE-MIT files for details.

Contributing

See CONTRIBUTING.md for the local checks and pull request workflow.