Skip to content

wc: add a UTF-8 character-count benchmark #15202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sylvestre merged 2 commits into from
Oct 9, 2026
Merged

wc: add a UTF-8 character-count benchmark #15202

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f6923e0
wc: add a UTF-8 character-count benchmark
darkraider01 Oct 8, 2026
9d308ce
wc: make character benchmarks use a consistent UTF-8 locale
darkraider01 Oct 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions src/uu/wc/BENCHMARKING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,12 @@ candidate as it's fairly large.
Use [`hyperfine`](https://github.com/sharkdp/hyperfine) to compare the
performance. For example, `hyperfine 'wc somefile' 'uuwc somefile'`.

Run the UTF-8 character-count benchmark with an explicit UTF-8 locale:

```shell
LC_ALL=C.UTF-8 cargo bench -p uu_wc --bench wc_bench -- wc_chars_utf8
```

If you want to get fancy and exhaustive, generate a table:

| | moby64.txt | odyssey256.txt | 25Mshortlines | /usr/bin/docker |
Expand Down
16 changes: 16 additions & 0 deletions src/uu/wc/benches/wc_bench.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,17 @@ fn wc_chars_large_line_count(bencher: Bencher, num_lines: usize) {
.bench_values(|args| black_box(uumain(args)));
}

#[divan::bench(args = [100_000])]
fn wc_chars_utf8(bencher: Bencher, num_lines: usize) {

@sylvestre sylvestre Oct 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please set LC_ALL in the bench itself, like join_bench.rs or sort_locale_utf8_bench.rs?
codspeed won't use the env from the doc

@darkraider01 darkraider01 Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

let temp_dir = tempfile::tempdir().unwrap();
let data = "hello ä € 💩\n".repeat(num_lines);
let file_path = create_test_file(data.as_bytes(), temp_dir.path());

bencher
.with_inputs(|| get_bench_args(&[&"-m", &file_path]).into_iter())
.bench_values(|args| black_box(uumain(args)));
}

/// Benchmark word counting on large line counts
#[divan::bench(args = [100_000])]
fn wc_words_large_line_count(bencher: Bencher, num_lines: usize) {
Expand Down Expand Up @@ -115,5 +126,10 @@ fn wc_lines_extreme_line_lengths(bencher: Bencher, (num_lines, line_len): (usize
}

fn main() {
// Set the locale before its first use, since character encoding is cached.
// SAFETY: The benchmark runner has not started any threads yet.
unsafe {
std::env::set_var("LC_ALL", "C.UTF-8");
}
divan::main();
}
Loading