wc: fix character counts for malformed UTF-8 input by darkraider01 · Pull Request #15139 · uutils/coreutils

darkraider01 · 2026-10-06T14:28:11Z

The fast path for wc -m and wc -cm counts some invalid or incomplete UTF-8 sequences as characters. This validates the input in UTF-8 locales and handles characters split across reads, with regression tests for those cases.
Related to #14928

darkraider01 · 2026-10-06T14:45:27Z

    let buf: &mut [u8] = &mut AlignedBuffer::default().data;
    let policy = SimdPolicy::detect();
    let simd_allowed = wc_simd_allowed(policy);
+    let validate_utf8 = COUNT_CHARS && is_utf8_locale();


This change only updates UTF-8 counting. The three C locale failures in wc/wc.pl remain and need a follow-up.

darkraider01 · 2026-10-06T14:45:56Z

+    let mut validate_with_simd = simd_allowed;
+    while !input.is_empty() {
+        let result = if validate_with_simd {
+            simdutf8::compat::from_utf8(input)


I added simdutf8 because scalar validation noticeably slowed down valid UTF-8 input. After the first invalid sequence in a chunk, validation switches to the standard library to avoid repeated SIMD setup.

github-actions · 2026-10-06T15:52:55Z

GNU testsuite comparison:

Skip an intermittent issue tests/date/resolution (fails in this run but passes in the 'main' branch)

codspeed · 2026-10-06T16:01:39Z

Merging this PR will regress 9 benchmarks

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 15 improved benchmarks
❌ 9 regressed benchmarks
✅ 375 untouched benchmarks
🆕 2 new benchmarks
⏩ 54 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`split_lines`	8.5 ms	11.1 ms	-23.5%
❌	Simulation	`split_numeric_suffix`	8.7 ms	11.4 ms	-23.08%
❌	Simulation	`wc_chars_large_line_count[100000]`	2.3 ms	2.8 ms	-15.57%
❌	Simulation	`five_38_bit_primes`	1.7 s	1.9 s	-12.55%
❌	Simulation	`tsort_complex_dag[50000]`	87.6 ms	97.1 ms	-9.79%
❌	Simulation	`tsort_tree_dag[(10, 3)]`	36.5 ms	39.4 ms	-7.33%
❌	Simulation	`sort_dictionary_order[500000]`	1.9 s	2.1 s	-6.59%
❌	Simulation	`tsort_wide_dag[100000]`	157.6 ms	167 ms	-5.65%
❌	Simulation	`tsort_linear_chain[1000000]`	1.9 s	2 s	-4.75%
⚡	Simulation	`three_39_bit_primes`	545.9 ms	255.9 ms	×2.1
⚡	Simulation	`check_sorted_utf8_locale`	491.1 ms	422.9 ms	+16.13%
⚡	Simulation	`sort_numeric_utf8_locale`	40.8 ms	36.1 ms	+12.99%
⚡	Simulation	`expand_many_lines[100000]`	106.2 ms	95 ms	+11.77%
⚡	Simulation	`merge_single_file_utf8_locale`	133.4 ms	119.4 ms	+11.76%
⚡	Simulation	`expand_custom_tabstops[50000]`	29.6 ms	26.9 ms	+10.22%
⚡	Simulation	`merge_pre_sorted_files_utf8_locale`	251.9 ms	238 ms	+5.81%
⚡	Simulation	`merge_pre_sorted_files`	252.5 ms	238.9 ms	+5.71%
⚡	Simulation	`sort_german_de_locale`	295 ms	282.1 ms	+4.6%
⚡	Simulation	`thirteen_39_bit_primes`	9.3 s	8.9 s	+4.44%
⚡	Simulation	`sort_case_sensitive[500000]`	334.9 ms	321 ms	+4.32%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing darkraider01:wc-utf8-character-counts (3a107c7) with main (0d4a305)}

54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

sylvestre · 2026-10-06T20:07:58Z


 const BUF_SIZE: usize = 64 * 1024;

+fn is_utf8_locale() -> bool {


please fix get_ctype_encoding() in uucore for locales without a suffix like en_IN instead of adding this unsafe locale code in wc

sylvestre · 2026-10-06T20:08:08Z

+            }
+            None => return 0,
+        }
+    }


codspeed shows a 39% regression, could you please look at it?
the is_ascii() pass followed by from_utf8 reads non-ASCII chunks twice

I looked into the CodSpeed report. The updated comparison shows about 18% lower efficiency for wc_chars_large_line_count; it also flags different runtime environments, so the exact size of the regression is uncertain. I'll look into it further.
Locally, removing is_ascii() slowed ASCII input, especially with SIMD disabled. A tail-first check helped some UTF-8 inputs but regressed others, so I reverted it. These experiments haven’t resolved the regression; the additional UTF-8 validation still adds work, and this remains open.

Okay so i digged deeper into it and benchmarked the PR against its base on my machine, with the same compiler and locale, alternating runs pinned to one CPU. The affected ASCII benchmark measured 0.764 ms on the base and 0.772 ms on the PR, within the variation between runs, so I couldn’t reproduce the reported regression locally.
There is a clear cost for valid UTF-8: that benchmark was about 68% slower. I also tried a scalar pass that validates and counts together, but it was substantially slower than the current SIMD approach, so I dropped it.
I’ve kept the current implementation for now. The local results don’t explain the CodSpeed regression, so I’m leaving this open.
Would you prefer further performance work in this PR, or a separate follow-up?

sylvestre · 2026-10-06T20:08:28Z

        }

-        // Fast paths that can be computed without Unicode decoding.
+        // Fast paths for byte, character, and line counts.


why change this comment?

I changed it because the character-counting path now validates UTF-8, so “without Unicode decoding” seemed less accurate. The new wording just describes which counts use these fast paths. Happy to keep the original comment if you prefer.

sylvestre · 2026-10-06T20:08:34Z

-            for (key, val) in &cmd_env {
+            // WASI reads the first duplicate, whereas Command::envs uses the last.
+            // Resolve overrides before forwarding them to the guest.
+            let wasm_env: BTreeMap<_, _> = cmd_env.iter().cloned().collect();


unrelated to wc, please move it into a separate PR

done! its in #15150 now

sylvestre · 2026-10-08T09:12:43Z

 rustix = { version = "1.1.4", default-features = false }
 self_cell = "1.0.4"
 selinux = "0.6"
+simdutf8 = "0.1.5"


is std from_utf8 really that much slower here?

I checked this directly on my machine, using the same UTF-8 benchmark and changing only the validator. With SIMD enabled, it took about 0.267 ms with simdutf8 and 1.325 ms with std::str::from_utf8 , roughly 5× slower across nine alternating runs. The counting code stayed the same, so I’ve kept simdutf8. This doesn’t explain the earlier CodSpeed ASCII regression, though.

sylvestre · 2026-10-08T09:12:54Z

-/// Return the character-type encoding (`LC_CTYPE`) deduced from the environment.
+/// Return the character-type encoding (`LC_CTYPE`) selected by the environment.
+/// Bare locale names are resolved through native locale data where available.
 pub fn get_ctype_encoding() -> UEncoding {


could you please move the uucore change into its own PR? same as the test helper

Moved it to #15199 and removed it from this PR. Bare-locale detection will depend on that landing first.

sylvestre · 2026-10-08T09:12:59Z

+    *CTYPE_ENCODING.get_or_init(|| {
+        let name = ["LC_ALL", "LC_CTYPE", "LANG"]
+            .iter()
+            .find_map(|key| std::env::var(key).ok().filter(|value| !value.is_empty()));


this duplicates the LC_ALL/LC_CTYPE/LANG lookup of get_locale_from_env, could it be shared?

Shared the lookup in #15199. Both paths now use the same environment lookup and parsing helpers, and get_ctype_encoding reads the environment only once.

sylvestre · 2026-10-08T09:13:04Z

+        // SAFETY: locale is live while CODESET is read. The previous thread locale
+        // is restored before the owned locale is freed.
+        unsafe {
+            let previous = libc::uselocale(locale);


could nl_langinfo_l be used here instead of switching the thread locale back and forth?

Yes, switched to nl_langinfo_l in #15199. It queries the locale object directly, so there’s no thread-locale switching anymore.

darkraider01 added 2 commits October 6, 2026 19:45

wc: record current UTF-8 character counting

1de1132

wc: count complete valid characters in UTF-8 locales

5968182

darkraider01 commented Oct 6, 2026

View reviewed changes

Merge branch 'main' into wc-utf8-character-counts

f91a876

darkraider01 added 3 commits October 6, 2026 21:26

wc: update fuzz lockfile for UTF-8 validation

bd1da7b

tests: preserve environment overrides in WASI

482cc3d

Merge branch 'main' into wc-utf8-character-counts

eedb43b

darkraider01 marked this pull request as draft October 6, 2026 16:04

darkraider01 mentioned this pull request Oct 6, 2026

wc: count characters as bytes in single-byte locales #15140

Draft

darkraider01 marked this pull request as ready for review October 6, 2026 16:41

Merge branch 'main' into wc-utf8-character-counts

7f4731c

sylvestre reviewed Oct 6, 2026

View reviewed changes

darkraider01 added 2 commits October 7, 2026 12:35

uucore: resolve character encodings for bare locale names

a07abe9

wc: move WASI environment overrides to a separate change

2a20f8f

darkraider01 mentioned this pull request Oct 7, 2026

tests: preserve environment overrides in WASI #15150

Merged

darkraider01 requested a review from sylvestre October 7, 2026 07:24

Merge branch 'main' into wc-utf8-character-counts

9b701aa

sylvestre reviewed Oct 8, 2026

View reviewed changes

darkraider01 mentioned this pull request Oct 8, 2026

uucore: detect character encodings for bare locale names #15199

Open

darkraider01 added 2 commits October 8, 2026 15:05

wc: split bare locale detection into a uucore change

3a107c7

wc: move the UTF-8 benchmark to its own PR

df791ac

darkraider01 mentioned this pull request Oct 8, 2026

wc: add a UTF-8 character-count benchmark #15202

Open


		const BUF_SIZE: usize = 64 * 1024;

		fn is_utf8_locale() -> bool {

Conversation

darkraider01 commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

codspeed Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 9 benchmarks

Performance Changes

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darkraider01 Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

darkraider01 commented Oct 6, 2026 •

edited

Loading

codspeed Bot commented Oct 6, 2026 •

edited

Loading

darkraider01 Oct 8, 2026 •

edited

Loading