Skip to content

fix[split]: keep surrogate pairs and combining marks together for an empty separator - #2171

Merged
raon0211 merged 4 commits into
toss:mainfrom
KooYS:fix/compat-split-unicode
Oct 30, 2026
Merged

raon0211 merged 4 commits into
toss:mainfrom
KooYS:fix/compat-split-unicode

Conversation

@KooYS

@KooYS KooYS commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

split in compat breaks emoji and accented characters when the separator is ''.
It just calls String.prototype.split, so the string is split by UTF-16 code units.
lodash handles this case differently: if the string has multi-byte characters, it splits by characters instead.

split('a😀b', '');
// es-toolkit: ['a', '\ud83d', '\ude00', 'b']
// lodash:     ['a', '😀', 'b']

split('Ame\u0301lie', '');
// es-toolkit: ['A', 'm', 'e', '\u0301', 'l', 'i', 'e']
// lodash:     ['A', 'm', 'é', 'l', 'i', 'e']
input lodash compat (before) compat (after)
split('a😀b', '') ['a', '😀', 'b'] ['a', '\ud83d', '\ude00', 'b'] ['a', '😀', 'b']
split('Ame\u0301lie', '') 6 items, 'é' kept 7 items, '\u0301' alone 6 items, 'é' kept
split('👨‍👩‍👧', '') ['👨‍👩‍👧'] 8 code units ['👨‍👩‍👧']
split('🇰🇷', '') ['🇰🇷'] 4 code units ['🇰🇷']
split('a😀b', '', 2) ['a', '😀'] ['a', '\ud83d'] ['a', '😀']

This call type-checks with @types/lodash in strict mode.
Grist uses it to highlight autocomplete matches, and the comment there says it expects diacritics to stay combined:
https://github.com/gristlabs/grist-core/blob/897d07997d0d101b7fe5ecfaca61bb5f4e3a6ce0/app/client/lib/ACIndex.ts#L273

With the current compat, highlighting the first 3 characters of "Ame\u0301lie" gives 'Ame' and '\u0301lie', so the accent ends up in the wrong part.

Changes

I reused the reUnicode pattern that is already in unicodeSize.ts (it is the same one lodash uses), and added _internal/unicodeToArray.ts that uses it, like lodash's _unicodeToArray.
split now uses it only when the separator is '' and the string has multi-byte characters. Other inputs work the same as before.

Tests

Added a test that fails before this change. The compat tests pass.
I also compared the results with lodash 4.18.1 for 308 input combinations, and they all match.

Benchmark results

Run with vitest bench on Node 22.14 (ops/sec).
The first row is the existing split.bench.ts. For the other two rows I used a temporary bench file, which is not included in this PR.

case main this branch lodash
'-' separator 2,626,812 2,869,407 6,496,327
ascii, '' 2,664,083 2,845,279 5,870,041
unicode, '' 1,225,899 365,102 423,759

The ascii cases are about the same. The unicode case is slower than main, but main was returning broken strings there, and the speed is now close to lodash.

Bundle size of compat/split goes from 388 B to 1,028 B because of the reUnicode pattern. compat/size already uses the same pattern, and lodash-es split is 4,219 B.
I also tried writing it as a loop instead of a regex. It matched lodash on 2,000,000 random strings, but the bundle was bigger (1,118 B), so I kept the regex.
The existing bundle-size snapshots don't change.

@vercel

vercel Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
es-toolkit Ready Ready Preview Oct 30, 2026 4:00am UTC

Request Review

@KooYS
KooYS marked this pull request as ready for review October 28, 2026 06:47
Comment thread src/compat/string/split.ts Resolved
Co-authored-by: Sojin Park <raon0211@gmail.com>
@raon0211
raon0211 merged commit 58a6592 into toss:main Oct 30, 2026
11 of 12 checks passed

This branch was successfully deployed

1 active deployment
Preview — c40486cb Deployed Oct 30, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants