Simplified Chinese doesn't split words or allow splitting words #278

Activity

1. HugoFara

   added

   bugSomething isn't workingSomething isn't working

   enhancementDevelop an existing featureDevelop an existing feature

   on Aug 23, 2026
2. HugoFara commented on Aug 23, 2026

   

   HugoFara

   on Aug 23, 2026

   Owner

   More actions

   Hi, for Chinese the easiest solution for now is to pass your text to an external parser to separate words and add spaces. You can have two chinese languages : one with space separation, and one without, depending of the text you want to read.

   If you are ready to use some experimental feature, there is built-in support for jieba, a parser for mandarin Chinese, see the external parsers page. However I don't have much time to maintain this feature, so expect some rough edges. PRs welcome!
3. HugoFara

   linked a pull request that will close this issuefix(language): make the parser setting real, and Chinese use jieba (#278) #281on Aug 23, 2026
4. HugoFara

   added a commit that references this issue on Aug 23, 2026

   fix(language): make the parser setting real, and Chinese use jieba (#278

   ...

   4c78c01
5. sarahmccuan commented on Aug 24, 2026

   

   sarahmccuan

   on Aug 24, 2026

   Author

   More actions

   I'll check it out, thanks.
6. HugoFara

   added 4 commits that reference this issue on Aug 27, 2026

   Merge pull request #281 from HugoFara/fix/278-wire-parser-registry

   ...

   10bb342

   Merge pull request #282 from HugoFara/feat/278-warn-when-a-text-parse…

   ...

   55ee5ac

   fix(text): judge a parse by its letters, not its length

   ...

   abeb9d2

   fix(text): split spaceless languages that have no tokenizer

   ...

   81a704c
7. HugoFara commented on Aug 31, 2026

   

   HugoFara

   on Aug 31, 2026

   Owner

   More actions

   Fixed on develop in 81a704c.

   Cause. A language written without spaces gives the regex tokenizer nothing to match on, so it takes the longest run of word characters it can find — with no gaps, that is the whole sentence:

   before:  [我昨天去了北京大学的图书馆看书] [今天天气很好]     2 words, longest 15 chars
   after:   [我] [昨] [天] [去] [了] [北] …                      21 words, longest 1 char
   

   That is the reported "no ability to select a character or character-grouping", and the second half of the title follows from the first — you cannot select a grouping out of a sentence that is already one token.

   Fix. Reaching StandardTextParser at all means no tokenizer ran (MeCab goes to JapaneseTextParser; an opted-in parser either handled the text or was unavailable). At that point a spaceless language has one way left to find boundaries, and it is the one the presets already encode: Chinese (Simplified), Chinese (Traditional) and Japanese are the only presets setting removeSpaces, all three also set makeCharacterWord, and no other preset sets it at all. getOptedInParserFromRow() already documents that it falls back on "their split-each-character setting"; getLanguageSettings() now makes that hold for a language that lost the flag or never had it.

   Verification. On a real corpus of 13 texts, the same Chinese text on two differently configured Chinese languages went from 78 and 103 words to 103 and 103; Thai went 8 → 19; French, German, Japanese, Korean and Hebrew were unchanged. 北京大学, 图书馆 and 天气 are all detected as multi-character terms once the text is segmented, which covers the "or allow splitting words" half. Four DB-backed regression tests added; PHPUnit 9073 passing, Cypress 304 passing, Psalm clean.

   One case deliberately not covered. A Chinese language whose LgRemoveSpaces is 0 still produces sentence-words — that is the language asserting its words are space-separated, and no fallback can infer around a statement the configuration makes about itself. Detecting it from token output alone is not reliable (one long word per sentence is indistinguishable from a legitimate one-word sentence), so it would need the config fact rather than a heuristic. The reported path — a language created from the Simplified Chinese preset, which sets removeSpaces: true — is covered.
8. HugoFara

   closed this as completedon Aug 31, 2026