Repository navigation
feat: UTF-8 validator product in lazy DFA (phase 3) - #184
Merged
Merged
Conversation
Go 1.27.2 uses export data version 5, golangci-lint v2.13 supports only version 4. CI failed on v0.12.27 merge commit.
Phase 3, step 13: validator state infrastructure. - State struct: add validatorState uint8 field - 9 validator states (VS0-VS8) from Björn Höhrmann's UTF-8 DFA, adapted - ValidatorTransition(vs, byte) → (next, quit): deterministic UTF-8 structural validation per byte. Quit on invalid sequences. - ComputeStateKeyFull: includes validatorState in hash (separate byte in FNV-1a, distinct from flags). Different validator states = different DFA states. - 41 unit tests: individual transitions, full UTF-8 sequences (2/3/4 byte valid, orphan/truncated/overlong/surrogate invalid), state key identity. No behavioral changes yet — determinize doesn't use validator state. That comes in task #13 (validator transitions in determinize).
Phase 3, steps 12-13 + task 15: replace temporary 3c quit with validator-informed transitions. Validator product (state.go): - 9 validator states (VS0-VS8) with ValidatorTransition(vs, byte) - ValidatorState field in State struct, included in ComputeStateKeyFull - 41 unit tests covering all transitions + full sequences + key identity Determinize (lazy.go): - At VS0: ASCII bytes → VS0 (stay in DFA). Lead bytes C2-F4 → VS1-VS8 (proceed through multi-byte chain). Invalid bytes → QuitState (PikeVM). - At VS1-VS8: valid continuations → next VS (stay in DFA). Invalid continuations → QuitState (PikeVM). - Replaces blanket 3c quit (ALL bytes >= 0x80 → quit) with targeted quit (only structurally invalid bytes → quit). EOI fallback (lazy.go): - needsEOIFallback: if validator state != VS0 at end of input, the DFA consumed lead bytes of a truncated sequence. Falls back to PikeVM for correct U+FFFD handling. Applied to all 6 forward search EOI paths. Result: DFA now processes valid non-ASCII text (Cyrillic, CJK) without quitting. Only structurally invalid UTF-8 bytes trigger PikeVM fallback.
Phase 3 correctness fix: truncated multi-byte sequences. Bug: validator only checked bytes >= 0x80. When validator was in S1-S8 (expecting continuation) and an ASCII byte arrived (e.g., 'x' after lead byte 0xC3), the code set nextValidatorState=VS0 without calling ValidatorTransition — no quit, NFA went to dead state, FFFD match lost. Fix: call ValidatorTransition for ALL bytes when hasInvalidUTF8Look, not just >= 0x80. ASCII byte in S1-S8 → validator returns quit → QuitState → PikeVM handles lead byte as U+FFFD correctly. Also added EOI fallback: needsEOIFallback checks if validator state != VS0 at end of input (truncated sequence at EOF). Applied to all 6 forward search EOI paths. New acceptance tests (6 cases): - .x on truncated 2/3/4-byte leads: \xc3x, \xe3x, \xe3\x81x, \xf0\x90x - [^a]x on truncated 2-byte: \xc3x - a.b on truncated mid-pattern: a\xc3b - FindAll variants for the above
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Benchmark ComparisonComparing Summary:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 3: UTF-8 validator product replaces temporary 3c quit in lazy DFA.
The temporary 3c fallback (quit on ALL bytes >= 0x80) caused 4-20x regression on Cyrillic/CJK text. The validator product tracks UTF-8 structural validity per byte (9-state automaton), quitting only on structurally invalid bytes. Valid non-ASCII text stays in DFA.
DFA state count (criterion 16: ≤ 2× vs 273ff75 baseline)
.+.+.+.+\S+\S+[^,]+ERROR.*\d+ms.*char_class(?m)^.*\.phpAll ≤ 2.00×. Zero cache clears on valid text. Max 76 states (28 KB).
Test plan
.xon\xc3x,\xe3x,\xe3\x81x,\xf0\x90x[^К]bon"Кb"= no match-raceclean, lint/gofmt clean.++17%, others within noise)Fixes #179