You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix: invalid UTF-8 bytes silently skipped instead of matching as U+FFFD #179
Invalid UTF-8 bytes (lead bytes without continuation, bare continuation bytes) are silently skipped instead of being treated as U+FFFD replacement characters of width 1.
Repro
re:=coregex.MustCompile(`(.)`)
fmt.Println(re.FindAllStringSubmatchIndex("\xf0", -1))
// coregex: [] — nothing matched// stdlib: [[0 1 0 1]] — U+FFFD at position 0, width 1re2:=coregex.MustCompile(`.`)
fmt.Println(re2.FindAllStringIndex("\xc3(", -1))
// coregex: [[1 2]] — skipped lead byte, matched '('// stdlib: [[0 1] [1 2]] — matched U+FFFD at 0, then '('
Why this matters
Go's fuzz mutator generates invalid UTF-8 constantly. With fuzz whitelists removed, 4 of 7 stdlib-comparison fuzz targets fail within seconds — all on this class:
Real-world occurrences: binary data in logs, truncated network buffers, protocol parsing, user-supplied input with encoding errors.
Expected behavior
Any byte that doesn't decode as valid UTF-8 should be treated as a single U+FFFD (width 1), matching ., \S, [^x], etc. This is the stdlib behavior and what unicode/utf8.DecodeRune returns for invalid sequences.
Current implementation
nfa/compile.go has a comment near invalidTrans stating that lead bytes 0xC2–0xF4 without valid continuation are intentionally not matched. This was a design choice for capture correctness, but it creates the largest class of stdlib divergence under fuzzing.
Related: #178 (tracking), #174 (negated classes — same byte-vs-rune family).
Clarification from review: NFA already handles 0x80–0xBF, C0/C1, F5–FF as single bytes. Only valid lead bytes C2–F4 without valid continuation are skipped. Half the invalid-byte table is already in place.
Additional acceptance criterion: \x{FFFD} in pattern must match invalid bytes. Currently \x{FFFD} on "a\xffb" returns nothing (stdlib: [1 2]). Same class, other direction — not just ./\S/[^x] should match invalid bytes, but explicit U+FFFD literal too.
Activity
kolkov commentedon Oct 9, 2026
Clarification from review: NFA already handles 0x80–0xBF, C0/C1, F5–FF as single bytes. Only valid lead bytes C2–F4 without valid continuation are skipped. Half the invalid-byte table is already in place.
Additional acceptance criterion:
\x{FFFD}in pattern must match invalid bytes. Currently\x{FFFD}on"a\xffb"returns nothing (stdlib:[1 2]). Same class, other direction — not just./\S/[^x]should match invalid bytes, but explicit U+FFFD literal too.fix: negated classes match runes not bytes (#180)
fix: anchored quit fallback, reverse context preservation, gofmt (#179)
feat: insert Look-gated U+FFFD fallback in NFA compiler (#179)
feat: evaluate LookInvalidUTF8 in PikeVM and BoundedBacktracker (#179)
feat: evaluate LookInvalidUTF8, acceptance tests, literal U+FFFD (#179)
test: remove D-class fuzz whitelists, add invalid UTF-8 seeds (#179)
fix: OnePass evaluates LookInvalidUTF8 at runtime, gofmt (#179)
docs: add #179 invalid UTF-8 to CHANGELOG [Unreleased]
feat: invalid UTF-8 bytes match as U+FFFD width 1 (#182)
fix: validator must check ALL bytes, not just >= 0x80 (#179)
docs: update CHANGELOG for #179 phase 3 validator product
docs: CHANGELOG — honest perf numbers for #179 validator
feat: UTF-8 validator product in lazy DFA (phase 3) (#184)