Skip to content

fix: invalid UTF-8 bytes silently skipped instead of matching as U+FFFD #179

Closed

Description

@kolkov

Invalid UTF-8 bytes (lead bytes without continuation, bare continuation bytes) are silently skipped instead of being treated as U+FFFD replacement characters of width 1.

Repro

re := coregex.MustCompile(`(.)`)
fmt.Println(re.FindAllStringSubmatchIndex("\xf0", -1))
// coregex: []                    — nothing matched
// stdlib:  [[0 1 0 1]]           — U+FFFD at position 0, width 1

re2 := coregex.MustCompile(`.`)
fmt.Println(re2.FindAllStringIndex("\xc3(", -1))
// coregex: [[1 2]]              — skipped lead byte, matched '('
// stdlib:  [[0 1] [1 2]]        — matched U+FFFD at 0, then '('

Why this matters

Go's fuzz mutator generates invalid UTF-8 constantly. With fuzz whitelists removed, 4 of 7 stdlib-comparison fuzz targets fail within seconds — all on this class:

FuzzFindAllStdlib         FAIL (5s)
FuzzFindSubmatchStdlib    FAIL (2s)  
FuzzFindAllSubmatchStdlib FAIL (0.4s)
FuzzSplitStdlib           FAIL (0.6s)

Real-world occurrences: binary data in logs, truncated network buffers, protocol parsing, user-supplied input with encoding errors.

Expected behavior

Any byte that doesn't decode as valid UTF-8 should be treated as a single U+FFFD (width 1), matching ., \S, [^x], etc. This is the stdlib behavior and what unicode/utf8.DecodeRune returns for invalid sequences.

Current implementation

nfa/compile.go has a comment near invalidTrans stating that lead bytes 0xC2–0xF4 without valid continuation are intentionally not matched. This was a design choice for capture correctness, but it creates the largest class of stdlib divergence under fuzzing.

Related: #178 (tracking), #174 (negated classes — same byte-vs-rune family).

Activity

  1. added
    priority: criticalRelease blocker, correctness regression
    type: bugSomething isn't working
    area: nfaThompson NFA, PikeVM, BoundedBacktracker
    on Oct 9, 2026
  2. kolkov commented on Oct 9, 2026

    @kolkov
    ContributorAuthor

    Clarification from review: NFA already handles 0x80–0xBF, C0/C1, F5–FF as single bytes. Only valid lead bytes C2–F4 without valid continuation are skipped. Half the invalid-byte table is already in place.

    Additional acceptance criterion: \x{FFFD} in pattern must match invalid bytes. Currently \x{FFFD} on "a\xffb" returns nothing (stdlib: [1 2]). Same class, other direction — not just ./\S/[^x] should match invalid bytes, but explicit U+FFFD literal too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: nfaThompson NFA, PikeVM, BoundedBacktrackerpriority: criticalRelease blocker, correctness regressiontype: bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions