Skip to content

feat: CompositeSequenceDFA 25x speedup + FindAllIndexCompact API - #84

Merged
kolkov merged 4 commits into
mainfrom
fix/overlap-detection-perf
Jan 14, 2026
Merged

kolkov merged 4 commits into
mainfrom
fix/overlap-detection-perf

Conversation

@kolkov

@kolkov kolkov commented Jan 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • CompositeSequenceDFA: 25x speedup for overlapping char class patterns like \w+[0-9]+
  • FindAllIndexCompact: Zero per-match allocations API (60974 → 9 allocations)

Performance

Engine Throughput vs Stdlib
CompositeSearcher (old) 56 MB/s 4.7x
CompositeSequenceDFA (new) 300 MB/s 25x

Implementation

  • NFA subset construction for correct overlap handling
  • Byte class reduction: 256 bytes → 3-8 equivalence classes
  • First-part skip optimization: O(1) check if position can start match
  • Loop unrolling: 4 bytes per iteration (Rust-inspired)

Test plan

  • go test ./... passes
  • golangci-lint run - 0 issues
  • Benchmarks show 25x speedup

Closes #83

Add FindAllIndexCompact/FindAllStringIndexCompact methods that return
[][2]int instead of [][]int, reducing allocations from N to 1 for
N matches.

Also fix per-match allocation in CompositeSearcher.matchAt by
pre-allocating matchLengths slice.

Benchmark results (1MB, 60974 matches):
- FindAllIndex:         48 MB/s, 60974 allocs
- FindAllIndexCompact:  51 MB/s, 9 allocs
- Compact + reuse:      73 MB/s, 0 allocs

Resolves performance concern in Issue #78 investigation.
Implement Rust-style DFA using NFA subset construction for patterns
like \w+[0-9]+ where character classes overlap.

Key improvements:
- Byte class reduction: 256 bytes -> 3-8 equivalence classes
- Subset construction: handles overlapping char classes correctly
- Single-pass matching: no backtracking needed

Benchmark results (1MB, \w+[0-9]+):
Before: 42 MB/s (CompositeSearcher backtracking)
After:  170 MB/s (CompositeSequenceDFA)
Speedup: 4x faster

vs stdlib: 12.9x faster (170 MB/s vs 13 MB/s)
vs Rust:   ~3x slower (170 MB/s vs 500 MB/s)
- Remove //go:nosplit (stack too large)
- Add first-part skip optimization (O(1) start position check)
- Implement 4-byte loop unrolling (Rust-inspired)

Benchmarks (1MB input):
- CompositeSearcher: 56 MB/s
- CompositeSequenceDFA: 260-300 MB/s (4.7-5.5x faster)

Related: #83
- Update CHANGELOG with v0.10.6 release notes
- Update README: CompositeSequenceDFA 25x speedup (was 5x)
- Fix linter issues: nolint for gocognit, gosec, unparam
- Remove unused nfaConfig type
- Fix staticcheck: remove unnecessary nil check
@github-actions

Copy link
Copy Markdown

Benchmark Comparison

Comparing main → PR #84

Summary: geomean 243.6n 243.4n -0.05%

⚠️ Potential regressions detected:

geomean                               ³                +0.00%               ³
geomean                               ³                +0.00%               ³
geomean                         ³                +0.00%               ³
geomean                         ³                +0.00%               ³
AhoCorasickVsStdlib/coregex_Count-4                     1.470µ ± ∞ ¹    1.486µ ± ∞ ¹     +1.09% (p=0.008 n=5)
AhoCorasickManyPatterns/coregex_25_patterns-4           71.80n ± ∞ ¹    80.36n ± ∞ ¹    +11.92% (p=0.008 n=5)
BranchDispatch_Stdlib/Digits-4                          133.5n ± ∞ ¹    135.8n ± ∞ ¹     +1.72% (p=0.016 n=5)
BranchDispatch_Stdlib/NoMatch-4                         77.26n ± ∞ ¹    77.85n ± ∞ ¹     +0.76% (p=0.032 n=5)
AnchoredAlt_ManyBranches_Stdlib/NoMatch-4               85.56n ± ∞ ¹    86.07n ± ∞ ¹     +0.60% (p=0.016 n=5)
FirstByteReject_Stdlib-4                                40.32n ± ∞ ¹    40.74n ± ∞ ¹     +1.04% (p=0.008 n=5)

Full results available in workflow artifacts. CI runners have ~10-20% variance.
For accurate benchmarks, run locally: ./scripts/bench.sh --compare

@kolkov
kolkov merged commit 9cb301e into main Jan 14, 2026
8 checks passed
@kolkov
kolkov deleted the fix/overlap-detection-perf branch January 14, 2026 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: CompositeSequenceDFA - Rust-style optimization for overlapping char classes

1 participant