re.MatchString Benchmark cost too much time

the benchmark cost 116.923s to finish, there is maybe a issue here:

func BenchmarkMatchStringCache_WithoutCache_coregex(b *testing.B) {
	runtime.GOMAXPROCS(4)

	var data = strings.Repeat("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36", 8)
	var re = coregex.MustCompile(`(?iU)\b(eval|system|exec|execute|passthru|shell_exec|phpinfo)\b`)

	b.ReportAllocs()
	b.ResetTimer()

	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			_ = re.MatchString(data)
		}
	})
}

result:

goos: darwin
goarch: arm64
pkg: [MY REPOSITORY]/internal/waf/utils
cpu: Apple M2 Max
BenchmarkMatchStringCache_WithoutCache_coregex-12            100        1148587620 ns/op        1737521223 B/op     1445 allocs/op
testing: BenchmarkMatchStringCache_WithoutCache_coregex-12 left GOMAXPROCS set to 4
PASS
ok      github.com/[MY REPOSITORY]/internal/waf/utils        116.923s

and with go stdlib regexp result:

goos: darwin
goarch: arm64
pkg: [MY REPOSITORY]/internal/waf/utils
cpu: Apple M2 Max
BenchmarkMatchStringCache_WithoutCache_regexp
testing: BenchmarkMatchStringCache_WithoutCache_regexp-12 left GOMAXPROCS set to 4
BenchmarkMatchStringCache_WithoutCache_regexp-12    	   91040	     12997 ns/op	       0 B/op	       0 allocs/op
PASS

Thank you for the report! This is a confirmed critical performance issue.

Root Cause

The problem is in our literal extractor — it currently rejects ALL case-insensitive patterns entirely (a blanket guard added for #87). Without extracted literals, the meta-engine has no prefilter candidates and falls back to lazy DFA, which suffers from state space explosion on (?i) patterns (181 NFA states → massive DFA cache thrashing).

For comparison, Rust's regex crate handles this pattern by:

Expanding case-insensitive literals into all folding variants (eval → EVAL, EVAl, EVaL, ... eval = 16 variants)
Trimming to 3-byte prefixes (60 total)
Using Teddy SIMD prefilter to scan — finds 0 candidates in your input → returns immediately in ~263ns

Our engine should do the same but currently skips step 1, so Teddy is never engaged.

Fix Plan

Literal extractor: Support case-folding expansion for (?i) alternation literals (like Rust does)
Strategy: Route expanded literals to Teddy/Aho-Corasick prefilter
Safety net: Add fallback from DFA to PikeVM for large NFAs without prefilter — prevents cache thrashing even when literal extraction isn't possible

Workaround (until fix is released)

You can manually expand the case variations and drop (?i):

// Instead of:
re := coregex.MustCompile(`(?iU)\b(eval|system|exec|execute|passthru|shell_exec|phpinfo)\b`)

// Use case-sensitive with explicit variations:
re := coregex.MustCompile(`\b(?i:eval|system|exec|execute|passthru|shell_exec|phpinfo)\b`)

If that doesn't help, the fastest workaround is to list lowercase only (if your input is predictable):

re := coregex.MustCompile(`\b(eval|system|exec|execute|passthru|shell_exec|phpinfo)\b`)

This routes to Teddy strategy (~200ns/op) since literals are extracted correctly without (?i).

We'll prioritize the fix. Thank you for the detailed benchmark — the RunParallel with GOMAXPROCS(4) test is especially valuable as it exposes DFA cache contention under concurrency.

feat: case-insensitive literal extraction with prefilter (Issue #137)

v0.12.13 addresses the match-case performance:

isMatchDFA now uses prefilter candidate loop with anchored DFA verification instead of full NFA scan. Your benchmark pattern on 1KB with match: 176μs → 27μs.

Aho-Corasick upgraded to v0.2.1 (DFA backend, 11-22x throughput).

FatTeddy AVX2 correctness fix (ANDL→ORL for 33-64 pattern lane combining).

Please try v0.12.13 — both match and no-match cases should be significantly faster than stdlib now.

Thank you, but sadly on my Apple M2 Max the match case became extremely slow with v0.12.13:
go.mod:

github.com/coregx/coregex v0.12.13 // indirect

benchmark:

func BenchmarkMatchStringCache_WithoutCache_matched_coregex(b *testing.B) {
	runtime.GOMAXPROCS(4)

	var data = strings.Repeat("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36 phpinfo", 8)
	var regex = coregex.MustCompile(`(?iU)\b(eval|system|exec|execute|passthru|shell_exec|phpinfo)\b`)

	b.ReportAllocs()
	b.ResetTimer()

	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			_ = regex.MatchString(data)
		}
	})
}

result:

goos: darwin
goarch: arm64
pkg: .../internal/waf/utils
cpu: Apple M2 Max
BenchmarkMatchStringCache_WithoutCache_matched_coregex
BenchmarkMatchStringCache_WithoutCache_matched_coregex-12    	     100	1016610276 ns/op	1737425287 B/op	     115 allocs/op
testing: BenchmarkMatchStringCache_WithoutCache_matched_coregex-12 left GOMAXPROCS set to 4
PASS

PS go version: go1.26.1 darwin/arm64

fix: isMatchDFA concurrent safety — prefilter rejection only (#137)

Merge pull request #145 from coregx/hotfix/concurrent-ismatch-safety

Fixed in v0.12.14.

Root cause: the prefilter candidate loop in isMatchDFA called e.dfa.SearchAtAnchored concurrently — shared lazy DFA is not thread-safe for concurrent state construction. On ARM64 without SIMD prefilters, every candidate triggered DFA lazy state build → cache corruption under RunParallel.

Fix: prefilter is now used for fast rejection only (single Find call). When a candidate is found, verification uses pooled PikeVM (thread-safe) instead of shared DFA. Also added TestConcurrentCaseInsensitivePrefilter (8 goroutines × 100 iterations) to catch this class of races.

go get github.com/coregx/coregex@v0.12.14

Thanks to your work, with 0.12.14, coregex performance under match condition is near with go regexp.

@tjbrains

@tjbrains Thanks! We'll fix it tomorrow; it should be noticeably closer to the Rust implementation. I'll let you know.

fix: restore FatTeddy for 33-64 patterns, AC only for >64

perf: per-goroutine DFA cache + AC prefilter + integrated loop (#146)

@tjbrains

@tjbrains v0.12.15 is released with major improvements for your use case:

Per-goroutine DFA cache (Rust approach) — each goroutine gets its own DFA cache via sync.Pool, eliminating all concurrent data races. No more shared mutable state.

Case	v0.12.14	v0.12.15	Improvement
IsMatch match 33B	1.69 us	0.44 us	3.8x faster
IsMatch no-match 33B	1.21 us	0.07 us	18x faster
Concurrent (8 goroutines)	data race fixed	0.45 us/op	stable

This was a large release with significant internal changes (DFA architecture split, 7 correctness fixes, integrated prefilter loop, strategy DFA caches). Please validate thoroughly in your environment — we've added a stdlib compatibility test (38/38 patterns match regexp) and ran full regex-bench CI (Benchmark + LangArena + DNA + Extreme) with no regressions, but with this many changes extra testing on your side would be appreciated.

go get github.com/coregx/coregex@v0.12.15

@tjbrains v0.12.15 is released with major improvements for your use case:

Per-goroutine DFA cache (Rust approach) — each goroutine gets its own DFA cache via sync.Pool, eliminating all concurrent data races. No more shared mutable state.

Results on your pattern (?iU)\b(eval|system|exec|execute|passthru|shell_exec|phpinfo)\b:

Case v0.12.14 v0.12.15 Improvement
IsMatch match 33B 1.69 us 0.44 us 3.8x faster
IsMatch no-match 33B 1.21 us 0.07 us 18x faster
Concurrent (8 goroutines) data race fixed 0.45 us/op stable
This was a large release with significant internal changes (DFA architecture split, 7 correctness fixes, integrated prefilter loop, strategy DFA caches). Please validate thoroughly in your environment — we've added a stdlib compatibility test (38/38 patterns match regexp) and ran full regex-bench CI (Benchmark + LangArena + DNA + Extreme) with no regressions, but with this many changes extra testing on your side would be appreciated.
go get github.com/coregx/coregex@v0.12.15

Amazing! I will do more test on coregex in my projects.

@tjbrains

@tjbrains Quick follow-up — v0.12.16 just released with a fix for a multiline pattern regression found on ARM64. If you use (?m)^ patterns, please update:

go get github.com/coregx/coregex@v0.12.16

@tjbrains

@tjbrains Quick follow-up — v0.12.16 just released with a fix for a multiline pattern regression found on ARM64. If you use (?m)^ patterns, please update:
go get github.com/coregx/coregex@v0.12.16

Got it!

Navigation Menu

Repository navigation

re.MatchString Benchmark cost too much time #137

Root Cause

Fix Plan

Workaround (until fix is released)

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

re.MatchString Benchmark cost too much time #137

Description

Activity

tjbrains commented on Mar 17, 2026

kolkov commented on Mar 17, 2026

Root Cause

Fix Plan

Workaround (until fix is released)

7 remaining items

tjbrains commented on Mar 19, 2026

tjbrains commented on Mar 19, 2026

kolkov commented on Mar 19, 2026

tjbrains commented on Mar 19, 2026

kolkov commented on Mar 19, 2026

kolkov commented on Mar 21, 2026

tjbrains commented on Mar 21, 2026

kolkov commented on Mar 21, 2026

tjbrains commented on Mar 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions