seg: 2-3x faster merge compression via Aho-Corasick matcher and cover-DP optimizations

[sudeepdino008]

   words
     │
     ├──────────────────────────► [raw .idt file]   (every word, verbatim)
     │                                   │
     │  Phase 1          Phase 2         │  re-read in Phase 3
     └► mine patterns ─► reduce to ──► dictionary
                         top-K            │
                                  Phase 3 ▼
                          cover each word: patterns + leftover gaps
                                          │
                                  Phase 4 ▼
                          Huffman-code the patterns & positions
                                          │
                                  Phase 5 ▼
                          bit-pack everything ─────► [.seg file]

Merging storage-domain snapshot files is single-threaded and compression-bound. This PR replaces the per-word pattern matcher and optimizes the cover DP; output is byte-identical (or marginally smaller, see notes).

Benchmarked on real storage files through DomainRoTx.mergeFiles (incl. .bt/.kvei build), Workers=1 as at chain-tip, merged content verified against an independent re-merge:

workload	main	this PR
bloatnet 16-step merge (766M in, 13.8M keys)	382.6s	135.0s (2.8x)
mainnet 6-file merge (1.61G in, 35.3M keys)	1204.2s	612.9s (2.0x)

Where the speedup comes from

CPU profile of a representative single-worker bloatnet 16-step merge, attributing the cover phase by sub-component:

phase	main	this PR
cover phase (matcher + DP)	241.2s	14.6s	16.5x
— matcher (FindLongestMatches)	186.2s	2.7s	~68x
— cover DP (excl. matcher)	55.0s	11.9s	~4.6x
SAIS dictionary extraction	89.2s	87.5s	untouched
.bt + I/O + gc	~19s	~17s	untouched
total	349.5s	119.5s	~2.9x

The matcher carries ~80% of the win, the cover-DP rewrite ~19%. End-to-end speedup is Amdahl-capped near 3x: SAIS dictionary extraction is untouched and is now ~79% of remaining merge time — parallelizing it (size-neutral) is the next lever (see Notes).

The matcher's responsibility is - given the 64K limit candidate pattern dictionary, run it through each word and find which pattern matches the word. The cover DP finds optimal subset of these matches which provides best compression.

Changes

• seg/patricia: Aho–Corasick matcher replaces per-word suffix-array matching (SAIS+LCP+bit-level patricia walk); same maximal-match set in one O(len) scan, resuming from the shared prefix with the previous word (merge keys are sorted)
• seg: coverWordByPatterns DP — single-match fast path; flat-slice deque replacing Ring; upper-bound skip; monotone-bound early exit with binary-search truncation; virtual initial window cells; position-code maps → arrays

Verification

• bloatnet output is byte-identical to main; mainnet output is 0.01% smaller — mainnet's dictionary contains prefix-nested patterns that the old matchers lose to a pre-existing patricia bug (patricia: Insert loses an existing key when its proper prefix is inserted later #21626) and the AC matcher finds; content verified equal in all runs
• FuzzLongestMatch extended with a brute-force oracle for the AC matcher (3.5M+ execs clean)
• make lint clean; db/seg, db/seg/patricia, db/state tests pass

Notes

• Remaining merge cost is dominated by dictionary learning (SAIS extraction), untouched here; sampling it regresses mainnet sizes (db/state: raise domain compression SamplingFactor to 4 #21639, closed) — parallelizing extraction is the size-neutral follow-up
• The earlier parallel existence-filter scan commit was dropped after review: a second concurrent cursor can fight experiment: merge on own mmap with madv_sequential #21482's sequential-view readahead on cold files
• Aho-Corasick can be parallelized by supplying ranges of words to each compression worker, rather than feed consecutive words to different workers -- when there is common prefix, AC can "resume" from previous word with common prefix, thereby doing less work. Supplying ranges of words exploit this. It'll be done in separate PR.

[sudeepdino008]

no idea what this all means; but results are good. Will check why in coming days

[sudeepdino008]

Updated: the original SamplingFactor 1→4 commit was unknowingly relying on a pre-existing bug — with SamplingFactor>1 only the first 16MB superstring was ever sampled (#21628). The branch now fixes the skip-phase accounting so SF=4 truly samples 25%, and keeps SF=4 for domains: 2.9x faster than full analysis, output within ±1%. Headline moves from 12.8x to an honest 8.5x (124s → 188s full merge; the accidental first-16MB dictionary was faster but head-biased on sorted keys).

[AskAlexSharov]

I advise separate PR:

• sampling change 1->4
• everything else
  Because first change is trivial and likely will give big speedup and we using sampling 4 on most of other files

[sudeepdino008]

Done — split per your advice:

• db/state: raise domain compression SamplingFactor to 4 #21639: SamplingFactor 1→4 one-liner (-30% alone; consistent with DefaultCfg's actual behavior)
• this PR: matcher/DP/btindex, all byte-identical output (382.6s → 129.0s alone; 25.4s with db/state: raise domain compression SamplingFactor to 4 #21639)
• the sampling skip-phase fix moved out: turns out SamplingFactor>1 only ever samples the first 16MB (seg: SamplingFactor>1 builds the pattern dictionary from the first 16MB only #21628) — fixing that changes dictionaries for all DefaultCfg users, so it needs its own measured PR

[sudeepdino008]

I advise separate PR:

• sampling change 1->4
• everything else
  Because first change is trivial and likely will give big speedup and we using sampling 4 on most of other files

sampling factor -> 4
it's a bug in the implementation - for sampling>1, it checks only first 16MB (doesn't scan the whole file). Interestingly, this leads to smaller file size (maybe because of smaller dictionary and resulting huffman encoding etc.).

21628: fix for doing the samplerate scan over file (right now it just considers the first 16mb superstring).

[sudeepdino008]

Mainnet validation (storage merge 8960→9085, 1.61 GiB in, 35.3M keys, inputs copied out of a live datadir, content verified in all runs):

	time	output
main (SF=1)	1204.2s	1,505,103,806 B
this PR (SF=1)	758.1s (1.6x)	1,504,946,332 B (-0.01%)

Two notes vs the bloatnet results:

• Speedup is 1.6x rather than 3x: at SF=1 the dictionary-learning scan (untouched here) is a bigger share on mainnet, and high-entropy mainnet data has fewer matches per word, so the matcher/DP were a smaller fraction to begin with.
• Output is not bit-identical on mainnet (it is on bloatnet): mainnet's dictionary contains prefix-nested patterns, which the old matchers lose to the patricia bug (patricia: Insert loses an existing key when its proper prefix is inserted later #21626) — the AC matcher finds them, giving marginally better compression. Content is verified equal.

[Copilot]

Pull request overview

This PR focuses on significantly speeding up storage-domain snapshot merges (single-threaded, compression-bound) while preserving byte-identical compressed output. It replaces the previous per-word suffix-array/patricia-based matching with an Aho–Corasick automaton and applies multiple DP/encoding hot-path optimizations, plus moves btindex existence-filter hashing off the critical path.

Changes:

• Add a new patricia Aho–Corasick automaton + per-goroutine matcher with prefix-resume caching for sorted-key streams.
• Optimize coverWordByPatterns (DP) and position-frequency accounting to reduce allocations and hot-path map lookups.
• Parallelize .kvei existence-filter hashing by scanning via a cloned seg.Reader in btindex.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
db/seg/seg_auto_rw.go	Adds Reader.CloneReader() to create an independent reader over the same decompressor for concurrent scans.
db/seg/patricia/testdata/fuzz/FuzzLongestMatch/34174f1279b62c1e	Adds fuzz corpus entry covering prefix-nested patterns (regression reproducer).
db/seg/patricia/patricia_fuzz_test.go	Extends fuzzing to validate AC matcher vs a brute-force oracle and checks warm/fresh matcher consistency.
db/seg/patricia/aho_corasick.go	Introduces the Aho–Corasick automaton + ACMatcher implementation with fail links and longest-match propagation.
db/seg/parallel_compress.go	Switches compressor matching to AC, adds DP fast-path and deque/pos-count optimizations, and reduces map pressure in position-code lookup.
db/datastruct/btindex/btree_index.go	Moves existence-filter hashing to a separate goroutine using a cloned reader to avoid serializing BT key-walk behind hashing.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[AskAlexSharov]

Let’s merge it after release/3.5 branch creation