[performance] cherry-pick: seg: 2-3x faster merge compression via Aho-Corasick matcher and cover-DP optimizations (#21625) - #21946
Merged
Conversation
…-DP optimizations (#21625) ``` words │ ├──────────────────────────► [raw .idt file] (every word, verbatim) │ │ │ Phase 1 Phase 2 │ re-read in Phase 3 └► mine patterns ─► reduce to ──► dictionary top-K │ Phase 3 ▼ cover each word: patterns + leftover gaps │ Phase 4 ▼ Huffman-code the patterns & positions │ Phase 5 ▼ bit-pack everything ─────► [.seg file] ``` Merging storage-domain snapshot files is single-threaded and compression-bound. This PR replaces the per-word pattern matcher and optimizes the cover DP; output is byte-identical (or marginally smaller, see notes). Benchmarked on real storage files through `DomainRoTx.mergeFiles` (incl. `.bt`/`.kvei` build), `Workers=1` as at chain-tip, merged content verified against an independent re-merge: | workload | main | this PR | |---|---|---| | bloatnet 16-step merge (766M in, 13.8M keys) | 382.6s | **135.0s (2.8x)** | | mainnet 6-file merge (1.61G in, 35.3M keys) | 1204.2s | **612.9s (2.0x)** | ## Where the speedup comes from CPU profile of a representative single-worker bloatnet 16-step merge, attributing the cover phase by sub-component: | phase | main | this PR | | |---|---|---|---| | cover phase (matcher + DP) | 241.2s | 14.6s | **16.5x** | | — matcher (`FindLongestMatches`) | 186.2s | 2.7s | ~68x | | — cover DP (excl. matcher) | 55.0s | 11.9s | ~4.6x | | SAIS dictionary extraction | 89.2s | 87.5s | untouched | | `.bt` + I/O + gc | ~19s | ~17s | untouched | | **total** | **349.5s** | **119.5s** | **~2.9x** | The matcher carries ~80% of the win, the cover-DP rewrite ~19%. End-to-end speedup is Amdahl-capped near 3x: SAIS dictionary extraction is untouched and is now ~79% of remaining merge time — parallelizing it (size-neutral) is the next lever (see Notes). The matcher's responsibility is - given the 64K limit candidate pattern dictionary, run it through each word and find which pattern matches the word. The cover DP finds optimal subset of these matches which provides best compression. ## Changes - `seg/patricia`: Aho–Corasick matcher replaces per-word suffix-array matching (SAIS+LCP+bit-level patricia walk); same maximal-match set in one O(len) scan, resuming from the shared prefix with the previous word (merge keys are sorted) - `seg`: `coverWordByPatterns` DP — single-match fast path; flat-slice deque replacing `Ring`; upper-bound skip; monotone-bound early exit with binary-search truncation; virtual initial window cells; position-code maps → arrays ## Verification - bloatnet output is byte-identical to main; mainnet output is 0.01% smaller — mainnet's dictionary contains prefix-nested patterns that the old matchers lose to a pre-existing patricia bug (#21626) and the AC matcher finds; content verified equal in all runs - `FuzzLongestMatch` extended with a brute-force oracle for the AC matcher (3.5M+ execs clean) - `make lint` clean; `db/seg`, `db/seg/patricia`, `db/state` tests pass ## Notes - Remaining merge cost is dominated by dictionary learning (SAIS extraction), untouched here; sampling it regresses mainnet sizes (#21639, closed) — parallelizing extraction is the size-neutral follow-up - The earlier parallel existence-filter scan commit was dropped after review: a second concurrent cursor can fight #21482's sequential-view readahead on cold files - Aho-Corasick can be parallelized by supplying ranges of words to each compression worker, rather than feed consecutive words to different workers -- when there is common prefix, AC can "resume" from previous word with common prefix, thereby doing less work. Supplying ranges of words exploit this. It'll be done in separate PR. --------- Co-authored-by: Sudeep Kumar <sudeep.kumar@erigon.tech> Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
AskAlexSharov
approved these changes
Jun 23, 2026
…rformance # Conflicts: # db/seg/patricia/patricia_fuzz_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #21625 to
performance.Replaces the per-word pattern matcher (Aho-Corasick) and optimizes the cover DP in
db/segmerge compression. Output byte-identical (or marginally smaller). Clean cherry-pick, noperformance-specific adaptations.Cover phase (matcher + DP) 241.2s → 14.6s (16.5x); SAIS dictionary extraction untouched.