btindex: standalone PrefixIndex with adaptive node distribution (40-62% faster)

[awskii]

New PrefixIndex search engine for domain .kv files, offered as an alternative to BpsTree lookups and gated behind ERIGON_USE_PREFIX_INDEX (default off).

Design

• [65536]prefixBucket{firstDI, endDI, nodes} — O(1) 2-byte-prefix lookup replaces the binary search over the pivot list
• up to 8 cached nodes per bucket, evenly spaced, for a second narrowing step
• L1 (256 entries) computed from L2 at build time, used when the exact 2-byte bucket is empty
• exact-match shortcut when a cached node equals the search key

Benchmarks

The earlier "40-62% faster" numbers came from a synthetic 1M-key file, which is the most favourable shape possible: 1M keys over 65536 buckets is ~15 keys per bucket, so the node cache nearly indexes every key individually. Re-measured on an idle 12-core box, warm page cache, main merged in so M=64 and interpolation search are both in the baseline, 3 runs per cell, keys sampled across the whole DI range.

Synthetic, 1M keys: Get -46% (734 -> 393 ns), Seek -52% (878 -> 418 ns).

Real mainnet domain files:

file	keys	Get BpsTree	Get PrefixIndex	Seek BpsTree	Seek PrefixIndex	PrefixIndex build
accounts.8192-8704	45.4M	1351 ns	1208 ns (-10.6%)	1780 ns	1057 ns (-40.6%)	1.23 s vs 2 ms
accounts.0-8192	338.9M	1860 ns	1583 ns (-14.9%)	2278 ns	1596 ns (-29.9%)	8.9 s vs 12 ms
code.8192-8704	9.8M	1123 ns	1098 ns (-2.2%)	1477 ns	1013 ns (-31.4%)	1.93 s vs ~0
storage.8960-9088	35.8M	1983 ns	2904 ns (+46.4%)	2371 ns	2942 ns (+24.1%)	4.2 s vs 2 ms

So: Seek is consistently faster, Get is roughly a wash, and storage regresses on both.

Why storage regresses. Its keys are address+slot, so every slot of one contract falls in a single 2-byte bucket. Probes drawn uniformly over keys land in big buckets in proportion to their size, so the effective window is set by the largest contracts, not by the 546-key mean. Accounts hold one key per address, so occupancy is even and the mean is representative — which is why accounts.0-8192 wins despite a higher mean occupancy (5171 keys/bucket) than storage.

TestNarrowingWindowByDensity measures the mechanism without timing noise. BpsTree's window is bounded by M; PrefixIndex's grows with bucket occupancy:

keys/bucket	BpsTree window	PrefixIndex window
48	62	29
781	62	93

Overlap with the recent .bt work

• Interpolation search (db/datastruct/btindex: interpolation search in BTree leaf #21813) runs in BpsTree.Get only, not Seek. PrefixIndex has neither. That is the asymmetry in every row above: Get is a wash, Seek is where PrefixIndex wins. Applying interpolation to Seek would capture most of that without a second index. Interpolation's stated benefit is cold-read page locality, and all of these numbers are warm.
• M 256 -> 64 (denser btree index for domains #22901) shrank BpsTree's window 4x while PrefixIndex's is unchanged. The files measured above still carry M=256 .bts; rebuilt at M=64 the gap narrows further.

Build cost

NewPrefixIndexWithNodes scans the whole .kv at open — 1-9 s per file warm, a full file read cold — against milliseconds for NewBpsTreeWithNodes, which reads its pivots straight from the .bt. Persisting buckets into the .bt (#21872) is a precondition for this to be usable at all, independent of the search-speed question.

Note also that with the flag on, BtIndex builds both engines; bplus is dead weight for lookups but is what makes the A/B benchmark possible in one process.

Fixes in this branch

• keys shorter than 2 bytes were dropped by record(), so they landed in no bucket and lookup() returned a DI range starting after them: Get reported not-found for keys that exist and Seek skipped past them. Commitment-domain keys are nibble paths and do get this short. TestPrefixIndexMatchesBpsTree is a differential test over mixed-length keys covering both constructors.
• addNode kept the first 8 pre-built nodes per bucket rather than evenly spaced ones, so a busy bucket's tail had nothing to narrow against. Fixing this moved accounts.0-8192 Get from -2.6% to -14.9% and storage from +72% to +46.4%.
• two redundant full .kv scans removed from NewPrefixIndexWithNodes.
• benchmarks wrote the .kv uncompressed and read it back through the compressed getter, so the file carried no pattern dictionary and the measured path was decompression-free.

What would actually compose

Keep all pivots, bucketed, so the window stays bounded by M like BpsTree, and put the O(1) bucket lookup in front of it — rather than a fixed 8-node cache whose quality degrades with bucket skew. Combined with persisting the buckets in the .bt, that keeps the Seek win without the build cost or the storage regression.

Co-Authored-By: Shuo shuo@erigon.dev

[awskii]

#21872 could encode nodes from prefix tree to avoid it during startup

[AskAlexSharov]

#21872 could encode nodes from prefix tree to avoid it during startup

Startup optimization is important. I guess it can be 3.8 headliner.

Currently i focusing on RAM usage reduction - because bloatnet did use 90% of RAM on 64G machine (now it using 20GB: 4gb domain existence, 1.3gb bt, 0.5gb compress dicts, 2gb exec. Everything else is: peaks during merge build of rs/bt/kv etc...)

FYI:

3.5G /erigon-data/snapshots/domain/v1.1-storage.928000-936000.bt
zstd shows compression ratio 4x

If you have some advise "what we can add into .bt file disk format - i can add it now. (don't understand from your message).

[Copilot]

Pull request overview

This PR introduces a new PrefixIndex search engine for domain .kv files (as a faster alternative to BpsTree lookups) and wires it into BtIndex behind an environment flag.

Changes:

• Add standalone PrefixIndex implementation with per-prefix buckets, cached nodes, and MatchCmp-based comparisons.
• Integrate PrefixIndex into BtIndex (conditional selection for Get/Seek/stats methods).
• Add extensive correctness tests and new benchmarks comparing PrefixIndex vs BpsTree.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
docs/plans/20260415-prefixindex-matchcmp-integration.md	Implementation plan + benchmark notes for MatchCmp integration.
db/datastruct/btindex/testhelpers_test.go	Adds controlled/minimal .kv generators and helpers used by new tests.
db/datastruct/btindex/prefix_index.go	New PrefixIndex implementation (bucket ranges, node narrowing, MatchCmp compare).
db/datastruct/btindex/prefix_index_test.go	Large correctness/concurrency suite for PrefixIndex + equivalence checks vs BpsTree.
db/datastruct/btindex/btree_index.go	Adds optional PrefixIndex search engine to BtIndex and routes calls when enabled.
db/datastruct/btindex/bpstree_bench_test.go	Adds benchmarks for PrefixIndex and comparison benchmarks.
common/dbg/dbg_env.go	Adds env-backed dbg.UsePrefixIndex toggle.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (5)

db/datastruct/btindex/prefix_index_test.go:1794

• BtIndexWriter keeps a pivot node for di==0 as well (see btree_index.go:264-266). Excluding di==0 here means the test doesn’t cover PrefixIndexWithNodes behavior with the first pivot present.

		if di > 0 && di%M == 0 {

db/datastruct/btindex/prefix_index_test.go:751

• BtIndexWriter keeps a pivot node for di==0 as well (see btree_index.go:264-266). Excluding di==0 here makes this test’s node slice diverge from the actual .bt layout.

		if di > 0 && di%M == 0 {

db/datastruct/btindex/prefix_index_test.go:100

• BtIndexWriter keeps a pivot node for di==0 as well (see btree_index.go:264-266). Dropping di==0 here means the "with nodes" path isn’t exercising the real on-disk layout and may miss edge cases around the first pivot.

This issue also appears in the following locations of the same file:

• line 751
• line 1794

		if di > 0 && di%M == 0 {

db/datastruct/btindex/prefix_index_test.go:1539

• This test uses t.Skip when the chosen gapKey overshoots the next key. That makes the test outcome data-dependent and can silently drop coverage. Prefer selecting a pair that satisfies the precondition (gap < next) and assert if none exists.

	mid := len(keys) / 2
	gap := gapKey(keys[mid])
	if bytes.Compare(gap, keys[mid+1]) >= 0 {
		// Gap overshot, skip this variant.
		t.Skip("gap key overshot next key")

db/datastruct/btindex/bpstree_bench_test.go:104

• BenchmarkPrefixIndexSeek reads a key from getter and then passes that slice into Seek using the same reader. seg.Reader.Next returns slices backed by the reader’s internal buffer, and Seek will reset/read via the same reader, so the search key can be overwritten mid-call and the probe stream is disrupted. Use a separate probe reader and copy the key into a stable buffer before calling Seek (no per-iteration alloc needed).

		key, _ = getter.Next(key[:0])
		getter.Skip()
		c, err := bt.search.Seek(getter, key)
		require.NoError(t, err)

[awskii]

closed in favour of #23841