btindex: add M to file, reduce RAM usage of Build()

[AskAlexSharov]

based on: #21777

• .bt file format change
• backwards compatible

In PR:

• reduce .Build() RAM usage by building ef_offsets eagerly off-heap and streaming nodes straight to the file (no big in-mem structs)
• Add M inside file metadata
• Removed di from file (node i has di = i*M, recomputed on read)
• metadata stored in Footer of file
• only the EF section is 4kb-aligned (for mmap/SIMD-friendly reads); nodes are byte-parsed, so left unaligned; footer is 8-byte aligned
• completely removed ETL usage: because now AddKey() can write directly to file, also Footer written at the end (when all required fields calculated).
• but keep 1 leading byte - just for legacy format detection

---

I realized that for building .bt we don't really need ETL (keys already always sorted, keys_count we know from decompressor, etc...). And I realized more - maybe other places in Erigon also can be more streaming-friendly (can build without ETL - in 1-pass).

To generalize idea we need move "header metadata" to "footer" - then if metadata storing keys_count we can write it at the end:

• Store metadata at Footer: wins at write-once by append-only use-cases.
• Store metadata at Header: wins at forward-stream consumers (sockets, tar to tape), mutable page-addressed files (SQLite, Postgres, InnoDB — when mutate/extend file Footer will change its offset).

Footer style file .bt:

[ leading_byte:u8 ]                                                      # 1 byte: legacy-format detection
[ nodes: (key_len:u16 | key)* ]                                          # variable length. unaligned (byte-parsed)
[ EF: elias-fano of all key offsets ]                                    # variable length. 4kb-aligned
[ footer: keys_count:u64 | M:u64 | ef_offset:u64 ]                       # 24 bytes. 8-byte-aligned
[ ANCHOR: footer_len:u32 | flags:u16 | format_version:u16 | magic:u64 ]  # 16 bytes. Fixed length

Build such files will be much more streaming-style-friendly. Don't need etl all incoming keys only to calculate keys_count (like we do now in .bt).

On file open:

• Validate the magic first (right format + not truncated) -> fail-fast. Not u16 - because probability of collision is high.
• format_version is next to the magic - then it will allow change ANCHOR format/len in the future.

(yes, exactly in our case: EF inside .bt requires to know key_count in advance - but we already have it - and maybe in future we can change it).

Also Footer allowing easy to add checksum to metadata fields in future, etc...

[Copilot]

Pull request overview

This PR updates the BtIndex on-disk format and build process to avoid mainnet-scale OOMs during B-tree index construction by (1) moving Elias–Fano (EF) build buffers off-heap and (2) streaming node records directly to the output instead of accumulating them in memory.

Changes:

• Switch EF construction in BtIndexWriter.Build() to eliasfano32.NewEliasFanoOffHeap(...) (mmap-backed temp file) to eliminate large Go-heap spikes.
• Change the index file layout to v1 ([version][nodesCount][nodes...][EF]) so node entries can be streamed during ETL Load.
• Update index open logic to support both v0 and v1 layouts, and tighten the test assertion to catch regressions where too many nodes are stored.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
db/datastruct/btindex/btree_index.go	Writes v1 header + streams nodes during Build; reads v0/v1 formats on open; uses off-heap EF builder.
db/datastruct/btindex/btree_index_test.go	Adds an upper bound assertion for the number of cached nodes.
db/datastruct/btindex/bps_tree.go	Removes node-list encoding helper; updates node decoding helper to return bytes consumed.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 21 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

db/datastruct/btindex/btree_index.go:40

• btindex no longer uses ETL collectors, but it still depends on db/etl solely for etl.BufIOSize. If the intent is to remove ETL coupling here, consider defining a local buffer size constant in btindex (or moving BufIOSize to a smaller shared package) so this package doesn’t need to import etl just for one value.

	"github.com/erigontech/erigon/common/dir"
	"github.com/erigontech/erigon/common/log/v3"
	"github.com/erigontech/erigon/common/murmur3"
	"github.com/erigontech/erigon/db/datastruct/existence"
	"github.com/erigontech/erigon/db/etl"
	"github.com/erigontech/erigon/db/recsplit/eliasfano32"
	"github.com/erigontech/erigon/db/seg"

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.