Tracking: Stability and Performance Optimization of Trillion-row Lance Tables #7968

Background & Goals

Targeting tables at the scale of one trillion rows / millions of fragments, this effort profiles Lance's stability and performance across metadata / read / write / index, and drives the corresponding optimizations to land.

This issue serves as the umbrella entry point, tracking the known problems and optimization tasks per module.

Test Dataset

Module 1: Lance-Core (Kernel)

Continuously being expanded.

Metadata

• Oversized manifest / inline-transaction redundancy: the manifest inlines the transaction by default, so under heavy-write workloads the fragment info is stored three times (once in the manifest struct + once in the inline transaction + once in the standalone _transactions/*.txn), ~1.6 GB per version. Large transactions are inlined into the manifest, doubling manifest size on large tables #7961
• O(n*m) fragment compares in build_manifest for Update/Delete O(n*m) fragment compares in build_manifest for Update/Delete #8209

Write

• Session sharing in the Fragment API: sharing currently only exists at the dataset level; Fragment-level operations cannot reuse it, which amplifies overhead when the many small operations. Support fragment session cache #7974

Read

• The Java blocking_dataset exposes more metadata information, such as Fragments statistics, to optimize the Read Plan process for engines like Spark. Add getFragmentStatistics() to the Java blocking_dataset API to expose per-fragment statistics #8071
• (TBD: profiling and optimization items for point lookups / range scans / projec reads at the hundred-billion scale)

Index

• Adapting zonemap-based partitioning to Compaction: Compaction may break the precondition that "cluster values are unique within a fragment," causing zonemap partition pruning to become ineffective — an adaptation scheme is needed.
• Bloom filter BinaryArray i32 offset overflow: i32 offset overflows on large BinaryArrays. bound read/write to prevent BinaryArray i32 offset overflow #7973
• zonemap / bloomfilter index parallelize zone statistics computation in scalar index training #8468

Compaction

• Compaction plan performance perf: CandidateBin.split_for_size is too slow for hundreds of thousands fragments #7967
• Compaction skip collect rowids to avoid oom during compaction commit perf(compaction)!: skip building row-address maps when index remapping is not needed #7778
• Support max_source_rows and max_source_bytes limits Support max_source_rows and max_source_bytes limits #8234

Module 2: Lance-Spark (Connector)

• Hot-read on the manifest during Append: the schema-fetch path triggers hot the large manifest.
• Hot-read on the manifest during Update / Add Columns / Compaction: the data mechanism causes repeated reads of the large manifest (DatasetCache vs. Session Share design choice).
• Driver OOM on Compaction commit: aggregating row id maps on the driver causes OOM.
• Write task count is not controllable: RequiresDistributionAndOrdering doe implement requiredNumPartitions(), so write parallelism can only be worked around indirectly via AQE parameters and cannot be specified directly, which easily leads oversized per-task buffering → off-heap OOM.
• The split plan takes more than two minutes under scenarios with large data volumes and zonemap indexes in Lance Spark.