You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support max_source_rows and max_source_bytes limits #8234
On large datasets compaction has to run incrementally — no single run can afford to
rewrite everything at once. Today the only knob for that is max_source_fragments.
It works well when fragments are similarly sized, but fragment sizes can vary a lot
in practice (a fragment may hold a handful of live rows or millions), so the same
fragment budget may translate into very different amounts of IO and rewrite time
across runs. Row- and byte-based budgets make the per-run workload easier to predict.
What
Add two planning budgets to CompactionOptions that complement max_source_fragments
by bounding a single run on the data volume dimension:
max_source_rows — counted as live rows (physical rows minus soft-deleted rows).
max_source_bytes — total size of the source fragments' data and overlay files,
read from sizes recorded in the manifest (no object-store metadata round trips).
Deletion files and files that only back dropped columns are not counted.
Behavior:
All configured max_source_* limits apply together: tasks are admitted in order
until adding the next task would exceed any one of them.
The limits are hard upper bounds: if the first task already exceeds a budget the
plan is empty and a warning is logged.
A budget of zero is rejected (None means no limit); validation is centralized in CompactionOptions::validate.
Exposed in Rust / Python / Java, plus the manifest config keys lance.compaction.max_source_rows and lance.compaction.max_source_bytes.
Activity
feat(compaction)!: support max_source_rows and max_source_bytes limits (