Skip to content

Support max_source_rows and max_source_bytes limits #8234

Open

Description

@zhangyue19921010

Why

On large datasets compaction has to run incrementally — no single run can afford to
rewrite everything at once. Today the only knob for that is max_source_fragments.
It works well when fragments are similarly sized, but fragment sizes can vary a lot
in practice (a fragment may hold a handful of live rows or millions), so the same
fragment budget may translate into very different amounts of IO and rewrite time
across runs. Row- and byte-based budgets make the per-run workload easier to predict.

What

Add two planning budgets to CompactionOptions that complement max_source_fragments
by bounding a single run on the data volume dimension:

  • max_source_rows — counted as live rows (physical rows minus soft-deleted rows).
  • max_source_bytes — total size of the source fragments' data and overlay files,
    read from sizes recorded in the manifest (no object-store metadata round trips).
    Deletion files and files that only back dropped columns are not counted.

Behavior:

  • All configured max_source_* limits apply together: tasks are admitted in order
    until adding the next task would exceed any one of them.
  • The limits are hard upper bounds: if the first task already exceeds a budget the
    plan is empty and a warning is logged.
  • A budget of zero is rejected (None means no limit); validation is centralized in
    CompactionOptions::validate.

Exposed in Rust / Python / Java, plus the manifest config keys
lance.compaction.max_source_rows and lance.compaction.max_source_bytes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions