Skip to content

feat: support update_columns for fragment operator #3539

Description

@SaintBacchus

Now the support fragment.merge_columns to add a new column

import lance
import pyarrow as pa
import pyarrow.compute as pc
table = pa.table({"a": [1, 2, 3, 4], "b": ["a", "b", "c", "d"]})
dataset = lance.write_dataset(table, "example")

dataset.to_table().to_pandas()

def double_a(batch: pa.RecordBatch) -> pa.RecordBatch:
    doubled = pc.multiply(batch["a"], 2)
    return pa.record_batch([doubled], ["a_doubled"])

fragments = []
for fragment in dataset.get_fragments():
    new_fragment, new_schema = fragment.merge_columns(double_a, columns=['a'])
    fragments.append(new_fragment)

operation = lance.LanceOperation.Merge(fragments, new_schema)
dataset = lance.LanceDataset.commit("example", operation,
                                    read_version=dataset.version)
dataset.to_table().to_pandas()

However many users want to update the values of the existing column. If we're going to archive this scenario, we should do three steps:

  1. add a column using the merge columns API
  2. delete the old column
  3. rename the new column into the old one.

Maybe it's better to provide a new API named update_columns to do these in one step.

The update operator in dataset API may not be efficient for the large lance dataset.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions