Now the support fragment.merge_columns to add a new column
import lance
import pyarrow as pa
import pyarrow.compute as pc
table = pa.table({"a": [1, 2, 3, 4], "b": ["a", "b", "c", "d"]})
dataset = lance.write_dataset(table, "example")
dataset.to_table().to_pandas()
def double_a(batch: pa.RecordBatch) -> pa.RecordBatch:
doubled = pc.multiply(batch["a"], 2)
return pa.record_batch([doubled], ["a_doubled"])
fragments = []
for fragment in dataset.get_fragments():
new_fragment, new_schema = fragment.merge_columns(double_a, columns=['a'])
fragments.append(new_fragment)
operation = lance.LanceOperation.Merge(fragments, new_schema)
dataset = lance.LanceDataset.commit("example", operation,
read_version=dataset.version)
dataset.to_table().to_pandas()
However many users want to update the values of the existing column. If we're going to archive this scenario, we should do three steps:
- add a column using the
merge columns API
- delete the old column
- rename the new column into the old one.
Maybe it's better to provide a new API named update_columns to do these in one step.
The update operator in dataset API may not be efficient for the large lance dataset.
Now the support
fragment.merge_columnsto add a new columnHowever many users want to update the values of the existing column. If we're going to archive this scenario, we should do three steps:
merge columnsAPIMaybe it's better to provide a new API named
update_columnsto do these in one step.