Skip to content

feat: add support for _partition metadata column - #2668

Merged
CTTY merged 6 commits into
apache:mainfrom
parthchandra:metadata-columns
Jul 29, 2026
Merged

feat: add support for _partition metadata column#2668
CTTY merged 6 commits into
apache:mainfrom
parthchandra:metadata-columns

Conversation

@parthchandra

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

Implements the _partition metadata column for table scans. This is a struct column whose type is the union of all partition fields across all partition specs (handling partition evolution). Each row gets the
partition values for its data file.

  • Adds compute_unified_partition_type() to compute the union of partition fields across all specs (equivalent to Java's Partitioning.partitionType())
  • Adds PartitionColumnConstant and build_partition_column_constant() for pre-computing the struct values per file
  • Adds ColumnSource::AddStructConstant variant to RecordBatchTransformer for materializing struct columns
  • Threads the unified partition type through scan planning and populates the constant in FileScanTask
  • Pipeline detects RESERVED_FIELD_ID_PARTITION in projected fields and injects the struct constant

Are these changes tested?

Because we do not have write support yet, I made the corresponding change to comet and then tested by adding tests in Comet which uses iceberg-java to write files and then iceberg-rust to read them back.
https://github.com/parthchandra/datafusion-comet/blob/iceberg-metadata-columns/spark/src/test/resources/sql-tests/iceberg/metadata_column_partition.sql

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for projecting metadata columns _pos, _spec_id, and _partition in table scan

5 participants