What is the problem the feature request solves?
Iceberg's metadata columns are currently not supported by Comet. A common use case for Iceberg (merge-on-read, copy-on-write) requires reading the metadata columns _file, _pos, _spec, and _partition. Iceberg scans automatically introduce these columns if required. Because these are not supported, these jobs simply fall back to Spark.
Describe the potential solution
Add support for these columns.
Additional context
_file, _pos, and, _spec are already supported by iceberg-rust. _partition is being reviewed.
What is the problem the feature request solves?
Iceberg's metadata columns are currently not supported by Comet. A common use case for Iceberg (merge-on-read, copy-on-write) requires reading the metadata columns
_file,_pos,_spec, and_partition. Iceberg scans automatically introduce these columns if required. Because these are not supported, these jobs simply fall back to Spark.Describe the potential solution
Add support for these columns.
Additional context
_file, _pos, and, _spec are already supported by iceberg-rust. _partition is being reviewed.