What is the problem the feature request solves?
The native Iceberg scan now preserves the sort order instead of throwing it away. Within each Spark partition it reads each already-sorted file as its own stream and k-way-merges them into one sorted stream, then tells Spark the output is sorted so redundant sorts can be dropped. It also reports Iceberg's key-grouped partitioning so
storage-partitioned joins can skip the shuffle.
Scope of this first version:
- Ordering reporting is limited to plain-column (identity) sort keys that are in the projection. Anything else falls back to an unordered read, so it's always correct.
- Two flags: spark.comet.scan.icebergNative.sortMerge.enabled (on by default) and spark.comet.scan.icebergNative.reportPartitioning.enabled (off by default).
- No changes to iceberg-rust.
Describe the potential solution
No response
Additional context
No response
What is the problem the feature request solves?
The native Iceberg scan now preserves the sort order instead of throwing it away. Within each Spark partition it reads each already-sorted file as its own stream and k-way-merges them into one sorted stream, then tells Spark the output is sorted so redundant sorts can be dropped. It also reports Iceberg's key-grouped partitioning so
storage-partitioned joins can skip the shuffle.
Scope of this first version:
Describe the potential solution
No response
Additional context
No response