Skip to content

feat: bump iceberg-rust past apache/iceberg-rust#3255 and narrow the nested schema evolution fallback - #7028

Draft
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:iceberg-rust-nested-evolution
Draft

andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:iceberg-rust-nested-evolution

Conversation

@andygrove

@andygrove andygrove commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #6934.
Closes #6790.

Rationale for this change

Since #6543, the native Iceberg scan falls back to Spark whenever a projected struct, list or map column has a nested field that some schema in the table's history lacks or names differently. That was needed because iceberg-rust matched a data file's nested fields to the table's by position (apache/iceberg-rust#2617). apache/iceberg-rust#3255 matches them by field id and fills absent optional fields with NULLs, so most of those fallbacks are no longer needed. The same bump picks up apache/iceberg-rust#3384, which applies a name mapping to nested fields and fixes #6790.

What changes are included in this PR?

  • Moves the iceberg and iceberg-storage-opendal pin to iceberg-rust b154603a. Cargo.lock changes only those three crates, and no Comet source change was needed to compile.
  • Replaces IcebergReflection.nestedFieldsAddedOrRenamed with structsWithNoFieldsInCommon. The scan now falls back only when a projected struct, at any depth, shares no field id with the same struct in some schema in the table's history. That is the one shape where the native read still differs from Spark: it returns a struct of NULLs or NULL depending on what else the scan reads, where Spark returns NULL for a top-level struct and a struct of NULLs for a nested one.
  • Runs that check on the task schema the scan reads (CometIcebergNativeScanMetadata.taskSchema, from perf: read only the nested fields Spark keeps in the native Iceberg scan #6725), instead of the table and scan schemas. With nested schema pruning on, that schema holds only the fields Spark keeps, so a read of only s.b falls back when a file predates s.b, while a read of s.a stays native.
  • Narrows the limitation bullet in docs/source/user-guide/latest/iceberg.md, and removes the migrated-table name mapping entry from parquet-iceberg-known-issues.md (Native Iceberg scan returns NULL for nested fields of migrated tables that use a name mapping #6790).
  • Notes in parquet-iceberg-known-issues.md, under Native Iceberg scan returns nested values that Spark reads as NULL in files without field ids or a name mapping #6794, a new failure mode. For data files with neither field ids nor a name mapping, iceberg-rust matches a struct's fields by position only when the counts are equal. A pruned read of some of a struct's fields now fails with "cannot reconcile struct fields by id" where it used to return the file's values (Spark returns NULL). A read of the whole struct is unchanged.

How are these changes tested?

In CometIcebergNativeSuite:

  • The four "falls back" nested-evolution tests (added to a struct, added inside an array or map, dropped and re-added, renamed) now assert native reads that match Spark. The self-join query allows two native scans. The VERSION AS OF tests added by perf: read only the nested fields Spark keeps in the native Iceberg scan #6725 now expect native scans too, with nested schema pruning on and off.
  • A new test covers the fallback shapes: top-level struct, list element, nested struct, and a pruned s.b read. A struct that kept one of its fields stays native.
  • A new test reads the struct, list and map fields of a SparkActions.snapshotTable table through its name mapping (Native Iceberg scan returns NULL for nested fields of migrated tables that use a name mapping #6790), including a pruned s.b read.
  • The test for data files without field ids now expects the pruned read to fail, as described above.

On Spark 4.1 with Iceberg 1.11, CometIcebergNativeSuite gives 131 passed, 0 failed, 1 canceled. I have not run the Spark matrix or the other Iceberg shards.

@andygrove andygrove changed the title feat: bump iceberg-rust past #3255 and narrow the nested schema evolution fallback feat: bump iceberg-rust past apache/iceberg-rust#3255 and narrow the nested schema evolution fallback Oct 15, 2026
@github-actions github-actions Bot added enhancement New feature or request area:writer Native Parquet writer area:scan Parquet scan / data reading area:Iceberg labels Oct 15, 2026
@andygrove
andygrove requested a review from comphead October 15, 2026 20:37
@andygrove andygrove added this to the 1.2.0 milestone Oct 15, 2026
… evolution fallback

Moves the iceberg-rust pin to b154603a, which matches a data file's nested
fields by field id (apache/iceberg-rust#3255) and applies the name mapping
to nested fields (apache/iceberg-rust#3384). The fallback added by apache#6543
now covers only a struct whose current fields were all added after a data
file was written, the one shape where the native read still differs from
Spark.

Closes apache#6934
Closes apache#6790
@andygrove
andygrove force-pushed the iceberg-rust-nested-evolution branch from 5fe8f8b to 1c51487 Compare October 17, 2026 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Iceberg area:scan Parquet scan / data reading area:writer Native Parquet writer enhancement New feature or request

Projects

None yet

1 participant