Repository navigation
Conversation
… evolution fallback Moves the iceberg-rust pin to b154603a, which matches a data file's nested fields by field id (apache/iceberg-rust#3255) and applies the name mapping to nested fields (apache/iceberg-rust#3384). The fallback added by apache#6543 now covers only a struct whose current fields were all added after a data file was written, the one shape where the native read still differs from Spark. Closes apache#6934 Closes apache#6790
andygrove
force-pushed
the
iceberg-rust-nested-evolution
branch
from
October 17, 2026 13:35
5fe8f8b to
1c51487
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #6934.
Closes #6790.
Rationale for this change
Since #6543, the native Iceberg scan falls back to Spark whenever a projected struct, list or map column has a nested field that some schema in the table's history lacks or names differently. That was needed because iceberg-rust matched a data file's nested fields to the table's by position (apache/iceberg-rust#2617). apache/iceberg-rust#3255 matches them by field id and fills absent optional fields with NULLs, so most of those fallbacks are no longer needed. The same bump picks up apache/iceberg-rust#3384, which applies a name mapping to nested fields and fixes #6790.
What changes are included in this PR?
icebergandiceberg-storage-opendalpin to iceberg-rustb154603a.Cargo.lockchanges only those three crates, and no Comet source change was needed to compile.IcebergReflection.nestedFieldsAddedOrRenamedwithstructsWithNoFieldsInCommon. The scan now falls back only when a projected struct, at any depth, shares no field id with the same struct in some schema in the table's history. That is the one shape where the native read still differs from Spark: it returns a struct of NULLs or NULL depending on what else the scan reads, where Spark returns NULL for a top-level struct and a struct of NULLs for a nested one.CometIcebergNativeScanMetadata.taskSchema, from perf: read only the nested fields Spark keeps in the native Iceberg scan #6725), instead of the table and scan schemas. With nested schema pruning on, that schema holds only the fields Spark keeps, so a read of onlys.bfalls back when a file predatess.b, while a read ofs.astays native.docs/source/user-guide/latest/iceberg.md, and removes the migrated-table name mapping entry fromparquet-iceberg-known-issues.md(Native Iceberg scan returns NULL for nested fields of migrated tables that use a name mapping #6790).parquet-iceberg-known-issues.md, under Native Iceberg scan returns nested values that Spark reads as NULL in files without field ids or a name mapping #6794, a new failure mode. For data files with neither field ids nor a name mapping, iceberg-rust matches a struct's fields by position only when the counts are equal. A pruned read of some of a struct's fields now fails with "cannot reconcile struct fields by id" where it used to return the file's values (Spark returns NULL). A read of the whole struct is unchanged.How are these changes tested?
In
CometIcebergNativeSuite:VERSION AS OFtests added by perf: read only the nested fields Spark keeps in the native Iceberg scan #6725 now expect native scans too, with nested schema pruning on and off.s.bread. A struct that kept one of its fields stays native.SparkActions.snapshotTabletable through its name mapping (Native Iceberg scan returns NULL for nested fields of migrated tables that use a name mapping #6790), including a pruneds.bread.On Spark 4.1 with Iceberg 1.11,
CometIcebergNativeSuitegives 131 passed, 0 failed, 1 canceled. I have not run the Spark matrix or the other Iceberg shards.