Bump iceberg-rust past apache/iceberg-rust#3255 and narrow the nested schema evolution fallback #6934

What is the problem the feature request solves?

Since #6543, which fixed #6504 and #6546, the native Iceberg scan falls back to Spark when a projected struct, list or map column has a nested field that some schema in the table's history lacks or names differently. CometScanRule finds these fields with IcebergReflection.nestedFieldsAddedOrRenamed. The fallback was needed because iceberg-rust matched a data file's nested fields to the table's by position, not by field id (apache/iceberg-rust#2617). A file written before a nested field was added failed the scan with Incorrect number of arrays for StructArray fields, and a nested field renamed since the file was written read back as NULL.

The check has to use the table's schema history, because a FileScanTask doesn't say which schema wrote its file. So once a table adds or renames a nested field, every scan that projects that column falls back for good, even after compaction has rewritten all the old files.

apache/iceberg-rust#3255 fixed the root cause. It merged on 2026-10-11 as 050ddfda, after the pin that #6851 moves to (1f3bc34). RecordBatchTransformer now builds a plan for each file that matches nested children by field id through structs, lists and maps. It fills absent optional children with nulls and applies renames and reorders. The same bump picks up apache/iceberg-rust#3384, which fixes #6790.

Describe the potential solution

After #6851 merges:

• Move the iceberg and iceberg-storage-opendal pin to iceberg-rust main. That was b154603a on 2026-10-13, 14 commits past 1f3bc34. iceberg-rust is still on arrow and parquet 59.2, and the lock change is only the three iceberg-rust crates.
• Replace nestedFieldsAddedOrRenamed with a narrower check for the one shape that still differs (below). That check would fall back when some schema in the table's history has a projected struct, at any depth, that shares none of its requested children with the current schema. If perf: read only the nested fields Spark keeps in the native Iceberg scan #6725 lands first, "requested" means the children that the pruned read asks for.
• Turn the four fallback tests in CometIcebergNativeSuite ("schema evolution - nested field ... falls back") into native reads that compare with Spark. The self-join query reads the table twice, so it needs a check that allows two native scans. Keep "dropped, moved, or promoted stays native", and add fallback tests for the shapes the narrower check keeps.
• Narrow the limitation bullet in docs/source/user-guide/latest/iceberg.md, and update the comments that cite Type casting error when reading files persisted with old schema for complex type iceberg-rust#2617 in CometScanRule and CometIcebergNativeSuite.
• Close Native Iceberg scan returns NULL for nested fields of migrated tables that use a name mapping #6790 with a test. The same range has fix(reader): apply name mapping to nested fields iceberg-rust#3384, which applies a name mapping to nested fields. A table made with SparkActions.snapshotTable, which sets the mapping, reads its struct, list and map fields correctly alongside a top-level column at b154603a. At 1f3bc34 they come back NULL.

What still differs at b154603a is a struct whose current children were all added after a data file was written. For example, a file is written with s STRUCT<a: INT> holding {a: 1} and NULL, and then ALTER TABLE adds s.b and drops s.a. iceberg-rust's projection selects no leaf under s for that file, and the result depends on what else the scan reads. With the #6504 check turned off, on Spark 4.1 and Iceberg 1.11:

So SELECT count(*) ... WHERE s IS NULL counts both of that file's rows on Spark, and only the NULL one natively. apache/iceberg-rust#3378 (open) is about the NULL when another column is read, and treats that NULL as the bug. Spark also returns NULL for a top-level struct, though. It returns a struct of NULLs only for a nested one, so an upstream fix that always returns a struct would break the case that matches today. A map whose value struct has this shape fails in Iceberg Java's own reader (IllegalArgumentException: ... is not in the store), so there is nothing to compare it with.

Additional context

Checked in a scratch build of #6851 with the pin moved to b154603a, on Spark 4.1 (Iceberg 1.11):

• It compiles with no Comet changes, and the 875 datafusion-comet lib tests pass.
• With the Native Iceberg scan fails on files written before a nested field was added, and since 1.1.0 null checks and explode hit it too #6504 check in place, the 14 Iceberg suites that don't need MinIO give 434 passed, 0 failed and 1 canceled, the same as at 1f3bc34.
• With the check turned off, every query in the four fallback tests matches Spark. Against 1f3bc34, the queries on added fields fail and the renamed fields read NULL.

Other changes in 1f3bc34..b154603a to check during the bump:

• fix(scan): reject ambiguous case-insensitive fields iceberg-rust#3070 makes a case-insensitive name lookup fail on an ambiguous name instead of picking one of the matches. iceberg-rust binds equality-delete predicates with the task's case_sensitive flag, and Comet sets that to false on every FileScanTask. From reading the code, an equality delete on a table whose column names differ only by case could now fail natively. Comet binds residual predicates case-sensitively, so those don't change.
• fix(arrow): strip metadata from nested fields and normalize container field names iceberg-rust#3303 changes strip_metadata_from_schema to normalize list and map field names. Neither Comet nor iceberg-rust's reader and writer call it.
• fix(transform): correct decimal and binary truncate literals iceberg-rust#3243 fixes truncate literals for decimal and binary in predicate projection. Comet's native writer truncates whole arrays, which it doesn't change.
• fix(arrow): decode short BYTE_ARRAY decimal row-group stats iceberg-rust#3364 decodes short BYTE_ARRAY decimal row-group bounds. Comet can't reach that until Push Iceberg residual predicates on decimal columns into the native scan #6854 pushes decimal predicates down.
• fix(spec): reject boolean literals for non-boolean types iceberg-rust#3334 rejects a boolean JSON literal for a non-boolean type.