Schema evolution issue with structs nested inside map fields (Parquet-backed Iceberg tables)

Apache Iceberg version

1.9.2 (latest release)

Query engine

Spark

Please describe the bug 🐞

At Wikimedia, we are increasingly adopting Iceberg in our data lake, but we’ve encountered a blocker that looks like a schema evolution issue when adding columns to structs nested within map fields (using Parquet).

I’ve put together a minimal reproducible example with Docker here: https://gitlab.wikimedia.org/aqu/iceberg_nested_map_test

Tested environments:
• Spark 3.3.2, Iceberg 1.6.1, Parquet 1.12.2, Java 17
• Spark 3.5.6, Iceberg 1.9.2, Parquet 1.13.1, Java 17

Description:
When I add a new column to a struct nested inside a map field, some subsequent SELECT queries fail with a field ID mismatch error. For example:

-- 1. Create table with map containing struct

CREATE TABLE test_map_struct_table (
page_id BIGINT,
revision_content_slots MAP<STRING, STRUCT<
content_body: STRING,
content_format: STRING,
content_model: STRING,
content_sha1: STRING,
content_size: BIGINT
>>,
revision_content_is_visible BOOLEAN,
wiki_id STRING
) USING iceberg;

-- 2. Insert test data

INSERT INTO test_map_struct_table VALUES
(12345, MAP('main', NAMED_STRUCT(
'content_body', 'This is the main content of the page',
'content_format', 'text/x-wiki',
'content_model', 'wikitext',
'content_sha1', 'abc123def456',
'content_size', 1024
)), true, 'enwiki');

-- 3. Add new column to struct within map

ALTER TABLE test_map_struct_table
ADD COLUMN revision_content_slots.value.content_origin_rev_id BIGINT;

-- 4. Attempt to read data → FAILS

SELECT page_id, revision_content_slots['main'].content_origin_rev_id
FROM test_map_struct_table
WHERE page_id = 12345;

This produces the following error (truncated):

java.lang.IllegalArgumentException: [revision_content_slots, key_value, key] required binary key (STRING) = 5 is not in the store:
[[page_id] required int64 page_id = 1] 1

Other queries (e.g., SELECT *) succeed.

⸻

Questions:

Is this expected behavior? Should adding columns to structs within maps work, or is this a known limitation?
Field ID management: Am I correct that Parquet files don’t store Iceberg field IDs, so resolution here is name-based?
Error clarity: The error message seems truncated — could it be improved to include the full schema context?
Are there recommended workarounds, such as:
• Creating a table with the new schema and moving data to it?
• Avoiding schema evolution in map-contained structs?
• Using specific configs or formats that handle this better?

I can provide a patch after discussing the feasibility with you.

Thanks a lot for your guidance!

Willingness to contribute

I can contribute a fix for this bug independently
I would be willing to contribute a fix for this bug with guidance from the Iceberg community
I cannot contribute a fix for this bug at this time

I .... am not sure here. Theoretically I don't think there should be a problem with doing this.
Parquet files written by Iceberg libraries do include field ids and these are used when reading. Iceberg starts with field ids, then if they are missing uses the default name mapping, and if that is missing uses positional mapping. See the Spec for notes on column projection
Possibly
Not sure, it sounds like there are real bugs here? In general though I recommend against using Map types in columnar data types since they can't be efficiently used for pruning. Now that we have Variants I think those are generally a better approach but ymmv

This is similar to #2962.

Yeah adding fields to structs which are map values should work. The project has a spark integration test which actually validates this https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAlterTable.java#L128

I locally modified the test to be similar to the provided example (it's a bit more simpler in terms of number of fields but still a map with string key and struct value with a field added to the struct later) but I can't actually repro the failure.


  @TestTemplate
  public void testAddColumnWithMap() {
    // use the implicit column name 'key' and 'value' to access member of map.
    // add column to value struct column
    sql("ALTER TABLE %s ADD COLUMN data2.value.c int", tableName);
    Types.StructType expectedSchema =
            Types.StructType.of(
                    NestedField.required(1, "id", Types.LongType.get()),
                    NestedField.optional(2, "data", Types.StringType.get()),
                    NestedField.optional(
                            3,
                            "data2",
                            Types.MapType.ofOptional(
                                    4,
                                    5,
                                    Types.StructType.of(NestedField.optional(6, "x", Types.IntegerType.get())),
                                    Types.StructType.of(
                                            NestedField.optional(7, "a", Types.IntegerType.get()),
                                            NestedField.optional(8, "b", Types.IntegerType.get()),
                                            NestedField.optional(9, "c", Types.IntegerType.get())))));


    sql("INSERT INTO %s VALUES (1, 'foo', map('main', named_struct('a', 10, 'b', 20, 'c', 21)))", tableName);
    List<Object[]> rows = sql("select data2['main'].c from %s where id = 1", tableName);
    assertThat(rows.size()).isEqualTo(1);
    assertEquals("expected rows", rows.get(0), row(21));
  }

Ah the issue may surface on reading the parquet files before schema evolution....let me test that.
Edit: Yeah that's it, OK I'll poke into what's the parquet read schema we're using when reading the older files...I feel like theoretically we should just be able to project the default value (null if it's not defined)

Yeah the projection that ends up getting produced for the older file looks incorrect

https://github.com/apache/iceberg/blob/main/parquet/src/main/java/org/apache/iceberg/parquet/ReadConf.java#L82

The file schema and the expected schema are expected, containing the projections from the filter and the actual row projections but the return value from Prunecolumns just contains the projection from the filter and then expectedly fails ....
Next step is to poke into that

I poked into this a bit more, I take back what I said about the projection being produced being incorrect. I do think it's expected that after PruneColumns is invoked it only returns the id column for this file.

I think the real issue is a mismatch between the model created in Spark for this case and the internal page readers. The internal page readers, will just project the id column based on pruning for that particular file. The Spark model produces a InternalRowReader for the ID column and then a map reader, where the underlying struct field reader in the value of the map is a null reader. Then when setting the page source to the model, the page reader for the ID expectedly cannot be set on the reader trying to read the whole map. I think if a nested struct field in a map is being projected that does not exist in the file, we should ideally create the null or default value reader, instead of creating the whole map reader, but I need to check this further.

Any updates on this? We are running into a similar issue

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

Re #14043 (comment)

This issue is not stale. It continues to affect the latest version of Iceberg as of this writing.

Tested this and I can no longer reproduce the issue with spark 4.1. This issue seems to affect spark (<4.1) with plain parquet too (IndexArrayOutOfBoundException). For spark < 4.1 setting spark.sql.optimizer.nestedSchemaPruning.enabled=false seems to workaround the issue and produce correct results. I'm not 100% sure but it could possibly have been fixed by apache/spark#52557 or one of its followups.

@nomoa

@nomoa can you share what versions of Iceberg, Parquet, and Java did you use for the non-repro?

@nomoa can you share what versions of Iceberg, Parquet, and Java did you use for the non-repro?

Java 17, Spark 4.1 + iceberg 1.12.0, the parquet version used is the one bundled with spark and iceberg which I believe is 1.17.1 for iceberg 1.12.0 and 1.16.0 for spark 4.1.

Quick note: I can reproduce the issue with iceberg 1.11.0 and spark 4.1 so my hypothesis about the linked PR in spark no longer holds. Perhaps fixed by #17320 then?

@amogh-jahagirdar

Hi @amogh-jahagirdar @RussellSpitzer, I'd like to help close this out. @nomoa saw it go away on 1.12.0 but still reproduced it on 1.11.0, which lines up with #17320. I'd rerun the Wikimedia repro on 1.11.0 and main for Spark 3.5 and 4.1. If it's fixed, I'll add a regression test with a map<string, struct> where a field is added to the value struct. If not, I'll look at how SparkParquetReaders builds the map value reader when the projected field is missing from the file. Does that sound right? Happy to take it.

Navigation Menu

Repository navigation

Schema evolution issue with structs nested inside map fields (Parquet-backed Iceberg tables) #14043

Apache Iceberg version

Query engine

Please describe the bug 🐞

Willingness to contribute

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Schema evolution issue with structs nested inside map fields (Parquet-backed Iceberg tables) #14043

Description

Apache Iceberg version

Query engine

Please describe the bug 🐞

Willingness to contribute

Activity

RussellSpitzer commented on Sep 11, 2025

xabriel commented on Sep 15, 2025

amogh-jahagirdar commented on Sep 16, 2025

amogh-jahagirdar commented on Sep 16, 2025

amogh-jahagirdar commented on Sep 16, 2025

amogh-jahagirdar commented on Sep 20, 2025

taxbit-xinghai commented on Nov 5, 2025

github-actions commented on May 5, 2026

xabriel commented on May 11, 2026

nomoa commented on Oct 8, 2026

xabriel commented on Oct 8, 2026

nomoa commented on Oct 8, 2026

nomoa commented on Oct 8, 2026

developer-rpai commented on Oct 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions