Repository navigation
SerializableTable.sortOrders() fails with ValidationException when a historical sort order references a dropped column #16519
Description
Apache Iceberg version
1.11.0 (latest release)
Query engine
None
Please describe the bug 🐞
Description
SerializableTable.sortOrders() throws a ValidationException when a table has a historical sort order that references a column that has since been dropped from the schema.
This was introduced in #15150, which added serialization of all sort orders (not just the default) to SerializableTable. During deserialization the new code binds every sort order against the current schema using strict validation, but historical sort orders may legitimately reference fields that no longer exist.
Root cause
Steps to reproduce
@Test
public void testSerializableTableSortOrdersWithDroppedColumn() {
// add an extra column and establish sort order 1 on id + ts
table.updateSchema().addColumn("ts", Types.LongType.get()).commit();
table.replaceSortOrder().asc("id").asc("ts").commit();
table.newAppend().appendFile(FILE_A).commit();
// switch to sort order 2 using only "id", then drop "ts"
// historical sort order 1 still references the dropped "ts" field
table.replaceSortOrder().asc("id").commit();
table.updateSchema().deleteColumn("ts").commit();
// SerializableTable.copyOf captures all sort orders at construction time
Table serialized = SerializableTable.copyOf(table);
// sortOrders() must return all historical orders without throwing a ValidationException,
// even though sort order 1 references the now-dropped "ts" field
assertThat(serialized.sortOrders()).hasSize(3); // unsorted(0), order-1(id+ts), order-2(id)
assertThat(serialized.sortOrder().fields()).hasSize(1); // current default is order-2
}
Expected behavior
sortOrders() should return all historical sort orders without throwing, binding non-default sort orders unchecked (as TableMetadataParser and PartitionSpec already do for the equivalent case).
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity
MonkeyCanCode commentedon May 21, 2026
Yes, got the same issue today and here is a reproducible:
CREATE TABLE polaris.db.sort_orphan_repro ( id bigint, time timestamp, source string, payload string ) USING iceberg PARTITIONED BY (days(time)) TBLPROPERTIES ('format-version' = '2'); ALTER TABLE polaris.db.sort_orphan_repro WRITE ORDERED BY time DESC NULLS LAST, source ASC NULLS LAST; INSERT INTO polaris.db.sort_orphan_repro VALUES (1, timestamp '2026-05-20 00:00:00', 'a', 'p1'); ALTER TABLE polaris.db.sort_orphan_repro DROP PARTITION FIELD days(time); ALTER TABLE polaris.db.sort_orphan_repro WRITE UNORDERED; ALTER TABLE polaris.db.sort_orphan_repro DROP COLUMN time; ALTER TABLE polaris.db.sort_orphan_repro ADD COLUMN time timestamp; ALTER TABLE polaris.db.sort_orphan_repro ADD PARTITION FIELD days(time); ALTER TABLE polaris.db.sort_orphan_repro WRITE ORDERED BY time DESC NULLS LAST, source ASC NULLS LAST; // this will fail INSERT INTO polaris.db.sort_orphan_repro (id, time, source, payload) VALUES (2, timestamp '2026-05-21 00:00:00', 'b', 'p2');Core: Fix SerializableTable.sortOrders() throwing on historical sort …
pallevam commentedon Jun 7, 2026
I would like to work on this bug. Please assign to me
yadavay-amzn commentedon Jun 7, 2026
@pallevam This has already been fixed by PR #16521 .
yadavay-amzn commentedon Jun 7, 2026
@aihuaxu Please close the issue as fixed. Thanks!
Core: Fix SerializableTable.sortOrders() throwing on historical sort …