Repository navigation
Iceberg Spark 4.2 feature tracking #17959
Description
Activity
Thanks @huan233usc for this, yes thought of a lot of DSV2 improvements pending Spark 4.2 directory creation. Everyone can also add more ideas as they come up (like any remaining item for ViewCatalog, etc)!
Yes, looking forward to 4.2 for a while. also need to add "Implement SupportsSchemaEvolution for automatic schema evolution- #17957
" into this list@huan233usc I see DSv2 table pinning and refresh semantics item SPARK-58389, apache/spark#57799, are targeted at Spark 4.3 — the new API isn't in the 4.2.0 artifact, so there's nothing to wire up in spark/v4.2 yet ?
i add @aokolnychyi to also validate the iceberg spark 4.2 behavior for official release
@huan233usc do you mind adding #16133 to the list? I'll add a fix soon.
Reacted by Xin Huang@hemanthboyina You’re right—thanks for catching this. I confirmed that the SPARK-58389 API is only available on master / Spark 4.3 and is not present in the 4.2.0 artifact, so there is nothing to wire up in
spark/v4.2. I’ll remove this item from the Spark 4.2 tracker and track it for 4.3 instead.Also add tests for existing behavior:
- DSv2 cache invalidation and refresh (spark#55416, spark#55536)
For apache/spark#55536 (SPARK-54022 / SPARK-56617, DSv2 CACHE TABLE read semantics), it is already mirrored in TestCachedTableRefresh from the 4.1 -> 4.2 work. One scenario (a session ALTER TABLE invalidating the cache) was
@Disabledpending SPARK-55631, which is fixed in 4.2.0, so #18298 re-enables it (now green under both the SparkCatalog and SparkSessionCatalog configs).For apache/spark#55416 (SPARK-54216, cache refresh with immutable Table instances), this PR was closed without merging and its JIRA is still open. The behavior looks already exercised by testCachedTableRefreshReflectsExternalChanges under the caching-catalog config, so I left it out; but I would be happy to add dedicated tests mirroring it if folks think that additional coverage is worth having.
I would like to take the DSv2 Transaction API item (SPARK-55855). I opened a dedicated sub-issue to track it: #18384.
Feature Request / Improvement
Spark 4.2 adds a number of features and optimizations that may benefit the Iceberg Spark connector. This issue tracks the corresponding Iceberg work. If you are interested in an item, please open a dedicated sub-issue and link it back here.
Also add tests for existing behavior:
Query engine
Spark
Willingness to contribute