Spark 4.1: Initial support for MERGE INTO schema evolution

[szehon-ho]

Fixes #5556

This is a first cut in implementing schema evolution for Spark MERGE INTO commands for Iceberg tables.

This was implemented in https://issues.apache.org/jira/browse/SPARK-54274 mostly in Spark 4.1.0. There are a few remaining features in Spark 4.1.1 that I can add test for, once we upgrade.

Unfortunately there are some issues with MERGE INTO WITH SCHEMA EVOLUTION if the target is a table branch.

[szehon-ho]

@amogh-jahagirdar @huaxingao can one of you help take a look?

[huaxingao]

Thanks @szehon-ho for the PR! Can we add a TODO or open a tracking issue for the branch limitation? Otherwise, LGTM.

[singhpk234]

LGTM as well, thanks a ton for driving it in both spark and here @szehon-ho !

It would be really nice to have a disabled test with a note with the limitation we are aware of, which we expect to get fixed in 4.1.1 (which seems already in motion here #14946)

[dramaticlly]

Thanks @szehon-ho ! I bet many would love to see this change in next release

[szehon-ho]

Thanks all, I just checked offline with @amogh-jahagirdar, we dont support alter schema for branches today.

That's the failure, it tries to alter table on the branch and fails, so it seems expected, and no need to create an issue

[huaxingao]

Thanks @szehon-ho for the PR! Thanks @singhpk234 @dramaticlly for the review!