Spark 4.1: Add tests for MERGE INTO schema evolution nested case

[varun-lakhyani]

Extends Tests for PR #14970

This PR creates a test for MERGE INTO schema evolution in nested case where source has fewer fields than target and by default it keeps the non existing fields in target intact and null for new source rows.

Spark Updates can be found at https://issues.apache.org/jira/browse/SPARK-54274
This specific feature https://issues.apache.org/jira/browse/SPARK-54621

It also renames an existing nested test like other tests so can be better understood
(testMergeWithSchemaEvolutionNestedStruct() -> testMergeWithSchemaEvolutionNestedStructSourceHasMoreFields())

[varun-lakhyani]

PTAL @huaxingao @singhpk234 @szehon-ho

I went through the discussions in Spark PRs related to this feature.
Please let me know if I missed anything.

[szehon-ho]

Yea , was going to do this, you beat me to it :)

The only missing part here is that there is a flag that needs to be set (as its disabled by default in Spark 4.1.x). We will probably turn it on in later version if there are no complaints on the behavior

[varun-lakhyani]

@szehon-ho Thanks for the original features and tests, This new test is a small tweak based on your existing one.

mergeNestedTypeCoercion flag is enabled for test and then unset it to default state afterwards,
Since 4.1.1 is is active in codebase, the test is now enabled, passing and ready for review.

Please let me know if any further changes are needed. @huaxingao @singhpk234

[varun-lakhyani]

PTAL @huaxingao @singhpk234

[szehon-ho]

looks mostly fine, but i wonder if we should hold the pr until 4.1.1 is supported in iceberg

[szehon-ho]

should be in try/catch. Do we have any utility in iceberg spark test cases to wrap code in spark configs?

[varun-lakhyani]

Yes correct, There is withSQLConf at spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/TestBase.java::186 which handles spark configs required for some action and It handles try block in its implementation.
I took reference from testMergeToWapBranch() at spark/v4.1/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java::2914 which implements similar thing.

Ready for review.

[szehon-ho]

sorry, just re-read and see test is running already against 4.1.1 , that's great. only one comment

[singhpk234]

LGTM as well, thanks @varun-lakhyani !

[szehon-ho]

nit, can we move this block in the with? its a bit disjointing to see the expectation before the merge :)

[varun-lakhyani]

iceberg/spark/v4.1/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java

Lines 2914 to 2946 in 4f57687

	public void testMergeToWapBranch() {
	assumeThat(branch).as("WAP branch only works for table identifier without branch").isNull();
	createAndInitTable("id INT", "{\"id\": -1}");
	ImmutableList<Object[]> originalRows = ImmutableList.of(row(-1));
	sql(
	"ALTER TABLE %s SET TBLPROPERTIES ('%s' = 'true')",
	tableName, TableProperties.WRITE_AUDIT_PUBLISH_ENABLED);
	spark.range(0, 5).coalesce(1).createOrReplaceTempView("source");
	ImmutableList<Object[]> expectedRows =
	ImmutableList.of(row(-1), row(0), row(1), row(2), row(3), row(4));
	withSQLConf(
	ImmutableMap.of(SparkSQLProperties.WAP_BRANCH, "wap"),
	() -> {
	sql(
	"MERGE INTO %s t USING source s ON t.id = s.id "
	+ "WHEN MATCHED THEN UPDATE SET *"
	+ "WHEN NOT MATCHED THEN INSERT *",
	tableName);
	assertEquals(
	"Should have expected rows when reading table",
	expectedRows,
	sql("SELECT * FROM %s ORDER BY id", tableName));
	assertEquals(
	"Should have expected rows when reading WAP branch",
	expectedRows,
	sql("SELECT * FROM %s.branch_wap ORDER BY id", tableName));
	assertEquals(
	"Should not modify main branch",
	originalRows,
	sql("SELECT * FROM %s.branch_main ORDER BY id", tableName));
	});

I felt the same way but found lot of places where this was followed as one referenced here.
Should I push the change with expected rows inside withSQLConf after MERGE command?

[szehon-ho]

i see, thanks for checking. well, i did it all this way in the other tests in this file, and we both agree, so id say let's just do it?

[varun-lakhyani]

Done. Updated in latest commit

[varun-lakhyani]

PTAL @szehon-ho.
Let me know if any other changes

[szehon-ho]

merged, thanks @varun-lakhyani , and all for reviews

[varun-lakhyani]

Thanks @huaxingao @singhpk234 @szehon-ho for reviews and merging