[GLUTEN-12569][CORE] Add enhanced DSv2 filter test suites for Spark 4.2

[akshaytayal]

What changes were proposed in this pull request?

Adds the enhanced DataSourceV2 filter test suites for Spark 4.2 in gluten-ut/spark42, wrapping the upstream Apache Spark suites with GlutenSQLTestsTrait:

• GlutenDataSourceV2EnhancedRuntimePartitionFilterSuite
• GlutenDataSourceV2EnhancedPartitionFilterSuite
• GlutenDataSourceV2EnhancedDeleteFilterSuite

and registers them in VeloxTestSettings.

Disabled (CI-confirmed)

2 tests in GlutenDataSourceV2EnhancedPartitionFilterSuite are disabled via VeloxTestSettings.exclude because Gluten changes the plan shape they assert on:

• case 9: partition filter pushed but returned in first pass is not re-pushed second pass
• nested identity partition: case 9 partition filter pushed but returned in first pass is not re-pushed second pass

Root cause: the tests assert a post-scan vanilla FilterExec remains, but Gluten replaces it with a columnar FilterExecTransformer / the InMemoryEnhancedPartitionFilterBatchScan falls back. The other two enhanced suites (Delete, RuntimePartition) pass. Follow-up to re-enable tracked with the DSv2 filter work (see #13181 umbrella).

How was this patch tested?

spark42 UT CI (groups 1-3 + slow extended/slow-hive) is green on this branch, which is rebased on top of #13163 (so it also carries the Spark 4.2 UT module + all its CI-green fixes).

Notes

• Stacked on [GLUTEN-12569][CORE] Add Spark 4.2 unit test module (gluten-ut/spark42) #13163 (Spark 4.2 UT module), which is stacked on the merged [VL] Spark 4.2: align profile dependency versions (Hadoop 3.5.0, Jackson 2.21.2, Guava, log4j, commons-lang3) #13161 and [GLUTEN-12569][VL] Add Spark 4.2 shim layer and build profile #13126.
• Corresponds to [GLUTEN-12569][CORE] Add enhanced DSv2 filter test suites for Spark 4.2 #13023.

(GLUTEN-12569)

Related issue: #12569

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[weiting-chen]

Adapt the two residual-filter assertions to mixed native execution

Target Location: GlutenDataSourceV2EnhancedPartitionFilterSuite.scala:21-23.

Problem: Enabling this inherited suite with Gluten also enables two case9 tests whose final assertions require a Spark FilterExec. Their retained string-equality filters can correctly offload to FilterExecTransformer, which is not a subclass of FilterExec. The in-memory scan falls back, but scan fallback does not force the parent filter to fall back. With the workflow's ANSI=false and native filters enabled, correct mixed execution can therefore fail these vanilla-only plan assertions.

Evidence:

class GlutenDataSourceV2EnhancedPartitionFilterSuite
  extends DataSourceV2EnhancedPartitionFilterSuite
  with GlutenSQLTestsTrait {}

The inherited "case 9" and nested identity case9 both use executedPlan.exists(_.isInstanceOf[FilterExec]). Gluten's filter transformer explicitly preserves the residual condition above a fallback child. This is source analysis, not an observed failure of these tests: the current group3 job stops at the parent's dependency error first.

Suggested Fix: Prefer Gluten variants of these two tests that retain all answer/pushdown checks and accept the intended native residual filter as well as a vanilla fallback filter. If this wrapper is deliberately scoped to Spark-filter/plugin integration instead, make that scope explicit without disabling the plugin:

import org.apache.spark.SparkConf

class GlutenDataSourceV2EnhancedPartitionFilterSuite
  extends DataSourceV2EnhancedPartitionFilterSuite
  with GlutenSQLTestsTrait {
  override def sparkConf: SparkConf =
    super.sparkConf.set("spark.gluten.sql.columnar.filter", "false")
}

Run the suite with SPARK_ANSI_SQL_MODE=false; a pass with ANSI fallback enabled would not exercise the problematic mixed-plan path.

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86