[GLUTEN-11550][VL] Enable GlutenDataFrameSubquerySuite for Spark 4.1

[baibaichen]

What changes are proposed in this pull request?

Fix struct join key validation and enable GlutenDataFrameSubquerySuite for Spark 4.1.

Spark 4.1 added isin(Dataset) API which creates struct IN subquery predicates. These are converted to BroadcastHashJoin with struct-typed join keys where field names may differ between sides (e.g., struct(a, b) vs struct(c, d)).

Changes:

1. JoinExecTransformer.scala — Remove struct field name comparison in sameType() to align with Spark's DataType.equalsStructurally(ignoreNullability=true) semantics. Spark's native HashJoin only checks structural type compatibility (field count + field types by position), not field names. This allows struct IN subqueries to be offloaded to Velox natively.

2. VeloxTestSettings.scala — Enable GlutenDataFrameSubquerySuite for Spark 4.1 (was previously disabled with // TODO: 4.x).

Evidence — Spark's join key validation does NOT check struct field names:

Spark HashJoin.scala uses DataType.equalsStructurally (identical in v4.0.1):

require(leftKeys.length == rightKeys.length &&
  leftKeys.map(_.dataType)
    .zip(rightKeys.map(_.dataType))
    .forall(types => DataType.equalsStructurally(types._1, types._2, ignoreNullability = true)),
  "Join keys from two sides should have same length and types")

DataType.equalsStructurally for StructType only compares field count + field types by position — no l.name == r.name:

case (StructType(fromFields), StructType(toFields)) =>
  fromFields.length == toFields.length &&
  fromFields.zip(toFields).forall { case (l, r) =>
    equalsStructurally(l.dataType, r.dataType, ignoreNullability) &&
    (ignoreNullability || l.nullable == r.nullable)
  }

Gluten's sameType() had an extra l.name.equalsIgnoreCase(r.name) check that was more restrictive than Spark's own validation.

How was this patch tested?

• GlutenDataFrameSubquerySuite on Spark 4.1: all 47 tests passed (struct IN subqueries offloaded to Velox natively)
• GlutenDataFrameSubquerySuite on Spark 4.0: all 40 tests passed (no regression)

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot

Related issue: #11550

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[baibaichen]

Run Gluten Clickhouse CI on x86

[baibaichen]

@zhouyuan please review this

[github-actions]

Run Gluten Clickhouse CI on x86

[zhouyuan]

👍