Skip to content

[MINOR] Remove Spark 3.2 tests - #11887

Merged
zhouyuan merged 1 commit into
apache:mainfrom
philo-he:remove-spark3.2-test
Apr 8, 2026
Merged

[MINOR] Remove Spark 3.2 tests#11887
zhouyuan merged 1 commit into
apache:mainfrom
philo-he:remove-spark3.2-test

Conversation

@philo-he

@philo-he philo-he commented Apr 8, 2026

Copy link
Copy Markdown
Member

What changes are proposed in this pull request?

Since Spark 3.2 support was dropped a few months ago, the related tests can be removed now.

How was this patch tested?

N/A

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions github-actions Bot added the CORE works for Gluten Core label Apr 8, 2026
@philo-he

philo-he commented Apr 8, 2026

Copy link
Copy Markdown
Member Author

@zhouyuan, could you take a look?

@github-actions

github-actions Bot commented Apr 8, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

zhouyuan
zhouyuan approved these changes Apr 8, 2026
View reviewed changes

@zhouyuan zhouyuan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@zhouyuan
zhouyuan merged commit 75b4323 into apache:main Apr 8, 2026
60 checks passed
malinjawi pushed a commit to malinjawi/incubator-gluten that referenced this pull request Apr 9, 2026
@philo-he Mohammad Linjawi
[MINOR] Remove Spark 3.2 tests (apache#11887)
037e2ef 
Since Spark 3.2 support was dropped a few months ago, the related tests can be removed now.

related: apache#11379
LuciferYang added a commit to LuciferYang/gluten that referenced this pull request Jul 15, 2026
@LuciferYang
[MINOR][CORE] Remove residual Spark 3.2 compatibility code in gluten-…
9e3714c 
…core

Spark 3.2 support was removed in prior PRs (apache#11351, apache#11687, apache#11731, apache#11887);
currently supported versions are Spark 3.3, 3.4, 3.5, 4.0, 4.1. A few symbols
that existed only as pre-Spark-3.3 shims are still around and dead code today.

1. `GlutenPlan.SupportsRowBasedCompatible` trait

   Introduced to provide `def supportsRowBased(): Boolean` for Spark < 3.3
   where `SparkPlan.supportsRowBased` did not exist yet. The default body was
   `throw new GlutenException("Illegal state: The method is not expected to
   be called")`. On Spark 3.3+, `SparkPlan.supportsRowBased` is native and
   every concrete GlutenPlan / ColumnarInputAdapter overrides it directly, so
   the trait's default was already unreachable. Drop the trait and its two
   mixin sites.

2. `SparkVersionUtil.gteSpark33`

   Always true after Spark 3.2 was dropped. Also drop the single caller
   guard in `canPropagateConvention` (Transitions), which no longer needs to
   skip UnionExec on Spark 3.2. `eqSpark33` and `comparedWithSpark33` are
   kept: they distinguish Spark 3.3 from 3.4+ (different `TaskContextImpl`
   ctor signature and different write planning API), which is unrelated to
   the Spark 3.2 residual concern.

3. `SparkPlanUtil.supportsRowBased` reflection

   The reflection was needed on Spark 3.2 because `SparkPlan.supportsRowBased`
   did not exist as a member yet; the same compiled artifact ran on 3.2 and
   3.3+ only by resolving the method reflectively at call time. Now that
   Spark 3.2 is dropped, a direct call `plan.supportsRowBased` compiles on
   all supported profiles and is strictly better (primitive Boolean instead
   of boxed, no per-call `getMethod` lookup, no `InvocationTargetException`
   wrapping). The 3 callers in `ConventionFunc` are on the planning hot
   path.

Verified via compile on Spark 3.3, 3.5, and 4.1 (scala-2.13) profiles, plus
gluten-core and gluten-substrait tests on Spark 3.5.
This was referenced Jul 15, 2026
Merged
Merged
LuciferYang added a commit to LuciferYang/gluten that referenced this pull request Jul 15, 2026
@LuciferYang
[MINOR] Align supported Spark version references
2aa6178 
Spark 3.2 was dropped by apache#11351/apache#11687/apache#11731/apache#11887; currently supported
versions per README.md / docs/index.md / pom.xml profiles are Spark 3.3,
3.4, 3.5, 4.0, and 4.1. Various docs, comments, and small code paths still
carry Spark 3.2 leftovers or predate the 4.0/4.1 additions. This PR aligns
them.

Docs:
- `docs/get-started/Velox.md`: version table + prose updated to
  3.3.1, 3.4.4, 3.5.5, 4.0.2, 4.1.1 (matching `<spark.version>` in each
  profile).
- `docs/velox-backend-limitations.md`: three section headers "For
  Spark3.2 and Spark3.3" reduced to "For Spark3.3"; the "spark3.2/3.3"
  runtime warning reduced to "spark3.3".
- `docs/get-started/VeloxQAT.md`: replaced hardcoded old version list
  with "all supported Spark versions" since the script iterates
  `SUPPORTED_SPARK_VERSIONS`.
- `docs/developers/HowToRelease.md`: removed the stale `spark-3.2.tar.gz`
  example line.
- `tools/gluten-it/README.md`: profile list updated to
  spark-3.3, spark-3.4, spark-3.5, spark-4.0, spark-4.1.
- `.github/ISSUE_TEMPLATE/bug.yml`: dropdown updated to
  Spark-3.3.x through Spark-4.1.x (removed Spark-3.2.x, added
  Spark-4.1.x, normalized case).

Comments / small code:
- `shims/spark33/.../OrcFileFormat.scala`: comment no longer references
  Spark 3.2 or a hypothetical shims-spark32.
- `backends-velox/.../RowToVeloxColumnarExec.scala`: removed
  `// For spark 3.2.` above `withNewChildInternal`; the override is the
  standard Spark 3.3+ API.
- `backends-velox/.../CudfNodeValidationRule.scala`: replaced
  `.find(_).isDefined` (used because Spark 3.2 lacked `TreeNode.exists`)
  with `.exists(_)`; dropped the stale comment.
- `backends-velox/.../VeloxAggregateFunctionsSuite.scala`: dropped
  stale "Spark 3.2 does not have this configuration" comment.
- `backends-velox/.../ArithmeticAnsiValidateSuite.scala`: comment
  narrowed from "Spark 3.2 and 3.3" to "Spark 3.3".
- `tools/gluten-it/common/.../SparkJvmOptions.java`: dropped the
  Spark-3.2 `ClassNotFoundException` fallback that returned "";
  consolidated all reflection exceptions into a single multi-catch.
LuciferYang added a commit to LuciferYang/gluten that referenced this pull request Jul 15, 2026
@LuciferYang
[MINOR] Align supported Spark version references
6f83361 
Spark 3.2 was dropped by apache#11351/apache#11687/apache#11731/apache#11887; currently supported
versions per README.md / docs/index.md / pom.xml profiles are Spark 3.3,
3.4, 3.5, 4.0, and 4.1. Various docs, comments, and small code paths still
carry Spark 3.2 leftovers or predate the 4.0/4.1 additions. This PR aligns
them.

Docs:
- `docs/get-started/Velox.md`: version table + prose updated to
  3.3.1, 3.4.4, 3.5.5, 4.0.2, 4.1.1 (matching `<spark.version>` in each
  profile).
- `docs/velox-backend-limitations.md`: three section headers "For
  Spark3.2 and Spark3.3" reduced to "For Spark3.3"; the "spark3.2/3.3"
  runtime warning reduced to "spark3.3".
- `docs/get-started/VeloxQAT.md`: replaced hardcoded old version list
  with "all supported Spark versions" since the script iterates
  `SUPPORTED_SPARK_VERSIONS`.
- `docs/developers/HowToRelease.md`: removed the stale `spark-3.2.tar.gz`
  example line.
- `tools/gluten-it/README.md`: profile list updated to
  spark-3.3, spark-3.4, spark-3.5, spark-4.0, spark-4.1.
- `.github/ISSUE_TEMPLATE/bug.yml`: dropdown updated to
  Spark-3.3.x through Spark-4.1.x (removed Spark-3.2.x, added
  Spark-4.1.x, normalized case).

Comments / small code:
- `shims/spark33/.../OrcFileFormat.scala`: comment no longer references
  Spark 3.2 or a hypothetical shims-spark32.
- `backends-velox/.../RowToVeloxColumnarExec.scala`: removed
  `// For spark 3.2.` above `withNewChildInternal`; the override is the
  standard Spark 3.3+ API.
- `backends-velox/.../CudfNodeValidationRule.scala`: replaced
  `.find(_).isDefined` (used because Spark 3.2 lacked `TreeNode.exists`)
  with `.exists(_)`; dropped the stale comment.
- `backends-velox/.../VeloxAggregateFunctionsSuite.scala`: dropped
  stale "Spark 3.2 does not have this configuration" comment.
- `backends-velox/.../ArithmeticAnsiValidateSuite.scala`: comment
  narrowed from "Spark 3.2 and 3.3" to "Spark 3.3".
- `tools/gluten-it/common/.../SparkJvmOptions.java`: dropped the
  Spark-3.2 `ClassNotFoundException` fallback that returned "";
  consolidated all reflection exceptions into a single multi-catch.
LuciferYang added a commit to LuciferYang/gluten that referenced this pull request Jul 16, 2026
@LuciferYang
[MINOR][CH] Remove residual Spark 3.2 branches from clickhouse tests
0c57b42 
Spark 3.2 was dropped by apache#11351 / apache#11687 / apache#11731 / apache#11887; the
`spark32` protected def in `GlutenClickHouseWholeStageTransformerSuite`
was defined as `sparkVersion.equals("3.2")` and has been dead since.
This PR removes the definition and prunes every reachable
`if (spark32) ...` / `if (!spark32) ...` / `${if (spark32) ... else ...}`
branch to keep only the Spark 3.3+ path.

Test-code changes (all in `backends-clickhouse/src/test/`):

- `GlutenClickHouseWholeStageTransformerSuite.scala`: drop the
  `protected def spark32` definition (always false).
- `GlutenClickHouseTPCHBucketSuite.scala`: `hasSortByCol = !spark32`
  collapses to `true` on all supported Sparks; version-gated
  `if (spark32) ...` branches removed.
- `GlutenClickHouseTPCHParquetBucketSuite.scala`,
  `GlutenClickHouseDeltaParquetWriteSuite.scala`,
  `GlutenClickHouseMergeTreeWriteSuite.scala`,
  `GlutenClickHouseMergeTreeOptimizeSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSWithRocksDBMetaSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnS3Suite.scala`,
  `GlutenClickHouseMergeTreePathBasedWriteSuite.scala`: every reachable
  `if (spark32) ... else ...` block, `if (!spark32) ...` guard, and
  inline `${if (spark32) "" else "SORTED BY (...)"}` interpolation is
  reduced to the Spark 3.3+ path (always emit `SORTED BY`).
- `GlutenClickHouseTPCDSParquetAQESuite.scala`,
  `GlutenClickHouseTPCDSParquetColumnarShuffleAQESuite.scala`:
  comments narrowed from "On Spark 3.2, ... on Spark 3.3, ..." to
  describe only the surviving Spark 3.3+ shape.
- `hive/GlutenClickHouseNativeWriteTableSuite.scala`: drop stale
  `// spark 3.2 without orc or parquet suffix` comment.

Main-code change (one file):

- `RowToCHNativeColumnarExec.scala`: drop the `// For spark 3.2.`
  comment above `withNewChildInternal`. The override is required by
  `TreeNode`'s API on every Spark version currently supported by
  Gluten, not a Spark 3.2-only quirk. Mirrors the same cleanup for
  `RowToVeloxColumnarExec` included in apache#12525.

Explicitly kept for a separate follow-up PR:

- `backends-clickhouse/.../ExtendedColumnPruning.scala:66-72` — the
  local `getAttributeToExtractValues` re-implementation exists because
  Spark 3.2's upstream signature was 2-arg. On 3.3+ it is 3-arg; the
  local copy could be replaced with a delegate. That is a real
  refactor, not a comment fix.
- `backends-clickhouse/.../CHColumnarWrite.scala:157` — the
  `bucketSpec` reflection was needed for Spark 3.2, may be replaceable
  with direct access on 3.3+. Also a real refactor.
- `CustomSum.scala:28` — historical provenance of a copied file, not
  a version gate; keep as-is.

Verified with `mvn -pl backends-clickhouse -am install -Pspark-3.3,
backends-clickhouse,delta` (SUCCESS) and `scalastyle:check spotless:check`
(SUCCESS).
LuciferYang added a commit to LuciferYang/gluten that referenced this pull request Jul 16, 2026
@LuciferYang
[MINOR][CH] Remove residual Spark 3.2 branches from clickhouse tests
f844047 
Spark 3.2 was dropped by apache#11351 / apache#11687 / apache#11731 / apache#11887; the
`spark32` protected def in `GlutenClickHouseWholeStageTransformerSuite`
was defined as `sparkVersion.equals("3.2")` and has been dead since.
This PR removes the definition and prunes every reachable
`if (spark32) ...` / `if (!spark32) ...` / `${if (spark32) ... else ...}`
branch to keep only the Spark 3.3+ path.

Test-code changes (all in `backends-clickhouse/src/test/`):

- `GlutenClickHouseWholeStageTransformerSuite.scala`: drop the
  `protected def spark32` definition (always false).
- `GlutenClickHouseTPCHBucketSuite.scala`: `hasSortByCol = !spark32`
  collapses to `true` on all supported Sparks; version-gated
  `if (spark32) ...` branches removed.
- `GlutenClickHouseTPCHParquetBucketSuite.scala`,
  `GlutenClickHouseDeltaParquetWriteSuite.scala`,
  `GlutenClickHouseMergeTreeWriteSuite.scala`,
  `GlutenClickHouseMergeTreeOptimizeSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSWithRocksDBMetaSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnS3Suite.scala`,
  `GlutenClickHouseMergeTreePathBasedWriteSuite.scala`: every reachable
  `if (spark32) ... else ...` block, `if (!spark32) ...` guard, and
  inline `${if (spark32) "" else "SORTED BY (...)"}` interpolation is
  reduced to the Spark 3.3+ path (always emit `SORTED BY`).
- `GlutenClickHouseTPCDSParquetAQESuite.scala`,
  `GlutenClickHouseTPCDSParquetColumnarShuffleAQESuite.scala`:
  comments narrowed from "On Spark 3.2, ... on Spark 3.3, ..." to
  describe only the surviving Spark 3.3+ shape.
- `hive/GlutenClickHouseNativeWriteTableSuite.scala`: drop stale
  `// spark 3.2 without orc or parquet suffix` comment.

Main-code change (one file):

- `RowToCHNativeColumnarExec.scala`: drop the `// For spark 3.2.`
  comment above `withNewChildInternal`. The override is required by
  `TreeNode`'s API on every Spark version currently supported by
  Gluten, not a Spark 3.2-only quirk. Mirrors the same cleanup for
  `RowToVeloxColumnarExec` included in apache#12525.

Explicitly kept for a separate follow-up PR:

- `backends-clickhouse/.../ExtendedColumnPruning.scala:66-72` — the
  local `getAttributeToExtractValues` re-implementation exists because
  Spark 3.2's upstream signature was 2-arg. On 3.3+ it is 3-arg; the
  local copy could be replaced with a delegate. That is a real
  refactor, not a comment fix.
- `backends-clickhouse/.../CHColumnarWrite.scala:157` — the
  `bucketSpec` reflection was needed for Spark 3.2, may be replaceable
  with direct access on 3.3+. Also a real refactor.
- `CustomSum.scala:28` — historical provenance of a copied file, not
  a version gate; keep as-is.

Verified with `mvn -pl backends-clickhouse -am install -Pspark-3.3,
backends-clickhouse,delta` (SUCCESS) and `scalastyle:check spotless:check`
(SUCCESS).
@LuciferYang LuciferYang mentioned this pull request Jul 16, 2026
Merged
zzcclp pushed a commit that referenced this pull request Jul 16, 2026
@LuciferYang
[MINOR][CH] Remove residual Spark 3.2 branches from clickhouse tests (#…
8226101 
…12532)

[MINOR][CH] Remove residual Spark 3.2 branches from clickhouse tests (#12532)

Spark 3.2 was dropped by #11351 / #11687 / #11731 / #11887; the
`spark32` protected def in `GlutenClickHouseWholeStageTransformerSuite`
was defined as `sparkVersion.equals("3.2")` and has been dead since.
This PR removes the definition and prunes every reachable
`if (spark32) ...` / `if (!spark32) ...` / `${if (spark32) ... else ...}`
branch to keep only the Spark 3.3+ path.

Test-code changes (all in `backends-clickhouse/src/test/`):

- `GlutenClickHouseWholeStageTransformerSuite.scala`: drop the
  `protected def spark32` definition (always false).
- `GlutenClickHouseTPCHBucketSuite.scala`: `hasSortByCol = !spark32`
  collapses to `true` on all supported Sparks; version-gated
  `if (spark32) ...` branches removed.
- `GlutenClickHouseTPCHParquetBucketSuite.scala`,
  `GlutenClickHouseDeltaParquetWriteSuite.scala`,
  `GlutenClickHouseMergeTreeWriteSuite.scala`,
  `GlutenClickHouseMergeTreeOptimizeSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSWithRocksDBMetaSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnS3Suite.scala`,
  `GlutenClickHouseMergeTreePathBasedWriteSuite.scala`: every reachable
  `if (spark32) ... else ...` block, `if (!spark32) ...` guard, and
  inline `${if (spark32) "" else "SORTED BY (...)"}` interpolation is
  reduced to the Spark 3.3+ path (always emit `SORTED BY`).
- `GlutenClickHouseTPCDSParquetAQESuite.scala`,
  `GlutenClickHouseTPCDSParquetColumnarShuffleAQESuite.scala`:
  comments narrowed from "On Spark 3.2, ... on Spark 3.3, ..." to
  describe only the surviving Spark 3.3+ shape.
- `hive/GlutenClickHouseNativeWriteTableSuite.scala`: drop stale
  `// spark 3.2 without orc or parquet suffix` comment.

Main-code change (one file):

- `RowToCHNativeColumnarExec.scala`: drop the `// For spark 3.2.`
  comment above `withNewChildInternal`. The override is required by
  `TreeNode`'s API on every Spark version currently supported by
  Gluten, not a Spark 3.2-only quirk. Mirrors the same cleanup for
  `RowToVeloxColumnarExec` included in #12525.

Explicitly kept for a separate follow-up PR:

- `backends-clickhouse/.../ExtendedColumnPruning.scala:66-72` — the
  local `getAttributeToExtractValues` re-implementation exists because
  Spark 3.2's upstream signature was 2-arg. On 3.3+ it is 3-arg; the
  local copy could be replaced with a delegate. That is a real
  refactor, not a comment fix.
- `backends-clickhouse/.../CHColumnarWrite.scala:157` — the
  `bucketSpec` reflection was needed for Spark 3.2, may be replaceable
  with direct access on 3.3+. Also a real refactor.
- `CustomSum.scala:28` — historical provenance of a copied file, not
  a version gate; keep as-is.

Verified with `mvn -pl backends-clickhouse -am install -Pspark-3.3,
backends-clickhouse,delta` (SUCCESS) and `scalastyle:check spotless:check`
(SUCCESS).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhouyuan zhouyuan zhouyuan approved these changes

Assignees

No one assigned

Labels

CORE works for Gluten Core

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@philo-he @zhouyuan