Skip to content

[CORE] Remove remaining Spark 3.2 compatibility code - #11731

Merged
philo-he merged 6 commits into
apache:mainfrom
QCLyu:qclyu
Mar 12, 2026
Merged

[CORE] Remove remaining Spark 3.2 compatibility code#11731
philo-he merged 6 commits into
apache:mainfrom
QCLyu:qclyu

Conversation

@QCLyu

@QCLyu QCLyu commented Mar 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Remove all lteSpark32 branches from shared code (dead code since Spark 3.2 is no longer supported)
  • Clean up SparkVersionUtil.scala (remove lteSpark32, lteSpark33)
  • Delete orphaned shims/spark32/ files and Spark 3.2-specific test branches
  • Rename KnownRowTypeForSpark33OrLater to KnownRowTypeWithDefault

Test plan

  • Build with Spark 3.3, 3.4, 3.5 profiles to verify no compilation errors
  • Run existing unit tests (no behavioral changes, only dead code removal)

Partially assisted by Claude Opus 4.6.

Closes #11379

@github-actions github-actions Bot added CORE works for Gluten Core VELOX labels Mar 10, 2026
@github-actions

github-actions Bot commented Mar 10, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@QCLyu QCLyu changed the title Qclyu Remove Spark 3.2 compatibility code Mar 10, 2026
@QCLyu

QCLyu commented Mar 10, 2026

Copy link
Copy Markdown
Contributor Author

Hi @philo-he , just thought carefully about your comment in the previous PR: #11495 (comment) This time I renamed KnownRowTypeForSpark33OrLater to KnownRowTypeWithDefault, instead of completely merging it with KnownRowType.

My rationale is "this: SparkPlan =>" created a self-type constraint: If you put this on KnownRowType directly, then any implementor of KnownRowType would have to be a SparkPlan. 

Please correct me if I'm wrong.

@QCLyu
QCLyu marked this pull request as draft March 10, 2026 05:25
@github-actions

github-actions Bot commented Mar 10, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@philo-he

philo-he commented Mar 10, 2026

Copy link
Copy Markdown
Member

Hi @philo-he , just thought carefully about your comment in the previous PR: #11495 (comment) This time I renamed KnownRowTypeForSpark33OrLater to KnownRowTypeWithDefault, instead of completely merging it with KnownRowType.

My rationale is "this: SparkPlan =>" created a self-type constraint: If you put this on KnownRowType directly, then any implementor of KnownRowType would have to be a SparkPlan. 

Please correct me if I'm wrong.

@QCLyu, the constraint is acceptable maybe. This trait was introduced to categorize Spark Plans based on the data format. So I would advise merging them. Thanks.

@github-actions

github-actions Bot commented Mar 11, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@QCLyu
QCLyu marked this pull request as ready for review March 11, 2026 15:16
@QCLyu

QCLyu commented Mar 11, 2026

Copy link
Copy Markdown
Contributor Author

@philo-he could you please review this PR?
One test failed because Query q64 did not have enough memory to build and broadcast the table to all worker nodes. Looks irrelevant.
cc @zhouyuan

philo-he
philo-he approved these changes Mar 12, 2026
View reviewed changes

@philo-he philo-he left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks.

@philo-he philo-he changed the title Remove Spark 3.2 compatibility code [CORE] Remove remaining Spark 3.2 compatibility code Mar 12, 2026
@philo-he
philo-he merged commit 5c8e55a into apache:main Mar 12, 2026
63 of 64 checks passed
LuciferYang added a commit to LuciferYang/gluten that referenced this pull request Jul 15, 2026
@LuciferYang
[MINOR][CORE] Remove residual Spark 3.2 compatibility code in gluten-…
9e3714c 
…core

Spark 3.2 support was removed in prior PRs (apache#11351, apache#11687, apache#11731, apache#11887);
currently supported versions are Spark 3.3, 3.4, 3.5, 4.0, 4.1. A few symbols
that existed only as pre-Spark-3.3 shims are still around and dead code today.

1. `GlutenPlan.SupportsRowBasedCompatible` trait

   Introduced to provide `def supportsRowBased(): Boolean` for Spark < 3.3
   where `SparkPlan.supportsRowBased` did not exist yet. The default body was
   `throw new GlutenException("Illegal state: The method is not expected to
   be called")`. On Spark 3.3+, `SparkPlan.supportsRowBased` is native and
   every concrete GlutenPlan / ColumnarInputAdapter overrides it directly, so
   the trait's default was already unreachable. Drop the trait and its two
   mixin sites.

2. `SparkVersionUtil.gteSpark33`

   Always true after Spark 3.2 was dropped. Also drop the single caller
   guard in `canPropagateConvention` (Transitions), which no longer needs to
   skip UnionExec on Spark 3.2. `eqSpark33` and `comparedWithSpark33` are
   kept: they distinguish Spark 3.3 from 3.4+ (different `TaskContextImpl`
   ctor signature and different write planning API), which is unrelated to
   the Spark 3.2 residual concern.

3. `SparkPlanUtil.supportsRowBased` reflection

   The reflection was needed on Spark 3.2 because `SparkPlan.supportsRowBased`
   did not exist as a member yet; the same compiled artifact ran on 3.2 and
   3.3+ only by resolving the method reflectively at call time. Now that
   Spark 3.2 is dropped, a direct call `plan.supportsRowBased` compiles on
   all supported profiles and is strictly better (primitive Boolean instead
   of boxed, no per-call `getMethod` lookup, no `InvocationTargetException`
   wrapping). The 3 callers in `ConventionFunc` are on the planning hot
   path.

Verified via compile on Spark 3.3, 3.5, and 4.1 (scala-2.13) profiles, plus
gluten-core and gluten-substrait tests on Spark 3.5.
This was referenced Jul 15, 2026
Merged
Merged
LuciferYang added a commit to LuciferYang/gluten that referenced this pull request Jul 15, 2026
@LuciferYang
[MINOR] Align supported Spark version references
2aa6178 
Spark 3.2 was dropped by apache#11351/apache#11687/apache#11731/apache#11887; currently supported
versions per README.md / docs/index.md / pom.xml profiles are Spark 3.3,
3.4, 3.5, 4.0, and 4.1. Various docs, comments, and small code paths still
carry Spark 3.2 leftovers or predate the 4.0/4.1 additions. This PR aligns
them.

Docs:
- `docs/get-started/Velox.md`: version table + prose updated to
  3.3.1, 3.4.4, 3.5.5, 4.0.2, 4.1.1 (matching `<spark.version>` in each
  profile).
- `docs/velox-backend-limitations.md`: three section headers "For
  Spark3.2 and Spark3.3" reduced to "For Spark3.3"; the "spark3.2/3.3"
  runtime warning reduced to "spark3.3".
- `docs/get-started/VeloxQAT.md`: replaced hardcoded old version list
  with "all supported Spark versions" since the script iterates
  `SUPPORTED_SPARK_VERSIONS`.
- `docs/developers/HowToRelease.md`: removed the stale `spark-3.2.tar.gz`
  example line.
- `tools/gluten-it/README.md`: profile list updated to
  spark-3.3, spark-3.4, spark-3.5, spark-4.0, spark-4.1.
- `.github/ISSUE_TEMPLATE/bug.yml`: dropdown updated to
  Spark-3.3.x through Spark-4.1.x (removed Spark-3.2.x, added
  Spark-4.1.x, normalized case).

Comments / small code:
- `shims/spark33/.../OrcFileFormat.scala`: comment no longer references
  Spark 3.2 or a hypothetical shims-spark32.
- `backends-velox/.../RowToVeloxColumnarExec.scala`: removed
  `// For spark 3.2.` above `withNewChildInternal`; the override is the
  standard Spark 3.3+ API.
- `backends-velox/.../CudfNodeValidationRule.scala`: replaced
  `.find(_).isDefined` (used because Spark 3.2 lacked `TreeNode.exists`)
  with `.exists(_)`; dropped the stale comment.
- `backends-velox/.../VeloxAggregateFunctionsSuite.scala`: dropped
  stale "Spark 3.2 does not have this configuration" comment.
- `backends-velox/.../ArithmeticAnsiValidateSuite.scala`: comment
  narrowed from "Spark 3.2 and 3.3" to "Spark 3.3".
- `tools/gluten-it/common/.../SparkJvmOptions.java`: dropped the
  Spark-3.2 `ClassNotFoundException` fallback that returned "";
  consolidated all reflection exceptions into a single multi-catch.
LuciferYang added a commit to LuciferYang/gluten that referenced this pull request Jul 15, 2026
@LuciferYang
[MINOR] Align supported Spark version references
6f83361 
Spark 3.2 was dropped by apache#11351/apache#11687/apache#11731/apache#11887; currently supported
versions per README.md / docs/index.md / pom.xml profiles are Spark 3.3,
3.4, 3.5, 4.0, and 4.1. Various docs, comments, and small code paths still
carry Spark 3.2 leftovers or predate the 4.0/4.1 additions. This PR aligns
them.

Docs:
- `docs/get-started/Velox.md`: version table + prose updated to
  3.3.1, 3.4.4, 3.5.5, 4.0.2, 4.1.1 (matching `<spark.version>` in each
  profile).
- `docs/velox-backend-limitations.md`: three section headers "For
  Spark3.2 and Spark3.3" reduced to "For Spark3.3"; the "spark3.2/3.3"
  runtime warning reduced to "spark3.3".
- `docs/get-started/VeloxQAT.md`: replaced hardcoded old version list
  with "all supported Spark versions" since the script iterates
  `SUPPORTED_SPARK_VERSIONS`.
- `docs/developers/HowToRelease.md`: removed the stale `spark-3.2.tar.gz`
  example line.
- `tools/gluten-it/README.md`: profile list updated to
  spark-3.3, spark-3.4, spark-3.5, spark-4.0, spark-4.1.
- `.github/ISSUE_TEMPLATE/bug.yml`: dropdown updated to
  Spark-3.3.x through Spark-4.1.x (removed Spark-3.2.x, added
  Spark-4.1.x, normalized case).

Comments / small code:
- `shims/spark33/.../OrcFileFormat.scala`: comment no longer references
  Spark 3.2 or a hypothetical shims-spark32.
- `backends-velox/.../RowToVeloxColumnarExec.scala`: removed
  `// For spark 3.2.` above `withNewChildInternal`; the override is the
  standard Spark 3.3+ API.
- `backends-velox/.../CudfNodeValidationRule.scala`: replaced
  `.find(_).isDefined` (used because Spark 3.2 lacked `TreeNode.exists`)
  with `.exists(_)`; dropped the stale comment.
- `backends-velox/.../VeloxAggregateFunctionsSuite.scala`: dropped
  stale "Spark 3.2 does not have this configuration" comment.
- `backends-velox/.../ArithmeticAnsiValidateSuite.scala`: comment
  narrowed from "Spark 3.2 and 3.3" to "Spark 3.3".
- `tools/gluten-it/common/.../SparkJvmOptions.java`: dropped the
  Spark-3.2 `ClassNotFoundException` fallback that returned "";
  consolidated all reflection exceptions into a single multi-catch.
LuciferYang added a commit to LuciferYang/gluten that referenced this pull request Jul 16, 2026
@LuciferYang
[MINOR][CH] Remove residual Spark 3.2 branches from clickhouse tests
0c57b42 
Spark 3.2 was dropped by apache#11351 / apache#11687 / apache#11731 / apache#11887; the
`spark32` protected def in `GlutenClickHouseWholeStageTransformerSuite`
was defined as `sparkVersion.equals("3.2")` and has been dead since.
This PR removes the definition and prunes every reachable
`if (spark32) ...` / `if (!spark32) ...` / `${if (spark32) ... else ...}`
branch to keep only the Spark 3.3+ path.

Test-code changes (all in `backends-clickhouse/src/test/`):

- `GlutenClickHouseWholeStageTransformerSuite.scala`: drop the
  `protected def spark32` definition (always false).
- `GlutenClickHouseTPCHBucketSuite.scala`: `hasSortByCol = !spark32`
  collapses to `true` on all supported Sparks; version-gated
  `if (spark32) ...` branches removed.
- `GlutenClickHouseTPCHParquetBucketSuite.scala`,
  `GlutenClickHouseDeltaParquetWriteSuite.scala`,
  `GlutenClickHouseMergeTreeWriteSuite.scala`,
  `GlutenClickHouseMergeTreeOptimizeSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSWithRocksDBMetaSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnS3Suite.scala`,
  `GlutenClickHouseMergeTreePathBasedWriteSuite.scala`: every reachable
  `if (spark32) ... else ...` block, `if (!spark32) ...` guard, and
  inline `${if (spark32) "" else "SORTED BY (...)"}` interpolation is
  reduced to the Spark 3.3+ path (always emit `SORTED BY`).
- `GlutenClickHouseTPCDSParquetAQESuite.scala`,
  `GlutenClickHouseTPCDSParquetColumnarShuffleAQESuite.scala`:
  comments narrowed from "On Spark 3.2, ... on Spark 3.3, ..." to
  describe only the surviving Spark 3.3+ shape.
- `hive/GlutenClickHouseNativeWriteTableSuite.scala`: drop stale
  `// spark 3.2 without orc or parquet suffix` comment.

Main-code change (one file):

- `RowToCHNativeColumnarExec.scala`: drop the `// For spark 3.2.`
  comment above `withNewChildInternal`. The override is required by
  `TreeNode`'s API on every Spark version currently supported by
  Gluten, not a Spark 3.2-only quirk. Mirrors the same cleanup for
  `RowToVeloxColumnarExec` included in apache#12525.

Explicitly kept for a separate follow-up PR:

- `backends-clickhouse/.../ExtendedColumnPruning.scala:66-72` — the
  local `getAttributeToExtractValues` re-implementation exists because
  Spark 3.2's upstream signature was 2-arg. On 3.3+ it is 3-arg; the
  local copy could be replaced with a delegate. That is a real
  refactor, not a comment fix.
- `backends-clickhouse/.../CHColumnarWrite.scala:157` — the
  `bucketSpec` reflection was needed for Spark 3.2, may be replaceable
  with direct access on 3.3+. Also a real refactor.
- `CustomSum.scala:28` — historical provenance of a copied file, not
  a version gate; keep as-is.

Verified with `mvn -pl backends-clickhouse -am install -Pspark-3.3,
backends-clickhouse,delta` (SUCCESS) and `scalastyle:check spotless:check`
(SUCCESS).
LuciferYang added a commit to LuciferYang/gluten that referenced this pull request Jul 16, 2026
@LuciferYang
[MINOR][CH] Remove residual Spark 3.2 branches from clickhouse tests
f844047 
Spark 3.2 was dropped by apache#11351 / apache#11687 / apache#11731 / apache#11887; the
`spark32` protected def in `GlutenClickHouseWholeStageTransformerSuite`
was defined as `sparkVersion.equals("3.2")` and has been dead since.
This PR removes the definition and prunes every reachable
`if (spark32) ...` / `if (!spark32) ...` / `${if (spark32) ... else ...}`
branch to keep only the Spark 3.3+ path.

Test-code changes (all in `backends-clickhouse/src/test/`):

- `GlutenClickHouseWholeStageTransformerSuite.scala`: drop the
  `protected def spark32` definition (always false).
- `GlutenClickHouseTPCHBucketSuite.scala`: `hasSortByCol = !spark32`
  collapses to `true` on all supported Sparks; version-gated
  `if (spark32) ...` branches removed.
- `GlutenClickHouseTPCHParquetBucketSuite.scala`,
  `GlutenClickHouseDeltaParquetWriteSuite.scala`,
  `GlutenClickHouseMergeTreeWriteSuite.scala`,
  `GlutenClickHouseMergeTreeOptimizeSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSWithRocksDBMetaSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnS3Suite.scala`,
  `GlutenClickHouseMergeTreePathBasedWriteSuite.scala`: every reachable
  `if (spark32) ... else ...` block, `if (!spark32) ...` guard, and
  inline `${if (spark32) "" else "SORTED BY (...)"}` interpolation is
  reduced to the Spark 3.3+ path (always emit `SORTED BY`).
- `GlutenClickHouseTPCDSParquetAQESuite.scala`,
  `GlutenClickHouseTPCDSParquetColumnarShuffleAQESuite.scala`:
  comments narrowed from "On Spark 3.2, ... on Spark 3.3, ..." to
  describe only the surviving Spark 3.3+ shape.
- `hive/GlutenClickHouseNativeWriteTableSuite.scala`: drop stale
  `// spark 3.2 without orc or parquet suffix` comment.

Main-code change (one file):

- `RowToCHNativeColumnarExec.scala`: drop the `// For spark 3.2.`
  comment above `withNewChildInternal`. The override is required by
  `TreeNode`'s API on every Spark version currently supported by
  Gluten, not a Spark 3.2-only quirk. Mirrors the same cleanup for
  `RowToVeloxColumnarExec` included in apache#12525.

Explicitly kept for a separate follow-up PR:

- `backends-clickhouse/.../ExtendedColumnPruning.scala:66-72` — the
  local `getAttributeToExtractValues` re-implementation exists because
  Spark 3.2's upstream signature was 2-arg. On 3.3+ it is 3-arg; the
  local copy could be replaced with a delegate. That is a real
  refactor, not a comment fix.
- `backends-clickhouse/.../CHColumnarWrite.scala:157` — the
  `bucketSpec` reflection was needed for Spark 3.2, may be replaceable
  with direct access on 3.3+. Also a real refactor.
- `CustomSum.scala:28` — historical provenance of a copied file, not
  a version gate; keep as-is.

Verified with `mvn -pl backends-clickhouse -am install -Pspark-3.3,
backends-clickhouse,delta` (SUCCESS) and `scalastyle:check spotless:check`
(SUCCESS).
@LuciferYang LuciferYang mentioned this pull request Jul 16, 2026
Merged
zzcclp pushed a commit that referenced this pull request Jul 16, 2026
@LuciferYang
[MINOR][CH] Remove residual Spark 3.2 branches from clickhouse tests (#…
8226101 
…12532)

[MINOR][CH] Remove residual Spark 3.2 branches from clickhouse tests (#12532)

Spark 3.2 was dropped by #11351 / #11687 / #11731 / #11887; the
`spark32` protected def in `GlutenClickHouseWholeStageTransformerSuite`
was defined as `sparkVersion.equals("3.2")` and has been dead since.
This PR removes the definition and prunes every reachable
`if (spark32) ...` / `if (!spark32) ...` / `${if (spark32) ... else ...}`
branch to keep only the Spark 3.3+ path.

Test-code changes (all in `backends-clickhouse/src/test/`):

- `GlutenClickHouseWholeStageTransformerSuite.scala`: drop the
  `protected def spark32` definition (always false).
- `GlutenClickHouseTPCHBucketSuite.scala`: `hasSortByCol = !spark32`
  collapses to `true` on all supported Sparks; version-gated
  `if (spark32) ...` branches removed.
- `GlutenClickHouseTPCHParquetBucketSuite.scala`,
  `GlutenClickHouseDeltaParquetWriteSuite.scala`,
  `GlutenClickHouseMergeTreeWriteSuite.scala`,
  `GlutenClickHouseMergeTreeOptimizeSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnHDFSWithRocksDBMetaSuite.scala`,
  `GlutenClickHouseMergeTreeWriteOnS3Suite.scala`,
  `GlutenClickHouseMergeTreePathBasedWriteSuite.scala`: every reachable
  `if (spark32) ... else ...` block, `if (!spark32) ...` guard, and
  inline `${if (spark32) "" else "SORTED BY (...)"}` interpolation is
  reduced to the Spark 3.3+ path (always emit `SORTED BY`).
- `GlutenClickHouseTPCDSParquetAQESuite.scala`,
  `GlutenClickHouseTPCDSParquetColumnarShuffleAQESuite.scala`:
  comments narrowed from "On Spark 3.2, ... on Spark 3.3, ..." to
  describe only the surviving Spark 3.3+ shape.
- `hive/GlutenClickHouseNativeWriteTableSuite.scala`: drop stale
  `// spark 3.2 without orc or parquet suffix` comment.

Main-code change (one file):

- `RowToCHNativeColumnarExec.scala`: drop the `// For spark 3.2.`
  comment above `withNewChildInternal`. The override is required by
  `TreeNode`'s API on every Spark version currently supported by
  Gluten, not a Spark 3.2-only quirk. Mirrors the same cleanup for
  `RowToVeloxColumnarExec` included in #12525.

Explicitly kept for a separate follow-up PR:

- `backends-clickhouse/.../ExtendedColumnPruning.scala:66-72` — the
  local `getAttributeToExtractValues` re-implementation exists because
  Spark 3.2's upstream signature was 2-arg. On 3.3+ it is 3-arg; the
  local copy could be replaced with a delegate. That is a real
  refactor, not a comment fix.
- `backends-clickhouse/.../CHColumnarWrite.scala:157` — the
  `bucketSpec` reflection was needed for Spark 3.2, may be replaceable
  with direct access on 3.3+. Also a real refactor.
- `CustomSum.scala:28` — historical provenance of a copied file, not
  a version gate; keep as-is.

Verified with `mvn -pl backends-clickhouse -am install -Pspark-3.3,
backends-clickhouse,delta` (SUCCESS) and `scalastyle:check spotless:check`
(SUCCESS).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@philo-he philo-he philo-he approved these changes

Assignees

No one assigned

Labels

CLICKHOUSE CORE works for Gluten Core VELOX

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Remove remaining Spark 3.2-specific compatibility code

2 participants

@QCLyu @philo-he