Repository navigation
Remove remaining Spark 3.2-specific compatibility code #11379
Description
Description
Some Spark 3.2-specific compatibility code remains in the common codebase:
#gluten-core/src/main/scala/org/apache/spark/util/SparkVersionUtil.scala
Defines lteSpark32 (used by other files)
#gluten-core/src/main/scala/org/apache/spark/util/SparkTaskUtil.scala
Lines 67-82: Spark 3.2-specific constructor path for TaskContext
#gluten-core/src/main/scala/org/apache/spark/util/SparkPlanUtil.scala
Lines 26-28: Spark 3.2-specific path for supportsRowBased
#gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/GlutenCostEvaluator.scala
Lines 33-35: Spark 3.2-specific CostEvaluator instantiation
#gluten-core/src/main/scala/org/apache/gluten/extension/columnar/transition/Convention.scala
Lines 174-184: Spark 3.2-specific row type handling
#backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala
Lines 911-919: Test that checks for Spark 3.2
These items are compatibility shims in shared code that can be cleaned up separately.
Also, need to clean up the shims/spark32 directory with:
shims/spark32/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnarArrayShim.java
shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFooterReaderShim.scala
Related Issue: #8960
Gluten version
None
Activity
QCLyu commentedon Jan 22, 2026
take
zhztheplayer commentedon Feb 11, 2026
Thanks for working on the 3.2 deprecation. Will #11495 also disable 3.2 CI?
QCLyu commentedon Feb 12, 2026
@zhztheplayer In the quoted issue, there is no Spark 3.2 CI in the current setup.
In .github/workflows/velox_backend_x86.yml the Spark matrix is [ "spark-3.3", "spark-3.4", "spark-3.5", "spark-4.0", "spark-4.1" ]. So there is no spark-3.2 job to disable. The pom.xml has Spark profiles for 3.3, 3.4, 3.5, 4.0, and 4.1, and no spark-3.2 profile. cc @philo-he in case I got confused.
zhztheplayer commentedon Feb 12, 2026
@QCLyu I found #11351 which I didn't notice earlier. I have no issues with further cleanups and please go ahead.
QCLyu commentedon Mar 2, 2026
Update: I was able to fix local build errors. Resuming the work to deprecate the rest of Spark 3.2 code.
FYI, after running into multiple rabbit holes, the solution to my local build was simple:
<!-- Skip javac entirely. In incremental mode (recompileMode=incremental), Zinc compiles both Scala and Java sources together, making maven-compiler-plugin redundant. Same approach as Apache Spark. --> <skipMain>false</skipMain> <skip>false</skip>I just switched "true" to "false" in the skipMain and skip commands above.
[MINOR] Remove Spark 3.2 tests (#11887)
[MINOR] Remove Spark 3.2 tests (apache#11887)
[CORE] Refactor common expression mappings out of Spark shims (#11941)