[GLUTEN-12807][CORE] Deprecate and remove Spark 3.3 support

[LuciferYang]

What changes are proposed in this pull request?

Removes the Spark 3.3 build surface. This covers three items in #12807: the Spark-33 unit tests, the Spark-33 shim source tree, and the building scripts. Two follow-ups remain, described at the end.

Seven directories go, 328 files and about 41k lines: shims/spark33, gluten-ut/spark33, backends-clickhouse/src-delta23, gluten-delta/src-delta23, backends-clickhouse/src-iceberg-spark33, gluten-iceberg/src-iceberg5, gluten-iceberg/src-iceberg3. The spark-3.3 profile is gone from the six poms that declared one, and the root pom's requireActiveProfile list now names only the four surviving versions, so -Pspark-3.3 fails with Missing spark version profile instead of quietly resolving to whatever the defaults are.

gluten-iceberg/src-iceberg3 needs a word since its name does not mention 3.3: no profile ever set iceberg.binary.version=3, so no build could reach it, and at the base commit its two files were byte-identical to src-iceberg5's. It was already dead, and src-iceberg5 was reachable only from the 3.3 profile, so both go together.

LICENSE carried twelve entries pointing into shims/spark33/ and src-delta23/. Those are removed, and the remaining 24 paths were checked to still resolve.

Four CI jobs ran only on Spark 3.3 and are the sole coverage for what they exercise, so they are retargeted to 3.5 rather than deleted: tpc-test-ubuntu-oom, tpc-test-ubuntu-randomkill, tpc-test-centos8-uniffle, tpc-test-ubuntu-2204-celeborn. For celeborn the gluten-it install command moved with it, and it now takes the profile from matrix.spark like the Gluten build two steps above it. spark-test-spark33 and spark-test-spark33-slow are deleted outright: 3.4 and 3.5 run the same profile set (-Piceberg -Pdelta -Phudi -Ppaimon), so no combination is orphaned. The shims33 change-detection flag and its path filter go with them.

The ClickHouse release chain hardcoded 3.3 end to end: ep/build-clickhouse/src/package.sh built only that version and bin/gluten.sh only recognised jars/spark33, so leaving it would have shipped a tarball whose launcher cannot find its own jars. Both move to 3.5, which also settles the # TODO: support spark 3.5 later sitting next to it. I have no ClickHouse environment, so that part rests on reading and bash -n. The ClickHouse CI triggers itself here (this PR touches pom.xml, backends-clickhouse/**, shims/** and more, which the trigger workflow's path filters match), but a CH maintainer's eyes on the packaging change would still help.

Four things reviewers should know.

install-spark-resources.sh no longer installs the 3.3 distribution, so docker_image.yml will rebuild the images and /opt/shims/spark33/spark_home disappears from them. Nothing in the repo references that path, but a branch that still does will break once the new image is published.

The ClickHouse CI is red on this PR for a reason that cannot be fixed from here. That job is a Jenkins build outside this repository (its workspace is .../gluten/gluten-ci/, and its parent pom is io.kyligence:devopslib), and it invokes mvn clean test -Pbackends-clickhouse -Pspark-3.3 -Pjava-8 ... -Dspark.test.home=/tmp/spark33. With the profile gone, enforce-spark-profile stops it before anything compiles: Missing spark version profile: -Pspark-<version>. Nothing in apache/gluten generates that command line, so whoever owns the job needs to move it to -Pspark-3.5 and a Spark 3.5 spark.test.home, which is the only version the ClickHouse backend has a profile for anyway. Worth noting that its reactor listing already shows Gluten Shims for Spark 3.5, so that environment appears to activate spark-3.5 from its own settings rather than from the command line. Because it fails at the first enforcer rule, that run says nothing yet about whether the ClickHouse changes in this PR are sound.

dev/release/build-release.sh had a "Java 8 is required" gate whose only purpose was the Spark 3.3 build directly below it. With that build gone nothing in the script needs Java 8 (the four remaining builds all pass -Pjava-17, and the script installs Corretto 17 a few lines later), while the gate still rejects a release manager on JDK 17, and it does so after the longest step in the script. The design note for this work listed it as out of scope; I removed it here instead, since this change is what made it vestigial. dev/release/package-release.sh needed the same treatment for a different reason: its SPARK_VERSIONS list still demanded a 3.3 bundle jar that can no longer be built, and it hard-fails on a missing jar before it packages anything, so a release cut would have produced zero artifacts.

Five test cases are pinned to 3.3 with testWithMaxSparkVersion / testWithSpecifiedSparkVersion, and those helpers skip the test() call entirely when no version matches, so from this PR until PR 3 they run nowhere and leave no skip marker: VeloxParquetWriteForHiveSuite's hive.exec.compress.output case, ClickhouseGlutenKafkaScanSuite's only case, and three in GlutenClickHouseNativeWriteTableSuite. Two carry TODOs saying they fail on 3.5, so each needs a decision (port or delete plus an issue) rather than a substitution, which is why they belong to PR 3.

Deliberately left alone: the nineteen ignoreClass entries in package/pom.xml under "The overridden class list by Gluten" are now inert, because all five classes they name were overridden only in shims/spark33, but banDuplicateClasses runs with fail=true and dropping the entries could surface a duplicate from a dependency jar, so that is its own call. tools/gluten-it/pom.xml is a standalone pom without requireActiveProfile, so -Pspark-3.3 there now warns and builds 3.5 rather than failing; adding the same gate would also outlaw the profile-less build that works today. And tools/workload/benchmark_velox/initialize.ipynb still provisions Spark 3.3.1 across five coupled lines including a symlink; I cannot test that notebook, so I left it rather than half-update it.

Follow-ups under #12807, both based on this one. PR 2 cleans up the shim APIs that exist only because 3.3 diverged from later versions, modelled on the Spark 3.2 cleanup in #11687: lift those methods into the common trait, inline the ones whose only remaining caller is trivial, drop getExtendedColumnarPostRules now that it is always empty, and turn the trait defaults that only 3.3 needed into abstract methods. PR 3 removes the residual version checks that are not shim APIs (eqSpark33 and its call sites, the dead AlignExpandOutputTypes, the NativeWritePostRule registration, the spark33 helper in the ClickHouse tests) and handles the five pinned cases above.

How was this patch tested?

Local verification, all against this branch:

check	result
mvn validate -Pspark-3.3	fails with Missing spark version profile, as intended
test-compile for 3.4, 3.5, 4.0, 4.1	all pass
full-feature builds (-Pceleborn,uniffle -Piceberg,delta,hudi,paimon)	pass
-Pbackends-clickhouse build	passes, which is what shows src-delta does not reference anything that lived only in src-delta23
dev/format-scala-code.sh --check	passes, no Missing spark version profile
docker buildx build --check on Dockerfile.ubuntu22-m2-cache	one NoEmptyContinuation warning, byte-for-byte the same warning the pre-change file produces
bash -n on every changed script	passes

One caveat on the local runs: the 3.5 builds needed -Pscala-2.13 to get around corrupted spark-*_2.12/3.5.5 jars in my local ~/.m2, and that combination has to skip -Ppaimon because upstream never published paimon-spark-3.5_2.13. The 2.12 path is what CI exercises.

The license header check needs a note, because I got it wrong at first. It inspects every file a PR changes, not only the files it adds, so removing the spark-3.3 profile from shims/pom.xml and the stale clean exclude from package/pom.xml put both under the check, and both carried the short Apache notice rather than the ASF header. They now carry the ASF header, matching gluten-ut/pom.xml and the root pom. I replaced the short block instead of stacking the ASF one above it, which is what check.py's own fix mode does and would have left two license blocks in each file. bash .github/workflows/util/check.sh <base> exits 0 locally after the change (it needs pip install regex).

Unit tests are what CI settles, not a local run. The ClickHouse packaging changes are the part I am least able to verify myself.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude claude-opus-5

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[LuciferYang]

Drive-by fix, unrelated to Spark 3.3, flagged here because it sits in the same RUN as the continuation fix below.

spark-4 is not a profile id. The root pom defines spark-4.0 and spark-4.1, so Maven printed The requested profile "spark-4" could not be activated because it does not exist. and carried on with the root defaults. enforce-spark-profile does not catch it either: dependency:go-offline invokes the plugin goal directly rather than running a lifecycle, and that execution binds to validate, so it never fires. I confirmed the command exits 0 with only the warning.

Falling back to the defaults gives a mix that is mostly 3.5: sparkshim.artifactId resolves to spark-sql-columnar-shims-spark35, sparkbundle.version to 3.5, delta.version to 3.3.2. Only spark.version happens to match, since the root default is already 4.0.2, so the Spark 4.0 jars did land in the cache while the Delta 4.0.1 and spark40 shim coordinates did not. That is the whole cost: no red build, just a cache image that does not hold what this stanza was meant to pre-populate for the 4.0 jobs.

Introduced in #11655. Happy to pull it into its own PR if you would rather keep this one to the removal.

[LuciferYang]

Split out to #12903 and reverted here, so this PR's only change to this file is the removal of the Spark 3.3 stanza plus the trailing ; \ that removal left dangling.

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

Pull request overview

Copilot reviewed 161 out of 369 changed files in this pull request and generated 1 comment.

[jackylee-ch]

cc @philo-he @zhouyuan

[LuciferYang]

A correction to my last comment: I blamed my own setup for the ClickHouseIcebergHiveTableSupport failure, and that was wrong. It is a build bug, and once I identified it I could verify the rest properly.

The iceberg and iceberg-test profiles both declare a build-helper-maven-plugin execution with the id add-iceberg-test-sources, at pom.xml:1861 and pom.xml:1912 on main. Maven merges executions by id, so when both profiles are active only one <sources> list survives. help:effective-pom with -Pbackends-clickhouse -Pspark-3.5 -Piceberg -Piceberg-test resolves that execution to src-iceberg-spark35/test/{scala,java} alone; the iceberg profile's src-iceberg/test/{scala,java} and src-iceberg10/test/{scala,java} are dropped.

For ClickHouse that combination is unbuildable by construction. ClickHouseIcebergHiveTableSupport lives in src-iceberg/test/scala, and both Java tests instantiate it, so the class leaves the source path at exactly the moment the tests join it. Giving the iceberg-test execution a distinct id and changing nothing else makes test-compile -Pbackends-clickhouse -Pspark-3.5 -Pscala-2.13 -Pspark-ut -Pdelta -Piceberg -Piceberg-test exit 0 and emit both ClickHouseIcebergHiveTableSupport.class and TestPositionDeletesTableGluten.class. So TestFlinkUpsert has not compiled since #9996 introduced this layout, and neither has its sibling in either copy. #9996 is also the commit that added the ported 3.5 sibling and left this file on 3.3.

Velox never hits the collision, which is why it has gone unnoticed: nothing there needs both directories at once. The src-iceberg/test suites run in the jobs that pass -Piceberg alone, and the 57 files under src-iceberg-spark34/test/java run in the three groups that pass both.

With the ids de-conflicted, a straight copy of TestFlinkUpsert into src-iceberg-spark35 fails on one symbol, org.apache.iceberg.flink.MiniClusterResource, used once for DISABLE_CLASSLOADER_CHECK_CONFIG. So "migrating the test to the 1.19 test API" was too strong: the file is already JUnit 5, with @TestTemplate and org.apache.iceberg.Parameters, and MiniFlinkClusterExtension is the replacement.

I would still keep all of it out of this PR, since fixing the profile ids changes what three Velox groups compile and would newly run a ClickHouse test that has never run. Filed as #12933 for the execution-id collision, and #12934 for the ClickHouse side, which covers the CI wiring, the MiniClusterResource port, and the Flink dependency block that has no user today.

[philo-he]

LGTM. Please rebase the code and resolve code conflicts. Thank you.

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

Pull request overview

Copilot reviewed 67 out of 412 changed files in this pull request and generated no new comments.

[LuciferYang]

LGTM. Please rebase the code and resolve code conflicts. Thank you.

done

[jackylee-ch]

@zhouyuan @zhztheplayer @zzcclp Any other comments on this PR?

I’ll leave it open for a couple more days to allow some time for additional feedback before merging.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[github-actions]

Run Gluten Clickhouse CI on x86

[philo-he]

I think it should be safe to merge these changes. Let's merge it to avoid easy conflict.

[prestodb-ci]

Rebase job