[VL] Support Spark 4.2.x #12569

Activity

1. malinjawi

   added

   enhancementNew feature or requestNew feature or request

   on Jul 19, 2026
2. wangyum commented on Jul 23, 2026

   

   wangyum

   on Jul 23, 2026

   Member

   More actions

   #12567 and #12556 These are for better compatibility with Spark 4.2, since our internal version has backported lots of Spark 4.2 features:
3. kapilks commented on Aug 25, 2026

   

   kapilks

   on Aug 25, 2026

   Contributor

   More actions

   Have we started with 4.2 support work?
4. weiting-chen commented on Aug 25, 2026

   

   weiting-chen

   on Aug 25, 2026

   Contributor

   More actions

   Apache Gluten has not started the work to support Spark 4.2 yet.
   At this stage, the community is gathering feedback from users and evaluating the demand and requirements for Spark 4.2 support. That said, we have already seen interest from several customers and users who would like to adopt Spark 4.2. Based on the feedback received so far, I believe Spark 4.2 support will likely become part of the roadmap in the near future, although no formal commitment or timeline has been announced yet.

   👍React with 👍1Reacted by Kapil Kumar Singh
5. manoj-ragupathy commented on Sep 14, 2026

   

   manoj-ragupathy

   on Sep 14, 2026

   Contributor

   More actions

   I'd like to pick this up. I've completed an initial Spark 4.2.0 bring-up for the Velox backend and would be glad to contribute it upstream if the approach looks right to the community.

   cc @weiting-chen @wangyum @baibaichen — following up on the "gathering feedback" note above.

   Proposed delivery — four sequenced PRs

   Mirroring the Spark 4.1 sequence (#11313 → #11347 → #11353 → #11380), which maps onto the proposed work list in the description:

   PR	Scope	Size
   1	Fix compatibility issues introduced by Spark 4.2 (version-agnostic only)	19 files
   2	spark-4.2 build profile + shims/spark42 layer + CI resources	44 files
   3	gluten-ut/spark42 unit test module + CI UT jobs	~1850 files
   4	Additional Spark 4.2 test suites	4 files

   Splitting this way keeps PR-1 and PR-2 small enough to review before the large gluten-ut move lands.

   Verification so far

   All four branches build against their target profiles. PR-1 is version-agnostic by construction and compile-verified on Spark 3.4, 3.5, 4.0 and 4.1 (it contains no spark42 paths), so it should be safe to merge independently of the rest.

   I should be clear about the limits: this is compile and build verification only. I have not been able to execute the unit tests locally, so the suites in PR-3/PR-4 are unproven at runtime — that needs CI (see blocker below).

   Both #12567 and #12556 are already in my base, and I did not redo them.

   Main compatibility deltas found

   Most of the Spark 4.2 delta is absorbed by the shim layer, but not all of it:

   • SPJ redesign — KeyGroupedPartitioning → KeyedPartitioning, StoragePartitionJoinParams and KeyGroupedPartitionedScan removed, BatchScanExec constructor changed, filteredPartitions now Seq[Option[InputPartition]], runtime filters pushed via PushDownUtils.pushRuntimeFilters.
   • postDriverMetrics now takes an explicit Array[CustomTaskMetric].
   • SampleExec.seed became Option[Long].
   • QueryTest — 4.2 hoists checkAnswer and friends into a new QueryTestBase trait that SharedSparkSession mixes in, which collided with GlutenQueryTest's own copies.
   • Netty 4.2.13 removed PlatformDependent.allocateDirectNoCleaner.
   • CharType, AppendDataExec, OverwriteByExpressionExec, BroadcastHashJoinExec and DataSourceV2ScanRelation all gained parameters, breaking positional extractor patterns.

   One behavioural degradation I want to flag explicitly for review: Spark 4.2 removed spjParams, and I found no scan-level equivalent for the partially-clustered "common partition values" (that logic moved into EnsureRequirements/GroupPartitionsExec). On 4.2 only, getCommonPartitionValues conservatively returns None, which disables the partially-clustered-distribution refinement; the base fully-clustered SPJ path is unaffected. Spark 3.4–4.1 behaviour is unchanged. I'd appreciate scrutiny on whether that's an acceptable starting point or needs a deeper port.

   Two things I need help with

   1. The 4.2 UT jobs cannot pass on a code change alone. -Dspark.test.home=/opt/shims/spark42/spark_home/ is baked into the CI container image at image-build time, and docker_image.yml only rebuilds on the Sunday cron. PR-2 adds the install-spark-resources.sh 4.2 lines to the Dockerfiles, but someone with committer rights would need to workflow_dispatch an image rebuild before the UT lanes in PR-3 can go green.
   2. PR-3 must be rebase-merged, not squashed — it uses the move/copy-back commit structure from [Discuss] Keep the commit history when adding a new spark version #11352 so git log --follow keeps history on the migrated suites. I verified --follow still traces through the move into pre-existing history.

   Also note the 4.2 UT lane drops -Pdelta: delta-spark_4.2, iceberg-spark-runtime-4.2_2.13 and hudi-spark4.2-bundle_2.13 do not exist on Maven Central yet, so those lanes can only be enabled once artifacts ship.

   Happy to adjust the split, ordering, or scope based on what reviewers prefer. Could this issue be assigned to me?

   Generated-by: GitHub Copilot CLI (Claude Opus 5)
6. manoj-ragupathy commented on Sep 14, 2026

   

   manoj-ragupathy

   on Sep 14, 2026

   Contributor

   More actions

   All four PRs are now open. I've opened the whole stack rather than waiting for each to merge, so the full shape of the port is visible for review — but the ordering still matters, and only the first is ready to merge.

   PR	Scope	State
   #13020	Version-agnostic compatibility fixes (no spark42 paths)	Ready for review
   #13021	spark-4.2 profile + shims/spark42 layer + CI resources	Draft — stacked on #13020
   #13022	gluten-ut/spark42 test module + CI UT jobs	Draft — stacked on #13021
   #13023	52 additional enhanced DSv2 filter tests	Draft — stacked on #13022

   Because GitHub requires PRs to target a branch in this repo, each stacked PR's diff currently also shows its parents' commits. Each description states exactly which top-N commits belong to it. I'll rebase and undraft each one as its parent merges, at which point the diffs shrink to their real size.

   The whole stack is rebased onto current main (9f6dcb5) as of now. #13020 is compile-verified on Spark 3.4 / 3.5 / 4.0 / 4.1, and the stack tip builds green on 4.2 (clean install -Pspark-ut).

   Repeating the two asks from my earlier comment, since they're what actually gates progress:

   1. A committer needs to workflow_dispatch docker_image.yml. [GLUTEN-12569][CORE] Add Spark 4.2 build profile and shim layer #13021 adds the Spark 4.2 resources to the Dockerfile, but the image only rebuilds on a Sunday cron. Until it's rebuilt, [GLUTEN-12569][CORE] Add Spark 4.2 unit test module (gluten-ut/spark42) #13022's UT lanes can't run regardless of code quality — so the test PRs have no CI signal yet. This is the single biggest blocker.
   2. [GLUTEN-12569][CORE] Add Spark 4.2 unit test module (gluten-ut/spark42) #13022 must be rebase-merged, not squashed, to preserve git log --follow through the spark41 → spark42 rename commit (per [Discuss] Keep the commit history when adding a new spark version #11352).

   And one design question I'd rather resolve early than after review: the getCommonPartitionValues → None degradation on 4.2 only, described in #13021. Happy to pursue a fuller solution if that's preferred over landing the gap.

   Generated-by: GitHub Copilot CLI (Claude Opus 5)