[GLUTEN-12569][CORE] Fix compatibility issues addressed in Spark 4.2 by manoj-ragupathy · Pull Request #13020 · apache/gluten

manoj-ragupathy · 2026-09-14T17:15:48Z

What changes are proposed in this pull request?

Pre-work for Spark 4.2.x support (#12569), following the pattern of #11313 in the Spark 4.1 bring-up: land the small, self-contained compatibility fixes first so the shim-layer and unit-test PRs that follow stay reviewable.

Every change here is version-agnostic — this PR adds no spark42 paths and no spark-4.2 profile. It only reshapes code that Spark 4.2 would otherwise break, in a way that works identically on Spark 3.4 through 4.1.

Change	Why Spark 4.2 needs it
Move `doPostDriverMetrics` out of `BatchScanExecTransformer` into each `BatchScanExecShim`	4.2 moved `postDriverMetrics` to `SupportsCustomDriverMetrics` and made the reported task metrics an explicit argument, so the call site has to differ per version
Add `getSampleSeed` to `SparkShims`	`SampleExec.seed` became `Option[Long]`, resolved lazily via `resolvedSeed`
Add `isKeyGroupedPartitioning` to `SparkShims`	`KeyGroupedPartitioning` was renamed to `KeyedPartitioning`
Match on type instead of position in `ConverterUtils` and `GlutenNoopWriterRule`	`CharType` gained `collation`; `AppendDataExec` and `OverwriteByExpressionExec` gained `tableName` and `transaction`, silently breaking positional extractor patterns
Make `GlutenQueryTest` extend Spark's `QueryTest`	4.2 hoisted `checkAnswer`, `checkDataset`, `assertCached` and friends into a new `QueryTestBase` trait that `SharedSparkSession` now mixes in; because `GlutenQueryTest` declared its own copies, suites mixing both inherited conflicting definitions. Extending `QueryTest` (an abstract class up to 4.1, a trait in 4.2) makes Gluten's versions genuine overrides on every supported version
Add `getBinaryView` stub to the `SpecializedGetters` compatibility shims	`SpecializedGetters` gained `getBinaryView` in 4.2. The return type is `Nothing` on purpose, matching the existing `getVariant`/`getGeography`/`getGeometry` stubs, so one definition works across the whole 3.x/4.x matrix
Use `ByteBuffer.allocateDirect` in `OnHeapFileSystemTest`	Spark 4.2 bumps Netty to 4.2.13, which removed `PlatformDependent.allocateDirectNoCleaner`
Ignore a `datasketches` duplicate class in `package/pom.xml`	Spark 4.2's `spark-catalyst` ships a patched copy of `ResourceImpl`, tripping the ban-duplicate-classes enforcer. Both artifacts are `provided`, so nothing extra is packaged

The two new SparkShims methods are implemented in all existing version shims (spark34, spark35, spark40, spark41), keeping gluten-substrait and backends-velox version-agnostic.

Part of #12569.

How was this patch tested?

No behaviour change is intended on any currently supported Spark version, so this relies on existing test coverage.

Compile-verified on all four supported profiles:

for cfg in "3.4 2.12" "3.5 2.12" "4.0 2.13" "4.1 2.13"; do
  set -- $cfg
  ./build/mvn -Pspark-$1 -Pscala-$2 -Pjava-17 -Pbackends-velox -Pspark-ut \
    clean test-compile -DskipTests
done

All four report BUILD SUCCESS. The GlutenQueryTest change is the highest-risk one here, since it affects every suite that mixes it with SharedSparkSession; it was verified on Spark 3.4 (the oldest supported version) as well as 4.1.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI (Claude Opus 5)

github-actions · 2026-09-14T17:16:18Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-09-14T19:30:49Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-09-18T16:41:46Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-09-18T16:44:49Z

Run Gluten Clickhouse CI on x86

Spark 4.2 added parameters to three case classes, which silently breaks positional extractor patterns: - CharType gained a collation parameter. - AppendDataExec / OverwriteByExpressionExec gained tableName and transaction. Match on type (and on the named field where a value is needed) instead, which is stable across all supported Spark versions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b4451c-d099-4965-871d-12781176d082 (cherry picked from commit b2a23c0)

Spark 4.2 hoisted checkAnswer, checkDataset, assertCached and friends out of QueryTest into a new QueryTestBase trait, which SharedSparkSession now mixes in. Because GlutenQueryTest declared its own copies of those members, every suite mixing both inherited conflicting definitions. Extending QueryTest (an abstract class up to 4.1, a trait in 4.2) makes Gluten's versions genuine overrides on every supported Spark version, and removes the duplicated-member conflict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b4451c-d099-4965-871d-12781176d082 (cherry picked from commit a0c9f51)

- Spark 4.2 bumps Netty to 4.2.13, which removed PlatformDependent.allocateDirectNoCleaner. Use ByteBuffer.allocateDirect, which works on every supported version. - Spark 4.2's spark-catalyst ships a patched copy of datasketches ResourceImpl, tripping the ban-duplicate-classes enforcer. Both artifacts are 'provided', so nothing extra is packaged; ignore it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b4451c-d099-4965-871d-12781176d082 (cherry picked from commit 5d4853d)

…rk 4.2 Spark 4.2 adds `getBinaryView` to `SpecializedGetters`, which makes `PlaceholderRow` (and any other `InternalRowSparkCompatible` subclass) fail to compile as non-abstract: BatchCarrierRow.scala:110: error: class PlaceholderRow needs to be abstract. Missing implementation for member of trait SpecializedGetters: def getBinaryView(x$1: Int): org.apache.spark.unsafe.types.BinaryView Handled with the existing cross-version seam: `SpecializedGettersSparkCompatible` gains a `getBinaryView` stub and `InternalRowSparkCompatible` overrides it. The `Nothing` return type conforms to whichever concrete type the running Spark version declares, so one definition covers the whole 3.x/4.x matrix -- exactly how getVariant/getGeography/getGeometry are already handled. Verified with clean builds (stale target/ classes previously masked this): spark-3.4, spark-3.5, spark-4.0, spark-4.1 and spark-4.2 all BUILD SUCCESS. (cherry picked from commit 7210d79)

Spark 4.2 moved postDriverMetrics to SupportsCustomDriverMetrics and made the reported task metrics an explicit argument. Move doPostDriverMetrics out of the version-agnostic BatchScanExecTransformer into each BatchScanExecShim so the call site can differ per version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8ae7d6bd-f561-417a-a635-40fe24dc67a5

Two Spark 4.2 renames that leak into version-agnostic code: - SampleExec.seed became Option[Long], resolved lazily via resolvedSeed. - KeyGroupedPartitioning was renamed to KeyedPartitioning. Both are now hidden behind SparkShims (getSampleSeed, isKeyGroupedPartitioning) so gluten-substrait and backends-velox stay version-agnostic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8ae7d6bd-f561-417a-a635-40fe24dc67a5

github-actions · 2026-09-23T02:08:23Z

Run Gluten Clickhouse CI on x86

baibaichen

LGTM

github-actions Bot added CORE works for Gluten Core VELOX labels Sep 14, 2026

manoj-ragupathy force-pushed the feature/before_42 branch from f4518ed to f673c8a Compare September 14, 2026 19:30

This was referenced Sep 14, 2026

[GLUTEN-12569][CORE] Add Spark 4.2 build profile and shim layer #13021

Draft

[GLUTEN-12569][CORE] Add Spark 4.2 unit test module (gluten-ut/spark42) #13022

Draft

[GLUTEN-12569][CORE] Add enhanced DSv2 filter test suites for Spark 4.2 #13023

Draft

manoj-ragupathy mentioned this pull request Sep 14, 2026

[VL] Support Spark 4.2.x #12569

Open

This was referenced Sep 15, 2026

[GLUTEN-12569][VL] Spark 4.2 support infrastructure (UT triage to follow) akshaytayal/gluten#1

Open

[GLUTEN-12569][CORE][VL] Spark 4.2 support infrastructure (build profile, spark42 shim, gluten-ut/spark42) #13025

Closed

manoj-ragupathy force-pushed the feature/before_42 branch from 6a12fa5 to 0ec9583 Compare September 18, 2026 16:44

MANOJ RAGUPATHY and others added 6 commits September 23, 2026 10:07

baibaichen force-pushed the feature/before_42 branch from 0ec9583 to 013b57c Compare September 23, 2026 02:07

baibaichen approved these changes Sep 23, 2026

View reviewed changes

baibaichen merged commit 7e309ee into apache:main Sep 23, 2026
56 checks passed

This was referenced Sep 24, 2026

[DONT MERGE][GLUTEN-12569][VL] Add Spark 4.2 build profile and shim layer (CI on latest main) akshaytayal/gluten#4

Open

[GLUTEN-12569][VL] Add Spark 4.2 shim layer and build profile #13126

Merged

Navigation Menu

Repository navigation

[GLUTEN-12569][CORE] Fix compatibility issues addressed in Spark 4.2 - #13020

[GLUTEN-12569][CORE] Fix compatibility issues addressed in Spark 4.2#13020
baibaichen merged 6 commits into
apache:mainfrom
manoj-ragupathy:feature/before_42

manoj-ragupathy commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

baibaichen left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

manoj-ragupathy commented Sep 14, 2026

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

baibaichen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants