Repository navigation
Conversation
Fixes CVE-2024-29857 (F2m curve high CPU DoS via crafted X.509), CVE-2024-30171 (RSA Bleichenbacher-style timing oracle), CVE-2024-30172 (Ed25519 infinite loop on crafted signature), CVE-2024-34447 (hostname handling). bcprov-jdk18on / bcpkix-jdk18on remain Java 8 compatible. BouncyCastle is only pulled in as a runtime dependency under the yarn profile. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Fixes CVE-2024-25710 (infinite-loop DoS via crafted DUMP file) and CVE-2024-26308 (OutOfMemory DoS via crafted Pack200 file). commons-compress 1.26.x requires Java 8 (minimum) and is API compatible with the streams Spark uses. core compiles on Java 8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Fixes CVE-2025-24970 (native crash via crafted SSL packet in SslHandler), CVE-2025-25193 (DoS via unsafe environment-file handling; completed fix in 4.1.119) and CVE-2024-29025 (HttpPostRequestDecoder OOM DoS). Patch-level bump within the 4.1.x line; API stable and Java 8 compatible. network-common compiles on Java 8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Fixes CVE-2024-7254 (StackOverflow DoS when parsing untrusted protobuf messages with deeply nested groups). Same fix Hadoop applied in HADOOP-19289. Updates protobuf.version in pom.xml and protoVersion in project/SparkBuild.scala (kept consistent per SPARK-41247). protoc 3.25.5 regenerates the status store types; protobuf-java 3.25.x is Java 8 compatible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Fixes CVE-2022-46751 (XML External Entity injection when parsing module metadata). The earlier SPARK-44968 hold referred to a HiveExternalCatalogVersionsSuite abort under Java 11/17 daily tests; this branch targets Java 8 and matches the later SPARK-44914 upgrade. core compiles on Java 8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Pin two transitive dependencies above their CVE-fixed versions via dependencyManagement: - gson 2.2.4 -> 2.10.1: CVE-2022-25647 (deserialization DoS) - jettison 1.1 -> 1.5.4: CVE-2022-40149/40150/45685/45693 (parser DoS) Both are backward and Java 8 compatible. dependency:tree confirms gson now resolves to 2.10.1 in core. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Fixes CVE-2023-26031 (relative library resolution leading to local privilege escalation) and picks up the YARN-10495 container-executor fix shipped in 3.3.5. 3.3.6 is the latest 3.3.x maintenance release and a drop-in for Spark 3.5; it supports Java 8. Also updates the hard-coded Hadoop fallbackVersion in IsolatedClientLoader.scala to match (per the pom.xml reminder comment). Verified: core, sql/catalyst, sql/core and sql/hive all compile on Java 8 with hadoop-client-api/runtime 3.3.6. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Reconciles dev/deps/spark-deps-hadoop-3-hive-2.3 with the CVE-driven version bumps in this branch (generated via dependency:build-classpath across all hadoop-3-hive-2.3 profiles, LC_ALL=C sorted): - commons-compress 1.23.0 -> 1.26.2 - netty 4.1.96.Final -> 4.1.119.Final (+ linux-riscv64 epoll native) - ivy 2.5.1 -> 2.5.2 - gson 2.2.4 -> 2.10.1, jettison 1.1 -> 1.5.4 (transitive pins) - hadoop 3.3.4 -> 3.3.6, which also brings aws-java-sdk-bundle 1.12.262 -> 1.12.367 and wildfly-openssl 1.0.7 -> 1.1.3.Final, and drops hadoop-openstack and jaxb-api transitively. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Fixes CVE-2025-55163 (Netty MadeYouReset HTTP/2 DDoS). Unlike the earlier Rapid-Reset issue, this is triggered by the server sending RST_STREAM in response to malformed control frames, so it was not covered by the 4.1.119 fixes. Patched in 4.1.124.Final. Patch-level bump within 4.1.x; Java 8 compatible. network-common full test suite passes on Java 8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Fixes CVE-2026-0636 (LDAP injection in LDAPStoreHelper, affects BC-JAVA 1.74 before 1.84) and CVE-2025-8916. bcprov/bcpkix-jdk18on 1.84 remain Java 8 compatible (jdk18on multi-release jar, min JDK 8). BouncyCastle is only a runtime dependency under the yarn profile; yarn module compiles on Java 8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Fixes CVE-2025-27821 (out-of-bounds write in the HDFS native client URI parser; affects Hadoop 3.2.0 through 3.4.1, fixed in 3.4.2). 3.4.3 is the latest 3.4.x. As a bonus, hadoop-aws 3.4.x moves to AWS SDK v2, dropping the transitive aws-java-sdk-bundle 1.12.x and its v1 CVEs. Hadoop branch 3.4 still supports Java 8. Extends IsolatedClientLoader.supportsHadoopShadedClient to recognize 3.4.x so the Hive isolated client keeps using hadoop-client-api/runtime, and updates the hard-coded fallbackVersion to 3.4.3. Verified on Java 8: core, sql/catalyst, sql/core, sql/hive and hadoop-cloud compile; core test sources compile; FileSuite passes (41 tests, 0 failures). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Reconciles the manifest with the round-2 CVE bumps (generated via
dependency:build-classpath across all profiles, LC_ALL=C sorted):
- netty 4.1.119.Final -> 4.1.124.Final
- hadoop 3.3.6 -> 3.4.3, which:
* replaces the AWS SDK v1 'aws-java-sdk-bundle' 1.12.367 with the
AWS SDK v2 'bundle' 2.35.4 (hadoop-aws v2 migration)
* bumps hadoop-shaded-guava 1.1.1 -> 1.5.0 and
wildfly-openssl 1.1.3.Final -> 2.2.5.Final
* adds optional hadoop-cloud connectors (hadoop-huaweicloud/esdk-obs,
analyticsaccelerator-s3, java-xmlbuilder) and bumps aliyun-sdk-oss.
(BouncyCastle is yarn-profile/runtime only and not in this manifest.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Holden Karau <holden@pigscanfly.ca>
…pat) The Netty 4.1.124 bump (CVE-2025-55163) breaks arrow-memory-netty 12.0.1: it reaches into io.netty.buffer.PooledByteBufAllocator's 'chunkSize' field, which Netty 4.1.120+ removed, so NettyAllocationManager.<clinit> throws NoSuchFieldError and ArrowUtilsSuite (all Arrow usage) aborts. The upstream fix (apache/arrow-java#728) only lands in Arrow 18+, which requires Java 11; Arrow v17 is the last Java 8 release, so Arrow cannot be upgraded under the Java 8 constraint. Instead, switch Spark from arrow-memory-netty to Arrow's Netty-free Unsafe allocation manager (arrow-memory-unsafe), which is Java 8 compatible and avoids the Netty internals entirely. Arrow stays 12.0.1, so the Arrow IPC wire format is unchanged and pyarrow needs no change. Spark code is allocator-agnostic (no references to the allocator classes). Verified: catalyst resolves arrow-memory-unsafe (no -netty); ArrowUtilsSuite passes; full catalyst/test has 0 aborted suites (was 1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Hadoop 3.4.3's hadoop-aws moved from AWS SDK v1 to v2, so the EOL
com.amazonaws:aws-java-sdk-bundle:1.11.375 is no longer on any classpath
and the Kubernetes integration-tests module failed to test-compile
("object amazonaws is not a member of package com"), surfacing in the
full SBT Test/compile job even though K8s itself is not being upgraded.
Mirror upstream Spark's own Hadoop-3.4 migration:
- Rewrite DepsTestsSuite getS3Client/createS3Bucket/createS3Object to the
AWS SDK v2 builder API (S3Client, AwsBasicCredentials, CreateBucketRequest,
PutObjectRequest + RequestBody, Region).
- Replace the v1 aws-java-sdk-bundle test dep with software.amazon.awssdk:bundle
(aws.java.sdk.v2.version=2.35.4, the version hadoop-aws:3.4.3 already pulls
in), declared in the main <dependencies> block instead of the hadoop-3
profile so both Maven and SBT test-compile resolve it.
AWS SDK v2 requires Java 8+ (bundle classes are major-version 52). The
assembly manifest is unchanged (test-scope dep in an optional module).
Verified: clean scala:testCompile under Java 8 (1.8.0_492) = BUILD SUCCESS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Holden Karau <holden@pigscanfly.ca>
…y 4.1.124 compat The CVE-2025-55163 Netty bump to 4.1.124.Final is incompatible with grpc-netty 1.56.0 (built for Netty 4.1.87). Per grpc-java's SECURITY.md, Netty 4.1.124 is supported only by grpc-java 1.75.x-1.76.x; running 1.56 against it breaks the Spark Connect server's HTTP/2 framing at runtime, resetting streams (client sees CANCELLED: Received RST_STREAM with error code 8, surfaced in the pyspark-connect CI job). Netty can't be downgraded (it is the CVE fix) and grpc-netty-shaded would re-bundle the vulnerable Netty, so bump grpc-java to 1.76.0. Align protobuf-java 3.25.5 -> 3.25.8 (what grpc 1.76 pulls). grpc-java 1.76 still supports Java 8. The Python grpcio is a separate implementation and needs no change (it interoperates over the wire). The 1.56->1.76 jump surfaced two transitive-dependency breakages, caught by a full Test/compile: - grpc-protobuf demotes grpc-protobuf-lite to runtime scope; SparkConnectService uses io.grpc.protobuf.lite.ProtoLiteUtils at compile time -> declare it explicitly at compile scope in connect/server. - io.grpc.inprocess was split into a separate grpc-inprocess artifact; the client test suites use InProcess{Channel,Server}Builder -> add grpc-inprocess at test scope in connect/client/jvm. Verified on Java 8 (1.8.0_492): connect common/server/client-jvm compile and test-compile clean (grpc stubs regenerated by protoc-gen-grpc-java 1.76); ReattachableExecuteSuite (real Netty grpc server, reattachable streaming) 15/15 passed, no RST_STREAM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Move to the last release of the sbt 1.9.x line for the CVE-remediation branch's CI. API/behavior-compatible within 1.9.x; the launcher jar resolves from Maven Central. appveyor-install-dependencies.ps1 no longer pins an sbt version, so nothing to update there. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
…ependencyOverrides The sbt build does not honor Maven <dependencyManagement> for transitive dependencies, so CVE-bumped libraries regressed to the older versions declared by transitive consumers. This is the same class of issue the existing DependencyOverrides object already patches for guava/xerces/jline/avro. Three fixes: - commons-compress: the override was left at the vulnerable 1.23.0 when pom.xml was bumped to 1.26.2, so the sbt build still pulled the version with CVE-2024-25710 / CVE-2024-26308. Bump to 1.26.2 to match pom.xml. - slf4j-api: breeze (via mllib-local) declares slf4j-api 1.7.5 transitively; sbt resolved that ancient version and CI failed fetching slf4j-api-1.7.5.jar from a partial .m2 (pom cached, jar evicted/never published). Force 2.0.7. - netty: transitive consumers (grpc/arrow) declare older netty (4.1.91/4.1.60.Final); force the full module set to the managed 4.1.124.Final so the sbt build resolves and does not regress past the Netty CVE bump. Verified with `build/sbt mllib-local/update` (now [success]; previously failed fetching slf4j-api-1.7.5.jar then netty 4.1.91/4.1.60 jars). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
…ment in pom.xml sbt does not honor Maven <dependencyManagement> for transitive deps, so CVE version bumps to slf4j/commons-compress/netty in pom.xml must also be mirrored in `object DependencyOverrides` in SparkBuild.scala. Add comments at each of those version properties (mirroring the existing SPARK-41247 protobuf note) so the next person bumping a version keeps the two builds in sync. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca>
Co-authored-by: Holden Karau <holden@pigscanfly.ca>
…3.18.0 (JDK 21) compat (#31)
* [SPARK-CVE][BUILD][3.5] Surface sbt errors in dev/mima instead of swallowing them dev/mima builds the tools and oldDeps classpaths via command substitution: TOOLS_CLASSPATH="$(build/sbt ... "export tools/fullClasspath" | grep jar | tail -n1)" All sbt output (including any [error] report) is captured into the variable, and with `set -e -o pipefail` any sbt failure aborts the script without printing anything. In CI the MIMA step then dies ~35s in showing only the two sbt-launcher stderr lines, which makes resolution failures undiagnosable. Buffer each export run in a temp file, check the exit status explicitly, and dump the sbt output to stderr when it fails (or when no classpath line is produced). Successful runs behave exactly as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdmNouFA7cCw3J11Xb7Djm * [SPARK-CVE][BUILD][3.5] Exclude netty native transport jars from the sbt build to fix MiMa The MIMA lint step has failed silently since the netty dependencyOverrides were introduced. Root cause, reproduced locally with a CI-equivalent ~/.m2: [error] (tools / update) FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:.../.m2/repository/io/netty/netty-transport-native-epoll/4.1.124.Final/netty-transport-native-epoll-4.1.124.Final.jar: not found [error] file:.../.m2/repository/io/netty/netty-transport-native-kqueue/4.1.124.Final/netty-transport-native-kqueue-4.1.124.Final.jar: not found Chain: selenium-remote-driver (a test dep of every module via the parent pom) declares netty-transport-native-epoll/kqueue WITHOUT a platform classifier; the dependencyOverrides force those modules to 4.1.124.Final; the lint job runs dev/test-dependencies.sh (full mvn install) right before dev/mima, which leaves the 4.1.124.Final POMs plus only the *classified* jars (linux-x86_64 etc.) in ~/.m2 - Maven never fetches the classifier-less jars. Coursier prefers the local Maven repo once a POM is present there and fails `update` with "not found" instead of falling back to Maven Central. No forced version can work here (any version Maven metadata-walks is POM-only locally), and un-forcing them re-breaks on selenium's own netty version the same way, so exclude the two native-transport modules from the sbt build entirely. They only carry the native .so (an optional perf feature - Spark defaults to NIO and no test selects EPOLL); the io.netty.channel.epoll/kqueue classes still come from netty-transport-classes-* which remain overridden to 4.1.124.Final. The Maven build and the published deps manifest are unchanged. Verified locally against a CI-equivalent poisoned ~/.m2: both dev/mima classpath exports and GenerateMIMAIgnore now succeed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdmNouFA7cCw3J11Xb7Djm * [SPARK-CVE][BUILD][3.5] Pin jopt-simple/geny/sourcecode to Maven-mediated versions for sbt Running the full dev/mima against a freshly Maven-populated ~/.m2 (what the lint job produces via dev/test-dependencies.sh when its docs-maven cache does not paper over it) surfaced three more updates failing on the same POM-present-but-jar-missing coursier failure mode as the netty native jars: [error] (sql-kafka-0-10 / update) ... jopt-simple-5.0.4.jar: not found [error] (streaming-kafka-0-10 / update) ... jopt-simple-5.0.4.jar: not found [error] (connect-client-jvm / update) ... geny_2.12-0.7.1.jar: not found [error] ... sourcecode_2.12-0.3.0.jar: not found In each case Maven's mediation resolves a different version than coursier's highest-wins does, so only the Maven-picked jar exists in ~/.m2: - The kafka module poms explicitly pin jopt-simple 3.2 (test) and Maven takes it over kafka_2.12's transitive 5.0.4 ("omitted for conflict with 3.2"). - In ammonite 2.5.9's tree (provided dep of connect-client-jvm) Maven picks geny 0.7.0 (via os-lib 0.8.0) and sourcecode 0.2.7 (via ammonite-terminal), while coursier picks 0.7.1 / 0.3.0. Pin all three in DependencyOverrides to the versions Maven mediates, per this object's existing purpose ("work around sbt's dependency resolution being different from Maven's"). CI currently only survives these because its accumulated docs-maven cache still contains the old jars; a cache eviction would have broken the MIMA step again. Verified: with these pins plus the netty native exclusion, the complete ./dev/mima (both classpath exports, GenerateMIMAIgnore, and mimaReportBinaryIssues over the whole build) passes locally against a CI-equivalent ~/.m2 with zero resolution errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdmNouFA7cCw3J11Xb7Djm --------- Co-authored-by: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This pull request updates a set of Spark 3.5 build and runtime dependencies to newer (often CVE-fixing) versions, and adjusts Spark’s build/test tooling and integration tests to remain compatible with those upgrades (notably Netty, Hadoop, Arrow, gRPC/Protobuf, and AWS SDK changes).
Changes:
- Bump core dependency versions (Hadoop 3.4.3, Netty 4.1.124.Final, Protobuf 3.25.8, gRPC 1.76.0, Ivy 2.5.2, commons-* updates, etc.) and mirror required overrides for the sbt build.
- Switch Arrow allocator dependency from
arrow-memory-nettytoarrow-memory-unsafeto avoid Netty internal API breakage. - Update Kubernetes integration tests from AWS SDK v1 to AWS SDK v2 and adjust the Maven test dependencies accordingly.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala | Updates fallback Hadoop version and shaded client gating for Hadoop 3.4+. |
| sql/api/pom.xml | Replaces Arrow Netty allocator with Unsafe allocator and documents the rationale. |
| resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala | Migrates MinIO/S3 test helper to AWS SDK v2 S3 client and request APIs. |
| resource-managers/kubernetes/integration-tests/pom.xml | Adds AWS SDK v2 bundle as a test dependency and removes the v1 bundle profile. |
| project/SparkBuild.scala | Updates protobuf/grpc versions and adds sbt dependency overrides/exclusions to match Maven’s managed dependency set. |
| project/build.properties | Bumps sbt version. |
| pom.xml | Updates dependency versions/properties and adds dependencyManagement pins (e.g., gson/jettison); switches Arrow memory module. |
| dev/mima | Improves error visibility when sbt classpath export fails. |
| dev/deps/spark-deps-hadoop-3-hive-2.3 | Refreshes the expected dependency listing to reflect upgrades. |
| connector/connect/server/pom.xml | Adds an additional gRPC artifact dependency to the Connect server module. |
| connector/connect/client/jvm/pom.xml | Adds grpc-inprocess as a test dependency for the Connect JVM client module. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+310
to
316
| val credentials = AwsBasicCredentials.create(accessKey, secretKey) | ||
| val s3client = S3Client.builder() | ||
| .credentialsProvider(StaticCredentialsProvider.create(credentials)) | ||
| .endpointOverride(URI.create(endPoint)) | ||
| .region(Region.of(REGION)) | ||
| .build() | ||
| s3client |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a397d43cb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| <log4j.version>2.20.0</log4j.version> | ||
| <!-- make sure to update IsolatedClientLoader whenever this version is changed --> | ||
| <hadoop.version>3.3.4</hadoop.version> | ||
| <hadoop.version>3.4.3</hadoop.version> |
There was a problem hiding this comment.
With this Hadoop bump, spark-hadoop-cloud no longer brings in hadoop-openstack: the regenerated dependency list dropped that jar, and a repo-wide rg hadoop-openstack only finds the Swift docs rather than another dependency declaration. Those docs (docs/storage-openstack-swift.md:38-39) tell users to include the hadoop-cloud module to get Swift support, so applications using swift://... through Spark's published hadoop-cloud artifact will now lose org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem at runtime unless the module is retained or the documented support is explicitly removed/updated.
Useful? React with 👍 / 👎.
HyukjinKwon
reviewed
Jul 7, 2026
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
Review: which of these upgrades are actually required for Spark 3.5?
I went through each bump with a reachability lens: does the CVE's vulnerable code path get exercised by Spark's own code, given what's on the 3.5 runtime classpath and how Spark uses the library? I also checked what upstream Apache Spark did on branch-3.5 (which is frozen: netty 4.1.96, grpc 1.56.0, protobuf 3.23.4, hadoop 3.3.4, ivy 2.5.1, commons-compress 1.23.0, gson 2.2.4, jettison 1.1 across every 3.5.0–3.5.8 release).
TL;DR — under a strict "only upgrade when absolutely required" bar, almost none of these clear it. The CVEs are either in code paths Spark never triggers, in libraries Spark doesn't drive (transitive via Hadoop/Hive), test-only, or opt-in cloud connectors. The big hadoop 3.3.4 -> 3.4.3 jump in particular pulls a large cascade (AWS SDK v1->v2 migration, arrow allocator swap, netty/guava churn, four net-new transitive connectors) for a security payoff that's essentially one MEDIUM local temp-dir CVE.
Not reachable — recommend dropping (pure version churn, not a Spark fix)
| Dep | CVE | Why it's not required |
|---|---|---|
| commons-lang3 3.12->3.18 | CVE-2025-48924 | Vuln is ClassUtils.getClass(String) unbounded recursion. Spark never calls it — only ClassUtils.isAssignable / getAllInterfaces, on internal class metadata. Upstream bumped this in 4.1.0 (SPARK-52778) as routine hygiene, not a CVE backport, never on 3.5. |
| gson 2.2.4->2.10.1 | CVE-2022-25647 | Transitive via Hive; not declared in 3.5 pom.xml. Spark's own JSON is Jackson/json4s — it never deserializes untrusted input through gson. Upstream never pinned it for this CVE. |
| jettison 1.1->1.5.4 | CVE-2022-40149/40150/45685/45693 | Transitive via Hadoop YARN. Zero jettison imports in Spark source; Spark never feeds it untrusted JSON/XML. It's Hadoop's parser, Hadoop's CVE. Only ever moved upstream as a side-effect of Hadoop bumps. |
| commons-compress 1.23->1.26.2 | CVE-2024-25710 / CVE-2024-26308 | Vulns are in the DUMP and Pack200 readers. Spark 3.5's only use is CountingOutputStream (a byte counter, output side) in EventLogFileWriters. It never parses DUMP/Pack200. Upstream 3.5 stayed on 1.23.0. |
| hadoop 3.3.4->3.4.3 | CVE-2023-26031, CVE-2025-27821, CVE-2024-23454 | The two HIGH CVEs are in native binaries Spark doesn't ship (YARN container-executor C binary; libhdfs native client). Spark ships the Java hadoop-client-api/runtime jars and uses the JVM FS client. Only CVE-2024-23454 (MEDIUM, local temp-dir perms) is jar-resident — and it doesn't need the 3.4 jump. Upstream held all of 3.5.x on 3.3.4 and only moved to 3.4.1 at the 4.0 major boundary; no Spark release ever used 3.4.3. |
| wildfly-openssl 1.0.7->2.2.5 | none | No CVE. Opt-in S3A OpenSSL channel mode only. This is purely a side-effect of the Hadoop jump. |
| aliyun-sdk-oss 3.13.0->3.13.2 | none | No CVE for the Java SDK. Opt-in oss:// connector. Side-effect of the Hadoop jump. |
| bouncycastle 1.77->1.84 | CVE-2024-30171/30172/34447 | Real CVEs, but on branch-3.5 BC is declared <scope>test</scope> in dependencyManagement (only used by MiniYARNCluster tests) — it is not on the runtime/distribution classpath (absent from dev/deps). Zero runtime exposure. Cheap bump, but not "required." |
Reachable in principle but already mitigated / Connect-only — weak justification
| Dep | CVE | Assessment |
|---|---|---|
| protobuf-java 3.23.4->3.25.8 | CVE-2024-7254 | This is the only CVE whose vulnerable path Spark ingests (Connect server parses client protobuf). But Spark 3.5 already caps it: SparkConnectService installs ProtoLiteUtils.marshallerWithRecursionLimit at spark.connect.grpc.marshallerRecursionLimit (default 1024, version("3.5.0"), SPARK-44587) on every inbound marshaller. CVE-2024-7254 is an uncontrolled-recursion StackOverflow — the exact path is already bounded. Connect-only, opt-in, trusted-network by design. |
| io.grpc 1.56->1.76 | CVE-2023-44487 (Rapid Reset) | Connect-server-only; not on the default classpath. HTTP/2 DoS against an exposed Connect endpoint — but Connect has no default auth and is trusted-network-by-design per Spark's security model, so an exposed endpoint already faces arbitrary-plan execution. Large version jump (1.56->1.76); if kept, Connect must be re-tested. |
| netty 4.1.96->4.1.124 | CVE-2024-29025, CVE-2025-24970, CVE-2025-55163 | Every one is unreachable in Spark's usage: the HTTP / HTTP2 CVEs need a netty HTTP server (Spark runs none — its RPC is custom binary framing); CVE-2025-24970 needs the native SSLEngine (tcnative is not shipped; SSLFactory defaults to JDK provider). Defensible only as low-risk hygiene on a core lib, not as an exploitable-vuln fix. |
If the real driver is SCA scanner compliance
Say so explicitly — that's a legitimate reason, but it's "silence the scanner," not "fix an exploitable hole in Spark," and the two justify very different risk budgets. For the transitive ones (gson, jettison, wildfly, aliyun) a narrow dependencyManagement pin closes the scanner finding without the Hadoop 3.4 cascade. For ivy, see the inline note — the fix version matters.
Net recommendation
Keep at most: ivy (real reachable XXE via --packages/spark.jars.ivySettings — but pin 2.5.3, not 2.5.2; see inline). Everything else is either unreachable, test-only, transitive-not-driven-by-Spark, or opt-in-connector churn that upstream 3.5 deliberately avoided. I'd drop the Hadoop 3.4.3 jump entirely and, with it, the arrow-allocator swap, AWS SDK v1->v2 migration, and the wildfly/aliyun/guava churn that ride on it.
Detailed per-line notes inline.
| daily tests; this branch targets Java 8 builds. Matches SPARK-44914. | ||
| --> | ||
| <ivy.version>2.5.1</ivy.version> | ||
| <ivy.version>2.5.2</ivy.version> |
There was a problem hiding this comment.
This is the one bump I'd keep — but pin 2.5.3, not 2.5.2.
CVE-2022-46751 (XXE) is genuinely reachable: Spark parses remote POMs via IBiblioResolver/ChainResolver and loads a user-supplied Ivy settings XML (spark.jars.ivySettings) in MavenUtils.scala, both XML paths Ivy 2.5.2 hardens. So unlike the rest, this one clears the reachability bar.
But 2.5.2 is exactly the version upstream reverted in SPARK-44968 — it broke HiveExternalCatalogVersionsSuite on Java 11/17 with "Multiple artifacts of the module log4j#log4j;1.2.17 are retrieved to the same file." The comment here says "this branch targets Java 8 builds," but that's fragile: it re-introduces a known-bad jar for anyone building/testing on 11/17. The clean fix landed at ivy 2.5.3 (SPARK-50794, Spark 4.0), which carries the same XXE fix without the 2.5.2 regression. Recommend 2.5.3.
| <log4j.version>2.20.0</log4j.version> | ||
| <!-- make sure to update IsolatedClientLoader whenever this version is changed --> | ||
| <hadoop.version>3.3.4</hadoop.version> | ||
| <hadoop.version>3.4.3</hadoop.version> |
There was a problem hiding this comment.
Recommend dropping the Hadoop jump — this is the biggest over-reach in the PR.
Reachability of the CVEs 3.4.x "fixes":
- CVE-2023-26031 (HIGH) — YARN
container-executor, a setuid C binary in a node-manager install. Not shipped in Spark'shadoop-client-api/runtimejars. - CVE-2025-27821 (HIGH) —
libhdfsnative client. Spark uses the JVMDistributedFileSystem, not the native client. (Also fixed only in 3.4.2, so even Spark 4.0's 3.4.1 wouldn't have it.) - CVE-2024-23454 (MEDIUM) —
RunJartemp-dir perms. This is in the shipped jar, but it's a local same-host disclosure, not remote, and does not require the 3.4 jump — a 3.3.6 patch bump or restrictivejava.io.tmpdiraddresses it.
Upstream context: every Spark 3.5.0–3.5.8 stayed on 3.3.4; the move to 3.4.1 (not 3.4.3) is SPARK-49550, fixVersion 4.0.0 only. No Spark release ever shipped 3.4.3. This minor-version jump is what drags in the AWS SDK v1->v2 migration, hadoop-shaded-guava 1.1.1->1.5.0, the arrow-allocator swap, and four net-new transitive connectors (esdk-obs-java, hadoop-huaweicloud, analyticsaccelerator-s3, java-xmlbuilder) — net-new attack surface, for a payoff of one MEDIUM local CVE. If specific transitive libs are flagged, pin them via dependencyManagement instead of moving Hadoop.
| <hadoop.version>3.4.3</hadoop.version> | ||
| <!-- SPARK-41247: When updating `protobuf.version`, also need to update `protoVersion` in `SparkBuild.scala` --> | ||
| <protobuf.version>3.23.4</protobuf.version> | ||
| <protobuf.version>3.25.8</protobuf.version> |
There was a problem hiding this comment.
protobuf 3.23.4 is technically < 3.25.5 so CVE-2024-7254 (uncontrolled-recursion StackOverflow) applies to the jar — and this is the only CVE in the PR whose vulnerable path Spark actually ingests (the Connect server parses client protobuf). But the path is already bounded in 3.5: SparkConnectService.methodWithCustomMarshallers wraps every inbound marshaller with ProtoLiteUtils.marshallerWithRecursionLimit(..., spark.connect.grpc.marshallerRecursionLimit) (default 1024, version("3.5.0"), SPARK-44587). The CVE is specifically about unbounded recursion; Spark caps it before the stack can overflow.
So this is Connect-only (not on the default classpath), opt-in, trusted-network-by-design (no default auth), and already mitigated. Worth keeping only if you ship+expose Connect and want defense-in-depth — not "absolutely required." Note the grpc 1.56->1.76 bump that rides with it is a large jump; re-test Connect if kept.
| compatible and Java 8 compatible. --> | ||
| <dependency> | ||
| <groupId>com.google.code.gson</groupId> | ||
| <artifactId>gson</artifactId> |
There was a problem hiding this comment.
gson is transitive via Hive — it's not declared in the 3.5 pom.xml today, so this adds a new managed pin. CVE-2022-25647's writeReplace deserialization path isn't reachable from Spark's own code (Spark's JSON is Jackson/json4s; the shaded Connect gson is separate). If a scanner is flagging the 2.2.4 jar, frame this as SCA-compliance, not a Spark fix. Otherwise it's droppable.
| <dependency> | ||
| <groupId>org.apache.arrow</groupId> | ||
| <artifactId>arrow-memory-netty</artifactId> | ||
| <artifactId>arrow-memory-unsafe</artifactId> |
There was a problem hiding this comment.
This allocator swap (arrow-memory-netty -> arrow-memory-unsafe) is a functional runtime change, not a version bump, and it diverges from the entire upstream lineage — Spark master and even 4.0 still use arrow-memory-netty. Arrow's allocator is on the hot path for columnar transfer and Python/R UDF (Arrow) execution, so switching allocation managers is a real behavioral change deserving its own justification and testing.
It only exists here to work around a NoSuchFieldError that the CVE-driven netty 4.1.124 bump triggers in arrow-memory-netty 12.0.1. That's the tail wagging the dog: since none of the netty CVEs are reachable in Spark's usage (no netty HTTP/HTTP2 server; native SSL/tcnative not shipped), dropping the forced netty bump removes the need for this swap entirely.
|
So TL;DR: I'd keep: ivy - CVE-2022-46751 XXE is genuinely reachable via --packages/spark.jars.ivySettings. But the PR pins 2.5.2, the exact version upstream reverted (SPARK-44968) for a Java 11/17 test break; the safe fix is 2.5.3 (SPARK-50794). |
…uffle fetch decoders FetchShuffleBlocks and FetchShuffleBlockChunks decode an outer element count and eagerly allocate a 2D array of that size. Check the count against the bytes actually received before allocating: each inner group carries its own 4-byte length prefix, so a valid count never exceeds readableBytes() / 4. Uses an explicit comparison since Objects.checkFromIndexSize is unavailable on Java 8. Backport of 9687793 Closes apache#58617 from holdenk/17-validate-fetch-counts-3.5. Lead-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com> Signed-off-by: Holden Karau <holden.karau@snowflake.com>
…Delegation Improve Transport Context Stream Manager Delegation by requiring authentication when enabled This is a backport of 3b47eee behind a feature flag since 3.5 is in extended LTS mode. Generated-By: Claude (multiple models) and Cursor with Kimi (K3) Closes apache#58612 from holdenk/f003-r2-branch-3.5. Lead-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com> Signed-off-by: Holden Karau <holden.karau@snowflake.com>
String index values are copied verbatim into store keys, so an embedded key separator (0x00) corrupts the index key structure and mixes entries across range scans and deletes. Fail fast with IllegalArgumentException in LevelDBTypeInfo and RocksDBTypeInfo instead. This is a backport of 23fdef7 Closes apache#58616 from holdenk/111-kvstore-index-validation-branch-3.5. Lead-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com> Signed-off-by: Holden Karau <holden.karau@snowflake.com>
Better handling for custom log names and URLs Backport of b3149c6 (Closes apache#58699) to branch-3.5. Two deviations from the master commit: the ui-test jest module does not exist on branch-3.5 so the JS unit test is not included, and the escapeHtml helper (added on master by 432adc7, not backported) is included here since formatLogsCells now depends on it. Custom log names can be weird, so let's validate that they're displayable before we display them. Invalid log URLs will be replaced with plain text. New unit tests Generated-by: Claude (mixture of models) and Cursor (Kimi K3) Closes apache#59026 from holdenk/f026-validate-custom-log-names-3.5-backport. Authored-by: Holden Karau <holden@pigscanfly.ca> Co-Authored-by: Holden Karau <holden.karau@snowflake.com> Signed-off-by: Holden Karau <holden.karau@snowflake.com>
…ory server ### What changes were proposed in this pull request? Backport of SPARK-59407 to branch-3.5, based on master commit 38258cb (apache#58700). Skip excessively long event log lines during History Server replay: `ReplayListenerBus` reads lines with a per-line byte cap (`spark.history.fs.eventLog.maxLineLength`, default 512m) instead of materializing unbounded lines; over-long lines are drained and skipped with a warning. Branch adaptations vs master: the config declares `.version("3.5.10")` and drops `.withBindingPolicy(...)` (ConfigBindingPolicy exists only on 4.2+); the new warning uses plain string interpolation since branch-3.5 predates the structured logging framework. ### Why are the changes needed? Individual corrupt or unexpectedly large event log lines can make loading a previous Spark job in the History Server slow or exhaust memory; bounding the per-line length keeps replay memory bounded. ### Does this PR introduce _any_ user-facing change? Yes: new configuration `spark.history.fs.eventLog.maxLineLength` (default 512m). Event log lines longer than the cap are skipped with a warning instead of being read into memory during replay; setting it to 0 or a negative value disables the limit. ### How was this patch tested? New test in `ReplayListenerSuite` covering that an over-long line is skipped while neighboring events still replay. Full core module test suite run locally on this branch, green. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude and Cursor Made with [Cursor](https://cursor.com) Closes apache#59058 from holdenk/f040-compressed-file-handling-branch-3.5. Lead-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com> Signed-off-by: Holden Karau <holden.karau@snowflake.com>
…TA_BYTE_ARRAY reader ### What changes were proposed in this pull request? Validate and fast fail invalid prefix lengths while reading parquet data. Backport 08d777a (note: upstream includes geo fix, but there is no geo read path in 3.5) ### Why are the changes needed? The prefix length in a DELTA_BYTE_ARRAY page is read from the file and was used to copy bytes out of the previous value without validation. Reject negative prefix lengths and prefix lengths larger than the previous value with a ParquetDecodingException, so corrupt pages fail the read deterministically instead of producing malformed values or hard-to-diagnose low-level exceptions. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? New unit test ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude (mixture of version) & Cursor (with Kimi K3). Closes apache#59050 from holdenk/33-parquet-delta-page-validation-branch-3.5. Lead-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com> Signed-off-by: Holden Karau <holden.karau@snowflake.com>
…s in UI ### What changes were proposed in this pull request? Make more side-effect causing behavior in the Spark Web UI POST v.s. GET/HEAD when not deployed on YARN. Also validate that the requests have a CSRF token. Even in YARN deployments GETs which have pre-fetch headers (`x-moz`: `prefecth`) will be rejected with 403. `app/kill`, `driver/kill`, `app/hold`, `app/resume` all require CSRF tokens now, however `worker/kill` does not as it is an exposed API for operators. Missing CSRF tokens will 403 & a HEAD/GET request will 405 unless deployed on YARN or fallback behavior is explicitly enabled. Backport of c4ef971 ### Why are the changes needed? Browser pre-fetch and agents making assumptions on POST v.s. GET "verbs." In general GETs should be side-effect free which is why things like crawlers browsers and agents may choose to prefect them which when the GET is not side-effect free causes undesired behavior. ### Does this PR introduce _any_ user-facing change? Yes new config flag, also pressing back on POST pages often behaves differently in browsers. ### How was this patch tested? New UI test, both "regular" in MasterWebUISuite and UISuite but also added additional selenium tests. Existing UI tests still function :) ### Was this patch authored or co-authored using generative AI tooling? Generated-By: Claude (mixutre of models) and Cursor (Kimi K3) Closes apache#59030 from holdenk/42-62-super-branch-3.5. Lead-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com> Signed-off-by: Holden Karau <holden.karau@snowflake.com>
Drop invalid links rather than break rendering Application and worker UI addresses come from external registrants and could contain bad/invalid info. Filter it out to prevent the UI from breaking. No New UIUtilSuite test Yes Generated-by: Cursor (Kimi K3) Closes apache#58622 from holdenk/improve-ui-link-rendering-part2. Lead-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com> Signed-off-by: Holden Karau <holden.karau@snowflake.com> (cherry picked from commit e91d22d) Signed-off-by: Holden Karau <holden.karau@snowflake.com>
Resolved pom.xml: upstream independently bumped Ivy to 2.5.3, past this branch's CVE bump to 2.5.2, so take upstream's and drop the now-stale comment explaining the 2.5.2 choice. dev/deps/spark-deps-hadoop-3-hive-2.3 is generated; resolved to upstream's version here and regenerated in a follow-up commit. Co-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com>
Generated with ./dev/test-dependencies.sh --replace-manifest. The only change against the branch's pre-merge manifest is Ivy 2.5.2 -> 2.5.3, which upstream bumped independently. Co-authored-by: Holden Karau <holden@pigscanfly.ca> Co-authored-by: Holden Karau <holden.karau@snowflake.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Upgrade Spark 3.5 dependencies with potential issues that we are not 100% sure about if they do not impact us.
Why are the changes needed?
These dependencies are very out of date, have some CVEs filed against them, and while we don't see a direct path to exploit it seems safer to update these since we are also not 100% sure there is not a path.
Does this PR introduce any user-facing change?
It should not.
How was this patch tested?
CI
Was this patch authored or co-authored using generative AI tooling?
Yes, claude in /goal mode to upgrade given a list of dependencies and instructions to run maven tests in between upgrades.