Repository navigation
Fix sbt dependency resolution issues with Maven-populated ~/.m2 - #33
Merged
Merged
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 97786b48a1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+1188
to
+1189
| ExclusionRule("io.netty", "netty-transport-native-epoll"), | ||
| ExclusionRule("io.netty", "netty-transport-native-kqueue")) |
There was a problem hiding this comment.
When Spark is built with sbt and a user sets spark.network.io.mode=EPOLL on Linux, these global exclusions remove the classified netty-transport-native-epoll artifacts that core/pom.xml and common/network-common/pom.xml declare explicitly; the remaining netty-transport-classes-epoll jar only supplies Java classes, so NettyUtils.createEventLoop(EPOLL, ...) reaches EpollEventLoopGroup without the JNI library and fails at startup. Please exclude only the classifier-less Selenium transitive dependency, not the whole module, so the linux-* classified jars stay on the sbt classpath.
Useful? React with 👍 / 👎.
Owner
Author
There was a problem hiding this comment.
Tried this empirically (twice) and the scoped exclusion doesn't hold: excluding the classifier-less module only beneath the selenium test deps still leaves core/update requesting the classifier-less netty-transport-native-epoll/kqueue-4.1.124.Final.jar against a Maven-populated ~/.m2 — the build's own pom-derived netty-transport-native entries (and their interaction with the forced netty versions in coursier) also resolve the default publication, and per-artifact/classifier exclusion isn't expressible in sbt (exclusions are module-level). Also worth noting: the classifier-less jars contain no .so (the natives live only in the classified jars), and in the sbt/coursier resolution the classified jars were arriving via netty-all's runtime deps rather than the pom's explicit classified entries. The Maven build — the path that produces release artifacts and the published deps manifest — is unaffected; the trade-off here is that an sbt-built dev/test classpath falls back to NIO if EPOLL is explicitly requested, which nothing in CI or the test tree does. Given a fully verified green dev/mima requires it, keeping the module-level exclusion.
Generated by Claude Code
holdenk
force-pushed
the
3.5-mini-upgrades
branch
from
July 2, 2026 21:31
6296fd2 to
9724c39
Compare
holdenk
force-pushed
the
claude/mima-failure-fix-jpzs1u
branch
from
July 2, 2026 21:34
93aa3d9 to
97786b4
Compare
…llowing them dev/mima builds the tools and oldDeps classpaths via command substitution: TOOLS_CLASSPATH="$(build/sbt ... "export tools/fullClasspath" | grep jar | tail -n1)" All sbt output (including any [error] report) is captured into the variable, and with `set -e -o pipefail` any sbt failure aborts the script without printing anything. In CI the MIMA step then dies ~35s in showing only the two sbt-launcher stderr lines, which makes resolution failures undiagnosable. Buffer each export run in a temp file, check the exit status explicitly, and dump the sbt output to stderr when it fails (or when no classpath line is produced). Successful runs behave exactly as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdmNouFA7cCw3J11Xb7Djm
…sbt build to fix MiMa The MIMA lint step has failed silently since the netty dependencyOverrides were introduced. Root cause, reproduced locally with a CI-equivalent ~/.m2: [error] (tools / update) FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:.../.m2/repository/io/netty/netty-transport-native-epoll/4.1.124.Final/netty-transport-native-epoll-4.1.124.Final.jar: not found [error] file:.../.m2/repository/io/netty/netty-transport-native-kqueue/4.1.124.Final/netty-transport-native-kqueue-4.1.124.Final.jar: not found Chain: selenium-remote-driver (a test dep of every module via the parent pom) declares netty-transport-native-epoll/kqueue WITHOUT a platform classifier; the dependencyOverrides force those modules to 4.1.124.Final; the lint job runs dev/test-dependencies.sh (full mvn install) right before dev/mima, which leaves the 4.1.124.Final POMs plus only the *classified* jars (linux-x86_64 etc.) in ~/.m2 - Maven never fetches the classifier-less jars. Coursier prefers the local Maven repo once a POM is present there and fails `update` with "not found" instead of falling back to Maven Central. No forced version can work here (any version Maven metadata-walks is POM-only locally), and un-forcing them re-breaks on selenium's own netty version the same way, so exclude the two native-transport modules from the sbt build entirely. They only carry the native .so (an optional perf feature - Spark defaults to NIO and no test selects EPOLL); the io.netty.channel.epoll/kqueue classes still come from netty-transport-classes-* which remain overridden to 4.1.124.Final. The Maven build and the published deps manifest are unchanged. Verified locally against a CI-equivalent poisoned ~/.m2: both dev/mima classpath exports and GenerateMIMAIgnore now succeed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdmNouFA7cCw3J11Xb7Djm
…ated versions for sbt
Running the full dev/mima against a freshly Maven-populated ~/.m2 (what the
lint job produces via dev/test-dependencies.sh when its docs-maven cache does
not paper over it) surfaced three more updates failing on the same
POM-present-but-jar-missing coursier failure mode as the netty native jars:
[error] (sql-kafka-0-10 / update) ... jopt-simple-5.0.4.jar: not found
[error] (streaming-kafka-0-10 / update) ... jopt-simple-5.0.4.jar: not found
[error] (connect-client-jvm / update) ... geny_2.12-0.7.1.jar: not found
[error] ... sourcecode_2.12-0.3.0.jar: not found
In each case Maven's mediation resolves a different version than coursier's
highest-wins does, so only the Maven-picked jar exists in ~/.m2:
- The kafka module poms explicitly pin jopt-simple 3.2 (test) and Maven takes
it over kafka_2.12's transitive 5.0.4 ("omitted for conflict with 3.2").
- In ammonite 2.5.9's tree (provided dep of connect-client-jvm) Maven picks
geny 0.7.0 (via os-lib 0.8.0) and sourcecode 0.2.7 (via ammonite-terminal),
while coursier picks 0.7.1 / 0.3.0.
Pin all three in DependencyOverrides to the versions Maven mediates, per this
object's existing purpose ("work around sbt's dependency resolution being
different from Maven's"). CI currently only survives these because its
accumulated docs-maven cache still contains the old jars; a cache eviction
would have broken the MIMA step again.
Verified: with these pins plus the netty native exclusion, the complete
./dev/mima (both classpath exports, GenerateMIMAIgnore, and
mimaReportBinaryIssues over the whole build) passes locally against a
CI-equivalent ~/.m2 with zero resolution errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HdmNouFA7cCw3J11Xb7Djm
holdenk
force-pushed
the
claude/mima-failure-fix-jpzs1u
branch
from
July 2, 2026 22:52
97786b4 to
771d3db
Compare
holdenk
added a commit
that referenced
this pull request
Jul 7, 2026
* [SPARK-CVE][BUILD][3.5] Surface sbt errors in dev/mima instead of swallowing them dev/mima builds the tools and oldDeps classpaths via command substitution: TOOLS_CLASSPATH="$(build/sbt ... "export tools/fullClasspath" | grep jar | tail -n1)" All sbt output (including any [error] report) is captured into the variable, and with `set -e -o pipefail` any sbt failure aborts the script without printing anything. In CI the MIMA step then dies ~35s in showing only the two sbt-launcher stderr lines, which makes resolution failures undiagnosable. Buffer each export run in a temp file, check the exit status explicitly, and dump the sbt output to stderr when it fails (or when no classpath line is produced). Successful runs behave exactly as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdmNouFA7cCw3J11Xb7Djm * [SPARK-CVE][BUILD][3.5] Exclude netty native transport jars from the sbt build to fix MiMa The MIMA lint step has failed silently since the netty dependencyOverrides were introduced. Root cause, reproduced locally with a CI-equivalent ~/.m2: [error] (tools / update) FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:.../.m2/repository/io/netty/netty-transport-native-epoll/4.1.124.Final/netty-transport-native-epoll-4.1.124.Final.jar: not found [error] file:.../.m2/repository/io/netty/netty-transport-native-kqueue/4.1.124.Final/netty-transport-native-kqueue-4.1.124.Final.jar: not found Chain: selenium-remote-driver (a test dep of every module via the parent pom) declares netty-transport-native-epoll/kqueue WITHOUT a platform classifier; the dependencyOverrides force those modules to 4.1.124.Final; the lint job runs dev/test-dependencies.sh (full mvn install) right before dev/mima, which leaves the 4.1.124.Final POMs plus only the *classified* jars (linux-x86_64 etc.) in ~/.m2 - Maven never fetches the classifier-less jars. Coursier prefers the local Maven repo once a POM is present there and fails `update` with "not found" instead of falling back to Maven Central. No forced version can work here (any version Maven metadata-walks is POM-only locally), and un-forcing them re-breaks on selenium's own netty version the same way, so exclude the two native-transport modules from the sbt build entirely. They only carry the native .so (an optional perf feature - Spark defaults to NIO and no test selects EPOLL); the io.netty.channel.epoll/kqueue classes still come from netty-transport-classes-* which remain overridden to 4.1.124.Final. The Maven build and the published deps manifest are unchanged. Verified locally against a CI-equivalent poisoned ~/.m2: both dev/mima classpath exports and GenerateMIMAIgnore now succeed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdmNouFA7cCw3J11Xb7Djm * [SPARK-CVE][BUILD][3.5] Pin jopt-simple/geny/sourcecode to Maven-mediated versions for sbt Running the full dev/mima against a freshly Maven-populated ~/.m2 (what the lint job produces via dev/test-dependencies.sh when its docs-maven cache does not paper over it) surfaced three more updates failing on the same POM-present-but-jar-missing coursier failure mode as the netty native jars: [error] (sql-kafka-0-10 / update) ... jopt-simple-5.0.4.jar: not found [error] (streaming-kafka-0-10 / update) ... jopt-simple-5.0.4.jar: not found [error] (connect-client-jvm / update) ... geny_2.12-0.7.1.jar: not found [error] ... sourcecode_2.12-0.3.0.jar: not found In each case Maven's mediation resolves a different version than coursier's highest-wins does, so only the Maven-picked jar exists in ~/.m2: - The kafka module poms explicitly pin jopt-simple 3.2 (test) and Maven takes it over kafka_2.12's transitive 5.0.4 ("omitted for conflict with 3.2"). - In ammonite 2.5.9's tree (provided dep of connect-client-jvm) Maven picks geny 0.7.0 (via os-lib 0.8.0) and sourcecode 0.2.7 (via ammonite-terminal), while coursier picks 0.7.1 / 0.3.0. Pin all three in DependencyOverrides to the versions Maven mediates, per this object's existing purpose ("work around sbt's dependency resolution being different from Maven's"). CI currently only survives these because its accumulated docs-maven cache still contains the old jars; a cache eviction would have broken the MIMA step again. Verified: with these pins plus the netty native exclusion, the complete ./dev/mima (both classpath exports, GenerateMIMAIgnore, and mimaReportBinaryIssues over the whole build) passes locally against a CI-equivalent ~/.m2 with zero resolution errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdmNouFA7cCw3J11Xb7Djm --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes dependency resolution failures in the sbt build when using a Maven-populated local repository (~/.m2). The issues stem from coursier's preference for the local Maven repo once a POM is present, which causes failures when only certain classifier variants of jars are available locally.
Key Changes
SparkBuild.scala:
jopt-simple,geny, andsourcecodeto versions that Maven's mediation selects, ensuring sbt resolves jars that actually exist in ~/.m2netty-transport-native-epollandnetty-transport-native-kqueuefrom the dependency overrides list (they are now excluded instead)ExcludedDependencies:
netty-transport-native-epollandnetty-transport-native-kqueueto prevent coursier from attempting to fetch classifier-less jars that don't exist in ~/.m2dev/mima:
export_classpath()helper functionImplementation Details
The root cause is that selenium-remote-driver (a test dependency) depends on netty native transport modules without platform classifiers. Maven only fetches the classifier variants (e.g., linux-x86_64), leaving the local repo with POMs but no classifier-less jars. When coursier prefers the local Maven repo (once a POM is present), it fails to find the classifier-less jars and aborts with "not found" errors.
The fix uses two complementary strategies:
https://claude.ai/code/session_01HdmNouFA7cCw3J11Xb7Djm