Skip to content

[SPARK-47234][BUILD] Upgrade Scala to 2.13.13 #45342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
panbingkun wants to merge 4 commits into from
Closed

[SPARK-47234][BUILD] Upgrade Scala to 2.13.13 #45342

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
74f3f9f
[SPARK-47234][BUILD] Upgrade Scala to 2.13.13
panbingkun Feb 29, 2024
3a5f5e7
Merge branch 'master' into SPARK-47234
panbingkun Mar 15, 2024
5df5f01
update ammonite to 3.0.0-M1
panbingkun Mar 15, 2024
c2f19e5
fix spark-deps-hadoop-3-hive-2.3
panbingkun Mar 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ jettison/1.5.4//jettison-1.5.4.jar
jetty-util-ajax/11.0.20//jetty-util-ajax-11.0.20.jar
jetty-util/11.0.20//jetty-util-11.0.20.jar
jline/2.14.6//jline-2.14.6.jar
jline/3.22.0//jline-3.22.0.jar
jline/3.24.1//jline-3.24.1.jar
jna/5.13.0//jna-5.13.0.jar
joda-time/2.12.7//joda-time-2.12.7.jar
jodd-core/3.5.2//jodd-core-3.5.2.jar
Expand Down Expand Up @@ -245,11 +245,11 @@ py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
rocksdbjni/8.11.3//rocksdbjni-8.11.3.jar
scala-collection-compat_2.13/2.7.0//scala-collection-compat_2.13-2.7.0.jar
scala-compiler/2.13.12//scala-compiler-2.13.12.jar
scala-library/2.13.12//scala-library-2.13.12.jar
scala-compiler/2.13.13//scala-compiler-2.13.13.jar
scala-library/2.13.13//scala-library-2.13.13.jar
scala-parallel-collections_2.13/1.0.4//scala-parallel-collections_2.13-1.0.4.jar
scala-parser-combinators_2.13/2.3.0//scala-parser-combinators_2.13-2.3.0.jar
scala-reflect/2.13.12//scala-reflect-2.13.12.jar
scala-reflect/2.13.13//scala-reflect-2.13.13.jar
scala-xml_2.13/2.2.0//scala-xml_2.13-2.2.0.jar
slf4j-api/2.0.12//slf4j-api-2.0.12.jar
snakeyaml-engine/2.7//snakeyaml-engine-2.7.jar
Expand Down
2 changes: 1 addition & 1 deletion docs/_config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ include:
SPARK_VERSION: 4.0.0-SNAPSHOT
SPARK_VERSION_SHORT: 4.0.0
SCALA_BINARY_VERSION: "2.13"
SCALA_VERSION: "2.13.12"
SCALA_VERSION: "2.13.13"
SPARK_ISSUE_TRACKER_URL: https://issues.apache.org/jira/browse/SPARK
SPARK_GITHUB_URL: https://github.com/apache/spark
# Before a new release, we should:
Expand Down
4 changes: 2 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@
<!-- managed up from 3.2.1 for SPARK-11652 -->
<commons.collections.version>3.2.2</commons.collections.version>
<commons.collections4.version>4.4</commons.collections4.version>
<scala.version>2.13.12</scala.version>
<scala.version>2.13.13</scala.version>
<scala.binary.version>2.13</scala.binary.version>
<scalatest-maven-plugin.version>2.2.0</scalatest-maven-plugin.version>
<!-- don't upgrade scala-maven-plugin to version 4.7.2 or higher, see SPARK-45144 for details -->
Expand Down Expand Up @@ -226,7 +226,7 @@
./python/pyspark/sql/pandas/utils.py, and ./python/setup.py too.
-->
<arrow.version>15.0.0</arrow.version>
<ammonite.version>2.5.11</ammonite.version>
<ammonite.version>3.0.0-M1</ammonite.version>

@dongjoon-hyun dongjoon-hyun Mar 15, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you update the dependency list? The other tests seem to pass.

+++ b/dev/pr-deps/spark-deps-hadoop-3-hive-2.3
@@ -139,7 +139,7 @@ jettison/1.5.4//jettison-1.5.4.jar
 jetty-util-ajax/11.0.20//jetty-util-ajax-11.0.20.jar
 jetty-util/11.0.20//jetty-util-11.0.20.jar
 jline/2.14.6//jline-2.14.6.jar
-jline/3.22.0//jline-3.22.0.jar
+jline/3.24.1//jline-3.24.1.jar

@dongjoon-hyun dongjoon-hyun Mar 15, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, it seems that we have two jline libraries at the same time.

@panbingkun panbingkun Mar 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me analyze it.

@LuciferYang LuciferYang Mar 15, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem, it has been checked before that their package names are different, so there will be no conflict

jline 2

image

jline 3

image

@panbingkun panbingkun Mar 15, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • jline-2.14.6.jar
    <groupId>jline</groupId>
    <artifactId>jline</artifactId>
    <version>2.14.6</version>
image image
  • jline-3.22.0.jar
    <groupId>org.jline</groupId>
    <artifactId>jline</artifactId>
    <version>3.22.0</version>
image

@panbingkun panbingkun Mar 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll see what depends on them. If they can be aligned, it would be great if only one is left.
Of course, that's another issue.

@panbingkun panbingkun Mar 15, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/apache/hive/blob/36b32612ba0a20d3d305c77a813aca62d7129ed0/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L41-L49
image
From the hive 2.3 version code, it seems that we cannot align here because its import of jline starts with jline....

@dongjoon-hyun dongjoon-hyun Mar 15, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me try in SPARK-47405 independently in the scope of upgrading spark-sql shell to use jline3 . If it's difficult, we can live with this as you confirmed that they are different packages now.


<!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
<leveldbjni.group>org.fusesource.leveldbjni</leveldbjni.group>
Expand Down