[SPARK-47234][BUILD] Upgrade Scala to 2.13.13 by panbingkun · Pull Request #45342 · apache/spark

panbingkun · 2024-02-29T23:41:51Z

What changes were proposed in this pull request?

The pr aims to upgrade scala from 2.13.12 to 2.13.13.

Why are the changes needed?

The new version bring some bug fixes:
fix: SliceIterator to drop a max of Int.MaxValue scala/scala#10525
Fixes issue with blocking and BatchingExecutor scala/scala#10528
The release notes as follows: https://github.com/scala/scala/releases/tag/v2.13.13

Does this PR introduce any user-facing change?

Yes, The scala version is changed from 2.13.12 to 2.13.13.

How was this patch tested?

Pass GA.
After the master is upgraded to this version 2.13.13, we need to continue to observe.

Was this patch authored or co-authored using generative AI tooling?

No.

panbingkun · 2024-03-01T00:53:07Z

From GA testing, it seems that ammonite has not yet supported scala 2.13.13.
com.lihaoyi:ammonite_2.13.13:2.5.11

A PR has been submitted to the upstream ammonite community to address this issue:
com-lihaoyi/Ammonite#1430

dongjoon-hyun · 2024-03-01T04:53:59Z

Thank you, @panbingkun !

dongjoon-hyun

Actually, according to the release note, we need to upgrade SBT too.

Users of sbt must upgrade to sbt 1.9.9 to use the console task to access the 2.13.13 REPL (sbt/sbt#7502); error message is NoSuchMethodError: org.jline.utils.AttributedString.fromAnsi

I also saw your SBT 1.9.9 test PR and its failure.

#45345

In the worst case, we cannot upgrade to Scala 2.13.13.

panbingkun · 2024-03-01T05:53:23Z

Actually, according to the release note, we need to upgrade SBT too.

Users of sbt must upgrade to sbt 1.9.9 to use the console task to access the 2.13.13 REPL (sbt/sbt#7502); error message is NoSuchMethodError: org.jline.utils.AttributedString.fromAnsi

I also saw your SBT 1.9.9 test PR and its failure.

[WIP][Only Test] Test sbt 1.9.9 #45345

In the worst case, we cannot upgrade to Scala 2.13.13.

Yes, but unfortunately, from the test results of another PR, it seems that sbt 1.9.9 still has issues starting from version sbt 1.9.4. (When there is a corrupted jar in Maven's local repo, it will fail)

dongjoon-hyun · 2024-03-01T06:19:18Z

Yes, that's what I meant exactly. Scala 2.13.13 requires SBT 1.9.9 but SBT 1.9.9 is not working properly. In short, there is no working combination in Scala 2.13.13 and this is the worst case where we are currently standing. I don't know if SBT 1.9.10 might solve every issues, but it's also unlikely.

Yes, but unfortunately, from the test results of another PR, it seems that sbt 1.9.9 still has issues starting from version sbt 1.9.4. (When there is a corrupted jar in Maven's local repo, it will fail)

panbingkun · 2024-03-01T07:34:00Z

Yes, that's what I meant exactly. Scala 2.13.13 requires SBT 1.9.9 but SBT 1.9.9 is not working properly. In short, there is no working combination in Scala 2.13.13 and this is the worst case where we are currently standing. I don't know if SBT 1.9.10 might solve every issues, but it's also unlikely.

Yes, but unfortunately, from the test results of another PR, it seems that sbt 1.9.9 still has issues starting from version sbt 1.9.4. (When there is a corrupted jar in Maven's local repo, it will fail)

I have already submitted a bug report to SBT: sbt/sbt#7506

But it seems to be a problem with the coursier .

cc @LuciferYang

LuciferYang · 2024-03-01T08:19:42Z

From GA testing, it seems that ammonite has not yet supported scala 2.13.13. com.lihaoyi:ammonite_2.13.13:2.5.11

A PR has been submitted to the upstream ammonite community to address this issue: com-lihaoyi/Ammonite#1430

In fact, com-lihaoyi/Ammonite#1421 is already in the process of upgrading, there are probably some tests that have not passed yet

dongjoon-hyun · 2024-03-01T16:56:31Z

Thank you for updates for both.

dongjoon-hyun · 2024-03-14T21:07:37Z

It seems that 3.0.0-M1 is released last week.

Could you rebase the PR? I guess the only remaining issue is SBT?

panbingkun · 2024-03-15T00:51:08Z

It seems that 3.0.0-M1 is released last week.

Could you rebase the PR? I guess the only remaining issue is SBT?

Okay, let me try it. Thanks for reminding me. ❤️

dongjoon-hyun · 2024-03-15T02:35:26Z

    -->
    <arrow.version>15.0.0</arrow.version>
-    <ammonite.version>2.5.11</ammonite.version>
+    <ammonite.version>3.0.0-M1</ammonite.version>


Could you update the dependency list? The other tests seem to pass.

+++ b/dev/pr-deps/spark-deps-hadoop-3-hive-2.3 @@ -139,7 +139,7 @@ jettison/1.5.4//jettison-1.5.4.jar jetty-util-ajax/11.0.20//jetty-util-ajax-11.0.20.jar jetty-util/11.0.20//jetty-util-11.0.20.jar jline/2.14.6//jline-2.14.6.jar -jline/3.22.0//jline-3.22.0.jar +jline/3.24.1//jline-3.24.1.jar

BTW, it seems that we have two jline libraries at the same time.

Let me analyze it.

No problem, it has been checked before that their package names are different, so there will be no conflict

jline 2

jline 3

jline-2.14.6.jar

<groupId>jline</groupId> <artifactId>jline</artifactId> <version>2.14.6</version>

jline-3.22.0.jar

<groupId>org.jline</groupId> <artifactId>jline</artifactId> <version>3.22.0</version>

I'll see what depends on them. If they can be aligned, it would be great if only one is left.
Of course, that's another issue.

https://github.com/apache/hive/blob/36b32612ba0a20d3d305c77a813aca62d7129ed0/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L41-L49

From the hive 2.3 version code, it seems that we cannot align here because its import of jline starts with jline....

Let me try in SPARK-47405 independently in the scope of upgrading spark-sql shell to use jline3 . If it's difficult, we can live with this as you confirmed that they are different packages now.

dongjoon-hyun · 2024-03-15T04:04:45Z

If CI passed, shall we land this to master first to investigate SBT issues further, @panbingkun and @LuciferYang ? It would be must faster to find any potential issues if this patch is in master branch because all people and Daily CIs will verify this.

panbingkun · 2024-03-15T05:25:33Z

If CI passed, shall we land this to master first to investigate SBT issues further, @panbingkun and @LuciferYang ? It would be must faster to find any potential issues if this patch is in master branch because all people and Daily CIs will verify this.

I agree with you!
Let me add to the description of this PR.

dongjoon-hyun

+1, LGTM. Thank you, @panbingkun and @LuciferYang .
Merged to master for Apache Spark 4.0.0.

[SPARK-47234][BUILD] Upgrade Scala to 2.13.13

74f3f9f

github-actions Bot added BUILD DOCS labels Feb 29, 2024

dongjoon-hyun reviewed Mar 1, 2024

View reviewed changes

panbingkun added 2 commits March 15, 2024 09:03

Merge branch 'master' into SPARK-47234

3a5f5e7

update ammonite to 3.0.0-M1

5df5f01

dongjoon-hyun reviewed Mar 15, 2024

View reviewed changes

fix spark-deps-hadoop-3-hive-2.3

c2f19e5

dongjoon-hyun changed the title ~~[WIP][SPARK-47234][BUILD] Upgrade Scala to 2.13.13~~ [SPARK-47234][BUILD] Upgrade Scala to 2.13.13 Mar 15, 2024

dongjoon-hyun marked this pull request as ready for review March 15, 2024 05:11

dongjoon-hyun approved these changes Mar 15, 2024

View reviewed changes

dongjoon-hyun closed this in 56cfc89 Mar 15, 2024

Conversation

panbingkun commented Feb 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

panbingkun commented Mar 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Mar 1, 2024

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Mar 1, 2024

Uh oh!

dongjoon-hyun commented Mar 1, 2024

Uh oh!

panbingkun commented Mar 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LuciferYang commented Mar 1, 2024

Uh oh!

dongjoon-hyun commented Mar 1, 2024

Uh oh!

dongjoon-hyun commented Mar 14, 2024

Uh oh!

panbingkun commented Mar 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

panbingkun Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

LuciferYang Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

panbingkun Mar 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panbingkun Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

panbingkun Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Mar 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Mar 15, 2024

Uh oh!

panbingkun commented Mar 15, 2024

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

panbingkun commented Feb 29, 2024 •

edited

Loading

panbingkun commented Mar 1, 2024 •

edited

Loading

panbingkun commented Mar 1, 2024 •

edited

Loading

panbingkun commented Mar 15, 2024 •

edited

Loading

panbingkun Mar 15, 2024 •

edited

Loading

dongjoon-hyun Mar 15, 2024 •

edited

Loading