Skip to content

NoSuchMethodError: java.lang.Object org.apache.spark.executor.TaskMetrics.withExternalAccums(scala.Function1) #1576

Closed
Closed
NoSuchMethodError: java.lang.Object org.apache.spark.executor.TaskMetrics.withExternalAccums(scala.Function1)#1576
Labels
bugSomething isn't working

Description

@mkgada
mkgada
opened on Mar 27, 2025

Describe the bug

Running on Spark 3.5.0, Java 11, scala 2.12.18 on GCP Dataproc Cluster.
Can confirm Comet initialization

Acc to Spark History Server- this failed on a .count() operation in my code
Here's the full stack trace

java.lang.NoSuchMethodError: 'java.lang.Object org.apache.spark.executor.TaskMetrics.withExternalAccums(scala.Function1)'
	at org.apache.spark.sql.comet.shims.ShimTaskMetrics$.getTaskAccumulator(ShimTaskMetrics.scala:30)
	at org.apache.spark.sql.comet.shims.ShimTaskMetrics.getTaskAccumulator(ShimTaskMetrics.scala)
	at org.apache.comet.parquet.BatchReader.init(BatchReader.java:350)
	at org.apache.comet.parquet.CometParquetFileFormat.$anonfun$buildReaderWithPartitionValues$1(CometParquetFileFormat.scala:177)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:217)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:279)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
	at org.apache.spark.sql.comet.CometScanExec$$anon$1.hasNext(CometScanExec.scala:257)
	at org.apache.comet.CometBatchIterator.hasNext(CometBatchIterator.java:50)
	at org.apache.comet.Native.executePlan(Native Method)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:129)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1$adapted(CometExecIterator.scala:127)
	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:157)
	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:127)
	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:148)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.comet.CometBatchIterator.hasNext(CometBatchIterator.java:50)
	at org.apache.comet.Native.executePlan(Native Method)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:129)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1$adapted(CometExecIterator.scala:127)
	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:157)
	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:127)
	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:148)
	at org.apache.spark.sql.comet.execution.shuffle.CometShuffleWriteProcessor.write(CometShuffleExchangeExec.scala:512)
	at org.apache.spark.sql.comet.shims.ShimCometShuffleWriteProcessor.write(ShimCometShuffleWriteProcessor.scala:35)
	at org.apache.spark.sql.comet.shims.ShimCometShuffleWriteProcessor.write$(ShimCometShuffleWriteProcessor.scala:28)
	at org.apache.spark.sql.comet.execution.shuffle.CometShuffleWriteProcessor.write(CometShuffleExchangeExec.scala:451)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
	at org.apache.spark.scheduler.Task.run(Task.scala:141)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:95)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Activity

  1. mkgada
    added
    bugSomething isn't working
    on Mar 27, 2025

  2. jinwenjie123 commented on Mar 27, 2025

    @jinwenjie123
    jinwenjie123
    on Mar 27, 2025

    I also encountered similar issue when I conducted the test for running spark3.4.4 in AWS. I fixed it by changing to spark3.4.3 since it looks like the issue is caused by shim. Maybe you can try something similar.

  3. mkgada commented on Mar 28, 2025

    @mkgada
    mkgada
    on Mar 28, 2025
    Author

    @jinwenjie123 appreciate your response! I am using one of the pre-built JARs, I will not be able to switch to Spark 3.4.x since our cluster was recently upgraded to 3.5.x and would like to stick to it

    @andygrove @alamb would y'all be able to help here? I saw that the prebuilt JARs were tested for 3.5.4 and upwards. Are they backwards compatible?

  4. alamb commented on Mar 28, 2025

    @alamb
    alamb
    on Mar 28, 2025
    Contributor

    @andygrove @alamb would y'all be able to help here? I saw that the prebuilt JARs were tested for 3.5.4 and upwards. Are they backwards compatible?

    Sorry I don't know much about comet or java errors

  5. andygrove commented on Mar 28, 2025

    @andygrove
    andygrove
    on Mar 28, 2025
    Member

    Comet 0.7.0 supported 3.5.0 through 3.5.4 but not 3.5.5 which had breaking changes to internal apis. I am on vacation this week but it looks like 3.5.5 support was added this week.

  6. parthchandra commented on Mar 29, 2025

    @parthchandra
    parthchandra
    on Mar 29, 2025
    Contributor

    This particular API is not a public API and we use it to so we can verify the metrics in tests. Maybe we can disable its use in non test environments?

  7. mkgada commented on Mar 31, 2025

    @mkgada
    mkgada
    on Mar 31, 2025
    Author

    @andygrove sorry for the confusion here, my cluster is on Spark 3.5.0, and Comet 0.7.0 prebuilt JAR did not work on it

    @parthchandra can you elaborate? Is there a opportunity to fix this through config?

  8. parthchandra commented on Apr 1, 2025

    @parthchandra
    parthchandra
    on Apr 1, 2025
    Contributor

    @andygrove sorry for the confusion here, my cluster is on Spark 3.5.0, and Comet 0.7.0 prebuilt JAR did not work on it

    @parthchandra can you elaborate? Is there a opportunity to fix this through config?

    Spark has something like this -

      /**
   * Indicates whether Spark is currently running unit tests.
   */
  def isTesting: Boolean = {
    // Scala's `sys.env` creates a ton of garbage by constructing Scala immutable maps, so
    // we directly use the Java APIs instead.
    System.getenv("SPARK_TESTING") != null || System.getProperty(IS_TESTING.key) != null
  }

    which is then used to enable some configurations by default when in unit tests. We could do the same.

  9. parthchandra commented on Apr 1, 2025

    @parthchandra
    parthchandra
    on Apr 1, 2025
    Contributor

    Also it appears the the API change in TaskMetrics has been backported to 3.4 as well.

  10. mkgada commented on Apr 1, 2025

    @mkgada
    mkgada
    on Apr 1, 2025
    Author

    Update: spun up another cluster on Spark 3.5.3 and used the same prebuilt Comet JAR 0.7.0

    Was able to get past the initial error documented here but now running into

    Caused by: java.lang.NoSuchMethodError: 'void org.apache.spark.shuffle.IndexShuffleBlockResolver.writeMetadataFileAndCommit(int, long, long[], long[], java.io.File)'
	at org.apache.spark.sql.comet.execution.shuffle.CometShuffleWriteProcessor.write(CometShuffleExchangeExec.scala:540)
	at org.apache.spark.sql.comet.shims.ShimCometShuffleWriteProcessor.write(ShimCometShuffleWriteProcessor.scala:35)
	at org.apache.spark.sql.comet.shims.ShimCometShuffleWriteProcessor.write$(ShimCometShuffleWriteProcessor.scala:28)
	at org.apache.spark.sql.comet.execution.shuffle.CometShuffleWriteProcessor.write(CometShuffleExchangeExec.scala:451)

  11. andygrove commented on Apr 1, 2025

    @andygrove
    andygrove
    on Apr 1, 2025
    Member

    Update: spun up another cluster on Spark 3.5.3 and used the same prebuilt Comet JAR 0.7.0

    Was able to get past the initial error documented here but now running into

    Caused by: java.lang.NoSuchMethodError: 'void org.apache.spark.shuffle.IndexShuffleBlockResolver.writeMetadataFileAndCommit(int, long, long[], long[], java.io.File)'

    I checked out Spark v3.5.3 and the method is defined as:

      def writeMetadataFileAndCommit(
      shuffleId: Int,
      mapId: Long,
      lengths: Array[Long],
      checksums: Array[Long],
      dataTmp: File): Unit = {

    It also looks like this method signature has not changed recently.

    Can you confirm that you are using the open-source Apache Spark v3.5.3 and not a GCP Dataproc version of Spark?

  12. mkgada commented on Apr 1, 2025

    @mkgada
    mkgada
    on Apr 1, 2025
    Author

    I am using a Spark image supplied by GCP Dataproc, thank you for checking this

    It is this one specifically - https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.2
    v2.2.50-debian12

    They usually install open source Spark, but I need to confirm this

  13. andygrove commented on Apr 1, 2025

    @andygrove
    andygrove
    on Apr 1, 2025
    Member

    I am assuming that the GCP version of Spark has some differences in these internal APIs

  14. mkgada commented on Apr 1, 2025

    @mkgada
    mkgada
    on Apr 1, 2025
    Author

    Thank you so much for looking into this though, @andygrove

    I will take this up with GCP folks

  15. mkgada commented on Apr 4, 2025

    @mkgada
    mkgada
    on Apr 4, 2025
    Author

    Looks like an issue stemming out of Spark, I was able to run my job to success by disabling comet shuffle manager

  16. andygrove commented on Jun 16, 2025

    @andygrove
    andygrove
    on Jun 16, 2025
    Member

    This issue was likely resolved #693 so will close for now. @mkgada feel free to reopen if this is still an issue

  17. andygrove
    closed this as completedon Jun 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions