Repository navigation
address failure caused by method signature change in SPARK-48791 #692
Description
Describe the bug
Comet BatchReader uses a private method in Spark TaskMetrics.externalAccums. In the latest codebase of Spark a fix for SPARK-48791 has changed the signature of this method which causes Comet to fail with
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 2) (localhost executor driver): java.lang.NoSuchMethodError: 'scala.collection.mutable.Buffer org.apache.spark.executor.TaskMetrics.externalAccums()'
Steps to reproduce
Build Spark from latest code base and tru to run the HiveParquetSuite with Comet enabled.
Expected behavior
The test should pass
Additional context
No response
Activity
kazuyukitanimura commentedon Aug 9, 2024
Reopening as there is a follow up PR that changes the method signature again
apache/spark@c3ba8fa
andygrove commentedon Apr 3, 2025
@parthchandra is this still an issue?
parthchandra commentedon Apr 3, 2025
Seems like a perennial issue. This signature changes in every release it appears (it is private after all).
#1576
kazuyukitanimura commentedon Apr 8, 2025
Fixed as a part of #1482
closing
mkgada commentedon Apr 8, 2025
Would this fix #1576 as well? @kazuyukitanimura
kazuyukitanimura commentedon Apr 8, 2025
I looked into #1576 and looks like @mkgada you are using Spark 3.5.0?
https://datafusion.apache.org/comet/user-guide/installation.html
We are currently only testing with 3.5.5 and 3.5.2-4 (partially)