PageRank causes java.util.NoSuchElementException #52

Activity

1. ghost assigned

   jegonzal

   on Nov 7, 2013
2. dcrankshaw commented on Nov 7, 2013

   

   dcrankshaw

   on Nov 7, 2013

   ContributorAuthor

   More actions

   Interestingly, #51 seems to fix the issue.
3. rxin commented on Nov 7, 2013

   

   rxin

   on Nov 7, 2013

   Member

   More actions

   simple code is less buggy :)
4. jegonzal commented on Nov 7, 2013

   

   jegonzal

   on Nov 7, 2013

   Contributor

   More actions

   Hmm, is that error in the master branch? Those line numbers don't seem to "line-up." I guess for now maybe it makes sense to merge #51 since it is simpler.
5. ankurdave

   closed this as completedon Nov 25, 2013
6. ccsevers commented on Jan 9, 2014

   

   ccsevers

   on Jan 9, 2014

   More actions

   I'm running into this issue in the latest version built off of #132

   The code I'm running is pretty simple:

       val input = sc.sequenceFile[VectorWritable, VectorWritable](inputPath, classOf[VectorWritable], classOf[VectorWritable])
   // not even parsing the vectors, just making some big graph
   val edges = input.flatMap {
         case (vec1, vec2) =>
           Seq(Edge(Random.nextLong, Random.nextLong, 1), Edge(Random.nextLong(),Random.nextLong(), 1))
       }
    val g = Graph.fromEdges(edges, 1)
    val cc = ConnectedComponents.run(g)
    cc.vertices.count()
   

   The error I see is this:
   4/01/09 10:38:46 WARN scheduler.TaskSetManager: Lost TID 400 (task 4.0:0)
   14/01/09 10:38:46 WARN scheduler.TaskSetManager: Loss was due to java.util.NoSuchElementException
   java.util.NoSuchElementException: End of stream
   at org.apache.spark.util.NextIterator.next(NextIterator.scala:83)
   at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:29)
   at org.apache.spark.graph.impl.RoutingTable$$anonfun$1.apply(RoutingTable.scala:34)
   at org.apache.spark.graph.impl.RoutingTable$$anonfun$1.apply(RoutingTable.scala:33)
   at org.apache.spark.rdd.RDD$$anonfun$1.apply(RDD.scala:449)
   at org.apache.spark.rdd.RDD$$anonfun$1.apply(RDD.scala:449)
   at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:34)
   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:242)
   at org.apache.spark.rdd.RDD.iterator(RDD.scala:231)
   at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:161)
   at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:102)
   at org.apache.spark.scheduler.Task.run(Task.scala:53)
   at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
   at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:50)
   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)

   This is running on a Hadoop 2.2.0 cluster with YARN 2.2.0. I've previously run against the same data set with Bagel and it works great.
7. ankurdave

   reopened this on Jan 9, 2014
8. ccsevers commented on Jan 9, 2014

   

   ccsevers

   on Jan 9, 2014

   More actions

   Just to make it simpler you can change the input to something like:
   val input = sc.parallelize(1 to 100000000, 400)
   and change the flatMap input appropriately and see the same error.

   edit: Also, oddly I can run it with say 11 splits on 10 works and it goes through. I see the same End of Stream error in the logs but it just fails that tasks and keeps going.
9. ankurdave commented on Jan 11, 2014

   

   ankurdave

   on Jan 11, 2014

   Member

   More actions

   Thanks for the report. I haven't yet been able to reproduce this. How many cores does Spark have in your configuration?
10. ccsevers commented on Jan 12, 2014

    

    ccsevers

    on Jan 12, 2014

    More actions

    @ankurdave I've tried it with 4-8 cores. I can try with 1 if you think it would help pin down what's going on.

    edit: Just to be clear, I mean 10-20 worker nodes with 4-8 cores each. (the machines have 24 cores each though).
11. pwendell

    closed this as completedin 4a805afon Jan 18, 2014
12. ankurdave

    reopened this on Jan 18, 2014
13. pwendell

    added a commit that references this issue on Mar 21, 2014

    [SPARK-1150] fix repo location in create script (re-open)

    ...

    This issue will close once commit fe195ae is merged into the 'master' branch.

    fe195ae
14. ankurdave

    added a commit that references this issue on Apr 9, 2014

    Handle empty partition iterators

    ...

    This issue will close once commit 7402177 is merged into the 'master' branch.

    7402177
15. pwendell

    closed this as completedin fe195aeon Apr 14, 2014
16. ankurdave

    reopened this on Apr 14, 2014
17. ankurdave

    added 2 commits that reference this issue on Apr 22, 2014

    Handle empty partition iterators

    ...

    This issue will close once commit 354d154 is merged into the 'master' branch.

    354d154

    Handle empty partition iterators

    ...

    This issue will close once commit e854542 is merged into the 'master' branch.

    e854542