-
Notifications
You must be signed in to change notification settings - Fork 15
Open
Description
Description
Spark exmaples which probably have string types input failed on current Sparkle.
Environment
current HerlettPackard/sparkle and replacing HewlettPackard/sparkle/alps with hvolos/alps
Reproducible example
Using CountVectorizerExample under spark repository
./bin/spark-submit \
--class org.apache.spark.examples.ml.CountVectorizerExample \
--master "spark://$(hostname):7077" \
--deploy-mode client \
--executor-memory 16G \
--conf "spark.executor.extraJavaOptions=-XX:MaxDirectMemorySize=16G" \
./examples/target/scala-2.11/jars/spark-examples_2.11-2.0.0.jar \
Steps to reproduce
- Build sparkle and start spark process
- Move spark directory (cd $SPARK_HOME) and executing example command
Result message(a part of driver stack trace)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1437)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1437)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
at scala.Option.foreach(Option.scala:257)
[...]
Caused by: java.lang.RuntimeException: serialize V for String value is not implemented
at com.hp.hpl.firesteel.shuffle.MapSHMShuffleStore.serializeVString(MapSHMShuffleStore.java:360)
at com.hp.hpl.firesteel.shuffle.MapSHMShuffleStore.serializeKVPair(MapSHMShuffleStore.java:386)
at org.apache.spark.shuffle.shm.ShmShuffleWriter.write(ShmShuffleWriter.scala:205)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Metadata
Metadata
Assignees
Labels
No labels