Обнаружена ошибка ниже при выполнении программы spark wordcount, пожалуйста, включите несколько индикаторов на этой
версии Spark: 2.4.0, работающей в автономном режиме
пакет com.ahshan.sparklearning
import org.apache.spark._
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
object WordCount {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("wordCount")
val sc = new SparkContext(conf)
// Load our input data.
val input = sc.textFile("file:///tmp/README.md")
// Split it up into words.
val words = input.flatMap(line => line.split(" "))
// Transform into pairs and count.
val counts = words.map(word => (word, 1)).reduceByKey{case (x, y) => x + y}
// Save the word count back out to a text file, causing evaluation.
counts.saveAsTextFile("file:///tmp/sparkout.txt")
}
}
sbt clean package
[info] Loading project definition from /root/SparkProject/project
[info] Loading settings for project sparkproject from build.sbt ...
[info] Set current project to learning-spark (in build file:/root/SparkProject/)
[success] Total time: 0 s, completed Feb 4, 2019 7:05:37 AM
[info] Updating ...
[info] Done updating.
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Compiling 1 Scala source to /root/SparkProject/target/scala-2.12/classes ...
[info] Done compiling.
[info] Packaging /root/SparkProject/target/scala-2.12/learning-spark_2.12-0.0.1.jar ...
[info] Done packaging.
[success] Total time: 8 s, completed Feb 4, 2019 7:05:45 AM
2019-02-04 07:02:54 ИНФОРМАЦИЯ BlockManagerInfo: 54 - Добавлено broadcast_0_piece0 в памяти на спарксервере: 39679 (размер: 22,9 КБ, бесплатно: 366,3 МБ) 2019-02-04 07:02: 54 INFO SparkContext: 54 - Создана трансляция 0 из textFile на WordCount.scala: 14 Исключение в потоке "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala / runtime / java8 / JFunction2 $ mcIII $ sp at com.Ахшан(NativeMethodAccessorImpl.java:62) в sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) в java.lang.reflect.Method.invoke (Method.java: 498) в org.apache.spark.deploy.JavaMainApplication.start (SparkApplication.scala: 52) в org.apache.spark.deploy.SparkSubmit.org $ apache $ spark $ deploy $ SparkSubmit $$ runMain (SparkSubmit.scala: 849) в org.apache.spark.deploy.SparkSubmit.doRunMain $ 1 (SparkSubmit.scala: 167) в org.apache.spark.deploy.SparkSubmit.submit (SparkSubmit.scala: 195) в org.apache.spark.deploy.SparkSubmit.scala: 933) at org.apache.spark.deploy.SparkSubmit.main (SparkSubmit.scala) Причина: java.lang.NoClassDefFoundError: scala / runtime / java8 / JFunction2 $ mcIII $ sp ... еще 14 причин: java.lang.ClassNotFoundException: scala.runtime.java8.JFunction2 $ mcIII $ sp в java.net.URLClassLoader.findClass (URLClassLoader.java:381) в java.lang.ClassLoader.loadClass (ClassLoader.java:424) в java.lan.ClassLoader.loadClass (ClassLoader.java:357) ... еще 14