Я получаю следующее исключение в java в Apache Beam, используя Spark в качестве бегуна. Я прокомментировал ошибочную строку.
PipelineOptions o = PipelineOptionsFactory.create();
SparkContextOptions options = o.as(SparkContextOptions.class);
SparkConf conf = new SparkConf().setMaster("local[4]").setAppName("test");
JavaSparkContext jsc = new JavaSparkContext(conf);
options.setRunner(SparkRunner.class);
options.setUsesProvidedSparkContext(true);
options.setProvidedSparkContext(jsc);
Pipeline pipeline = Pipeline.create(options); //Error in this line
Выше код и ниже ошибка !!
3639 [main] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@18324f97{/metrics/json,null,AVAILABLE,@Spark}
Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method SparkRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
at com.svayamtech.bean_app.Wordcount_lambda.main(Wordcount_lambda.java:42)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 4 more
Caused by: java.lang.NoClassDefFoundError: org/apache/beam/runners/core/construction/PipelineResources
at org.apache.beam.runners.spark.SparkRunner.fromOptions(SparkRunner.java:129)
... 9 more
Caused by: java.lang.ClassNotFoundException: org.apache.beam.runners.core.construction.PipelineResources
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 10 more
3690 [Thread-1] INFO org.apache.spark.SparkContext - Invoking stop() from shutdown hook
3696 [Thread-1] INFO org.spark_project.jetty.server.AbstractConnector - Stopped Spark@78f9ed3e{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
Spark Runner не работает? Кто-нибудь может мне помочь в этой проблеме?