Я работаю над интеграцией Apache Spark со Springboot и CassandaraDB.Я предоставил следующий файл конфигурации для конфигурирования спарк и кассандры.
@Configuration
@PropertySource("classpath:application.properties")
public class SparkConfig {
private static Logger log = LoggerFactory.getLogger(DashboardRestServicesApplication.class.getName());
@Value("${spark.master}")
private String sparkMaster;
@Value("${spring.data.cassandra.keyspace-name}")
private String cassandraKeyspace;
@Value("${cassandra.table}")
private String cassandraTable;
@Value("${spring.data.cassandra.contact-points}")
private String cassandraHost;
@Value("${spring.data.cassandra.port}")
private String cassandraPort;
@Value("${spring.data.cassandra.username}")
private String username;
@Value("${spring.data.cassandra.password}")
private String password;
@Bean
public SparkConf sparkConf() {
SparkConf conf = new SparkConf(true)
.set("spark.cassandra.connection.host",cassandraHost)
.set("spark.cassandra.connection.port", cassandraPort)
.set("spark.cassandra.auth.username", username)
.set("spark.cassandra.auth.password", password)
.set("spark.submit.deployMode", "client")
.setMaster(sparkMaster)
.setAppName("DashboardSparkService");
return conf;
}
@Bean
public JavaSparkContext javaSparkContext() {
log.info("Connecting to spark with master Url: {}, and cassandra host: {}",
sparkMaster, cassandraHost);
JavaSparkContext javaSparkContext = new JavaSparkContext(sparkConf());
log.debug("spark context created");
return javaSparkContext;
}
@Bean
public SparkSession sparkSession() {
return SparkSession
.builder()
.sparkContext(javaSparkContext().sc())
.appName("DashboardSparkService")
.getOrCreate();
}
}
Вот мой сервис:
@Autowired
private JavaSparkContext javaSparkContext;
@Autowired
private TaskSummarySparkRepo taskSummarySparkRepo;
public int getAllOrders() {
//JavaSparkContext javaSparkContext = sparkConfig.javaSparkContext();
return taskSummarySparkRepo.getAllOrders(javaSparkContext);
}
А вот мой репо:
@Service
public class TaskSummarySparkRepo {
@Value("${spring.data.cassandra.keyspace-name}")
private String CassandraKeyspace;
@Value("${cassandra.table}")
private String CassandraTable;
public int getAllOrders(JavaSparkContext javaSparkContext) {
JavaRDD<TaskSummary> rdd = javaFunctions(javaSparkContext)
.cassandraTable(CassandraKeyspace, CassandraTable,
mapRowTo(TaskSummary.class));
return (int) rdd.count();
}
}
Все выглядит хорошо, так как его сборка успешно, но когда я пытаюсь запустить jar, он выдает следующее сообщение об ошибке:
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'taskSummarySparkService': Unsatisfied dependency expressed through field 'javaSparkContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'javaSparkContext' defined in class path resource [com/spectrum/dashboard/config/SparkConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.spark.api.java.JavaSparkContext]: Factory method 'javaSparkContext' threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/spark/network/util/ByteUnit
И когда я пытаюсь запустить его как приложение Spring-boot из sts, он дает следующееошибка:
Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'javaSparkContext' defined in class path resource [com/spectrum/dashboard/config/SparkConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.spark.api.java.JavaSparkContext]: Factory method 'javaSparkContext' threw exception; nested exception is java.lang.NoClassDefFoundError: org/spark_project/guava/collect/MapMaker
РЕДАКТИРОВАТЬ: Вот зависимости файла pom, которые я использую для проекта:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-cassandra</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/com.datastax.cassandra/cassandra-driver-core -->
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.5.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.datastax.cassandra/cassandra-driver-mapping -->
<!-- <dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-mapping</artifactId>
<version>3.5.0</version>
</dependency>
-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector -->
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.11</artifactId>
<version>2.0.8</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.2.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
Есть ли какие-либо проблемы с конфигурацией или это из-за некоторых проблем с зависимостямиинъекции?Любая помощь приветствуется.