Я пытаюсь создать два dataframe и присоединиться к ним, используя метод dataframe.join.
Вот код scala:
import org.apache.spark.sql.SparkSession
import org.apache.spark.SparkConf
object RuleExecutor {
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf().setAppName(AppConstants.AppName).setMaster("local")
val sparkSession = SparkSession.builder().appName(AppConstants.AppName).config(sparkConf).enableHiveSupport().getOrCreate()
import sparkSession.sql
sql(s"CREATE DATABASE test")
sql ("CREATE TABLE test.box_width (id INT, width INT)") // Create table box_width
sql ("INSERT INTO test.box_width VALUES (1,1), (2,2)") // Insert data in box_width
sql ("CREATE TABLE test.box_length (id INT, length INT)") // Create table box_length
sql ("INSERT INTO test.box_length VALUES (1,10), (2,20)") // Insert data in box_length
val widthDF = sql("select * from test.box_width") // Get DF for table box_width
val lengthDF = sql("select * from test.box_length") // Get DF for table box_length
val dimensionDF = lengthDF.join(widthDF, "id"); // Joining
dimensionDF.show();
}
}
Но при выполнении кода я получаю следующую ошибку:
Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1062)…..
Caused by: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)……
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)……
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)……
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)…
Caused by: org.datanucleus.api.jdo.exceptions.ClassNotPersistenceCapableException: The class "org.apache.hadoop.hive.metastore.model.MVersionTable" is not persistable. This means that it either hasnt been enhanced, or that the enhanced version of the file is not in the CLASSPATH (or is hidden by an unenhanced version), or the Meta-Data/annotations for the class are not found.
NestedThrowables:
org.datanucleus.exceptions.ClassNotPersistableException: The class "org.apache.hadoop.hive.metastore.model.MVersionTable" is not persistable. This means that it either hasnt been enhanced, or that the enhanced version of the file is not in the CLASSPATH (or is hidden by an unenhanced version), or the Meta-Data/annotations for the class are not found.
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:473)……
Caused by: org.datanucleus.exceptions.ClassNotPersistableException: The class "org.apache.hadoop.hive.metastore.model.MVersionTable" is not persistable. This means that it either hasnt been enhanced, or that the enhanced version of the file is not in the CLASSPATH (or is hidden by an unenhanced version), or the Meta-Data/annotations for the class are not found.
at org.datanucleus.ExecutionContextImpl.assertClassPersistable(ExecutionContextImpl.java:5113)……
Версии, которые я использую:
Scala = 2.11
Spark-hive = 2.2.2
Maven-org-spark-project-hive_hive-metastore = 1.x
DataNucleus = 5.x
Как решитьЭта проблема? полный журнал список зависимостей
Спасибо