У меня есть задание Kafka Streaming Consumer, которое сохраняет данные в таблице Hive.
Проблема в том, что иногда из-за куста работа не выполняется, и ее приходится перезапускать снова.
Это происходит со всеми потоковыми заданиями потребителей, которые пытаются получить доступ к Hive.
Не в состоянии отследить причину.
Ценю ваши предложения.
Ниже приведен журнал ошибок высокого уровня
INFO ApplicationMaster: Starting the user application in a separate Thread
INFO ApplicationMaster: Waiting for spark context initialization...
WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Waiting 1 seconds before next connection attempt.
INFO metastore: Trying to connect to metastore with URI thrift://<server2:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Waiting 1 seconds before next connection attempt.
INFO metastore: Trying to connect to metastore with URI thrift://<server2:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Waiting 1 seconds before next connection attempt.
WARN Hive: Failed to access metastore. This class should not accessed in runtime.
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: java.lang.reflect.InvocationTargetException
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: Peer indicated failure: DIGEST-MD5: IO error acquiring password
INFO metastore: Trying to connect to metastore with URI thrift://<server2:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Waiting 1 seconds before next connection attempt.
INFO metastore: Trying to connect to metastore with URI thrift://<server2:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Waiting 1 seconds before next connection attempt.
INFO metastore: Trying to connect to metastore with URI thrift://<server2:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Trying to connect to metastore with URI thrift://<server1:port1>
WARN metastore: Failed to connect to the MetaStore Server...
INFO metastore: Waiting 1 seconds before next connection attempt.
ERROR ApplicationMaster: User class threw exception: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
Caused by: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: java.lang.reflect.InvocationTargetException
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: Peer indicated failure: DIGEST-MD5: IO error acquiring password