Spark Master узел не подходит в datastax dse - PullRequest
Новая
       80

Spark Master узел не подходит в datastax dse

0 голосов
/ 19 октября 2019

Любая помощь будет высоко оценена. См. Жирный курсив, который кажется основной ошибкой

Я настроил 6 узлов кластера dse, dc1 (для рабочих нагрузок cassandra) и dc2 (для рабочих нагрузок искры аналитики). Оба dcs имеют 3 узла, и они работают и показываютисправен с "dsetool ring"

Попытка получить ошибку ниже:

dse spark ИЛИ dse -u cassandra -p cassandra spark

Ошибка: 

The log file is at /root/.spark-shell.log
WARN  2019-10-19 08:27:21,816 org.apache.spark.deploy.SparkNodeConfiguration: Failed to fetch dynamic configuration from DSE, retrying in 2 seconds...
ERROR 2019-10-19 08:27:25,170 org.apache.spark.deploy.DseSparkSubmitBootstrapper: Failed to start or submit Spark application because of java.io.IOException: Failed to fetch dynamic configuration from DSE - see details in the log file(s): /root/.spark-shell.log
ERROR 2019-10-19 08:27:25,352 
org.apache.spark.deploy.DseSparkSubmitBootstrapper: Failed to cancel delegation token

Уже можно проверитьв dse_leases.leases dse выбрал одного мастера для выбора cassandra@cqlsh:dse_leases> select * from dse_leases.leases;

 name              | dc    | duration_ms | epoch | holder

-------------------+-------+-------------+-------+--------------

 Leader/master/6.0 | spark |       30000 |  1091 | 172.31.31.232

Но когда я зашел в /var/log/spark/master.log, я получаю ошибку ниже: 

DEBUG 2019-10-19 08:12:16,084 ServiceRunner.java:205 - Service SPARK-MASTER set state: NOT_STARTED -> SHUTTING_DOWN
DEBUG 2019-10-19 08:12:16,084 ServiceRunner.java:197 - Service SPARK-MASTER state is: SHUTTING_DOWN
DEBUG 2019-10-19 08:12:16,084 ServiceRunner.java:197 - Service SPARK-MASTER state is: SHUTTING_DOWN

DEBUG 2019-10-19 08:12:16,084 ServiceRunner.java:219 - Service SPARK-MASTER FAILED changing state: NOT_STARTED -> SHUTTING_DOWN
DEBUG 2019-10-19 08:12:16,084 ServiceRunner.java:205 - Service SPARK-MASTER set state: SHUTTING_DOWN -> NOT_STARTED
WARN  2019-10-19 08:12:16,084 Logging.scala:47 - Not overriding system property java.library.path
***WARN  2019-10-19 08:12:16,085 Logging.scala:47 - Not overriding system property java.library.path
ERROR 2019-10-19 08:12:16,085 Logging.scala:72 - Failed to stop redirecting UI
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_SERIAL
        at org.apache.cassandra.service.StorageProxy.getPaxosParticipants(StorageProxy.java:327)***
        at org.apache.cassandra.service.StorageProxy.readWithPaxos(StorageProxy.java:1357)
        at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1303)
        at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1276)
        at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:371)
        at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:579)
        at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:312)
        at org.apache.cassandra.cql3.QueryProcessor.lambda$processStatement$4(QueryProcessor.java:266)
        at io.reactivex.internal.operators.single.SingleDefer.subscribeActual(SingleDefer.java:36)
        at io.reactivex.Single.subscribe(Single.java:2700)
        at io.reactivex.Single.blockingGet(Single.java:2153)
        at org.apache.cassandra.concurrent.TPCUtils.blockingGet(TPCUtils.java:75)
...