Я пытаюсь создать сеанс spark, используя apache livy, но я не могу создать сеанс на кластере spark, и я не получаю никаких полезных журналов.
Есть ли способ, которым я могу может найти действительную причину этой ошибки.
Любая помощь будет признательна.
Журналы, которые я получаю в пользовательском интерфейсе livy: -
Session 7 unexpectedly reached final status 'dead'. See logs:
stdout:
stderr:
20/05/03 14:44:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/05/03 14:44:55 INFO client.RMProxy: Connecting to ResourceManager at streamflux-master/172.17.0.2:8032
20/05/03 14:44:55 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
20/05/03 14:44:55 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (10240 MB per container)
20/05/03 14:44:55 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
20/05/03 14:44:55 INFO yarn.Client: Setting up container launch context for our AM
20/05/03 14:44:55 INFO yarn.Client: Setting up the launch environment for our AM container
20/05/03 14:44:55 INFO yarn.Client: Preparing resources for our AM container
20/05/03 14:44:55 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
20/05/03 14:44:57 INFO yarn.Client: Uploading resource file:/tmp/spark-f25d699f-f00d-400d-bd47-9a42adf1fd01/__spark_libs__8419058456894841499.zip -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/__spark_libs__8419058456894841499.zip
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/rsc-jars/livy-rsc-0.6.0-incubating.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/livy-rsc-0.6.0-incubating.jar
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/rsc-jars/netty-all-4.0.37.Final.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/netty-all-4.0.37.Final.jar
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/rsc-jars/livy-thriftserver-session-0.6.0-incubating.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/livy-thriftserver-session-0.6.0-incubating.jar
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/rsc-jars/livy-api-0.6.0-incubating.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/livy-api-0.6.0-incubating.jar
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/repl_2.11-jars/livy-core_2.11-0.6.0-incubating.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/livy-core_2.11-0.6.0-incubating.jar
20/05/03 14:44:58 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/repl_2.11-jars/commons-codec-1.9.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/commons-codec-1.9.jar
20/05/03 14:44:59 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/livy/repl_2.11-jars/livy-repl_2.11-0.6.0-incubating.jar -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/livy-repl_2.11-0.6.0-incubating.jar
20/05/03 14:44:59 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/spark/R/lib/sparkr.zip#sparkr -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/sparkr.zip
20/05/03 14:44:59 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/spark/python/lib/pyspark.zip -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/pyspark.zip
20/05/03 14:44:59 INFO yarn.Client: Uploading resource file:/home/streamflux/usr/local/spark/python/lib/py4j-0.10.7-src.zip -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/py4j-0.10.7-src.zip
20/05/03 14:44:59 WARN yarn.Client: Same name resource file:///home/streamflux/usr/local/spark/python/lib/pyspark.zip added multiple times to distributed cache
20/05/03 14:44:59 WARN yarn.Client: Same name resource file:///home/streamflux/usr/local/spark/python/lib/py4j-0.10.7-src.zip added multiple times to distributed cache
20/05/03 14:44:59 INFO yarn.Client: Uploading resource file:/tmp/spark-f25d699f-f00d-400d-bd47-9a42adf1fd01/__spark_conf__6956912608326152031.zip -> hdfs://streamflux-master:8020/user/streamflux/.sparkStaging/application_1588509039643_0015/__spark_conf__.zip
20/05/03 14:45:00 INFO spark.SecurityManager: Changing view acls to: streamflux
20/05/03 14:45:00 INFO spark.SecurityManager: Changing modify acls to: streamflux
20/05/03 14:45:00 INFO spark.SecurityManager: Changing view acls groups to:
20/05/03 14:45:00 INFO spark.SecurityManager: Changing modify acls groups to:
20/05/03 14:45:00 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(streamflux); groups with view permissions: Set(); users with modify permissions: Set(streamflux); groups with modify permissions: Set()
20/05/03 14:45:01 INFO yarn.Client: Submitting application application_1588509039643_0015 to ResourceManager
20/05/03 14:45:01 INFO impl.YarnClientImpl: Submitted application application_1588509039643_0015
20/05/03 14:45:01 INFO yarn.Client: Application report for application_1588509039643_0015 (state: ACCEPTED)
20/05/03 14:45:01 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1588517101092
final status: UNDEFINED
tracking URL: http://streamflux-master:8088/proxy/application_1588509039643_0015/
user: streamflux
20/05/03 14:45:01 INFO util.ShutdownHookManager: Shutdown hook called
20/05/03 14:45:01 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-f25d699f-f00d-400d-bd47-9a42adf1fd01
20/05/03 14:45:01 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-028c83f0-9c34-4f2c-9466-a1fbcd03ac6b
YARN Diagnostics: Application killed by user.