Я пытаюсь развернуть Spark-приложение (Java) в Spark Engine как часть службы службы аналитики в Bluemix. Я следовал за шагом упоминания здесь
https://console.bluemix.net/docs/services/AnalyticsEngine/WCE-CLI.html#analytics-engine-command-line-interface
После выполнения всех шагов и во время spark-submit я получаю следующую ошибку
<code>C:\Users\IBM_ADMIN\eclipse-workspace\mySparkApp\target>bx ae spark-submit --className mySparkApp.Main mySparkApp-0.0.1-S
NAPSHOT.jar
Current user is 'clsadmin'
Password>
Contacting endpoint 'https://159.122.220.119:8443'...
FAILED
Server call failed. Message: '<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /gateway/default/livy/v1/batches. Reason:
<pre> Server Error
Работает на причале: // <час />
'
Я могу получить доступ к серверу ambari и вижу, что сервисы spark работают и работают. Я также могу получить доступ к конечной точке API livy из браузера
https://chs -uvi-769-mn001.bi.services.eu-gb.bluemix.net: 8443 / шлюз / по умолчанию / Ливий / v1 / партии
{"from":0,"total":5,"sessions":[{"id":0,"state":"dead","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["java.lang.Exception: No YARN application is found with tag livy-batch-0-t8fc4ebv in 60 seconds. Please check your cluster status, it is may be very busy.","com.cloudera.livy.utils.SparkYarnApp.com$cloudera$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:248) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:245) scala.Option.getOrElse(Option.scala:120) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:245) com.cloudera.livy.Utils$$anon$1.run(Utils.scala:95)"]},{"id":1,"state":"dead","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["java.lang.Exception: No YARN application is found with tag livy-batch-1-1olxdmt5 in 60 seconds. Please check your cluster status, it is may be very busy.","com.cloudera.livy.utils.SparkYarnApp.com$cloudera$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:248) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:245) scala.Option.getOrElse(Option.scala:120) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:245) com.cloudera.livy.Utils$$anon$1.run(Utils.scala:95)"]},{"id":2,"state":"dead","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["java.lang.Exception: No YARN application is found with tag livy-batch-2-xbjzpkbp in 60 seconds. Please check your cluster status, it is may be very busy.","com.cloudera.livy.utils.SparkYarnApp.com$cloudera$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:248) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:245) scala.Option.getOrElse(Option.scala:120) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:245) com.cloudera.livy.Utils$$anon$1.run(Utils.scala:95)"]},{"id":3,"state":"dead","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["java.lang.Exception: No YARN application is found with tag livy-batch-3-sbilpm4a in 60 seconds. Please check your cluster status, it is may be very busy.","com.cloudera.livy.utils.SparkYarnApp.com$cloudera$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:248) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:245) scala.Option.getOrElse(Option.scala:120) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:245) com.cloudera.livy.Utils$$anon$1.run(Utils.scala:95)"]},{"id":4,"state":"dead","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["java.lang.Exception: No YARN application is found with tag livy-batch-4-rvlzpl8o in 60 seconds. Please check your cluster status, it is may be very busy.","com.cloudera.livy.utils.SparkYarnApp.com$cloudera$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:248) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:245) scala.Option.getOrElse(Option.scala:120) com.cloudera.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:245) com.cloudera.livy.Utils$$anon$1.run(Utils.scala:95)"]}]}
Я не могу понять, в чем проблема. Я включил трассировку bluemix, и вывод в режиме отладки для этой же команды
<code>C:\Users\IBM_ADMIN\eclipse-workspace\mySparkApp\target>SET BLUEMIX_TRACE=true
C:\Users\IBM_ADMIN\eclipse-workspace\mySparkApp\target>bx ae spark-submit --className mySparkApp.Main mySparkApp-0.0.1-S
NAPSHOT.jar
Current user is 'clsadmin'
Password>
DEBUG: INPUTS PROVIDED
file mySparkApp-0.0.1-SNAPSHOT.jar
proxyUser NOTSET
className mySparkApp.Main
inputArgs []
jars []
pyFiles []
files []
driverMemory NOTSET
driverCores -1
executorMemory NOTSET
executorCores -1
numExecutors -1
archives []
queue NOTSET
name NOTSET
conf map[]
asynchronous false
upload false
DEBUG: PROCESSED INPUTS
file mySparkApp-0.0.1-SNAPSHOT.jar
proxyUser NOTSET
className mySparkApp.Main
inputArgs []
jars []
pyFiles []
files []
driverMemory NOTSET
driverCores -1
executorMemory NOTSET
executorCores -1
numExecutors -1
archives []
queue NOTSET
name NOTSET
conf map[]
asynchronous false
upload false
Contacting endpoint 'https://159.122.220.119:8443'...
←[1;39mREQUEST:←[0m [2018-05-03T16:52:33+05:30]
POST /gateway/default/livy/v1/batches HTTP/1.1
Host: 159.122.220.119:8443
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
X-Requested-By: livy
{"file":"mySparkApp-0.0.1-SNAPSHOT.jar","className":"mySparkApp.Main"}
←[1;39mRESPONSE:←[0m [2018-05-03T16:52:34+05:30] ←[1;39mElapsed:←[0m 1382ms
HTTP/1.1 500 Server Error
Connection: close
Content-Length: 321
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html; charset=ISO-8859-1
Date: Thu, 03 May 2018 11:22:52 GMT
Date: Thu, 03 May 2018 11:22:52 GMT
Server: Jetty(9.2.16.v20160414)
Set-Cookie: JSESSIONID=vyw2g1lyyjmo2d2spzs5pnd1;Path=/gateway/default;Secure;HttpOnly
Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Wed, 02-May-2018 11:22:52 GMT
Strict-Transport-Security: max-age=31536000
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /gateway/default/livy/v1/batches. Reason:
<pre> Server Error
Работает на причале: // <ч />
НЕ УДАЛОСЬ
Ошибка вызова сервера. Сообщение: '
Ошибка 500 Ошибка сервера
HTTP ERROR 500
Проблема с доступом к / gateway / default / livy / v1 / batches. Причина:
Server Error
Powered by Jetty: //
'
НЕ УДАЛОСЬ
Ошибка вызова сервера. Сообщение: '← [1; 31 м
Ошибка 500 Ошибка сервера
ОШИБКА HTTP 500
Проблема с доступом к / gateway / default / livy / v1 / batches. Причина:
Server Error
Powered by Jetty: //
← [0m»
Кто-нибудь может понять, в чем проблема? любая помощь или указатель?