Я проследил за блогом, который представляет Spark как веб-сервис с использованием Akka HTTP.
Код работает нормально, когда я запускаю его независимо, но не удается, когда я пытался подключить его к автономному кластеру Spark. Я провел некоторый анализ, и многие предложили правильно установить запись узла и файл spark-env.sh Я сделал все это и до сих пор, это не работает.
Примечание : Моя версия Spark и версия Spark в моем pom также совпадают, и я могу отправить задание в один кластер с помощью «spark-submit».
Файл хоста
10.93.125.98 master
10.93.125.97 cluster-node-2
10.93.125.99 cluster-node-3
Файл Spark Slaves
master
cluster-node-2
cluster-node-3
Spark Env File
export SPARK_MASTER_HOST=master
Scala Code
object SparkFactory {
val spark: SparkSession = SparkSession.builder
.master(AppConfig.sparkMaster)
.appName(AppConfig.sparkAppName)
.getOrCreate
val sc = spark.sparkContext
val sparkConf = sc.getConf
}
Полный код доступен в репозитории GitHub ниже
https://github.com/spoddutur/spark-as-service-using-embedded-server
Получение приведенной ниже ошибки при попытке подключиться к искровому кластеру. Любые указатели будут высоко оценены.
[service-user@10.93.125.98 cloud]$ java -jar spark-with-scala-0.0.1-SNAPSHOT-allinone.jar -m spark://master:7077
Press RETURN to stop...
[INFO] [07/10/2019 15:53:49.205] [WebServer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(WebServer)] Server online at http://master:8001/
^C[service-user@10.93.125.98 cloud]$ java -jar spark-with-scala-0.0.1-SNAPSHOT-allinone.jar -m spark://master:7077
Press RETURN to stop...
[INFO] [07/10/2019 15:56:15.096] [WebServer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(WebServer)] Server online at http://master:8001/
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/07/10 15:56:27 INFO SparkContext: Running Spark version 2.4.3
19/07/10 15:56:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/07/10 15:56:28 INFO SparkContext: Submitted application: spark-with-scala
19/07/10 15:56:28 INFO SecurityManager: Changing view acls to: service-user
19/07/10 15:56:28 INFO SecurityManager: Changing modify acls to: service-user
19/07/10 15:56:28 INFO SecurityManager: Changing view acls groups to:
19/07/10 15:56:28 INFO SecurityManager: Changing modify acls groups to:
19/07/10 15:56:28 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(service-user); groups with view permissions: Set(); users with modify permissions: Set(service-user); groups with modify permissions: Set()
19/07/10 15:56:29 INFO Utils: Successfully started service 'sparkDriver' on port 39006.
19/07/10 15:56:29 INFO SparkEnv: Registering MapOutputTracker
19/07/10 15:56:29 INFO SparkEnv: Registering BlockManagerMaster
19/07/10 15:56:29 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/07/10 15:56:29 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/07/10 15:56:29 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-f77fe9f4-dbb8-49db-bbf2-9fca39e18e11
19/07/10 15:56:30 INFO MemoryStore: MemoryStore started with capacity 127.2 MB
19/07/10 15:56:30 INFO SparkEnv: Registering OutputCommitCoordinator
19/07/10 15:56:30 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/07/10 15:56:30 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://master:4040
19/07/10 15:56:30 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:56:31 INFO TransportClientFactory: Successfully created connection to master/10.93.125.98:7077 after 79 ms (0 ms spent in bootstraps)
[INFO] [07/10/2019 15:56:47.256] [WebServer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(WebServer)] Request timeout encountered for request [GET /count Empty]
19/07/10 15:56:50 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:57:10 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:57:30 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
19/07/10 15:57:30 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
19/07/10 15:57:30 INFO SparkUI: Stopped Spark web UI at http://master:4040
19/07/10 15:57:30 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40701.
19/07/10 15:57:30 INFO NettyBlockTransferService: Server created on master:40701
19/07/10 15:57:31 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/07/10 15:57:31 INFO StandaloneSchedulerBackend: Shutting down all executors
19/07/10 15:57:31 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/07/10 15:57:31 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
19/07/10 15:57:31 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/07/10 15:57:31 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO MemoryStore: MemoryStore cleared
19/07/10 15:57:31 INFO BlockManager: BlockManager stopped
19/07/10 15:57:31 INFO BlockManagerMasterEndpoint: Registering block manager master:40701 with 127.2 MB RAM, BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManagerMaster: BlockManagerMaster stopped
19/07/10 15:57:31 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/07/10 15:57:31 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
at scala.Option.getOrElse(Option.scala:138)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
at akka.actor.Actor.aroundReceive(Actor.scala:539)
at akka.actor.Actor.aroundReceive$(Actor.scala:537)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
at akka.actor.ActorCell.invoke(ActorCell.scala:581)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
at akka.dispatch.Mailbox.run(Mailbox.scala:229)
at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
19/07/10 15:57:31 INFO SparkContext: SparkContext already stopped.
19/07/10 15:57:31 INFO SparkContext: Successfully stopped SparkContext
Uncaught error from thread [WebServer-akka.actor.default-dispatcher-5]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[WebServer]
java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
at akka.actor.Actor.aroundReceive(Actor.scala:539)
at akka.actor.Actor.aroundReceive$(Actor.scala:537)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
at akka.actor.ActorCell.invoke(ActorCell.scala:581)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
at akka.dispatch.Mailbox.run(Mailbox.scala:229)
at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
at scala.Option.getOrElse(Option.scala:138)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
... 57 more
19/07/10 15:57:31 INFO ShutdownHookManager: Shutdown hook called
[ERROR] [SECURITY][07/10/2019 15:57:31.461] [WebServer-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(WebServer)] Uncaught error from thread [WebServer-akka.actor.default-dispatcher-5]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[WebServer]
java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
at akka.actor.Actor.aroundReceive(Actor.scala:539)
at akka.actor.Actor.aroundReceive$(Actor.scala:537)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
at akka.actor.ActorCell.invoke(ActorCell.scala:581)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
at akka.dispatch.Mailbox.run(Mailbox.scala:229)
at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
at scala.Option.getOrElse(Option.scala:138)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
... 57 more