У меня есть скрипт оболочки, который запускает приложение spark с использованием spark-submit.
Сегодня я обнаружил, что spark запускается очень медленно.Журнал выглядит следующим образом:
18/10/07 06:00:53 WARN SparkConf: In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
18/10/07 06:00:53 WARN SparkConf: Detected deprecated memory fraction settings: [spark.shuffle.memoryFraction, spark.storage.memoryFraction, spark.storage.unrollFraction]. As of Spark 1.6, execution and storage memory management are unified. All memory fractions used in the old model are now deprecated and no longer read. If you wish to use the old memory management, you may explicitly enable `spark.memory.useLegacyMode` (not recommended).
18/10/07 06:00:54 INFO log: Logging initialized @47766ms
18/10/07 06:00:54 INFO Server: jetty-9.2.z-SNAPSHOT
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4a481728{/jobs,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4ace284d{/jobs/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@77fa9381{/jobs/job,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@76d72b5b{/jobs/job/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7cac93fe{/stages,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@479b5066{/stages/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@64deb58f{/stages/stage,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2b1cd7bc{/stages/stage/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@33379242{/stages/pool,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2e32fc22{/stages/pool/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3f5156a6{/storage,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@72bdd7c7{/storage/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7d17ee50{/storage/rdd,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@44fdce3c{/storage/rdd/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@712c5463{/environment,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6d33a66e{/environment/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2f14b0f6{/executors,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5b20706{/executors/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@16aed49f{/executors/threadDump,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@66de00f2{/executors/threadDump/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6c479fdf{/static,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@79d82f66{/,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@19b206ee{/api,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@ffd4cba{/jobs/job/kill,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5f1483fd{/stages/stage/kill,null,AVAILABLE,@Spark}
18/10/07 06:00:54 INFO ServerConnector: Started Spark@21263314{HTTP/1.1}{10.1.237.119:22640}
18/10/07 06:00:54 INFO Server: Started @47955ms
18/10/07 06:06:46 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5d5574c7{/metrics/json,null,AVAILABLE,@Spark}
18/10/07 06:06:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2328ec73{/SQL,null,AVAILABLE,@Spark}
18/10/07 06:06:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@58021c3c{/SQL/json,null,AVAILABLE,@Spark}
18/10/07 06:06:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7bda01da{/SQL/execution,null,AVAILABLE,@Spark}
18/10/07 06:06:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5c997de8{/SQL/execution/json,null,AVAILABLE,@Spark}
18/10/07 06:06:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@15d58530{/static/sql,null,AVAILABLE,@Spark}
18/10/07 06:06:48 WARN SessionState: load mapred-default.xml, HIVE_CONF_DIR env not found!
18/10/07 06:06:50 WARN SessionState: load mapred-default.xml, HIVE_CONF_DIR env not found!
Из журнала требуется около 6 минут для запуска какого-либо сервера?
18/10/07 06:00:54 INFO Server: Started @47955ms
18/10/07 06:06:46 INFO ContextHandler: Started
Я хотел бы спросить, что здесь может случиться, почему такмного времени?