ContextCleaner: убрал аккумулятор, что это значит в scala spark?

Question

Когда я запускаю свою искровую программу, я вижу этот вывод, и чтобы замедлить до конца, что это значит из этого контекста?

19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 82
19/04/01 15:34:24 INFO ContextCleaner: Cleaned shuffle 0
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 69
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 30
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 40
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 61
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 41
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 52
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 29
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 31
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 57
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 60
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 87
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 79
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 78
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 84
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 34
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 49
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 75
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 88
19/04/01 15:34:24 INFO ContextCleaner: Cleaned accumulator 48

Депс Я использую

name := "BigData"

version := "0.1"

scalaVersion := "2.11.12"

libraryDependencies += "com.github.tototoshi" %% "scala-csv" % "1.3.5"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0"

// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0"
// https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc
libraryDependencies += "com.microsoft.sqlserver" % "mssql-jdbc" % "6.1.0.jre8"
libraryDependencies += "com.databricks" % "spark-xml_2.11" % "0.4.1"

// https://mvnrepository.com/artifact/com.typesafe.akka/akka-actor
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.5.19"
// https://mvnrepository.com/artifact/com.typesafe.akka/akka-http
libraryDependencies += "com.typesafe.akka" %% "akka-http" % "10.1.5"
// https://mvnrepository.com/artifact/com.typesafe.akka/akka-stream
libraryDependencies += "com.typesafe.akka" %% "akka-stream" % "2.5.19"

// https://mvnrepository.com/artifact/org.apache.livy/livy-core
libraryDependencies += "org.apache.livy" %% "livy-core" % "0.5.0-incubating"

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.9.4"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.4"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.9.4"
// https://mvnrepository.com/artifact/net.liftweb/lift-json
libraryDependencies += "net.liftweb" %% "lift-json" % "3.2.0"

// https://mvnrepository.com/artifact/org.json4s/json4s-jackson
libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.6.5"

// https://mvnrepository.com/artifact/org.json4s/json4s-native
libraryDependencies += "org.json4s" %% "json4s-native" % "3.6.5"

// https://mvnrepository.com/artifact/oracle/xdb

//libraryDependencies += "oracle" % "xdb" % "1.0"

Answer 1

ContextCleaner работает на драйвере. Он создается и сразу запускается при запуске SparkContext. Поток очистителя контекста, который очищает RDD, состояния перемешивания и широковещания, аккумуляторы (с использованием метода keepCleaning). context-cleaner-period-gc запрашивает сборщик мусора JVM. Периодические запуски запускаются при запуске ContextCleaner и прекращаются при остановке ContextCleaner.