Я надеюсь, что следующие строки SBT и их комментарии будут достаточны для объяснения вашего вопроса.
// The version of your project itself.
// You can change this value whenever you want,
// e.g. everytime you make a production release.
version := "0.1.0"
// The Scala version your project uses for compile.
// If you use spark, you can only use a 2.11.x version.
// Also, because Spark includes its own Scala in runtime
// I recommend you use the same one;
//you can check which one your Spark instance uses in the spark-shell.
scalaVersion := "2.11.12"
// The spark version the project uses for compile.
// Because you wont generate an uber jar with Spark included,
// but deploy your jar to an spark cluster instance.
// This version must match with the remote one, unless you want weird bugs...
val SparkVersion = "2.3.1"
// Note, I use a val with the Spark version
// to make it easier to include several Spark modules in my project,
// this way, if I want/have to change the Spark version,
// I only have to modify one line,
// and avoid strange erros because I changed some versions, but not others.
// Also note the 'Provided' modifier at the end,
// it indicates SBT that it shouldn't include the Spark bits in the generated jar
// neither in package nor assembly tasks.
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % SparkVersion % Provided,
"org.apache.spark" %% "spark-sql" % SparkVersion % Provided,
)
// Exclude Scala from the assembly jar, because spark already includes it.
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
Вам также следует позаботиться о версии SBT, то есть о версии SBT, используемой в вашем проекте.,Вы устанавливаете его в файле "project / build.properties" .
sbt.version=1.2.3
Примечание: Я использую плагин sbt-assembly , чтобы создать банку со всеми включенными зависимостями, кроме Spark и Scala.Это полезно, если вы используете другие библиотеки, например MongoSparkConnector.