Я новичок в Scala, я пытаюсь запустить модель в Scala, но сталкиваюсь с некоторыми проблемами:
Это файл:
package com.salesforce.hw.titanic
import com.salesforce.op._
import com.salesforce.op.features.FeatureBuilder
import com.salesforce.op.features.types._
import com.salesforce.op.readers.DataReaders
import com.salesforce.op.stages.impl.classification._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.log4j.{Level, LogManager}
/**
* A minimal Titanic Survival example with TransmogrifAI
*/
object OpTitanicMini {
case class Passenger
(
id: Long,
survived: Double,
pClass: Option[Long],
name: Option[String],
sex: Option[String],
age: Option[Double],
sibSp: Option[Long],
parCh: Option[Long],
ticket: Option[String],
fare: Option[Double],
cabin: Option[String],
embarked: Option[String]
)
def main(args: Array[String]): Unit = {
LogManager.getLogger("com.salesforce.op").setLevel(Level.ERROR)
implicit val spark = SparkSession.builder.config(new SparkConf()).getOrCreate()
import spark.implicits._
// Read Titanic data as a DataFrame
val pathToData = Option(args(0))
val passengersData = DataReaders.Simple.csvCase[Passenger](pathToData, key = _.id.toString).readDataset().toDF()
// Automated feature engineering
val (survived, features) = FeatureBuilder.fromDataFrame[RealNN](passengersData, response = "survived")
val featureVector = features.toSeq.autoTransform()
// Automated feature selection
val checkedFeatures = survived.sanityCheck(featureVector, checkSample = 1.0, removeBadFeatures = true)
// Automated model selection
val (pred, raw, prob) = BinaryClassificationModelSelector().setInput(survived, checkedFeatures).getOutput()
val model = new OpWorkflow().setInputDataset(passengersData).setResultFeatures(pred).train()
println("Model summary:\n" + model.summaryPretty())
}
}
Когда я пытаюсь запустить его, я получаю эту ошибку:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/LogManager
at com.salesforce.hw.titanic.OpTitanicMini$.main(OpTitanicMini.scala:72)
at com.salesforce.hw.titanic.OpTitanicMini.main(OpTitanicMini.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.LogManager
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 2 more
Я попытался взглянуть на эту проблему и нашел сообщение в блоге , я попытался сказать, что написано в этом сообщении:
Мой файл log4j.properties выглядит так:
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.Remoting=ERROR
log4j.logger.org.eclipse.jetty=ERROR
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
# Change this to set Hadoop log level
log4j.logger.org.apache.hadoop=ERROR
# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN
# Change this to set Spark log level
log4j.logger.org.apache.spark=ERROR
# Breeze
log4j.logger.breeze.optimize=FATAL
# BLAS & LAPACK
log4j.logger.com.github.fommil.netlib=ERROR
# TransmogrifAI logging
log4j.logger.com.salesforce.op=INFO
log4j.logger.com.salesforce.op.utils.spark.OpSparkListener=OFF
# Helloworld logging
log4j.logger.com.salesforce.hw=INFO
Я попытался выполнить шаги, упомянутые в сообщении в блоге, но все еще сталкиваюсь с той же проблемой. Как мне решить эту проблему?