Я использую официальную Рекомендацию в качестве теста.
Я сделал эти шаги успешно:
- сервер событий установлен в Docker-контейнере. (Успешно)
- config eventdata, метаданные и все вещи хранятся в mysql. (Успешно)
- обучить и развернуть сервер в другом док-контейнере. (Успешно)
- автономный кластер искры установлен в другом контейнере. (Успешно)
- создать новое приложение. (Успешно)
- импортировать достаточно событийных данных. (Успешно)
Когда я тренируюсь и разворачиваюсь следующим образом, все в порядке, как описано в документации:
pio train
pio deploy
Но когда я использую кластер искр, обучаю и развертываю следующим образом, train в порядке (новая модель сохранена в mysql), но deploy не удалась.
pio train -v engine.json -- --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1
pio deploy -v engine.json --feedback --event-server-ip predictionevent --event-server-port 7070 --accesskey Th7k5gE5yEu9ZdTdM6KdAj0InDrLNJQ1U3qEBy7dbMnYgTxWx5ALNAa2hKjqaHSK -- --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1
Весь журнал:
[INFO] [Runner$] Submission command: /spark/bin/spark-submit --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1 --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/PredictionIO/lib/mysql-connector-java-5.1.46.jar,file:/ebsa/app/cf/target/scala-2.11/template-scala-parallel-recommendation_2.11-0.1-SNAPSHOT.jar,file:/ebsa/app/cf/target/scala-2.11/template-scala-parallel-recommendation-assembly-0.1-SNAPSHOT-deps.jar,file:/PredictionIO/lib/spark/pio-data-elasticsearch-assembly-0.12.1.jar,file:/PredictionIO/lib/spark/pio-data-hbase-assembly-0.12.1.jar,file:/PredictionIO/lib/spark/pio-data-jdbc-assembly-0.12.1.jar,file:/PredictionIO/lib/spark/pio-data-localfs-assembly-0.12.1.jar,file:/PredictionIO/lib/spark/pio-data-s3-assembly-0.12.1.jar,file:/PredictionIO/lib/spark/pio-data-hdfs-assembly-0.12.1.jar --files file:/PredictionIO/conf/log4j.properties --driver-class-path /PredictionIO/conf:/PredictionIO/lib/mysql-connector-java-5.1.46.jar --driver-java-options -Dpio.log.dir=/root file:/PredictionIO/lib/pio-assembly-0.12.1.jar --engine-id org.example.recommendation.RecommendationEngine --engine-version 0387c097c02018fa29109a8990b03d163249be00 --engine-variant file:/ebsa/app/cf/engine.json --verbosity 0 --json-extractor Both --env PIO_ENV_LOADED=1,PIO_STORAGE_SOURCES_MYSQL_PASSWORD=***,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/root/.pio_store,PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://***:3306/predictionio,PIO_HOME=/PredictionIO,PIO_FS_ENGINESDIR=/root/.pio_store/engines,PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=MYSQL,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=MYSQL,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_MYSQL_USERNAME=***,PIO_FS_TMPDIR=/root/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=MYSQL,PIO_CONF_DIR=/PredictionIO/conf
[INFO] [Engine] Extracting datasource params...
[INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be used.
[INFO] [Engine] Datasource params: (,DataSourceParams(cf,None))
[INFO] [Engine] Extracting preparator params...
[INFO] [Engine] Preparator params: (,Empty)
[INFO] [Engine] Extracting serving params...
[INFO] [Engine] Serving params: (,Empty)
[INFO] [log] Logging initialized @6069ms
[INFO] [Server] jetty-9.3.z-SNAPSHOT
[INFO] [Server] Started @6184ms
[WARN] [Utils] Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
[INFO] [AbstractConnector] Started ServerConnector@2b53840a{HTTP/1.1,[http/1.1]}{0.0.0.0:4041}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@422ad5e2{/jobs,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1b3ab4f9{/jobs/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1c8f6c66{/jobs/job,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@151732fb{/jobs/job/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@40ed1802{/stages,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@feb098f{/stages/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@31e739bf{/stages/stage,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7f42e06e{/stages/stage/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2efd2f21{/stages/pool,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@316cda31{/stages/pool/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@17d2b075{/storage,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@310b2b6f{/storage/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6b5ab2f2{/storage/rdd,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6b2dd3df{/storage/rdd/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@73c48264{/environment,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5bcec67e{/environment/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7a2fce12{/executors,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4bb1b96b{/executors/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1f66d8e1{/executors/threadDump,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@3421debd{/executors/threadDump/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@68b7d0ef{/static,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@319642db{/,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@35bfa1bb{/api,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2eda4eeb{/jobs/job/kill,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@309dcdf3{/stages/stage/kill,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@6a2d867d{/metrics/json,null,AVAILABLE,@Spark}
[INFO] [Engine$] EngineWorkflow.train
[INFO] [Engine$] DataSource: org.example.recommendation.DataSource@5db3d57c
[INFO] [Engine$] Preparator: org.example.recommendation.Preparator@395f52ed
[INFO] [Engine$] AlgorithmList: List(org.example.recommendation.ALSAlgorithm@26e0d39c)
[INFO] [Engine$] Data sanity check is on.
[INFO] [Engine$] org.example.recommendation.TrainingData does not support data sanity check. Skipping check.
[INFO] [Engine$] org.example.recommendation.PreparedData does not support data sanity check. Skipping check.
[WARN] [BLAS] Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
[WARN] [BLAS] Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
[INFO] [Engine$] org.apache.spark.mllib.recommendation.ALSModel does not support data sanity check. Skipping check.
[INFO] [Engine$] EngineWorkflow.train completed
[INFO] [Engine] engineInstanceId=0ac606dc-9959-40f8-9f40-d32354ebf221
[WARN] [TaskSetManager] Stage 1403 contains a task of very large size (1217 KB). The maximum recommended task size is 100 KB.
[WARN] [TaskSetManager] Stage 1404 contains a task of very large size (1767 KB). The maximum recommended task size is 100 KB.
[INFO] [CoreWorkflow$] Inserting persistent model
[INFO] [CoreWorkflow$] Updating engine instance
[INFO] [CoreWorkflow$] Training completed successfully.
[INFO] [AbstractConnector] Stopped Spark@2b53840a{HTTP/1.1,[http/1.1]}{0.0.0.0:4041}
[INFO] [Runner$] Submission command: /spark/bin/spark-submit --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1 --class org.apache.predictionio.workflow.CreateServer --jars file:/PredictionIO/lib/mysql-connector-java-5.1.46.jar,file:/ebsa/app/cf/target/scala-2.11/template-scala-parallel-recommendation_2.11-0.1-SNAPSHOT.jar,file:/ebsa/app/cf/target/scala-2.11/template-scala-parallel-recommendation-assembly-0.1-SNAPSHOT-deps.jar,file:/PredictionIO/lib/spark/pio-data-elasticsearch-assembly-0.12.1.jar,file:/PredictionIO/lib/spark/pio-data-hbase-assembly-0.12.1.jar,file:/PredictionIO/lib/spark/pio-data-jdbc-assembly-0.12.1.jar,file:/PredictionIO/lib/spark/pio-data-localfs-assembly-0.12.1.jar,file:/PredictionIO/lib/spark/pio-data-s3-assembly-0.12.1.jar,file:/PredictionIO/lib/spark/pio-data-hdfs-assembly-0.12.1.jar --files file:/PredictionIO/conf/log4j.properties --driver-class-path /PredictionIO/conf:/PredictionIO/lib/mysql-connector-java-5.1.46.jar --driver-java-options -Dpio.log.dir=/root file:/PredictionIO/lib/pio-assembly-0.12.1.jar --engineInstanceId 0ac606dc-9959-40f8-9f40-d32354ebf221 --engine-variant file:/ebsa/app/cf/engine.json --ip 0.0.0.0 --port 8000 --event-server-ip predictionevent --event-server-port 7070 --accesskey Th7k5gE5yEu9ZdTdM6KdAj0InDrLNJQ1U3qEBy7dbMnYgTxWx5ALNAa2hKjqaHSK --feedback --json-extractor Both --env PIO_ENV_LOADED=1,PIO_STORAGE_SOURCES_MYSQL_PASSWORD=***,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/root/.pio_store,PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://***:3306/predictionio,PIO_HOME=/PredictionIO,PIO_FS_ENGINESDIR=/root/.pio_store/engines,PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=MYSQL,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=MYSQL,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_MYSQL_USERNAME=***,PIO_FS_TMPDIR=/root/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=MYSQL,PIO_CONF_DIR=/PredictionIO/conf
[WARN] [WorkflowUtils$] Non-empty parameters supplied to org.example.recommendation.Preparator, but its constructor does not accept any arguments. Stubbing with empty parameters.
[WARN] [WorkflowUtils$] Non-empty parameters supplied to org.example.recommendation.Serving, but its constructor does not accept any arguments. Stubbing with empty parameters.
[INFO] [log] Logging initialized @6953ms
[INFO] [Server] jetty-9.3.z-SNAPSHOT
[INFO] [Server] Started @7086ms
[WARN] [Utils] Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
[INFO] [AbstractConnector] Started ServerConnector@d8ed4d9{HTTP/1.1,[http/1.1]}{0.0.0.0:4041}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@307b5956{/jobs,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@62d50094{/jobs/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@8a644df{/jobs/job,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5a9054e7{/jobs/job/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@15402e55{/stages,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@295b1de5{/stages/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@e7ac843{/stages/stage,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@da15f73{/stages/stage/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@506a8fc2{/stages/pool,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@fc4cf4d{/stages/pool/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@255cef05{/storage,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4e8f6bce{/storage/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4e6427d4{/storage/rdd,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5fca5109{/storage/rdd/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2acbd47f{/environment,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@39004878{/environment/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@785b7109{/executors,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@f0fce80{/executors/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@19ab67fc{/executors/threadDump,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@644558a3{/executors/threadDump/json,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@40fa6a20{/static,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@238b2adb{/,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4bbba0ce{/api,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@3d1e4c06{/jobs/job/kill,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@70f8bf47{/stages/stage/kill,null,AVAILABLE,@Spark}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@558311ee{/metrics/json,null,AVAILABLE,@Spark}
[INFO] [Engine] Using persisted model
[INFO] [Engine] Custom-persisted model detected for algorithm org.example.recommendation.ALSAlgorithm
[ERROR] [OneForOneStrategy] empty collection
Не знаю почему.
Подробнее:
Вместо использования автономного кластера, установленного в другом контейнере док-станции, я запускаю локальный кластер с искрой в том же контейнере с сервером train & deploy (@ user2906838 , упомянутый ), он успешно работает.
Я не понимаю, почему это произошло. Я не могу использовать местную искру, это очень странно.
Подробнее:
папка / tmp, в двух разных ситуациях размер файла различен.
Успешное изображение
ошибка изображения
Подробнее:
Это интересно. Я нахожу данные модели в контейнере искрового рабочего.
изображение рабочего искра