Запущенная докерная искра с мезо ломает мезос - PullRequest
Новая
       14

Запущенная докерная искра с мезо ломает мезос

0 голосов
/ 12 марта 2019

Я пытался запустить докер Jupyter Notebook (версия Ubuntu 16.04) на DC / OS, используя Spark на Mesos. Python выводил множество бесполезных сообщений об ошибках, но после подключения к контейнеру и попытки запустить задание spark-submit из контейнера я получил много ошибок о проблемах соединения.

Драйвер Spark не может правильно подключиться к Mesos, похоже, что в большинстве случаев достаточно установить LIBPROCESS_IP. Однако в моем случае его использование полностью приводит к зависанию Mesos.

Это то, что я запускаю внутри контейнера докера: 

export LIBPROCESS_ADVERTISE_IP=172.16.6.105; export SPARK_HOME=spark-2.3.2-bin-hadoop2.6; export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64; export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so; export LIBPROCESS_IP=172.19.0.4; ./spark-2.3.2-bin-hadoop2.6/bin/spark-submit --master mesos://leader.mesos:5050 --class org.apache.spark.examples.SparkPi https://downloads.mesosphere.com/spark/assets/spark-examples_2.11-2.0.1.jar 30

Драйвер Spark зависает на этой части: 

I0312 07:18:13.722151  3764 sched.cpp:232] Version: 1.2.3
I0312 07:18:13.732707  3758 sched.cpp:336] New master detected at master@172.16.6.103:5050
I0312 07:18:13.733749  3758 sched.cpp:352] No credentials provided. Attempting to register without authentication

На этом шаге Месос зависает. Нет никакого доступа к пользовательскому интерфейсу вообще, и постстартовые проверки DCOS показывают ошибки.

Я проверил логи Mesos, и вот что я вижу: 

mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.911664 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.911737 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.911801 32335 master.cpp:3038] Subscribing framework Spark Pi with checkpointing disabled and capabilities [  ]
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.911841 32335 master.cpp:3048] Framework f1731b0f-a064-434f-8f15-2225a57ce2de-0014 (Spark Pi) at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534 already subscribed, resending acknowledgement
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912062 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912149 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912243 32335 master.cpp:3038] Subscribing framework Spark Pi with checkpointing disabled and capabilities [  ]
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912281 32335 master.cpp:3048] Framework f1731b0f-a064-434f-8f15-2225a57ce2de-0014 (Spark Pi) at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534 already subscribed, resending acknowledgement
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912369 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912441 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912499 32335 master.cpp:3038] Subscribing framework Spark Pi with checkpointing disabled and capabilities [  ]
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912534 32335 master.cpp:3048] Framework f1731b0f-a064-434f-8f15-2225a57ce2de-0014 (Spark Pi) at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534 already subscribed, resending acknowledgement
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912771 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912860 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912921 32335 master.cpp:3038] Subscribing framework Spark Pi with checkpointing disabled and capabilities [  ]
mar 12 08:19:36 centos-master-01 mesos-master[32324]: I0312 08:19:31.912957 32335 master.cpp:3048] Framework f1731b0f-a064-434f-8f15-2225a57ce2de-0014 (Spark Pi) at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534 already subscribed, resending acknowledgement

И я тоже иногда вижу это: 

mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.638309   855 master.cpp:3038] Subscribing framework Spark Pi with checkpointing disabled and capabilities [  ]
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.638342   855 master.cpp:3048] Framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139 already subscribed, resending acknowledgement
mar 12 08:35:13 centos-master-01 mesos-master[837]: W0312 08:35:11.638381   855 master.hpp:2322] Master attempted to send message to disconnected framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.638442   855 master.cpp:3038] Subscribing framework Spark Pi with checkpointing disabled and capabilities [  ]
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.638475   855 master.cpp:3048] Framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139 already subscribed, resending acknowledgement
mar 12 08:35:13 centos-master-01 mesos-master[837]: W0312 08:35:11.638514   855 master.hpp:2322] Master attempted to send message to disconnected framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.638572   855 master.cpp:3038] Subscribing framework Spark Pi with checkpointing disabled and capabilities [  ]
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.638605   855 master.cpp:3048] Framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139 already subscribed, resending acknowledgement
mar 12 08:35:13 centos-master-01 mesos-master[837]: W0312 08:35:11.638644   855 master.hpp:2322] Master attempted to send message to disconnected framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.638715   855 master.cpp:3038] Subscribing framework Spark Pi with checkpointing disabled and capabilities [  ]
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.638751   855 master.cpp:3048] Framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139 already subscribed, resending acknowledgement
mar 12 08:35:13 centos-master-01 mesos-master[837]: W0312 08:35:11.638790   855 master.hpp:2322] Master attempted to send message to disconnected framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.638847   855 master.cpp:3038] Subscribing framework Spark Pi with checkpointing disabled and capabilities [  ]
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.638881   855 master.cpp:3048] Framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139 already subscribed, resending acknowledgement
mar 12 08:35:13 centos-master-01 mesos-master[837]: W0312 08:35:11.638921   855 master.hpp:2322] Master attempted to send message to disconnected framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.638978   855 master.cpp:3038] Subscribing framework Spark Pi with checkpointing disabled and capabilities [  ]
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.639011   855 master.cpp:3048] Framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139 already subscribed, resending acknowledgement
mar 12 08:35:13 centos-master-01 mesos-master[837]: W0312 08:35:11.639060   855 master.hpp:2322] Master attempted to send message to disconnected framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.639118   855 master.cpp:3038] Subscribing framework Spark Pi with checkpointing disabled and capabilities [  ]
mar 12 08:35:13 centos-master-01 mesos-master[837]: I0312 08:35:11.639153   855 master.cpp:3048] Framework e40238eb-4b82-4883-be2c-54103b84dea0-0009 (Spark Pi) at scheduler-0ae862ca-bf59-4f80-8d95-9d244c796547@172.16.6.105:35139 already subscribed, resending acknowledgement

Это повторяется все время. Когда я останавливаю драйвер, Mesos все еще не работает и все время выводит эти сообщения: 

mar 12 08:19:49 centos-master-01 mesos-master[32324]: I0312 08:19:41.871507 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:49 centos-master-01 mesos-master[32324]: I0312 08:19:41.871595 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:49 centos-master-01 mesos-master[32324]: I0312 08:19:41.871671 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:49 centos-master-01 mesos-master[32324]: I0312 08:19:41.871744 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:49 centos-master-01 mesos-master[32324]: I0312 08:19:41.871811 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:49 centos-master-01 mesos-master[32324]: I0312 08:19:41.871911 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:49 centos-master-01 mesos-master[32324]: I0312 08:19:41.871979 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:49 centos-master-01 mesos-master[32324]: I0312 08:19:41.872048 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534
mar 12 08:19:49 centos-master-01 mesos-master[32324]: I0312 08:19:41.872140 32335 master.cpp:2958] Received SUBSCRIBE call for framework 'Spark Pi' at scheduler-e6d4dc88-8470-4519-967d-c86c2fee1c39@172.16.6.105:38534

Похоже, что Spark Driver спамит Mesos вызовами Subscribe так быстро, что Mesos не справляется с их обработкой. Пробовал Spark 2.3.2 и 2.4.0, те же результаты.

Я попытался подключить Spark к Spark Mesos Dispatcher, однако получаю обычную ошибку подключения даже при установленных переменных LIBPROCESS: 

E0312 08:01:55.658208  4874 process.cpp:2431] Failed to shutdown socket with fd 279: Transport endpoint is not connected
E0312 08:01:55.658838  4874 process.cpp:2431] Failed to shutdown socket with fd 279: Transport endpoint is not connected
E0312 08:01:55.659353  4874 process.cpp:2431] Failed to shutdown socket with fd 279: Transport endpoint is not connected
E0312 08:01:55.660073  4874 process.cpp:2431] Failed to shutdown socket with fd 279: Transport endpoint is not connected
E0312 08:01:55.660650  4874 process.cpp:2431] Failed to shutdown socket with fd 279: Transport endpoint is not connected
E0312 08:01:55.661358  4874 process.cpp:2431] Failed to shutdown socket with fd 279: Transport endpoint is not connected
E0312 08:01:55.662775  4874 process.cpp:2431] Failed to shutdown socket with fd 279: Transport endpoint is not connected
E0312 08:01:55.663313  4874 process.cpp:2431] Failed to shutdown socket with fd 279: Transport endpoint is not connected
E0312 08:01:55.663964  4874 process.cpp:2431] Failed to shutdown socket with fd 279: Transport endpoint is not connected
E0312 08:01:55.664711  4874 process.cpp:2431] Failed to shutdown socket with fd 279: Transport endpoint is not connected

У кого-нибудь была такая проблема? Как я могу решить это?

1 Ответ

1 голос
/ 12 марта 2019

Я запустил искру на мезо, используя docker compose. У меня уже был образ докера с установленным на нем Mesos и я настроил кластер mesos, например, я определил Master и работников. Затем я написал эти докеры сочинения для хозяина и раба. Они работали без ошибок.

Мастер сочинения: 

version: '3.7'
services:
  master:
   image: ubuntu_mesos_spark
   command: bash -c "sleep 40; /home/mesos-1.7.0/build/bin/mesos-master.sh --ip=150.20.11.136 --work_dir=/var/run/mesos --hostname=x.x.x.x"  ##hostname : IP of the master node
   restart: always
   network_mode: host
   environment:
    - MESOS_HOSTNAME="150.20.11.136"
    - MESOS_QUORUM=1
    - MESOS_LOG_DIR=/var/log/mesos
   expose:
    - 5050
    - 4040
    - 7077
    - 8080
   ports:
    - 5050:5050
    - 4040:4040
    - 7077:7077
    - 8080:8080

Составь раб: 

  version: '3.7'
  services: 
    slave:
      image: ubuntu_mesos_spark
      command: bash -c "sleep 40; /home/mesos-1.7.0/build/bin/mesos-slave.sh 
      --master=150.20.11.136:5050 --work_dir=/var/run/mesos 
      --systemd_enable_support=false"
      restart: always
      privileged: true
      network_mode: host
      environment:
      - MESOS_HOSTNAME="150.20.11.157"
      - MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins #also in Dockerfile
      - MESOS_LOG_DIR=/var/log/mesos
      - MESOS_LOGGING_LEVEL=INFO
      expose:
      - 5051
      ports:
      - 5051:5051

Надеюсь, это было полезно.

...