Мы используем Spark 2.2.0 и Mesos для управления ресурсами. Недавно для Spark, представивших приложения в режиме кластера, мы начали видеть REASON_COMMAND_EXECUTOR_FAILED в журналах диспетчера.
Также после просмотра этого журнала мы заметили, что задания очень часто ставятся в очередь, хотя в Mesos достаточно ресурсов.
Вот логи из приложения Диспетчера
18/04/25 10:11:05 INFO MesosClusterScheduler: Reviving Offers.
18/04/25 10:24:23 INFO MesosClusterScheduler: Reviving Offers.
18/04/25 10:24:56 INFO MesosClusterScheduler: Reviving Offers.
18/04/25 10:26:25 INFO MesosClusterScheduler: Reviving Offers.
18/04/25 10:36:12 INFO MesosClusterScheduler: Reviving Offers.
18/04/25 10:37:12 INFO MesosClusterScheduler: Reviving Offers.
18/04/25 10:39:25 INFO MesosClusterScheduler: Reviving Offers.
18/04/25 11:02:45 INFO MesosClusterScheduler: Reviving Offers.
18/04/25 11:39:50 INFO MesosClusterScheduler: Reviving Offers.
18/04/25 11:43:44 INFO MesosClusterScheduler: Reviving Offers.
18/04/25 14:20:30 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425110245-0052 state=TASK_RUNNING message= reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:20:32 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425114344-0054 state=TASK_RUNNING message= reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:20:33 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425113950-0053 state=TASK_RUNNING message= reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:20:36 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425102625-0048 state=TASK_RUNNING message= reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:20:36 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425103612-0049 state=TASK_RUNNING message= reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:20:38 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425103925-0051 state=TASK_RUNNING message= reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:20:42 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425103712-0050 state=TASK_RUNNING message= reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:21:16 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425110245-0052 state=TASK_FINISHED message=Container exited with status 0 reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:21:19 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425113950-0053 state=TASK_FINISHED message=Container exited with status 0 reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:21:21 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425114344-0054 state=TASK_FINISHED message=Container exited with status 0 reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:21:45 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425102625-0048 state=TASK_FINISHED message=Container exited with status 0 reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:21:46 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425103612-0049 state=TASK_FINISHED message=Container exited with status 0 reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:21:47 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425103712-0050 state=TASK_FINISHED message=Container exited with status 0 reason=REASON_COMMAND_EXECUTOR_FAILED
18/04/25 14:24:12 INFO MesosClusterScheduler: Received status update: taskId=driver-20180425103925-0051 state=TASK_FINISHED message=Container exited with status 0 reason=REASON_COMMAND_EXECUTOR_FAILED
Я не смог найти причину этого сообщения. Хотя работа удалась - эта ошибка приходит. Но нас беспокоит то, что рабочие места ставятся в очередь.