У меня есть Dag, который потерпел неудачу из-за ошибки памяти, но он не смог пометить задачу как сбойную и, таким образом, остановить DAG. Это журнал:
[2020-01-13 11:38:25,850] {bash_operator.py:128} INFO - MemoryError: Unable to allocate 551. MiB for an array with shape (66, 1093488) and data type float64
[2020-01-13 11:38:26,423] {bash_operator.py:132} INFO - Command exited with return code 1
[2020-01-13 11:38:26,444] {taskinstance.py:1058} ERROR - Bash command failed
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airflow/models/taskinstance.py", line 930, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.7/dist-packages/airflow/operators/bash_operator.py", line 136, in execute
raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
[2020-01-13 11:38:26,447] {taskinstance.py:1089} INFO - Marking task as FAILED.
[2020-01-13 11:38:27,242] {logging_mixin.py:112} INFO - [2020-01-13 11:38:27,242] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982968 s
[2020-01-13 11:38:32,284] {logging_mixin.py:112} INFO - [2020-01-13 11:38:32,283] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982814 s
[2020-01-13 11:38:37,319] {logging_mixin.py:112} INFO - [2020-01-13 11:38:37,319] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.98348 s
[2020-01-13 11:38:42,474] {logging_mixin.py:112} INFO - [2020-01-13 11:38:42,473] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982813 s
[2020-01-13 11:38:47,530] {logging_mixin.py:112} INFO - [2020-01-13 11:38:47,529] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.981145 s
[2020-01-13 11:38:52,587] {logging_mixin.py:112} INFO - [2020-01-13 11:38:52,586] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.983146 s
[2020-01-13 11:38:57,644] {logging_mixin.py:112} INFO - [2020-01-13 11:38:57,643] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982219 s
[2020-01-13 11:39:02,960] {logging_mixin.py:112} INFO - [2020-01-13 11:39:02,960] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.981545 s
[2020-01-13 11:39:08,020] {logging_mixin.py:112} INFO - [2020-01-13 11:39:08,020] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.980606 s
[2020-01-13 11:39:13,075] {logging_mixin.py:112} INFO - [2020-01-13 11:39:13,074] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.983574 s
[2020-01-13 11:39:18,127] {logging_mixin.py:112} INFO - [2020-01-13 11:39:18,127] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.981776 s
[2020-01-13 11:39:23,210] {logging_mixin.py:112} INFO - [2020-01-13 11:39:23,209] {local_task_job.py:124} WARNING - Time since last heartbeat(0.03 s) < heartrate(5.0 s), sleeping for 4.965117 s
[2020-01-13 11:39:28,242] {logging_mixin.py:112} INFO - [2020-01-13 11:39:28,242] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.983009 s
[2020-01-13 11:39:33,304] {logging_mixin.py:112} INFO - [2020-01-13 11:39:33,304] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.981632 s
[2020-01-13 11:39:38,358] {logging_mixin.py:112} INFO - [2020-01-13 11:39:38,358] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.983045 s
[2020-01-13 11:39:43,415] {logging_mixin.py:112} INFO - [2020-01-13 11:39:43,415] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982969 s
[2020-01-13 11:39:48,467] {logging_mixin.py:112} INFO - [2020-01-13 11:39:48,467] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982669 s
[2020-01-13 11:39:53,513] {logging_mixin.py:112} INFO - [2020-01-13 11:39:53,513] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.983276 s
[2020-01-13 11:39:58,562] {logging_mixin.py:112} INFO - [2020-01-13 11:39:58,562] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982814 s
[2020-01-13 11:40:03,620] {logging_mixin.py:112} INFO - [2020-01-13 11:40:03,620] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.981938 s
[2020-01-13 11:40:08,687] {logging_mixin.py:112} INFO - [2020-01-13 11:40:08,687] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.98037 s
[2020-01-13 11:40:13,930] {logging_mixin.py:112} INFO - [2020-01-13 11:40:13,930] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.980573 s
[2020-01-13 11:40:18,982] {logging_mixin.py:112} INFO - [2020-01-13 11:40:18,982] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.983803 s
[2020-01-13 11:40:24,045] {logging_mixin.py:112} INFO - [2020-01-13 11:40:24,045] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.981002 s
[2020-01-13 11:40:29,103] {logging_mixin.py:112} INFO - [2020-01-13 11:40:29,103] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.980058 s
[2020-01-13 11:40:34,175] {logging_mixin.py:112} INFO - [2020-01-13 11:40:34,175] {local_task_job.py:124} WARNING - Time since last heartbeat(0.03 s) < heartrate(5.0 s), sleeping for 4.973125 s
[2020-01-13 11:40:39,218] {logging_mixin.py:112} INFO - [2020-01-13 11:40:39,218] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.98339 s
[2020-01-13 11:40:44,309] {logging_mixin.py:112} INFO - [2020-01-13 11:40:44,309] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982201 s
[2020-01-13 11:40:49,362] {logging_mixin.py:112} INFO - [2020-01-13 11:40:49,362] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.98393 s
[2020-01-13 11:40:54,427] {logging_mixin.py:112} INFO - [2020-01-13 11:40:54,426] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.983137 s
[2020-01-13 11:40:59,479] {logging_mixin.py:112} INFO - [2020-01-13 11:40:59,479] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.984136 s
[2020-01-13 11:41:04,555] {logging_mixin.py:112} INFO - [2020-01-13 11:41:04,555] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.98192 s
[2020-01-13 11:41:09,611] {logging_mixin.py:112} INFO - [2020-01-13 11:41:09,611] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982419 s
[2020-01-13 11:41:14,668] {logging_mixin.py:112} INFO - [2020-01-13 11:41:14,668] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.98168 s
[2020-01-13 11:41:19,720] {logging_mixin.py:112} INFO - [2020-01-13 11:41:19,719] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982789 s
[2020-01-13 11:41:24,778] {logging_mixin.py:112} INFO - [2020-01-13 11:41:24,777] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982629 s
[2020-01-13 11:41:29,834] {logging_mixin.py:112} INFO - [2020-01-13 11:41:29,834] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.98336 s
[2020-01-13 11:41:34,888] {logging_mixin.py:112} INFO - [2020-01-13 11:41:34,887] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.98313 s
[2020-01-13 11:41:39,944] {logging_mixin.py:112} INFO - [2020-01-13 11:41:39,943] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.983736 s
[2020-01-13 11:41:45,001] {logging_mixin.py:112} INFO - [2020-01-13 11:41:45,000] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982984 s
[2020-01-13 11:41:50,093] {logging_mixin.py:112} INFO - [2020-01-13 11:41:50,093] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.98285 s
[2020-01-13 11:41:55,158] {logging_mixin.py:112} INFO - [2020-01-13 11:41:55,158] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.983758 s
[2020-01-13 11:42:00,215] {logging_mixin.py:112} INFO - [2020-01-13 11:42:00,215] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.983343 s
[2020-01-13 11:42:05,266] {logging_mixin.py:112} INFO - [2020-01-13 11:42:05,266] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982753 s
[2020-01-13 11:42:10,334] {logging_mixin.py:112} INFO - [2020-01-13 11:42:10,334] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.980572 s
[2020-01-13 11:42:15,400] {logging_mixin.py:112} INFO - [2020-01-13 11:42:15,400] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.975424 s
[2020-01-13 11:42:20,454] {logging_mixin.py:112} INFO - [2020-01-13 11:42:20,454] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982312 s
[2020-01-13 11:42:25,534] {logging_mixin.py:112} INFO - [2020-01-13 11:42:25,534] {local_task_job.py:124} WARNING - Time since last heartbeat(0.03 s) < heartrate(5.0 s), sleeping for 4.974131 s
[2020-01-13 11:42:30,559] {logging_mixin.py:112} INFO - [2020-01-13 11:42:30,559] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982918 s
[2020-01-13 11:42:35,638] {logging_mixin.py:112} INFO - [2020-01-13 11:42:35,638] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.977553 s
[2020-01-13 11:42:40,696] {logging_mixin.py:112} INFO - [2020-01-13 11:42:40,696] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.979204 s
[2020-01-13 11:42:45,756] {logging_mixin.py:112} INFO - [2020-01-13 11:42:45,756] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.981878 s
[2020-01-13 11:42:50,809] {logging_mixin.py:112} INFO - [2020-01-13 11:42:50,809] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.983122 s
[2020-01-13 11:42:55,860] {logging_mixin.py:112} INFO - [2020-01-13 11:42:55,860] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982893 s
[2020-01-13 11:43:00,925] {logging_mixin.py:112} INFO - [2020-01-13 11:43:00,924] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.975923 s
[2020-01-13 11:43:05,975] {logging_mixin.py:112} INFO - [2020-01-13 11:43:05,975] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.981564 s
[2020-01-13 11:43:11,044] {logging_mixin.py:112} INFO - [2020-01-13 11:43:11,043] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.978843 s
[2020-01-13 11:43:16,102] {logging_mixin.py:112} INFO - [2020-01-13 11:43:16,102] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.980535 s
[2020-01-13 11:43:21,156] {logging_mixin.py:112} INFO - [2020-01-13 11:43:21,156] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.980601 s
[2020-01-13 11:43:26,207] {logging_mixin.py:112} INFO - [2020-01-13 11:43:26,207] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982849 s
[2020-01-13 11:43:31,263] {logging_mixin.py:112} INFO - [2020-01-13 11:43:31,262] {local_task_job.py:124} WARNING - Time since last heartbeat(0.02 s) < heartrate(5.0 s), sleeping for 4.982943 s
Несколько минут go, по той же причине произошла ошибка, и он был помечен как неудачный. Кажется, что-то мой планировщик иногда не работает должным образом (то есть см. Этот связанный вопрос Поток воздуха складывается в последнем задании DAG после нескольких выполнений ) Как я мог решить эту проблему?
Спасибо