AirLag ETL dag имеет ошибку каждый день
Наша установка воздушного потока использует CeleryExecutor.Конфиги параллелизма были
# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
parallelism = 32
# The number of task instances allowed to run concurrently by the scheduler
dag_concurrency = 16
# Are DAGs paused by default at creation
dags_are_paused_at_creation = True
# When not using pools, tasks are run in the "default pool",
# whose size is guided by this config element
non_pooled_task_slot_count = 128
# The maximum number of active DAG runs per DAG
max_active_runs_per_dag = 16
[celery]
# This section only applies if you are using the CeleryExecutor in
# [core] section above
# The app name that will be used by celery
celery_app_name = airflow.executors.celery_executor
# The concurrency that will be used when starting workers with the
# "airflow worker" command. This defines the number of task instances that
# a worker will take, so size up your workers based on the resources on
# your worker box and the nature of your tasks
celeryd_concurrency = 16
У нас есть dag, который выполняется ежедневно.Он имеет около 21 задачи параллельно, следуя шаблону, который определяет, существуют ли данные в формате hdf, затем спит 10 минут и, наконец, загружается в s3.
В некоторых задачах возникла следующая ошибка:
2019-05-12 00:00:46,209 INFO - Executor reports wh_hdfs_to_s3.check_hdfs_data_dct_order_item_15 execution_date=2019-05-11 04:00:00+00:00 as failed for try_number 1
2019-05-12 00:00:46,212 ERROR - Executor reports task instance <TaskInstance: wh_hdfs_to_s3.check_hdfs_data_dct_order_item_15 2019-05-11 04:00:00+00:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally?
2019-05-12 00:00:46,212 INFO - Filling up the DagBag from /opt/DataLoader/airflow/dags/wh_hdfs_to_s3.py
2019-05-12 00:00:46,425 INFO - Using connection to: id: wh_aws_mysql. Host: db1.prod.coex.us-east-1.aws.owneriq.net, Port: None, Schema: WAREHOUSE_MOST, Login: whuser, Password: XXXXXXXX, extra: {}
2019-05-12 00:00:46,557 ERROR - Executor reports task instance <TaskInstance: wh_hdfs_to_s3.check_hdfs_data_dct_order_item_15 2019-05-11 04:00:00+00:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally?
None
2019-05-12 00:00:46,558 INFO - Marking task as UP_FOR_RETRY
2019-05-12 00:00:46,561 WARNING - section/key [smtp/smtp_user] not found in config
2019-05-12 00:00:46,640 INFO - Sent an alert email to [u'wh-report-admin@owneriq.com']
2019-05-12 00:00:46,679 INFO - Executor reports wh_hdfs_to_s3.check_hdfs_data_tbldimmostlineitem_105 execution_date=2019-05-11 04:00:00+00:00 as failed for try_number 1
2019-05-12 00:00:46,682 ERROR - Executor reports task instance <TaskInstance: wh_hdfs_to_s3.check_hdfs_data_tbldimmostlineitem_105 2019-05-11 04:00:00+00:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally?
2019-05-12 00:00:46,682 INFO - Filling up the DagBag from /opt/DataLoader/airflow/dags/wh_hdfs_to_s3.py
2019-05-12 00:00:46,686 INFO - Using connection to: id: wh_aws_mysql. Host: db1.prod.coex.us-east-1.aws.owneriq.net, Port: None, Schema: WAREHOUSE_MOST, Login: whuser, Password: XXXXXXXX, extra: {}
2019-05-12 00:00:46,822 ERROR - Executor reports task instance <TaskInstance: wh_hdfs_to_s3.check_hdfs_data_tbldimmostlineitem_105 2019-05-11 04:00:00+00:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally?
None
2019-05-12 00:00:46,822 INFO - Marking task as UP_FOR_RETRY
2019-05-12 00:00:46,826 WARNING - section/key [smtp/smtp_user] not found in config
2019-05-12 00:00:46,902 INFO - Sent an alert email to [u'wh-report-admin@owneriq.com']
2019-05-12 00:00:46,918 INFO - Executor reports wh_hdfs_to_s3.check_hdfs_data_tbldimdatasourcetag_135 execution_date=2019-05-11 04:00:00+00:00 as success for try_number 1
2019-05-12 00:00:46,921 INFO - Executor reports wh_hdfs_to_s3.check_hdfs_data_flight_69 execution_date=2019-05-11 04:00:00+00:00 as success for try_number 1
2019-05-12 00:00:46,923 INFO - Executor reports wh_hdfs_to_s3.check_hdfs_data_tbldimariamode_93 execution_date=2019-05-11 04:00:00+00:00 as success for try_number 1
Этот тип ошибки происходит случайно в этих задачах.Когда возникает эта ошибка, состояние экземпляра задачи немедленно устанавливается на up_for_retry, и в рабочих узлах не регистрируется журнал.После некоторых попыток они выполняются и в конечном итоге завершаются.
Эта проблема иногда дает нам большую задержку ETL.Кто-нибудь знает, как решить эту проблему?