Иногда возникает ошибка при использовании BashOperator в потоке воздуха - PullRequest
Новая
       5

Иногда возникает ошибка при использовании BashOperator в потоке воздуха

1 голос
/ 30 марта 2020

Я хочу запустить файл Rscript в контейнере docker для использования воздушного потока.

тест пройден успешно

, но когда работает планировщик воздушного потока, это не удалось

I Изменено разрешение каталога на выполнение работ (755) и проверен путь к каталогу.

Проверено двумя способами ниже

1) 

pref_model=BashOperator(
task_id='al_pref_model',
bash_command='sh /home/deploy/work/airflow/dags/fra_rcode_run.sh 2>&1',
dag=dag)

#fra_rcode_run.sh 
sudo docker exec -it bir bash -c "Rscript /home/biservice/fra/fra_Rcode.R"

2) 

al_pref_model=BashOperator(
task_id='al_pref_model',
bash_command='sudo docker exec -it bir bash -c \'Rscript /home/biservice/fra/fra_Rcode.R \'',
dag=dag)

это журналы ошибок 

[2020-03-30 10:41:32,141] {bash_operator.py:91} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_ID=v1_gamebi_rf_model
AIRFLOW_CTX_TASK_ID=al_pref_model
AIRFLOW_CTX_EXECUTION_DATE=2020-03-26T08:26:35.545495+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2020-03-26T08:26:35.545495+00:00
[2020-03-30 10:41:32,141] {bash_operator.py:105} INFO - Temporary script location: /tmp/airflowtmps88dimow/al_pref_modeldnp70z3t
[2020-03-30 10:41:32,142] {bash_operator.py:115} INFO - Running command: sh /home/deploy/work/airflow/dags/fra_rcode_run.sh 2>&1
[2020-03-30 10:41:32,149] {bash_operator.py:124} INFO - Output:
[2020-03-30 10:41:32,184] {bash_operator.py:128} INFO - the input device is not a TTY
[2020-03-30 10:41:32,187] {bash_operator.py:132} INFO - Command exited with return code 1
[2020-03-30 10:41:32,198] {taskinstance.py:1051} ERROR - Bash command failed
Traceback (most recent call last):
  File "/home/deploy/work/myenv/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 926, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/home/deploy/work/myenv/lib/python3.6/site-packages/airflow/operators/bash_operator.py", line 136, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
[2020-03-30 10:41:32,199] {taskinstance.py:1082} INFO - Marking task as FAILED.
[2020-03-30 10:41:32,216] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model Traceback (most recent call last):
[2020-03-30 10:41:32,216] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model   File "/home/deploy/work/myenv/bin/airflow", line 32, in <module>
[2020-03-30 10:41:32,216] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model     args.func(args)
[2020-03-30 10:41:32,216] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model   File "/home/deploy/work/myenv/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2020-03-30 10:41:32,216] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model     return f(*args, **kwargs)
[2020-03-30 10:41:32,216] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model   File "/home/deploy/work/myenv/lib/python3.6/site-packages/airflow/bin/cli.py", line 522, in run
[2020-03-30 10:41:32,216] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model     _run(args, dag, ti)
[2020-03-30 10:41:32,216] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model   File "/home/deploy/work/myenv/lib/python3.6/site-packages/airflow/bin/cli.py", line 440, in _run
[2020-03-30 10:41:32,216] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model     pool=args.pool,
[2020-03-30 10:41:32,216] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model   File "/home/deploy/work/myenv/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
[2020-03-30 10:41:32,217] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model     return func(*args, **kwargs)
[2020-03-30 10:41:32,217] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model   File "/home/deploy/work/myenv/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 926, in _run_raw_task
[2020-03-30 10:41:32,217] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model     result = task_copy.execute(context=context)
[2020-03-30 10:41:32,217] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model   File "/home/deploy/work/myenv/lib/python3.6/site-packages/airflow/operators/bash_operator.py", line 136, in execute
[2020-03-30 10:41:32,217] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model     raise AirflowException("Bash command failed")
[2020-03-30 10:41:32,217] {base_task_runner.py:115} INFO - Job 290: Subtask al_pref_model airflow.exceptions.AirflowException: Bash command failed
[2020-03-30 10:41:34,094] {logging_mixin.py:95} INFO - [[34m2020-03-30 10:41:34,092[0m] {[34mlocal_task_job.py:[0m105} INFO[0m - Task exited with return code 1[0m

Я жду вашей помощи

Спасибо

...