Я пытаюсь запустить SqlSensor локально под docker на компьютере с Windows 10, он работает в Linux, но у меня возникают ошибки при попытке запустить тот же простой DAG локально.
Причина, по которой я пытаюсьустановить это так, чтобы я мог разрабатывать локально и тестировать, чтобы ускорить цикл разработки.
Ошибка из журнала Airflow:
[2018-05-22 08:27:04,929] {{models.py:1428}} INFO - Executing <Task(SqlSensor): limits_test> on 2018-05-21 08:00:00
[2018-05-22 08:27:04,929] {{base_task_runner.py:115}} INFO - Running: ['bash', '-c', 'airflow run sql-sensor-test-dag limits_test 2018-05-21T08:00:00 --job_id 8 --raw -sd DAGS_FOLDER/sql_sensor_test.py']
[2018-05-22 08:27:05,685] {{base_task_runner.py:98}} INFO - Subtask: [2018-05-22 08:27:05,684] {{__init__.py:45}} INFO - Using executor CeleryExecutor
[2018-05-22 08:27:05,749] {{base_task_runner.py:98}} INFO - Subtask: [2018-05-22 08:27:05,749] {{models.py:189}} INFO - Filling up the DagBag from /usr/local/airflow/dags/sql_sensor_test.py
[2018-05-22 08:27:05,791] {{cli.py:374}} INFO - Running on host 0f8e7a60dbab
[2018-05-22 08:27:05,858] {{base_task_runner.py:98}} INFO - Subtask: [2018-05-22 08:27:05,858] {{base_hook.py:80}} INFO - Using connection to: LABCHGVA-SQL295
[2018-05-22 08:27:05,888] {{base_task_runner.py:98}} INFO - Subtask: [2018-05-22 08:27:05,888] {{sensors.py:111}} INFO - Poking: SELECT max(snapshot_id) FROM limits_run
[2018-05-22 08:27:05,896] {{base_task_runner.py:98}} INFO - Subtask: [2018-05-22 08:27:05,896] {{base_hook.py:80}} INFO - Using connection to: LABCHGVA-SQL295
[2018-05-22 08:27:05,924] {{models.py:1595}} ERROR - Connection to the database failed for an unknown reason.
Traceback (most recent call last):
File "pymssql.pyx", line 635, in pymssql.connect (pymssql.c:10734)
File "_mssql.pyx", line 1902, in _mssql.connect (_mssql.c:21821)
File "_mssql.pyx", line 638, in _mssql.MSSQLConnection.__init__ (_mssql.c:6594)
_mssql.MSSQLDriverException: Connection to the database failed for an unknown reason.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1493, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.6/site-packages/airflow/operators/sensors.py", line 78, in execute
while not self.poke(context):
File "/usr/local/lib/python3.6/site-packages/airflow/operators/sensors.py", line 112, in poke
records = hook.get_records(self.sql)
File "/usr/local/lib/python3.6/site-packages/airflow/hooks/dbapi_hook.py", line 106, in get_records
with closing(self.get_conn()) as conn:
File "/usr/local/lib/python3.6/site-packages/airflow/hooks/mssql_hook.py", line 43, in get_conn
port=conn.port)
File "pymssql.pyx", line 644, in pymssql.connect (pymssql.c:10892)
pymssql.InterfaceError: Connection to the database failed for an unknown reason.
Использование этого образа Docker:
FROM puckel/docker-airflow:1.9.0-2
USER root
RUN apt-get update
RUN apt-get install freetds-dev -yqq && \
pip install apache-airflow[mssql]
USER airflow
И следующий простой DAG:
from datetime import timedelta
import airflow
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.sensors import SqlSensor
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'catchup': False,
'start_date': airflow.utils.dates.days_ago(1),
'email': ['myemail@company.com'],
'email_on_failure': True,
'email_on_retry': True,
'retries': 10,
'retry_delay': timedelta(minutes=15),
'sla': timedelta(hours=3)
}
dag = DAG(
'sql-sensor-test-dag',
default_args=default_args,
description='Sensor tests',
schedule_interval='0 8 * * *'
# schedule_interval='@once'
)
with dag:
sql_sensor = SqlSensor(
task_id='limits_test',
conn_id='bpeak_limits_ro',
sql="SELECT max(snapshot_id) FROM limits_run"
)
done = DummyOperator(task_id='done')
sql_sensor >> done