Я пытаюсь развернуть на эластичном бобовом стебле проект django, который использует периодические задачи с сельдереем, используя SQS.
Я более или менее следовал приведенным здесь инструкциям:
Как запустить работника сельдерея с приложением Django, масштабируемым с помощью AWS Elastic Beanstalk?
При развертывании в eb периодические задачи не выполняются.
Проверка удара сельдереяжурнал, все кажется правильным:
celery beat v4.2.1 (windowlicker) is starting.
__ - ... __ - _
LocalTime -> 2019-01-27 09:48:16
Configuration ->
. broker -> sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
. loader -> celery.loaders.app.AppLoader
. scheduler -> django_celery_beat.schedulers.DatabaseScheduler
. logfile -> [stderr]@%INFO
. maxinterval -> 5.00 seconds (5s)
/opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
[2019-01-27 09:48:44,659: INFO/MainProcess] beat: Starting...
[2019-01-27 09:48:44,660: INFO/MainProcess] Writing entries...
[2019-01-27 09:48:44,809: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2019-01-27 09:48:44,809: INFO/MainProcess] Writing entries...
[2019-01-27 09:48:49,865: INFO/MainProcess] Writing entries...
[2019-01-27 09:49:00,409: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:50:00,050: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:51:00,045: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:51:50,543: INFO/MainProcess] Writing entries...
[2019-01-27 09:52:00,048: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:53:00,045: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
но, похоже, есть проблема с работником сельдерея.Это ошибка, которую я получаю в файле celery-worker.log:
/opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
[2019-01-27 09:49:12,513: INFO/MainProcess] Connected to sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
[2019-01-27 09:49:12,950: WARNING/MainProcess] /opt/python/run/venv/local/lib/python3.6/site-packages/celery/fixups/django.py:200: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-01-27 09:49:12,951: INFO/MainProcess] celery@ip-172-31-23-20 ready.
[2019-01-27 09:49:12,952: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.',)
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 20, in get_client
return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
self.blueprint.start(self)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
return self.obj.start()
File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 317, in start
blueprint.start(self)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 593, in start
c.loop(*c.loop_args())
File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/loops.py", line 91, in asynloop
next(loop)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 291, in create_loop
item()
File "/opt/python/run/venv/local/lib/python3.6/site-packages/vine/promises.py", line 163, in __call__
return self.throw()
File "/opt/python/run/venv/local/lib/python3.6/site-packages/vine/promises.py", line 160, in __call__
retval = fun(*final_args, **final_kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 316, in _schedule_queue
queue, callback=promise(self._loop1, (queue,)),
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 332, in _get_bulk_async
return self._get_async(queue, maxcount, callback=callback)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 342, in _get_async
qname, count=count, connection=self.asynsqs,
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 436, in asynsqs
region=self.region
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/sqs/connection.py", line 27, in __init__
**kwargs
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/connection.py", line 178, in __init__
**http_client_params)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/connection.py", line 151, in __init__
self._httpclient = http_client or get_client()
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 22, in get_client
client = hub._current_http_client = Client(hub, **kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 13, in Client
return CurlClient(hub, **kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/curl.py", line 43, in __init__
raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.
-------------- celery@ip-172-31-23-20 v4.2.1 (windowlicker)
---- **** -----
--- * *** * -- Linux-4.14.88-72.76.amzn1.x86_64-x86_64-with-glibc2.3.4 2019-01-27 09:49:12
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: raiseflags:0x7fbf06309f28
- ** ---------- .> transport: sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. check_users_inboxes
. periodic_send
. raiseflags.celery.debug_task
. scrapeURLs
. spreadsheets_update
. sum_two_numbers
Я искал ответы, и у некоторых людей, похоже, возникла та же проблема.Я пытался развернуть разные версии сельдерея и комбу, но не могу решить.
Это мой файл require.txt:
amqp==2.4.0
anyjson==0.3.3
beautifulsoup4==4.7.1
billiard==3.5.0.5
boto==2.49.0
bs4==0.0.1
cachetools==3.0.0
celery==4.2.1
certifi==2018.11.29
chardet==3.0.4
Django==2.1.5
django-celery-beat==1.4.0
django-celery-results==1.0.4
django-timezone-field==3.0
google-api-python-client==1.7.7
google-auth==1.6.2
google-auth-httplib2==0.0.3
httplib2==0.12.0
idna==2.8
kombu==4.2.2.post1
oauth2client==4.1.3
psycopg2==2.7.7
pyasn1==0.4.5
pyasn1-modules==0.2.3
python-dateutil==2.7.5
pycurl==7.43.0 --global-option="--with-nss"
python-crontab==2.3.6
pytz==2018.9
requests==2.21.0
rsa==4.0
six==1.12.0
soupsieve==1.7.3
stripe==2.18.1
uritemplate==3.0.0
urllib3==1.24.1
vine==1.2.0
Если я ssh подключусь к экземпляру и наберу "python -c 'import pycurl'", все будет в порядке, проблем нет.
Кто-нибудь знает, как это решить?Буду признателен за любую помощь.
Большое спасибо.