Я уже пробовал другие решения на сайте, и они не работали , также многие старые вопросы, касающиеся устаревшего пакета django -celery-beat. Я использую Django 3 и Celery 4, что, согласно документации последнего, должно означать, что мне больше не нужно ничего для запуска Celery с Django.
Мой проект структурирован итак:
proj/
|- app1/
|- app2/
|- proj/
|- settings.py
|- ...
|- celery.py
|- tasks.py
(я также пытался вставить tasks.py
в одно из приложений.)
celery.py:
import os
from celery import Celery
from celery.schedules import crontab
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "proj.settings")
app = Celery("proj")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
@app.on_after_configure.connect
def setup_periodic_tasks(sender, **_):
from .tasks import debug_task
sender.add_periodic_task(crontab(), debug_task.apply_async())
tasks.py:
from celery import shared_task
@shared_task
def debug_task():
print("Task executed.")
Я пытаюсь запустить проект:
py manage.py runserver
, celery -A proj.celery beat
, celery -A proj.celery worker
.
В Ubuntu я запускал все три команды с отдельными сеансами screen
, чтобы запускать все на одном и том же терминале, если это важно, но я не смог найти никакой приемлемой альтернативы для Windows, поэтому все три выполняются в отдельных терминалах, открытых после того, как команда предыдущего уже выполнялась. Я попытался поместить «beat» и «worker» сразу после «celery», если это имело значение.
Я также попробовал то же самое с помощью функции запуска PyCharm (для сервера Django) и двух отдельных экземпляров терминала для двух других команды. Если есть лучшая процедура, которой нужно следовать, не стесняйтесь предложить ее.
Вывод удара сельдерея:
celery beat v4.4.0 (cliffs) is starting.
__ - ... __ - _
LocalTime -> 2020-03-14 16:48:32
Configuration ->
. broker -> amqp://guest:**@localhost:5672//
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]@%DEBUG
. maxinterval -> 5.00 minutes (300s)
[2020-03-14 16:48:32,042: DEBUG/MainProcess] Setting default socket timeout to 30
[2020-03-14 16:48:32,044: INFO/MainProcess] beat: Starting...
[2020-03-14 16:48:32,288: DEBUG/MainProcess] Current schedule:
<ScheduleEntry: celery.backend_cleanup celery.backend_cleanup() <crontab: 0 4 * * * (m/h/d/dM/MY)>
[2020-03-14 16:48:32,288: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2020-03-14 16:48:32,292: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
Вывод работника сельдерея с ошибкой:
[2020-03-14 16:49:50,319: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2020-03-14 16:49:50,326: DEBUG/MainProcess] | Worker: Building graph...
[2020-03-14 16:49:50,327: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Pool, Autoscaler, Beat, StateDB, Consumer}
[2020-03-14 16:49:50,472: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2020-03-14 16:49:50,473: DEBUG/MainProcess] | Consumer: Building graph...
[2020-03-14 16:49:50,593: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Agent, Events, Heart, Mingle, Tasks, Control, Gossip, event loop}
-------------- celery@MY-LATPTOPS-NAME v4.4.0 (cliffs)
--- ***** -----
-- ******* ---- Windows-10-10.0.18362-SP0 2020-03-14 16:49:50
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: proj:0x3f91ef8
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. proj.tasks.debug_task
[2020-03-14 16:49:50,600: DEBUG/MainProcess] | Worker: Starting Pool
[2020-03-14 16:49:50,614: WARNING/MainProcess] e:\path_to\my_project\venv38\lib\site-packages\billiard\pool.py:1022: UserWarning: Soft timeouts are not supported: on this platform: It does not have the SIGUSR1 signal.
warnings.warn(UserWarning(
[2020-03-14 16:49:50,736: DEBUG/MainProcess] ^-- substep ok
[2020-03-14 16:49:50,737: DEBUG/MainProcess] | Worker: Starting Consumer
[2020-03-14 16:49:50,738: DEBUG/MainProcess] | Consumer: Starting Connection
[2020-03-14 16:49:51,250: DEBUG/MainProcess] recreated blocker with handle 24
[2020-03-14 16:49:51,251: DEBUG/MainProcess] recreated blocker with handle 32
[2020-03-14 16:49:51,253: DEBUG/MainProcess] recreated blocker with handle 20
[2020-03-14 16:49:51,256: DEBUG/MainProcess] recreated blocker with handle 24
[2020-03-14 16:49:51,267: DEBUG/MainProcess] recreated blocker with handle 20
[2020-03-14 16:49:51,268: DEBUG/MainProcess] recreated blocker with handle 24
[2020-03-14 16:49:51,305: INFO/SpawnPoolWorker-1] child process 3316 calling self.run()
[2020-03-14 16:49:51,309: DEBUG/MainProcess] recreated blocker with handle 24
[2020-03-14 16:49:51,310: DEBUG/MainProcess] recreated blocker with handle 28
[2020-03-14 16:49:51,315: INFO/SpawnPoolWorker-3] child process 12052 calling self.run()
[2020-03-14 16:49:51,318: INFO/SpawnPoolWorker-4] child process 13968 calling self.run()
[2020-03-14 16:49:51,360: INFO/SpawnPoolWorker-2] child process 6336 calling self.run()
[2020-03-14 16:49:52,776: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [WinError 10061] No connection could be made because the target machine actively refused it.
Trying again in 2.00 seconds...
Как вы можете видеть, задачи видны с рабочего, но никогда не выполняются.
Я думал, что это как-то связано с брандмауэром, поэтому я повторил попытку после его полного отключения, но ничего не изменилось.
Ошибка явно Windows связана, но мне нужно, чтобы это также работало на экземпляре Ubuntu-18.04, поэтому, если есть что-то, что мне нужно изменить между двумя системами, пожалуйста, не стесняйтесь упоминать об этом.
ОБНОВЛЕНИЕ
Довольно неловко, кажется, я забыл настроить RabbitMQ на этой машине. Выполнение этого избавило от сообщения об ошибке, но задача все еще не выполняется должным образом. Рабочий процесс теперь тикает примерно каждую секунду, и начальный вывод (аналогичный показанному выше) правильно показывает задачу отладки, но его вывод никогда не отображается ни на одном из терминалов (сервер Django, бит сельдерея и работник сельдерея ) даже после нескольких минут активности.
Как я могу подтвердить, выполняется ли задача или нет?
Вот начальный вывод celery beat -A proj.celery -l DEBUG
:
celery beat v4.4.0 (cliffs) is starting.
__ - ... __ - _
LocalTime -> 2020-03-19 10:08:34
Configuration ->
. broker -> amqp://guest:**@localhost:5672//
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]@%DEBUG
. maxinterval -> 5.00 minutes (300s)
[2020-03-19 10:08:34,789: DEBUG/MainProcess] Setting default socket timeout to 30
[2020-03-19 10:08:34,790: INFO/MainProcess] beat: Starting...
[2020-03-19 10:08:35,109: DEBUG/MainProcess] Current schedule:
<ScheduleEntry: celery.backend_cleanup celery.backend_cleanup() <crontab: 0 4 * * * (m/h/d/dM/MY)>
[2020-03-19 10:08:35,109: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2020-03-19 10:08:35,111: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
Это вывод celery worker -A proj.celery -l DEBUG
:
[2020-03-19 10:08:11,288: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2020-03-19 10:08:11,296: DEBUG/MainProcess] | Worker: Building graph...
[2020-03-19 10:08:11,297: DEBUG/MainProcess] | Worker: New boot order: {StateDB, Beat, Timer, Hub, Pool, Autoscaler, Consumer}
[2020-03-19 10:08:11,475: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2020-03-19 10:08:11,475: DEBUG/MainProcess] | Consumer: Building graph...
[2020-03-19 10:08:11,613: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Heart, Mingle, Gossip, Agent, Tasks, Control, event loop}
-------------- celery@MY-LAPTOPS-NAME v4.4.0 (cliffs)
--- ***** -----
-- ******* ---- Windows-10-10.0.18362-SP0 2020-03-19 10:08:11
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: proj:0x4261e50
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. proj.tasks.debug_task
[2020-03-19 10:08:11,646: DEBUG/MainProcess] | Worker: Starting Pool
[2020-03-19 10:08:11,773: DEBUG/MainProcess] ^-- substep ok
[2020-03-19 10:08:11,773: DEBUG/MainProcess] | Worker: Starting Consumer
[2020-03-19 10:08:11,774: DEBUG/MainProcess] | Consumer: Starting Connection
[2020-03-19 10:08:11,842: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'bas
ic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct
_reply_to': True}, 'cluster_name': 'rabbit@MY-LAPTOPS-NAME', 'copyright': 'Copyright (c) 2007-2020 Pivotal Software, Inc.', 'information': 'Licensed under the MPL 1.1. Website:
https://rabbitmq.com', 'platform': 'Erlang/OTP 22.3', 'product': 'RabbitMQ', 'version': '3.8.3'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2020-03-19 10:08:11,850: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2020-03-19 10:08:11,855: DEBUG/MainProcess] ^-- substep ok
[2020-03-19 10:08:11,856: DEBUG/MainProcess] | Consumer: Starting Events
[2020-03-19 10:08:11,911: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'bas
ic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct
_reply_to': True}, 'cluster_name': 'rabbit@MY-LAPTOPS-NAME', 'copyright': 'Copyright (c) 2007-2020 Pivotal Software, Inc.', 'information': 'Licensed under the MPL 1.1. Website:
https://rabbitmq.com', 'platform': 'Erlang/OTP 22.3', 'product': 'RabbitMQ', 'version': '3.8.3'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2020-03-19 10:08:11,924: DEBUG/MainProcess] ^-- substep ok
[2020-03-19 10:08:11,945: DEBUG/MainProcess] | Consumer: Starting Heart
[2020-03-19 10:08:11,946: DEBUG/MainProcess] using channel_id: 1
[2020-03-19 10:08:11,961: DEBUG/MainProcess] Channel open
[2020-03-19 10:08:12,021: DEBUG/MainProcess] ^-- substep ok
[2020-03-19 10:08:12,021: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2020-03-19 10:08:12,029: DEBUG/MainProcess] | Consumer: Starting Mingle
[2020-03-19 10:08:12,030: INFO/MainProcess] mingle: searching for neighbors
[2020-03-19 10:08:12,030: DEBUG/MainProcess] using channel_id: 1
[2020-03-19 10:08:12,074: DEBUG/MainProcess] Channel open
[2020-03-19 10:08:12,141: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'bas
ic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct
_reply_to': True}, 'cluster_name': 'rabbit@MY-LAPTOPS-NAME', 'copyright': 'Copyright (c) 2007-2020 Pivotal Software, Inc.', 'information': 'Licensed under the MPL 1.1. Website:
https://rabbitmq.com', 'platform': 'Erlang/OTP 22.3', 'product': 'RabbitMQ', 'version': '3.8.3'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2020-03-19 10:08:12,148: DEBUG/MainProcess] using channel_id: 1
[2020-03-19 10:08:12,157: DEBUG/MainProcess] Channel open
[2020-03-19 10:08:12,461: DEBUG/MainProcess] recreated blocker with handle 292
[2020-03-19 10:08:12,462: DEBUG/MainProcess] recreated blocker with handle 296
[2020-03-19 10:08:12,497: DEBUG/MainProcess] recreated blocker with handle 20
[2020-03-19 10:08:12,498: DEBUG/MainProcess] recreated blocker with handle 24
[2020-03-19 10:08:12,504: DEBUG/MainProcess] recreated blocker with handle 20
[2020-03-19 10:08:12,505: DEBUG/MainProcess] recreated blocker with handle 24
[2020-03-19 10:08:12,514: INFO/SpawnPoolWorker-1] child process 9188 calling self.run()
[2020-03-19 10:08:12,554: INFO/SpawnPoolWorker-4] child process 9072 calling self.run()
[2020-03-19 10:08:12,562: INFO/SpawnPoolWorker-2] child process 1284 calling self.run()
[2020-03-19 10:08:12,602: DEBUG/MainProcess] recreated blocker with handle 20
[2020-03-19 10:08:12,606: DEBUG/MainProcess] recreated blocker with handle 24
[2020-03-19 10:08:12,665: INFO/SpawnPoolWorker-3] child process 13412 calling self.run()
[2020-03-19 10:08:13,030: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9840000000003783 secs.
[2020-03-19 10:08:13,244: INFO/MainProcess] mingle: all alone
[2020-03-19 10:08:13,245: DEBUG/MainProcess] ^-- substep ok
[2020-03-19 10:08:13,246: DEBUG/MainProcess] | Consumer: Starting Gossip
[2020-03-19 10:08:13,246: DEBUG/MainProcess] using channel_id: 2
[2020-03-19 10:08:13,248: DEBUG/MainProcess] Channel open
[2020-03-19 10:08:13,267: DEBUG/MainProcess] ^-- substep ok
[2020-03-19 10:08:13,267: DEBUG/MainProcess] | Consumer: Starting Tasks
[2020-03-19 10:08:13,486: DEBUG/MainProcess] ^-- substep ok
[2020-03-19 10:08:13,487: DEBUG/MainProcess] | Consumer: Starting Control
[2020-03-19 10:08:13,488: DEBUG/MainProcess] using channel_id: 3
[2020-03-19 10:08:13,495: DEBUG/MainProcess] Channel open
[2020-03-19 10:08:13,518: DEBUG/MainProcess] ^-- substep ok
[2020-03-19 10:08:13,519: DEBUG/MainProcess] | Consumer: Starting event loop
[2020-03-19 10:08:13,522: INFO/MainProcess] celery@MY-LAPTOPS-NAME ready.
[2020-03-19 10:08:13,522: DEBUG/MainProcess] basic.qos: prefetch_count->16
[2020-03-19 10:08:14,023: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2020-03-19 10:08:15,027: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2020-03-19 10:08:16,028: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
...