Get RecursionError / KeyError пытается добавить периодическую задачу в ритме сельдерея - PullRequest
Новая
       27

Get RecursionError / KeyError пытается добавить периодическую задачу в ритме сельдерея

0 голосов
/ 28 августа 2018

Я пытаюсь настроить некоторые периодические задачи с битом сельдерея на Джанго. Я использую следующую настройку: 

python3.6
django==1.11.10
celery==4.2.1
kombu==4.2.1

Мой рабочий график запускается в контейнере Docker через docker-compose и запускается следующей командой: 

celery -A proj beat --loglevel=debug -s /src/celerybeat-schedule

Без периодических задач все в порядке, и я получаю следующие строки в консоли: 

[2018-08-28 17:06:05,261: DEBUG/MainProcess] Setting default socket timeout to 30
[2018-08-28 17:06:05,267: INFO/MainProcess] beat: Starting...
[2018-08-28 17:06:05,298: DEBUG/MainProcess] Current schedule:

[2018-08-28 17:06:05,298: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2018-08-28 17:06:05,299: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.

Но когда я пытаюсь добавить периодическое задание (сразу после настройки Celery): 

@app.on_after_configure.connect
def setup_periodic_tasks(sender):
    sender.add_periodic_task(10.0, test.s('hello'), name='add every 10')


@app.task()
def test(arg):
    print(arg)

Я получаю KeyError со следующими строками: 

[2018-08-28 17:11:56,666: WARNING/MainProcess] Traceback (most recent call last):
[2018-08-28 17:11:56,666: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
[2018-08-28 17:11:56,666: WARNING/MainProcess] return obj.__dict__[self.__name__]
[2018-08-28 17:11:56,666: WARNING/MainProcess] KeyError
[2018-08-28 17:11:56,666: WARNING/MainProcess] :
[2018-08-28 17:11:56,667: WARNING/MainProcess] 'tz'

тогда те же ошибки с 'timezone' и 'data'

, а затем МНОГО ЭТОГО (RecursionError): 

[2018-08-28 17:11:57,050: WARNING/MainProcess] File "/usr/local/lib/python3.6/collections/__init__.py", line 987, in __getitem__
[2018-08-28 17:11:57,050: WARNING/MainProcess] if key in self.data:
[2018-08-28 17:11:57,051: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/collections.py", line 126, in __getattr__
[2018-08-28 17:11:57,051: WARNING/MainProcess] return self[k]
[2018-08-28 17:11:57,051: WARNING/MainProcess] File "/usr/local/lib/python3.6/collections/__init__.py", line 987, in __getitem__
[2018-08-28 17:11:57,052: WARNING/MainProcess] if key in self.data:
[2018-08-28 17:11:57,052: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/collections.py", line 126, in __getattr__
[2018-08-28 17:11:57,052: WARNING/MainProcess] return self[k]

Окончательные строки: 

[2018-08-28 17:11:57,115: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
[2018-08-28 17:11:57,116: WARNING/MainProcess] module = imp(module_name, package=package, **kwargs)
[2018-08-28 17:11:57,116: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/loaders/base.py", line 98, in import_from_cwd
[2018-08-28 17:11:57,117: WARNING/MainProcess] package=package,
[2018-08-28 17:11:57,117: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 104, in import_from_cwd
[2018-08-28 17:11:57,117: WARNING/MainProcess] return imp(module, package=package)
[2018-08-28 17:11:57,118: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/loaders/base.py", line 92, in import_module
[2018-08-28 17:11:57,118: WARNING/MainProcess] return importlib.import_module(module, package=package)
[2018-08-28 17:11:57,118: WARNING/MainProcess] File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
[2018-08-28 17:11:57,119: WARNING/MainProcess] return _bootstrap._gcd_import(name[level:], package, level)
[2018-08-28 17:11:57,119: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 994, in _gcd_import
[2018-08-28 17:11:57,121: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 968, in _find_and_load
[2018-08-28 17:11:57,122: WARNING/MainProcess] RecursionError
[2018-08-28 17:11:57,123: WARNING/MainProcess] :
[2018-08-28 17:11:57,123: WARNING/MainProcess] maximum recursion depth exceeded

Не могу найти решение этой проблемы около недели, поэтому я здесь. Я готов предоставить вам столько настроек, сколько сможете.

Спасибо:)

Очень большое обновление, извините, купить, может быть, это может быть важно. Вот что происходит между KeyErrors и запуском рекурсии: 

[2018-08-28 17:46:18,467: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
[2018-08-28 17:46:18,468: WARNING/MainProcess] return obj.__dict__[self.__name__]
[2018-08-28 17:46:18,468: WARNING/MainProcess] KeyError
[2018-08-28 17:46:18,468: WARNING/MainProcess] :
[2018-08-28 17:46:18,468: WARNING/MainProcess] 'data'
[2018-08-28 17:46:18,469: WARNING/MainProcess] During handling of the above exception, another exception occurred:
[2018-08-28 17:46:18,469: WARNING/MainProcess] Traceback (most recent call last):
[2018-08-28 17:46:18,469: WARNING/MainProcess] File "/usr/local/bin/celery", line 11, in <module>
[2018-08-28 17:46:18,470: WARNING/MainProcess] sys.exit(main())
[2018-08-28 17:46:18,470: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/__main__.py", line 16, in main
[2018-08-28 17:46:18,470: WARNING/MainProcess] _main()
[2018-08-28 17:46:18,470: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 322, in main
[2018-08-28 17:46:18,471: WARNING/MainProcess] cmd.execute_from_commandline(argv)
[2018-08-28 17:46:18,471: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
[2018-08-28 17:46:18,473: WARNING/MainProcess] super(CeleryCommand, self).execute_from_commandline(argv)))
[2018-08-28 17:46:18,473: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 275, in execute_from_commandline
[2018-08-28 17:46:18,474: WARNING/MainProcess] return self.handle_argv(self.prog_name, argv[1:])
[2018-08-28 17:46:18,475: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in handle_argv
[2018-08-28 17:46:18,476: WARNING/MainProcess] return self.execute(command, argv)
[2018-08-28 17:46:18,476: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 420, in execute
[2018-08-28 17:46:18,477: WARNING/MainProcess] ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
[2018-08-28 17:46:18,477: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 279, in run_from_argv
[2018-08-28 17:46:18,477: WARNING/MainProcess] sys.argv if argv is None else argv, command)
[2018-08-28 17:46:18,478: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 363, in handle_argv
[2018-08-28 17:46:18,478: WARNING/MainProcess] return self(*args, **options)
[2018-08-28 17:46:18,479: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 238, in __call__
[2018-08-28 17:46:18,479: WARNING/MainProcess] ret = self.run(*args, **kwargs)
[2018-08-28 17:46:18,480: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/bin/beat.py", line 109, in run
[2018-08-28 17:46:18,480: WARNING/MainProcess] return beat().run()
[2018-08-28 17:46:18,481: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/apps/beat.py", line 81, in run
[2018-08-28 17:46:18,481: WARNING/MainProcess] self.start_scheduler()
[2018-08-28 17:46:18,482: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/apps/beat.py", line 109, in start_scheduler
[2018-08-28 17:46:18,482: WARNING/MainProcess] service.start()
[2018-08-28 17:46:18,483: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/beat.py", line 588, in start
[2018-08-28 17:46:18,483: WARNING/MainProcess] interval = self.scheduler.tick()
[2018-08-28 17:46:18,484: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/beat.py", line 293, in tick
[2018-08-28 17:46:18,484: WARNING/MainProcess] self.populate_heap()
[2018-08-28 17:46:18,484: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/beat.py", line 267, in populate_heap
[2018-08-28 17:46:18,485: WARNING/MainProcess] is_due, next_call_delay = entry.is_due()
[2018-08-28 17:46:18,485: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/beat.py", line 132, in is_due
[2018-08-28 17:46:18,485: WARNING/MainProcess] return self.schedule.is_due(self.last_run_at)
[2018-08-28 17:46:18,486: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/schedules.py", line 163, in is_due
[2018-08-28 17:46:18,486: WARNING/MainProcess] last_run_at = self.maybe_make_aware(last_run_at)
[2018-08-28 17:46:18,487: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/schedules.py", line 78, in maybe_make_aware
[2018-08-28 17:46:18,487: WARNING/MainProcess] return maybe_make_aware(dt, self.tz)
[2018-08-28 17:46:18,488: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/kombu/utils/objects.py", line 44, in __get__
[2018-08-28 17:46:18,488: WARNING/MainProcess] value = obj.__dict__[self.__name__] = self.__get(obj)
[2018-08-28 17:46:18,488: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/schedules.py", line 90, in tz
[2018-08-28 17:46:18,489: WARNING/MainProcess] return self.app.timezone
[2018-08-28 17:46:18,489: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/kombu/utils/objects.py", line 44, in __get__
[2018-08-28 17:46:18,489: WARNING/MainProcess] value = obj.__dict__[self.__name__] = self.__get(obj)
[2018-08-28 17:46:18,490: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/app/base.py", line 1260, in timezone
[2018-08-28 17:46:18,491: WARNING/MainProcess] if not conf.timezone:
[2018-08-28 17:46:18,491: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 129, in timezone
[2018-08-28 17:46:18,491: WARNING/MainProcess] return self.first('timezone', 'time_zone')
[2018-08-28 17:46:18,492: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/collections.py", line 447, in first
[2018-08-28 17:46:18,492: WARNING/MainProcess] return first(None, (self.get(key) for key in keys))
[2018-08-28 17:46:18,493: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/functional.py", line 84, in first
[2018-08-28 17:46:18,493: WARNING/MainProcess] None,
[2018-08-28 17:46:18,493: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/functional.py", line 82, in <genexpr>
[2018-08-28 17:46:18,494: WARNING/MainProcess] (v for v in evaluate_promises(it) if (
[2018-08-28 17:46:18,494: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/functional.py", line 69, in evaluate_promises
[2018-08-28 17:46:18,494: WARNING/MainProcess] for value in it:
[2018-08-28 17:46:18,495: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/collections.py", line 447, in <genexpr>
[2018-08-28 17:46:18,495: WARNING/MainProcess] return first(None, (self.get(key) for key in keys))
[2018-08-28 17:46:18,496: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/collections.py", line 452, in get
[2018-08-28 17:46:18,496: WARNING/MainProcess] return self[key]
[2018-08-28 17:46:18,497: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/collections.py", line 429, in __getitem__
[2018-08-28 17:46:18,497: WARNING/MainProcess] return getitem(k)
[2018-08-28 17:46:18,498: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/collections.py", line 278, in __getitem__
[2018-08-28 17:46:18,498: WARNING/MainProcess] return mapping[_key]
[2018-08-28 17:46:18,499: WARNING/MainProcess] File "/usr/local/lib/python3.6/collections/__init__.py", line 987, in __getitem__
[2018-08-28 17:46:18,499: WARNING/MainProcess] if key in self.data:
[2018-08-28 17:46:18,500: WARNING/MainProcess] File "/usr/local/lib/python3.6/site-packages/celery/utils/collections.py", line 126, in __getattr__
[2018-08-28 17:46:18,500: WARNING/MainProcess] return self[k]

Я также пытаюсь настроить tz / timezone для celery_app: 

app.conf.timezone = pytz.timezone('UTC')
app.conf.tz = pytz.timezone('UTC')
app.timezone = pytz.timezone('UTC')
app.tz = pytz.timezone('UTC')

И у меня тоже есть настройки TZ в Django ...

Я также нашел начало журналов, но это действительно последняя часть, которую я имею: (

[2018-08-28 18:01:56,536: DEBUG/MainProcess] Setting default socket timeout to 30
[2018-08-28 18:01:56,537: INFO/MainProcess] beat: Starting...
[2018-08-28 18:01:56,558: DEBUG/MainProcess] Current schedule:
<ScheduleEntry: add every 10 somewhere_api.celery.test('hello') <freq: 10.00 seconds>
[2018-08-28 18:01:56,558: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2018-08-28 18:01:56,572: CRITICAL/MainProcess] beat raised exception <class 'RecursionError'>: RecursionError('maximum recursion depth exceeded',)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'tz'

1 Ответ

0 голосов
/ 29 августа 2018

Я нашел ошибку, и решение , чтобы быть осторожным . В одном из моих унаследованных приложений Django неожиданно появился другой экземпляр Celery, который пытался запустить другие настройки и сломал все.

...