У меня есть API, который запускает задачу сельдерея через result = task.delay(), а затем ждет результата через result.get(timeout=5).В настоящее время я пишу тест производительности, который выполняет эту задачу очень часто.Он отлично работает на моей локальной машине, но демонстрирует странное поведение при выполнении внутри нашей виртуальной машины разработки.Приблизительно после 90-92 выполнений время ожидания result.get(timeout=5) истекает, даже если задача выполняется в течение нескольких миллисекунд.
Кажется, что результат отсутствует в бэкэнде результата.Я использую RabbitMQ в качестве брокера сообщений для обоих направлений:
celery_broker_url = pyamqp://guest@localhost//
celery_result_backend = rpc://
Может кто-нибудь дать мне подсказку о том, как исследовать эту проблему дальше?Можно ли проверить, передан ли результат в бэкэнд результата?В журнале RabbitMQ нет записей:
-- Logs begin at Wed 2019-01-30 16:49:24 UTC, end at Thu 2019-01-31 14:01:46 UTC. --
-- No entries --
Вот полная трассировка стека на случай, если это поможет:
[2019-01-31 13:56:42,313] ERROR in app: Exception on /user/lmhsqs/register [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/celery/backends/async.py", line 255, in _wait_for_pending
on_interval=on_interval):
File "/usr/local/lib/python3.6/dist-packages/celery/backends/async.py", line 54, in drain_events_until
raise socket.timeout()
socket.timeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/dist-packages/connexion/decorators/decorator.py", line 66, in wrapper
response = function(request)
File "/usr/local/lib/python3.6/dist-packages/connexion/decorators/validation.py", line 122, in wrapper
response = function(request)
File "/usr/local/lib/python3.6/dist-packages/connexion/decorators/validation.py", line 293, in wrapper
return function(request)
File "/usr/local/lib/python3.6/dist-packages/connexion/decorators/decorator.py", line 42, in wrapper
response = function(request)
File "/usr/local/lib/python3.6/dist-packages/connexion/decorators/parameter.py", line 219, in wrapper
return function(**kwargs)
File "/mynedata/lib/api/apicalls.py", line 73, in register_user
res_to_return = result.get(timeout=5)
File "/usr/local/lib/python3.6/dist-packages/celery/result.py", line 224, in get
on_message=on_message,
File "/usr/local/lib/python3.6/dist-packages/celery/backends/async.py", line 188, in wait_for_pending
for _ in self._wait_for_pending(result, **kwargs):
File "/usr/local/lib/python3.6/dist-packages/celery/backends/async.py", line 259, in _wait_for_pending
raise TimeoutError('The operation timed out.')
celery.exceptions.TimeoutError: The operation timed out.
127.0.0.1 - - [2019-01-31 13:56:42] "POST /user/lmhsqs/register HTTP/1.1" 500 388 5.050726
[2019-01-31 13:56:47,374] ERROR in app: Exception on /user/lmhsqs/login [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/celery/backends/async.py", line 255, in _wait_for_pending
on_interval=on_interval):
File "/usr/local/lib/python3.6/dist-packages/celery/backends/async.py", line 54, in drain_events_until
raise socket.timeout()
socket.timeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/dist-packages/connexion/decorators/decorator.py", line 66, in wrapper
response = function(request)
File "/usr/local/lib/python3.6/dist-packages/connexion/decorators/validation.py", line 122, in wrapper
response = function(request)
File "/usr/local/lib/python3.6/dist-packages/connexion/decorators/validation.py", line 293, in wrapper
return function(request)
File "/usr/local/lib/python3.6/dist-packages/connexion/decorators/decorator.py", line 42, in wrapper
response = function(request)
File "/usr/local/lib/python3.6/dist-packages/connexion/decorators/parameter.py", line 219, in wrapper
return function(**kwargs)
File "/mynedata/lib/api/apicalls.py", line 123, in login_user
res = result.get(timeout=5)
File "/usr/local/lib/python3.6/dist-packages/celery/result.py", line 224, in get
on_message=on_message,
File "/usr/local/lib/python3.6/dist-packages/celery/backends/async.py", line 188, in wait_for_pending
for _ in self._wait_for_pending(result, **kwargs):
File "/usr/local/lib/python3.6/dist-packages/celery/backends/async.py", line 259, in _wait_for_pending
raise TimeoutError('The operation timed out.')
celery.exceptions.TimeoutError: The operation timed out.