Рабочий Celet Eventlet django .db.utils.DatabaseError в Docker - PullRequest
Новая
       41

Рабочий Celet Eventlet django .db.utils.DatabaseError в Docker

0 голосов
/ 17 марта 2020

У меня есть веб-приложение, которое локально запускается с docker -compose. Он содержит Django, подключенный к PostgreSQL, Celery и RabbitMQ. Каждый раз, когда я запускаю shared_task, я получаю следующую ошибку DatabaseError. 

worker_1  | [2020-03-17 20:03:40,931: ERROR/MainProcess] Signal handler <bound method 

DjangoWorkerFixup.on_task_prerun of <celery.fixups.django.DjangoWorkerFixup object at 0x7fa4dbf3ead0>> raised: DatabaseError("DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 140346062341232 and this is thread id 140345963084624.")
worker_1  | Traceback (most recent call last):
worker_1  |   File "/usr/local/lib/python3.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send
worker_1  |     response = receiver(signal=self, sender=sender, **named)
worker_1  |   File "/usr/local/lib/python3.7/site-packages/celery/fixups/django.py", line 166, in on_task_prerun
worker_1  |     self.close_database()
worker_1  |   File "/usr/local/lib/python3.7/site-packages/celery/fixups/django.py", line 177, in close_database
worker_1  |     return self._close_database()
worker_1  |   File "/usr/local/lib/python3.7/site-packages/celery/fixups/django.py", line 186, in _close_database
worker_1  |     conn.close()
worker_1  |   File "/usr/local/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner
worker_1  |     return func(*args, **kwargs)
worker_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 286, in close
worker_1  |     self.validate_thread_sharing()
worker_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 558, in validate_thread_sharing
worker_1  |     % (self.alias, self._thread_ident, _thread.get_ident())
worker_1  | django.db.utils.DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 140346062341232 and this is thread id 140345963084624.

Файл needs.txt выглядит следующим образом. 

-i https://pypi.org/simple
asgiref==3.2.3
argh==0.26.2
celery==4.4.0
django-cors-headers==3.2.0
django==3.0
djangorestframework-jwt==1.11.0
djangorestframework==3.10.3
django-filter==2.2.0
django-storages==1.9.1
eventlet==0.25.1
gunicorn==19.9.0
pyjwt==1.7.1
pytz==2019.3
pyyaml==5.3
sqlparse==0.3.0
stripe==2.42.0
watchdog==0.10.2

docker -создать файл 

version: '3.7'

services:
    django:
        build: ./
        command: python manage.py runserver 0.0.0.0:8000
        depends_on:
            - db
            - broker
        volumes:
            - ./:/usr/src/app/
        ports:
            - 8000:8000
        env_file:
            - ./.env.dev
    worker:
        build:
            context: .
            dockerfile: Dockerfile.celery
        command: watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- celery worker --app=wranglab_backend.celery --pool=eventlet --concurrency=500 -l info
        volumes:
            - ./:/usr/src/app
        depends_on:
            - broker
        env_file:
            - ./.env.dev
    db:
        image: postgres:12.0-alpine
        volumes:
            - postgres_data:/var/lib/postgresql/data/
            - postgres_data_backups:/backups
        env_file:
            - ./.env.dev.db
    broker:
        image: rabbitmq:latest
        env_file:
            - ./.env.dev
        ports:
            - 5672:5672
volumes:
    postgres_data: {}
    postgres_data_backups: {}

Dockerfile 

FROM python:3.7-alpine

WORKDIR /usr/src/app

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

RUN apk update && apk add libpq
RUN apk add postgresql-dev gcc python3-dev musl-dev

RUN pip install --upgrade pip

RUN apk --no-cache add musl-dev linux-headers g++
RUN pip install numpy && \
    pip install pandas && \
    pip install psycopg2

COPY ./requirements.txt /usr/src/app/requirements.txt
RUN pip install -r requirements.txt

COPY ./entrypoint.sh /usr/src/app/entrypoint.sh

COPY . /usr/src/app/

ENTRYPOINT ["/usr/src/app/entrypoint.sh"]

Dockerfile.celery 

FROM python:3.7-alpine

WORKDIR /usr/src/app

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

RUN apk update && apk add libpq
RUN apk add postgresql-dev gcc python3-dev musl-dev

RUN pip install --upgrade pip

RUN apk --no-cache add musl-dev linux-headers g++
RUN pip install numpy && \
    pip install pandas && \
    pip install psycopg2

COPY ./requirements.txt /usr/src/app/requirements.txt
RUN pip install -r requirements.txt

COPY . /usr/src/app/

Eventlet импортируется в проекты init , в manage.py и в wsgi.py как следует. 

import eventlet
eventlet.monkey_patch()

celery.py 

from __future__ import absolute_import

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'wranglab_backend.settings')
app = Celery("wranglab_backend")
app.config_from_object('django.conf:settings', namespace="CELERY")
app.autodiscover_tasks()

Я пытался выполнить задачу, подобную следующей, в приложениях tasks.py, а также в проектах tasks.py, но получаю ошибка каждый раз. Я надеюсь, что кто-то может помочь мне. 

@shared_task()
def test_print_task(say):
    time.sleep(6)
    print(say)
    time.sleep(3)
    return say
...