ТЛ; др
Как я могу сериализовать приложение Flask или контекст запроса, или подмножество этого контекста (т. Е. Все, что может быть успешно сериализовано), чтобы я мог получить доступ к этому контексту из другого процесса , а не поток?
Длинная версия
У меня есть некоторые функции, которым требуется доступ к контексту запроса Flask или контексту приложения, которые я хочу запускать в фоновом режиме.
Flask имеет встроенный @copy_current_request_context декоратор, чтобы обернуть функцию в копию контекста запроса, чтобы вы могли запустить ее в другом потоке:
from threading import Thread
from flask import Flask, request, copy_current_request_context
app = Flask(__name__)
@app.route('/')
def index():
request.foo = 'bar'
@copy_current_request_context
def baz():
print(request.foo)
thr = Thread(target=baz)
thr.start()
return 'ok'
Хотя Flask не предоставляет встроенный декоратор для копирования контекста приложения, он предоставляет механизм для этого - решение описано в Access flask.g внутри гринлета
Что я пробовал
Прежде всего, я надеялся решить с Пиклом. Это то, что используется concurrent.futures.ProcessPoolExecutor. К сожалению, Pickle завершается ошибкой из-за присутствия объектов блокировки потоков внутри контекста приложения:
>>> with app.test_request_context('/'):
... pickle.dumps(appctx)
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
TypeError: can't pickle _thread.lock objects
Он также не может копировать оформленные / упакованные функции:
>>> with app.test_request_context('/'):
... bar = copy_current_request_context(pow)
... pickle.dumps(bar)
...
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
_pickle.PicklingError: Can't pickle <function pow at 0x110bee8c8>: it's not the same object as builtins.pow
Далее я попробовал dill. Он проходит два вышеупомянутых теста, но не может сериализовать множество других вещей, которые могут оказаться в контексте. Контексты приложений особенно склонны к тому, чтобы расширения связывались друг с другом: SQLAlchemy - отличный пример. Вот что происходит, когда вы используете SQLAlchemy и пытаетесь сериализовать контекст вашего приложения:
>>> from flask_sqlalchemy import SQLAlchemy
>>> db = SQLAlchemy(app)
>>> with app.test_request_context('/'):
... from flask.globals import _app_ctx_stack
... appctx = _app_ctx_stack.top
... dill.dumps(appctx)
...
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 294, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 287, in dump
pik.dump(obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 816, in save_list
self._batch_appends(obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 843, in _batch_appends
save(tmp[0])
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1386, in save_function
obj.__dict__), obj=obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce
save(args)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 786, in save_tuple
save(element)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
save(element)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1129, in save_cell
pickler.save_reduce(_create_cell, (f,), obj=obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce
save(args)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
save(element)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1330, in save_type
StockPickler.save_global(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 957, in save_global
(obj, module_name, name)) from None
_pickle.PicklingError: Can't pickle <class 'sqlalchemy.orm.session.SignallingSession'>: it's not found as sqlalchemy.orm.session.SignallingSession
Итак: есть ли способ обойти это? Одна мысль, которая у меня была, заключалась в том, что создание нового процесса «как раз вовремя», то есть непосредственно перед тем, как я хочу запустить фоновое задание, может означать, что мне даже не нужно вообще копировать контекст, он должен уже иметь его, пока форк происходит после того, как контекст настроен. Однако я не знаю, как заставить это работать с ProcessPoolExecutor, где процессы - это долгоживущий пул. Единственная идея, которая у меня есть, - каким-то образом принудительно завершать рабочие процессы в конце каждой задачи, но я уверен, что это приведет к поломке пула.