У меня проблемы с использованием распределенного LocalCluster, и я попытался создать минимальный пример.Но я даже не могу этого сделать (я использую Python 3.6, распределенный 1.23.3, торнадо 5.1.1).Если я создаю файл Python test.py с содержанием:
from distributed import LocalCluster
cluster = LocalCluster()
и выполняю файл с:
python test.py
, я получаю очень длинное сообщение об ошибке:
...
distributed.nanny - WARNING - Worker process 841 was killed by unknown signal
distributed.nanny - WARNING - Restarting worker
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/user/venv/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/user/venv/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/user/venv/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
Traceback (most recent call last):
File "test.py", line 7, in <module>
cluster = LocalCluster(n_workers=2)
File "/home/user/venv/lib/python3.6/site-packages/distributed/deploy/local.py", line 141, in __init__
self.start(ip=ip, n_workers=n_workers)
File "/home/user/venv/lib/python3.6/site-packages/distributed/deploy/local.py", line 171, in start
self.sync(self._start, **kwargs)
File "/home/user/venv/lib/python3.6/site-packages/distributed/deploy/local.py", line 164, in sync
return sync(self.loop, func, *args, **kwargs)
File "/home/user/venv/lib/python3.6/site-packages/distributed/utils.py", line 277, in sync
six.reraise(*error[0])
File "/home/user/venv/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/home/user/venv/lib/python3.6/site-packages/distributed/utils.py", line 262, in f
result[0] = yield future
File "/home/user/venv/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/home/user/venv/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/home/user/venv/lib/python3.6/site-packages/distributed/deploy/local.py", line 191, in _start
yield [self._start_worker(**self.worker_kwargs) for i in range(n_workers)]
File "/home/user/venv/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/home/user/venv/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/user/venv/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/user/venv/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-10, started daemon)>
distributed.nanny - WARNING - Worker process 849 was killed by unknown signal
/usr/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 12 leaked semaphores to clean up at shutdown
len(cache))
Exception ignored in: <bound method LocalCluster.__del__ of LocalCluster('tcp://127.0.0.1:38903', workers=0, ncores=0)>
Traceback (most recent call last):
File "/home/user/venv/lib/python3.6/site-packages/distributed/deploy/local.py", line 340, in __del__
File "/home/user/venv/lib/python3.6/site-packages/distributed/deploy/local.py", line 291, in close
File "/home/user/venv/lib/python3.6/site-packages/distributed/utils.py", line 427, in run_sync
File "/home/user/venv/lib/python3.6/site-packages/distributed/utils.py", line 272, in sync
tornado.util.TimeoutError: timed out after 20 s.
Странно то, что если я просто открою оболочку Python и введу в две строки, все будет работать, как и ожидалось.
Обновление:
, поместив все в функциюработает:
from distributed import LocalCluster
def main():
cluster = LocalCluster()
if __name__ == "__main__":
main()
все еще не понимает, почему первая версия дает сбой.