В длительных вычислительных тестах я сталкиваюсь с периодическими ошибками торнадо.В приведенном ниже примере кажется, что мои узлы в .223 и .198 потеряли свои потоки торнадо, что приводит к многочисленным зависимым не найденным ошибкам.Насколько я могу судить, у меня не было никаких проблем с сетью, которые могли бы привести к разрыву соединений.
Однако кажется, что оба узла восстанавливают свои соединения и продолжают обрабатывать данные.Я также не сразу вижу влияние этих периодических неудач на мои результаты.
Tornado версии 6.0.2 работает на всех узлах.
У меня есть пара вопросов:
Мне кажется, я видел некоторые старые комментарии github, которыеПредполагаемое понижение скорости торнадо может решить эту проблему стабильности потока.Это точно?
Может ли планировщик Dask компенсировать эти потерянные проблемы с зависимостями, или, скорее всего, я потерял некоторые данные здесь?Из сообщений мне не ясно, разрешил ли планировщик проблемы после сообщений "запрашивающего планировщика".
2019-06-13 06:12:04,844 - INFO - 192.168.0.198 - distributed.worker - ERROR - failed during get data with tcp://192.168.0.198:33643 -> tcp://192.168.0.223:35739
2019-06-13 06:12:04,857 - INFO - 192.168.0.198 - Traceback (most recent call last):
2019-06-13 06:12:04,868 - INFO - 192.168.0.198 - File "/usr/local/lib/python3.7/dist-packages/distributed/comm/tcp.py", line 194, in read
2019-06-13 06:12:04,875 - INFO - 192.168.0.198 - n_frames = yield stream.read_bytes(8)
2019-06-13 06:12:04,881 - INFO - 192.168.0.198 - File "/home/dan/.local/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
2019-06-13 06:12:04,889 - INFO - 192.168.0.198 - value = future.result()
2019-06-13 06:12:04,899 - INFO - 192.168.0.198 - tornado.iostream.StreamClosedError: Stream is closed
2019-06-13 06:12:04,906 - INFO - 192.168.0.198 -
2019-06-13 06:12:04,911 - INFO - 192.168.0.198 - During handling of the above exception, another exception occurred:
2019-06-13 06:12:04,918 - INFO - 192.168.0.198 -
2019-06-13 06:12:04,925 - INFO - 192.168.0.198 - Traceback (most recent call last):
2019-06-13 06:12:04,934 - INFO - 192.168.0.198 - File "/usr/local/lib/python3.7/dist-packages/distributed/worker.py", line 1114, in get_data
2019-06-13 06:12:04,941 - INFO - 192.168.0.198 - response = yield comm.read(deserializers=serializers)
2019-06-13 06:12:04,949 - INFO - 192.168.0.198 - File "/home/dan/.local/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
2019-06-13 06:12:04,958 - INFO - 192.168.0.198 - value = future.result()
2019-06-13 06:12:04,964 - INFO - 192.168.0.198 - File "/home/dan/.local/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
2019-06-13 06:12:04,972 - INFO - 192.168.0.198 - yielded = self.gen.throw(*exc_info) # type: ignore
2019-06-13 06:12:04,983 - INFO - 192.168.0.198 - File "/usr/local/lib/python3.7/dist-packages/distributed/comm/tcp.py", line 214, in read
2019-06-13 06:12:04,989 - INFO - 192.168.0.198 - convert_stream_closed_error(self, e)
2019-06-13 06:12:04,996 - INFO - 192.168.0.198 - File "/usr/local/lib/python3.7/dist-packages/distributed/comm/tcp.py", line 139, in convert_stream_closed_error
2019-06-13 06:12:05,015 - INFO - 192.168.0.198 - raise CommClosedError("in %s: %s: %s" % (obj, exc.__class__.__name__, exc))
2019-06-13 06:12:05,024 - INFO - 192.168.0.198 - distributed.comm.core.CommClosedError: in <closed TCP>: TimeoutError: [Errno 110] Connection timed out
2019-06-13 06:12:05,034 - INFO - 192.168.0.198 - distributed.core - INFO - Lost connection to 'tcp://192.168.0.223:39930': in <closed TCP>: TimeoutError: [Errno 110] Connection timed out
2019-06-13 06:12:13,551 - INFO - 192.168.0.223 - distributed.worker - ERROR - Worker stream died during communication: tcp://192.168.0.198:33643
2019-06-13 06:12:13,558 - INFO - 192.168.0.223 - Traceback (most recent call last):
2019-06-13 06:12:13,569 - INFO - 192.168.0.223 - File "/usr/local/lib/python3.7/dist-packages/distributed/comm/tcp.py", line 204, in read
2019-06-13 06:12:13,580 - INFO - 192.168.0.223 - n = yield stream.read_into(frame)
2019-06-13 06:12:13,589 - INFO - 192.168.0.223 - File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 729, in run
2019-06-13 06:12:13,596 - INFO - 192.168.0.223 - value = future.result()
2019-06-13 06:12:13,603 - INFO - 192.168.0.223 -
tornado.iostream.StreamClosedError: Stream is closed
2019-06-13 06:12:13,612 - INFO - 192.168.0.223 -
2019-06-13 06:12:13,620 - INFO - 192.168.0.223 - During handling of the above exception, another exception occurred:
2019-06-13 06:12:13,629 - INFO - 192.168.0.223 -
2019-06-13 06:12:13,637 - INFO - 192.168.0.223 - Traceback (most recent call last):
2019-06-13 06:12:13,646 - INFO - 192.168.0.223 - File "/usr/local/lib/python3.7/dist-packages/distributed/worker.py", line 1827, in gather_dep
2019-06-13 06:12:13,656 - INFO - 192.168.0.223 - self.rpc, deps, worker, who=self.address
2019-06-13 06:12:13,668 - INFO - 192.168.0.223 - File
"/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 729, in run
2019-06-13 06:12:13,675 - INFO - 192.168.0.223 - value = future.result()
2019-06-13 06:12:13,682 - INFO - 192.168.0.223 - File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 736, in run
2019-06-13 06:12:13,692 - INFO - 192.168.0.223 - yielded = self.gen.throw(*exc_info) # type: ignore
2019-06-13 06:12:13,698 - INFO - 192.168.0.223 - File "/usr/local/lib/python3.7/dist-packages/distributed/worker.py", line 2975, in get_data_from_worker
2019-06-13 06:12:13,707 - INFO - 192.168.0.223 - max_connections=max_connections,
2019-06-13 06:12:13,717 - INFO - 192.168.0.223 - File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 729, in run
2019-06-13 06:12:13,726 - INFO - 192.168.0.223 - value = future.result()
2019-06-13 06:12:13,733 - INFO - 192.168.0.223 - File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 736, in run
2019-06-13 06:12:13,742 - INFO - 192.168.0.223 - yielded = self.gen.throw(*exc_info) # type: ignore
2019-06-13 06:12:13,747 - INFO - 192.168.0.223 - File "/usr/local/lib/python3.7/dist-packages/distributed/core.py", line 533, in send_recv
2019-06-13 06:12:13,756 - INFO - 192.168.0.223 - response = yield comm.read(deserializers=deserializers)
2019-06-13 06:12:13,768 - INFO - 192.168.0.223 - File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 729, in run
2019-06-13 06:12:13,774 - INFO - 192.168.0.223 - value = future.result()
2019-06-13 06:12:13,780 - INFO - 192.168.0.223 - File
"/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 736, in run
2019-06-13 06:12:13,787 - INFO - 192.168.0.223 - yielded = self.gen.throw(*exc_info) # type: ignore
2019-06-13 06:12:13,797 - INFO - 192.168.0.223 - File "/usr/local/lib/python3.7/dist-packages/distributed/comm/tcp.py", line 214, in read
2019-06-13 06:12:13,808 - INFO - 192.168.0.223 - convert_stream_closed_error(self, e)
2019-06-13 06:12:13,817 - INFO - 192.168.0.223 - File "/usr/local/lib/python3.7/dist-packages/distributed/comm/tcp.py", line 139, in convert_stream_closed_error
2019-06-13 06:12:13,826 - INFO - 192.168.0.223 - raise CommClosedError("in %s: %s: %s" % (obj, exc.__class__.__name__, exc))
2019-06-13 06:12:13,835 - INFO - 192.168.0.223 - distributed.comm.core.CommClosedError: in <closed TCP>: ConnectionResetError: [Errno 104] Connection reset by peer
2019-06-13 06:12:13,842 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-b0b4738d264f6945f9c834a3eb3034f2', 86)
2019-06-13 06:12:13,851 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-064b03b4644b5196efffcd76a028ce78', 43)
2019-06-13 06:12:13,862 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-201b152cd6c6dbd35b7c09c271f00ef3', 88)
2019-06-13 06:12:13,873 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-e6da691f1332a148e3e2abf33d036106', 12)
2019-06-13 06:12:13,883 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-bfe19a2cd99841edf872025d33204dbc', 43) 0 . Asking scheduler
2019-06-13 06:12:13,890 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-f3f97c94e9d195250bed969135e41ee9', 86) 0 . Asking scheduler
2019-06-13 06:12:13,895 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-2d8b882a55ba0db37bd7959314810fc0', 13)
2019-06-13 06:12:13,904 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-1527be94e500bd3cb01d884223f52210', 40)
2019-06-13 06:12:13,914 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('mean-737e7a910e1de114c8b4109983dbebeb', 88) 0 . Asking scheduler
2019-06-13 06:12:13,923 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-6fc3f47fe126860c91c1a0c601e483ee', 11) 0 . Asking scheduler
2019-06-13 06:12:13,930 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('mean-1caf67e6c53a8a35be6b367dd2687246', 109)
2019-06-13 06:12:13,937 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-8e0d5626cfc4e3a0971047a0d7f7fdd4', 47)
2019-06-13 06:12:13,948 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-e08fe4aba5a8b99a2c3f6690dc258aec', 142)
2019-06-13 06:12:13,961 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-fe826f8a391ff3cc519f65f200b13282', 79)
2019-06-13 06:12:13,968 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('mean-223748d53a0e3a60721a82115f08178f', 15)
2019-06-13 06:12:13,977 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('mean-8af791e9c72b08f2fc764c94b10e6d25', 46)
2019-06-13 06:12:13,987 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-40595528939aa11ca3505d750f921423', 12) 0 . Asking scheduler
2019-06-13 06:12:13,993 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('mean-423629b8390850c12815796d088a7cd6', 40) 0 . Asking scheduler
2019-06-13 06:12:14,000 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-cb2280fb80816498cd2429d3b60cbc09', 108) 0 . Asking scheduler
2019-06-13 06:12:14,005 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('mean-f425ebcb6fb41c8c95ac7d7ad1209ac1', 47) 0 . Asking scheduler
2019-06-13 06:12:14,016 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-6020fd4bf2d56d763924528fa46be27c', 141) 0 . Asking scheduler
2019-06-13 06:12:14,026 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-212e79d68bebc52dc86551c484598d29', 79) 0 . Asking scheduler
2019-06-13 06:12:14,033 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-977fcdbac2d640e2f242a88537140a4c', 14) 0 . Asking scheduler
2019-06-13 06:12:14,040 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-012780092f000510cbe3d4a9ed813194', 46) 0 . Asking scheduler
2019-06-13 06:12:14,050 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('mean-27e0d68ac0c66864af593b534f232d93', 29)
2019-06-13 06:12:14,058 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('mean-f425ebcb6fb41c8c95ac7d7ad1209ac1', 46)
2019-06-13 06:12:14,065 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-b57b008a86d935f61c54c000a8535d62', 25)
2019-06-13 06:12:14,077 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-9c7b905830729af7395a9778e6b807f5', 112)
2019-06-13 06:12:14,086 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-c3b86359854b7f86bb2111890487f427', 157)
2019-06-13 06:12:14,092 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('mean-737e7a910e1de114c8b4109983dbebeb', 211)
2019-06-13 06:12:14,101 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-assign-1c2bd60c87f4fe74b9c54c364bb3fd8f', 212)
2019-06-13 06:12:14,110 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-4af340f090ff1e780520c3618c798634', 28) 0 . Asking scheduler
2019-06-13 06:12:14,119 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-a7950c21569bb1d9533655f871321b1a', 46) 0 . Asking scheduler
2019-06-13 06:12:14,126 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('mean-be8bb79a4adf62d69f39d194e25b2df6', 25) 0 . Asking scheduler
2019-06-13 06:12:14,134 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('mean-1caf67e6c53a8a35be6b367dd2687246', 112) 0 . Asking scheduler
2019-06-13 06:12:14,140 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-b8cbd7837d50c10aba44671d8f767303', 156) 0 . Asking scheduler
2019-06-13 06:12:14,147 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-eb21c3c486ecd0d5758d63ded2910773', 210) 0 . Asking scheduler
2019-06-13 06:12:14,153 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-25d3df1bf96f66bc1dc5967bbfbc24a9', 135)
2019-06-13 06:12:14,158 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-416a20249223e365842fc7386a6db4be', 211) 0 . Asking scheduler
2019-06-13 06:12:14,164 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('mean-d321ea9f08c74d5eb64561e67e869add', 36)
2019-06-13 06:12:14,170 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-78512a0b254b482f8ca8a615b78c05b4', 178)
2019-06-13 06:12:14,176 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('mean-21829096928db207e23145630b92b0e2', 56)
2019-06-13 06:12:14,180 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-bee6654dce2c9e3250bef45789acb7c1', 159)
2019-06-13 06:12:14,186 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('assign-2855ba1fb595dd6ecbfe09420dbfddba', 135) 0 . Asking scheduler
2019-06-13 06:12:14,193 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-8d5b96e15874d9dda0f69b82f1a1f435', 35) 0 . Asking scheduler
2019-06-13 06:12:14,199 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('assign-eb2d64ec455669feda959a7c21802d93', 41)
2019-06-13 06:12:14,206 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('mean-5044be10738d686aafe2d9e7aeeebd2c', 178) 0 . Asking scheduler
2019-06-13 06:12:14,214 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-ed45cd64a0eaf787115fae07d8d8cf12', 55) 0 . Asking scheduler
2019-06-13 06:12:14,223 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-b57b008a86d935f61c54c000a8535d62', 50)
2019-06-13 06:12:14,229 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('assign-eb2d64ec455669feda959a7c21802d93', 85)
2019-06-13 06:12:14,236 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-bda4235b17d00f3527990e4592829047', 159) 0 . Asking scheduler
2019-06-13 06:12:14,241 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-1a3ff2c2b415f4bc6705d501934ab112', 98)
2019-06-13 06:12:14,246 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('mean-aef54c920fa11d66ec603c6525875d75', 97)
2019-06-13 06:12:14,255 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-b0f2100a7deecb80731aa81ea982c54f', 95)
2019-06-13 06:12:14,260 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('diff-c8a8df3066ce89ef0ac06a0c23afd3c8', 41) 0 . Asking scheduler
2019-06-13 06:12:14,267 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('mean-be8bb79a4adf62d69f39d194e25b2df6', 50) 0 . Asking scheduler
2019-06-13 06:12:14,274 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('diff-c8a8df3066ce89ef0ac06a0c23afd3c8', 85) 0 . Asking scheduler
2019-06-13 06:12:14,280 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-85d008658ebdbd81c0140221baf01ad1', 192)
2019-06-13 06:12:14,287 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('mean-f8b5f64dfb109317445bfd2d9ecafa8c', 98) 0 . Asking scheduler
2019-06-13 06:12:14,297 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-da56931a1da627cf39daba246bf3413a', 96) 0 . Asking scheduler
2019-06-13 06:12:14,305 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('mean-d321ea9f08c74d5eb64561e67e869add', 95) 0 . Asking scheduler
2019-06-13 06:12:14,312 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('mean-6fa3b45560f75a79fb25329a31564dde', 57)
2019-06-13 06:12:14,322 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('mean-6e31ceb783c3a2ecf26036670608452c', 192) 0 . Asking scheduler
2019-06-13 06:12:14,329 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-e08fe4aba5a8b99a2c3f6690dc258aec', 114)
2019-06-13 06:12:14,337 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-90346e005d3c26277f30fc0263cd8539', 15)
2019-06-13 06:12:14,352 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-4d02a28a999d610c2c06cc97ed002e6d', 57) 0 . Asking scheduler
2019-06-13 06:12:14,361 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-a38092d9f5ac33cd112272a3b164984a', 114) 0 . Asking scheduler
2019-06-13 06:12:14,370 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-d9e53d6dad5decc7777b83481a0ec014', 14) 0 . Asking scheduler
2019-06-13 06:12:14,378 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-8e0d5626cfc4e3a0971047a0d7f7fdd4', 60)
2019-06-13 06:12:14,385 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-064b03b4644b5196efffcd76a028ce78', 202)
2019-06-13 06:12:14,395 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-5533f763aebd36dd2a1bc41b3de990db', 93)
2019-06-13 06:12:14,403 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('assign-82b172e353731751aea9edf240a42f86', 60) 0 . Asking scheduler
2019-06-13 06:12:14,412 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-416a20249223e365842fc7386a6db4be', 201) 0 . Asking scheduler
2019-06-13 06:12:14,418 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-fece67fb58d46fdd434d8f22aca9fd39', 93) 0 . Asking scheduler
2019-06-13 06:12:14,429 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-d42bee87555b1f02ea35bfaf0bb0dbeb', 123)
2019-06-13 06:12:14,436 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-8f64f8f5f0e48a96e283fd2bdd6bff29', 123) 0 . Asking scheduler
2019-06-13 06:12:14,443 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('assign-c88e84256917d4aab1fb8be2f7235732', 41)
2019-06-13 06:12:14,450 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('diff-a36f84d92694c92da1a01f9f014dc311', 41) 0 . Asking scheduler
2019-06-13 06:12:14,459 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('assign-repartition-split-repartition-merge-2ce020c17721b0c63c6abae 37e473892', 497)
2019-06-13 06:12:14,470 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('drop-907f50cb223e8d5307ba3d7ce5dcdc87', 142)
2019-06-13 06:12:14,479 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('diff-cec38660d2a8386b5f930133d9dc945c', 203) 0 . Asking scheduler
2019-06-13 06:12:14,485 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('assign-c22cdabd8a06076694ca31c2f811a25a', 142) 0 . Asking scheduler
2019-06-13 06:12:14,490 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-51dc7180c0dee821337e796114b09faf', 202)
2019-06-13 06:12:14,497 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-33c5e2499cfef883647ca42ce462278e', 202) 0 . Asking scheduler
2019-06-13 06:12:14,502 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-a36f84d92694c92da1a01f9f014dc311', 85)
2019-06-13 06:12:14,509 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('mean-f8b5f64dfb109317445bfd2d9ecafa8c', 161)
2019-06-13 06:12:14,517 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-331d49d3e50a5ebbdd3acc95f6d040c7', 84) 0 . Asking scheduler
2019-06-13 06:12:14,526 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('overlap-prepend-015906c0ce5c098e253086f4a258b57f', 160) 0 . Asking scheduler
2019-06-13 06:12:14,532 - INFO - 192.168.0.223 - distributed.worker - INFO - Can't find dependencies for key ('diff-ab25950d21b5e436710d391157ad5d0f', 112)
2019-06-13 06:12:14,539 - INFO - 192.168.0.223 - distributed.worker - INFO - Dependent not found: ('getitem-21950e2c8f920570ed2977b10db3924b', 112) 0 . Asking scheduler