Я использовал данные сканирования scrapy-redis и redis для запроса dupefilter, после сканирования около 3000k запросов, внезапно, паук не может получить или поместить запрос в ключ dupefilter, независимо от того, перезапускаю ли я spider (не очищает ключ) или redis, он просто перестанет ползать. Я хочу знать, есть ли у Redis какой-то предел?
Я пробовал много раз, и всегда номер запроса составляет около 3000 КБ, он должен быть связан, пожалуйста, помогите мне!
шоу редис-сервера:
[8096] 10 Jan 16:29:52 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:29:52 - 5 clients connected (0 slaves), 28787472 bytes in use
[8096] 10 Jan 16:29:57 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:29:57 - 5 clients connected (0 slaves), 28754120 bytes in use
[8096] 10 Jan 16:30:02 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:02 - 5 clients connected (0 slaves), 28585192 bytes in use
[8096] 10 Jan 16:30:07 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:07 - 5 clients connected (0 slaves), 27473288 bytes in use
[8096] 10 Jan 16:30:12 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:12 - 5 clients connected (0 slaves), 26759176 bytes in use
[8096] 10 Jan 16:30:17 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:17 - 5 clients connected (0 slaves), 28058760 bytes in use
[8096] 10 Jan 16:30:22 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:22 - 5 clients connected (0 slaves), 26286480 bytes in use
[8096] 10 Jan 16:30:24 * 10000 changes in 60 seconds. Saving...
[8096] 10 Jan 16:30:24 * Foregroud saving started by pid 8096
[8096] 10 Jan 16:30:24 * DB saved on disk
[8096] 10 Jan 16:30:24 * Background saving terminated with success
[8096] 10 Jan 16:30:27 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:27 - 5 clients connected (0 slaves), 25357000 bytes in use
[8096] 10 Jan 16:30:32 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:32 - 5 clients connected (0 slaves), 25802800 bytes in use
[8096] 10 Jan 16:30:37 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:37 - 5 clients connected (0 slaves), 23736768 bytes in use
[8096] 10 Jan 16:30:42 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:42 - 5 clients connected (0 slaves), 23954720 bytes in use
[8096] 10 Jan 16:30:47 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:47 - 5 clients connected (0 slaves), 22061280 bytes in use
[8096] 10 Jan 16:30:52 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:52 - 5 clients connected (0 slaves), 21993872 bytes in use
[8096] 10 Jan 16:30:57 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:30:57 - 5 clients connected (0 slaves), 22569824 bytes in use
[8096] 10 Jan 16:31:03 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:03 - 5 clients connected (0 slaves), 21005816 bytes in use
[8096] 10 Jan 16:31:08 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:08 - 5 clients connected (0 slaves), 20970824 bytes in use
[8096] 10 Jan 16:31:13 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:13 - 5 clients connected (0 slaves), 19985472 bytes in use
[8096] 10 Jan 16:31:18 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:18 - 5 clients connected (0 slaves), 19297080 bytes in use
[8096] 10 Jan 16:31:23 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:23 - 5 clients connected (0 slaves), 19221384 bytes in use
[8096] 10 Jan 16:31:28 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:28 - 5 clients connected (0 slaves), 16717416 bytes in use
[8096] 10 Jan 16:31:30 * 10000 changes in 60 seconds. Saving...
[8096] 10 Jan 16:31:30 * Foregroud saving started by pid 8096
[8096] 10 Jan 16:31:31 * DB saved on disk
[8096] 10 Jan 16:31:31 * Background saving terminated with success
[8096] 10 Jan 16:31:33 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:33 - 5 clients connected (0 slaves), 16596904 bytes in use
[8096] 10 Jan 16:31:38 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:38 - 5 clients connected (0 slaves), 17379168 bytes in use
[8096] 10 Jan 16:31:43 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:43 - 5 clients connected (0 slaves), 17069712 bytes in use
[8096] 10 Jan 16:31:48 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:48 - 5 clients connected (0 slaves), 15774568 bytes in use
[8096] 10 Jan 16:31:53 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:53 - 5 clients connected (0 slaves), 14270800 bytes in use
[8096] 10 Jan 16:31:58 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:31:58 - 5 clients connected (0 slaves), 14275816 bytes in use
[8096] 10 Jan 16:32:03 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:32:03 - 5 clients connected (0 slaves), 10735304 bytes in use
[8096] 10 Jan 16:32:08 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:32:08 - 5 clients connected (0 slaves), 10739464 bytes in use
[8096] 10 Jan 16:32:13 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:32:13 - 5 clients connected (0 slaves), 10825088 bytes in use
[8096] 10 Jan 16:32:18 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:32:18 - 5 clients connected (0 slaves), 10661624 bytes in use
[8096] 10 Jan 16:32:23 - Accepted 127.0.0.1:53991
[8096] 10 Jan 16:32:23 - Accepted 127.0.0.1:53992
[8096] 10 Jan 16:32:23 - DB 0: 1 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:32:23 - 7 clients connected (0 slaves), 7264584 bytes in use
[8096] 10 Jan 16:32:28 - DB 0: 1 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:32:28 - 7 clients connected (0 slaves), 7298664 bytes in use
[8096] 10 Jan 16:32:34 - DB 0: 1 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:32:34 - 7 clients connected (0 slaves), 7318112 bytes in use
[8096] 10 Jan 16:32:39 - DB 0: 1 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:32:39 - 7 clients connected (0 slaves), 7332752 bytes in use
[8096] 10 Jan 16:32:44 - DB 0: 1 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:32:44 - 7 clients connected (0 slaves), 7342720 bytes in use
[8096] 10 Jan 16:32:44 - Client closed connection
[8096] 10 Jan 16:32:44 - Client closed connection
[8096] 10 Jan 16:32:44 - Client closed connection
[8096] 10 Jan 16:32:44 - Client closed connection
[8096] 10 Jan 16:32:44 - Client closed connection
[8096] 10 Jan 16:32:44 - Client closed connection
[8096] 10 Jan 16:32:49 - DB 0: 1 keys (0 volatile) in 4 slots HT.
[8096] 10 Jan 16:32:49 - 1 clients connected (0 slaves), 4964808 bytes in use
[8096] 10 Jan 16:32:54 - DB 0: 1 keys (0 volatile) in 4 slots HT.
и когда появляется ошибка, внезапно подключаются еще два клиента, я не знаю, почему они возникают
а им я закрываю паук и клиент закрыл соединение