Я пытаюсь запустить веб-паук, который получает все URL для определенного URL.Прямо сейчас он возвращает около 64 URL, когда я знаю, что есть еще сотни и тысячи.Кто-нибудь знает, почему он заканчивает раньше?
class MySpider(BaseSpider):
custom_settings = {
'AUTOTHROTTLE_ENABLED': True,
'DOWNLOAD_DELAY': 1.5
}
name = 'www.shopgoodwill.com'
allowed_domains = ['www.shopgoodwill.com']
start_urls = [
'https://www.shopgoodwill.com'
]
def __init__(self, alexa_site_id, *args, **kwargs):
super(MySpider, self).__init__(*args, **kwargs)
self.alexa_site_id = alexa_site_id
def parse(self, response):
le = LinkExtractor()
for link in le.extract_links(response):
yield Request(link.url, callback=self.parse_item)
Вот результаты, которые я отметил, говорит: request_depth_max: 1, но у меня DEPTH_LIMIT = 0 в настройках
2019-02-19 23:31:03 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 14739,
'downloader/request_count': 32,
'downloader/request_method_count/GET': 32,
'downloader/response_bytes': 336986,
'downloader/response_count': 32,
'downloader/response_status_count/200': 23,
'downloader/response_status_count/302': 9,
'dupefilter/filtered': 11,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 2, 19, 23, 31, 3, 824302),
'log_count/DEBUG': 38,
'log_count/INFO': 22,
'memusage/max': 108908544,
'memusage/startup': 108908544,
'offsite/domains': 5,
'offsite/filtered': 5,
'request_depth_max': 1,
'response_received_count': 23,
'scheduler/dequeued': 32,
'scheduler/dequeued/memory': 32,
'scheduler/enqueued': 32,
'scheduler/enqueued/memory': 32,
'start_time': datetime.datetime(2019, 2, 19, 23, 30, 4, 918201)}
2019-02-19 23:31:03 [scrapy.core.engine] INFO: Spider closed (finished)
Closing spider (finished)
Dumping Scrapy stats:
{'downloader/request_bytes': 14739,
'downloader/request_count': 32,
'downloader/request_method_count/GET': 32,
'downloader/response_bytes': 336986,
'downloader/response_count': 32,
'downloader/response_status_count/200': 23,
'downloader/response_status_count/302': 9,
'dupefilter/filtered': 11,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 2, 19, 23, 31, 3, 824302),
'log_count/DEBUG': 38,
'log_count/INFO': 22,
'memusage/max': 108908544,
'memusage/startup': 108908544,
'offsite/domains': 5,
'offsite/filtered': 5,
'request_depth_max': 1,
'response_received_count': 23,
'scheduler/dequeued': 32,
'scheduler/dequeued/memory': 32,
'scheduler/enqueued': 32,
'scheduler/enqueued/memory': 32,
'start_time': datetime.datetime(2019, 2, 19, 23, 30, 4, 918201)}
Spider closed (finished)