Scrapy не выдает URL-адреса веб-сайтов с ошибками поиска DNS.

Question

У меня есть список URL, который перенаправляется на другой URL в текстовом файле. Я хочу получить все перенаправленные URL-адреса. Так что я сделал паук, который открывает URL-адрес из текстового файла. Теперь я получаю сообщение об ошибке для немногих, например «Ошибка поиска DNS» или «Нет маршрута». Я проверил эти URL-адреса непосредственно в браузере и обнаружил, что эти URL-адреса получают «IP-адрес не найден ошибка». Однако я хочу scrap для каждого перенаправленного URL-адреса независимо от ошибки. Какие-нибудь решения для этого?

Вот паук, которого я побежал

import scrapy

class AwesomeSpiderSpider(scrapy.Spider):
    name = 'web_uk'
# opening the list of urls that gets redirected.
    f = open("urls.txt")
    start_urls = [url.strip() for url in f.readlines()]
    f.close()


    def parse(self,response):

        item = {}
        item['Web Address'] = response.url
        yield item

Вот вывод

2019-07-04 03:02:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://gatesidedevelopments.com/> (referer: None) ['cached']
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://honka.com/gb/en/>
{'Web Address': 'https://honka.com/gb/en/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.dapconstruction.co.uk/>
{'Web Address': 'https://www.dapconstruction.co.uk/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.virtueprojects.com>
{'Web Address': 'http://www.virtueprojects.com'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://waynemoore.com/>
{'Web Address': 'https://waynemoore.com/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.avenuenorth.co.uk/>
{'Web Address': 'http://www.avenuenorth.co.uk/'}
2019-07-04 03:02:03 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk/robots.txt> (failed 1 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:03 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.mic.uk.com>
{'Web Address': 'http://www.mic.uk.com'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.vlconstruction.co.uk/>
{'Web Address': 'https://www.vlconstruction.co.uk/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.whitehalloflondon.co.uk>
{'Web Address': 'http://www.whitehalloflondon.co.uk'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.vandthomes.com>
{'Web Address': 'http://www.vandthomes.com'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.atlanticdwellings.com/>
{'Web Address': 'https://www.atlanticdwellings.com/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.rgfhomeimprovements.co.uk>
{'Web Address': 'http://www.rgfhomeimprovements.co.uk'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://leonoc.co.uk/>
{'Web Address': 'https://leonoc.co.uk/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.home-refurbishments.co.uk/>
{'Web Address': 'http://www.home-refurbishments.co.uk/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://gatesidedevelopments.com/>
{'Web Address': 'https://gatesidedevelopments.com/'}
2019-07-04 03:02:07 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/robots.txt> (failed 1 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:07 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
2019-07-04 03:02:11 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk/robots.txt> (failed 2 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.45 Safari/537.36
2019-07-04 03:02:12 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/robots.txt> (failed 2 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:12 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
2019-07-04 03:02:17 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.galkivconstruction.co.uk/robots.txt> (failed 3 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:17 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http://www.galkivconstruction.co.uk/robots.txt>: DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
Traceback (most recent call last):
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 975, in startConnectionAttempts
    "no results for hostname lookup: {}".format(self._hostStr)
DNSLookupError: DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:17 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36
2019-07-04 03:02:19 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.rrawlins.co.uk/robots.txt> (failed 3 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:19 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http://www.rrawlins.co.uk/robots.txt>: An error occurred while connecting: 113: No route to host.
Traceback (most recent call last):
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
ConnectError: An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:19 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
2019-07-04 03:02:22 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/> (failed 1 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:22 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36
2019-07-04 03:02:23 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk> (failed 1 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:23 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31
2019-07-04 03:02:29 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk> (failed 2 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:29 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
2019-07-04 03:02:29 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/> (failed 2 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:29 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36
2019-07-04 03:02:34 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.galkivconstruction.co.uk> (failed 3 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:34 [scrapy.core.scraper] ERROR: Error downloading <GET http://www.galkivconstruction.co.uk>
Traceback (most recent call last):
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 975, in startConnectionAttempts
    "no results for hostname lookup: {}".format(self._hostStr)
DNSLookupError: DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:36 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.rrawlins.co.uk/> (failed 3 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:36 [scrapy.core.scraper] ERROR: Error downloading <GET http://www.rrawlins.co.uk/>
Traceback (most recent call last):
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
ConnectError: An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:36 [scrapy.core.engine] INFO: Closing spider (finished)

Answer 1

Вам необходимо использовать errback (см. Использование ошибочных ответов для перехвата исключений при обработке запроса ).

Answer 2

Наконец-то это сработало после добавления errback.

import scrapy
from scrapy.spidermiddlewares.httperror import HttpError
from twisted.internet.error import DNSLookupError
from twisted.internet.error import TimeoutError, TCPTimedOutError

class HouzzSpiderSpider(scrapy.Spider):
    name = 'web_uk'
    f = open("urls.txt")
    start_urls = [url.strip() for url in f.readlines()]
    f.close()


# this is just to no retry errors for this example spider
    custom_settings = {
        'RETRY_ENABLED': False
    }

    def start_requests(self):
        for u in self.start_urls:
            yield scrapy.Request(u, callback=self.parse_web, errback=self.errback_web, dont_filter=True)


    def parse_web(self, response):

        self.logger.info('Got successful response from {}'.format(response.url))
        # do something useful here...

        item = {}
        item['Web Address']= response.request.url
        yield item
    def errback_web(self, failure):
        # log all failures
        self.logger.error(repr(failure))
        item ={}
        item['Web Address']= failure.request.url
        yield item