Что вызывает эту ошибку?Отсутствующая схема в URL запроса: h

Question

Когда я пытаюсь сканировать свою веб-страницу, она выдает мне вывод, но появляется какая-то ошибка:

ValueError: Missing scheme in request url: h 

books2.py

class Books1Spider(Spider):
    name = 'books1'

    allowed_domains = ['books.toscrape.com']
    start_urls = ['http://books.toscrape.com/']
    headers = {
            "Host": "localhost",
            "Connection": "keep-alive",
            "Cache-Control": "max-age=0",
            "Upgrade-Insecure-Requests": "1",
            "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36",
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
            "DNT": "1",
            "Accept-Encoding": "gzip, deflate, sdch",
            "Accept-Language":"en-US,en;q=0.8"
        } 


    def parse_book(self,response):
        title = response.xpath('//h1/text()').extract_first()
        price = response.xpath('.//*[@class="price_color"]/text()').extract_first()

        image_urls = response.xpath('.//img/@src').extract_first()
        image_urls = image_urls.replace('../..','http://books.toscrape.com/')


        rating = response.xpath('//*[contains(@class,"star-rating")]/@class').extract_first()
        rating = rating.replace('star-rating','')

        description = response.xpath('//*[@id="product_description"]/following-sibling::p/text()').extract_first()

yield { 'title':title,
        'price':price,
        'image_urls':image_urls,
        'rating':rating,
        'description': description,
        }

Ожидаемый результат:

{'rating': u' Five', 'price': u'\xa352.29', 'description': u'Scott Pilgrim\'s life is totally sweet. He\'s 23 years old, he\'s in a rockband, he\'s "between jobs" and he\'s dating a cute high school girl. Nothing could possibly go wrong, unless a seriously mind-blowing, dangerously fashionable, rollerblading delivery girl named Ramona Flowers starts cruising through his dreams and sailing by him at parties. Will Scott\'s awesome life get Scott Pilgrim\'s life is totally sweet. He\'s 23 years old, he\'s in a rockband, he\'s "between jobs" and he\'s dating a cute high school girl. Nothing could possibly go wrong, unless a seriously mind-blowing, dangerously fashionable, rollerblading delivery girl named Ramona Flowers starts cruising through his dreams and sailing by him at parties. Will Scott\'s awesome life get turned upside-down? Will he have to face Ramona\'s seven evil ex-boyfriends in battle? The short answer is yes. The long answer is Scott Pilgrim, Volume 1: Scott Pilgrim\'s Precious Little Life ...more', 'image_urls': u'http://books.toscrape.com//media/cache/97/27/97275841c81e66d53bf9313cba06f23e.jpg', 'title': u"Scott Pilgrim's Precious Little Life (Scott Pilgrim #1)"}

Фактический результат:

2019-02-07 16:06:54 [scrapy.core.scraper] ERROR: Error processing {'rating': u' Five', 'price': u'\xa352.29', 'description': u'Scott Pilgrim\'s life is totally sweet. He\'s 23 years old, he\'s in a rockband, he\'s "between jobs" and he\'s dating a cute high school girl. Nothing could possibly go wrong, unless a seriously mind-blowing, dangerously fashionable, rollerblading delivery girl named Ramona Flowers starts cruising through his dreams and sailing by him at parties. Will Scott\'s awesome life get Scott Pilgrim\'s life is totally sweet. He\'s 23 years old, he\'s in a rockband, he\'s "between jobs" and he\'s dating a cute high school girl. Nothing could possibly go wrong, unless a seriously mind-blowing, dangerously fashionable, rollerblading delivery girl named Ramona Flowers starts cruising through his dreams and sailing by him at parties. Will Scott\'s awesome life get turned upside-down? Will he have to face Ramona\'s seven evil ex-boyfriends in battle? The short answer is yes. The long answer is Scott Pilgrim, Volume 1: Scott Pilgrim\'s Precious Little Life ...more', 'image_urls': u'http://books.toscrape.com//media/cache/97/27/97275841c81e66d53bf9313cba06f23e.jpg', 'title': u"Scott Pilgrim's Precious Little Life (Scott Pilgrim #1)"}
Traceback (most recent call last):
  File "/home/divum/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/divum/venv/local/lib/python2.7/site-packages/scrapy/pipelines/media.py", line 79, in process_item
    requests = arg_to_iter(self.get_media_requests(item, info))
  File "/home/divum/venv/local/lib/python2.7/site-packages/scrapy/pipelines/images.py", line 155, in get_media_requests
    return [Request(x) for x in item.get(self.images_urls_field, [])]
  File "/home/divum/venv/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
    self._set_url(url)
 File "/home/divum/venv/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 62, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: h

Answer 1

Вы извлекаете image_urls как u'…'.Значение image_urls должно быть списком: [u'…'].

. В вашем коде переключите:

image_urls = response.xpath('.//img/@src').extract_first()
image_urls = image_urls.replace('../..','http://books.toscrape.com/')

на

image_url = response.xpath('.//img/@src').extract_first()
image_urls = [image_url.replace('../..','http://books.toscrape.com/')]

Answer 2

Похоже, что вы пропустили или передали некоторые неверные данные в запросе вызова.

Перейдите на все ваши запросы и узнайте, что URL-адрес, который вы передаете, имеет правильную форму.

Попробуйте использоватьRequest(response.urljoin(url), ...) Чтобы он покрывал любую плохую схему URL.