Я настраиваю простого паука для загрузки изображений из xkcd, вот код, который у меня есть:
паук:
import scrapy
from scrapy.loader import ItemLoader
from test_im.items import TestImItem
class SpiderSpider(scrapy.Spider):
name = 'spider_'
allowed_domains = ['xkcd.com/']
start_urls = ['http://xkcd.com//']
def parse(self, response):
test_item = TestImItem()
relative_url = response.xpath('//*[@id="comic"]//@src').extract_first()
image_urls = (response.urljoin(relative_url) )
print (image_urls)
test_item['image_urls'] = image_urls
yield test_item
товар:
import scrapy
class TestImItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
image_urls = scrapy.Field()
параметры настройки по:
ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
IMAGES_STORE = '/home/luis/Documentos/proyectos/test_im/test_im/images/'
Я получаю следующую ошибку:
2019-07-08 21:25:13 [scrapy.core.scraper] ERROR: Error processing {'image_urls': 'https://imgs.xkcd.com/comics/trained_a_neural_net.png'}
Traceback (most recent call last):
File "/home/luis/anaconda3/lib/python3.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/home/luis/anaconda3/lib/python3.7/site-packages/scrapy/pipelines/media.py", line 79, in process_item
requests = arg_to_iter(self.get_media_requests(item, info))
File "/home/luis/anaconda3/lib/python3.7/site-packages/scrapy/pipelines/images.py", line 155, in get_media_requests
return [Request(x) for x in item.get(self.images_urls_field, [])]
File "/home/luis/anaconda3/lib/python3.7/site-packages/scrapy/pipelines/images.py", line 155, in <listcomp>
return [Request(x) for x in item.get(self.images_urls_field, [])]
File "/home/luis/anaconda3/lib/python3.7/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
self._set_url(url)
File "/home/luis/anaconda3/lib/python3.7/site-packages/scrapy/http/request/__init__.py", line 62, in _set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: h
Насколько я понимаю, 'ValueError: отсутствует схема в запросе url: h' означает, что URL изображения неверен.
Но я могу открыть его в браузере без проблем.
'image_urls': 'https://imgs.xkcd.com/comics/trained_a_neural_net.png'