Использование селекторов CSS и Xpath с Scrapy

Question

Я следую официальному учебнику Scrapy, где я должен очистить данные от http://quotes.toscrape.com,, в учебнике показано, как очистить данные следующим пауком:

class QuotesSpiderCss(scrapy.Spider):
    name = "quotes_css"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        quotes = response.css('div.quote')
        for quote in quotes:
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('small.author::text').extract_first(),
                'tags': quote.css('div.tags::text').extract()
            }

Затем ползатьпаук в JSON-файл возвращает то, что указано:

[
{"text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d", "author": "Albert Einstein", "tags": ["\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        "]},
{"text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d", "author": "J.K. Rowling", "tags": ["\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n        "]},
{"text": "\u201cThere are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.\u201d", "author": "Albert Einstein", "tags": ["\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        "]},
...]

Я пытаюсь написать тот же Spider, используя xpath вместо css:

class QuotesSpiderXpath(scrapy.Spider):
    name = 'quotes_xpath'
    start_urls = [
        'http://quotes.toscrape.com/page/1/'
    ]

    def parse(self, response):
        quotes = response.xpath('//div[@class="quote"]')
        for quote in quotes:
            yield {
                'text': quote.xpath("//span[@class='text']/text()").extract_first(),
                'author': quote.xpath("//small[@class='author']/text()").extract_first(),
                'tags': quote.xpath("//div[@class='tags']/text()").extract()
            }

Но этот паук возвращает мнесписок с такой же цитатой:

[
{"text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d", "author": "Albert Einstein", "tags": ["\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        "]},
{"text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d", "author": "Albert Einstein", "tags": ["\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        "]},
{"text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d", "author": "Albert Einstein", "tags": ["\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n        ", "\n            Tags:\n            ", " \n            \n            ", "\n            \n            ", "\n            \n            ", "\n            \n        "]},
...]

Заранее спасибо!

Answer 1

Причина, по которой вы всегда получаете одинаковые кавычки, в том, что вы не используете относительный XPath.См. документацию .

Добавьте точку префикса к вашим операторам XPath, как в следующем методе разбора:

def parse(self, response):
    quotes = response.xpath('//div[@class="quote"]')
    for quote in quotes:
        yield {
            'text': quote.xpath(".//span[@class='text']/text()").extract_first(),
            'author': quote.xpath(".//small[@class='author']/text()").extract_first(),
            'tags': quote.xpath(".//div[@class='tags']/text()").extract()
        }