Скребок информации об акциях из Business Insider с помощью Scrapy - PullRequest
Новая
       5

Скребок информации об акциях из Business Insider с помощью Scrapy

0 голосов
/ 26 мая 2020

Я пытаюсь вытащить поля «Имя», «Последняя цена» и «%» для каждой акции со следующего сайта: https://markets.businessinsider.com/index/components/s&p_500

Однако я не получить никаких данных, хотя я подтвердил, что мои XPath работают в консоли Chrome для этих полей.

Для справки, я использовал это руководство: https://realpython.com/web-scraping-with-scrapy-and-mongodb/

items.py

from scrapy.item import Item, Field

class InvestmentItem(Item):
    ticker = Field()
    name = Field()
    px = Field()
    pct = Field()

investment_spider.py

from scrapy import Spider
from scrapy.selector import Selector
from investment.items import InvestmentItem

class InvestmentSpider(Spider):
    name = "investment"
    allowed_domains = ["markets.businessinsider.com"]
    start_urls = [
            "https://markets.businessinsider.com/index/components/s&p_500",
            ]

    def parse(self, response):
        stocks = Selector(response).xpath('//*[@id="index-list-container"]/div[2]/table/tbody/tr')

        for stock in stocks:
            item = InvestmentItem()
            item['name'] = stock.xpath('td[1]/a/text()').extract()[0]
            item['px'] = stock.xpath('td[2]/text()[1]').extract()[0]
            item['pct'] = stock.xpath('td[5]/span[2]').extract()[0]

            yield item

вывод из консоли: 

...
2020-05-26 00:08:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://markets.businessinsider.com/robots.txt> (referer: None)
2020-05-26 00:08:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://markets.businessinsider.com/index/components/s&p_500> (referer: None)
2020-05-26 00:08:33 [scrapy.core.engine] INFO: Closing spider (finished)
2020-05-26 00:08:33 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
...
2020-05-26 00:08:33 [scrapy.core.engine] INFO: Spider closed (finished)

Ответы [ 2 ]

1 голос
/ 26 мая 2020

XPATH версия 

    def parse(self, response):

        rows = response.xpath('//*[@id="index-list-container"]/div[2]/table/tr')
        for row in rows:
            yield{
                'name' : row.xpath('td[1]/a/text()').extract(),
                'price':row.xpath('td[2]/text()[1]').extract(),
                'pct':row.xpath('td[5]/span[2]/text()').extract(),
                'datetime':row.xpath('td[7]/span[2]/text()').extract(),
            }

CSS версия 

    def parse(self, response):

        table = response.css('div#index-list-container table.table-small') 
        rows = table.css('tr') 

        for row in rows:
            name = row.css("a::text").get()
            high_low = row.css('td:nth-child(2)::text').get()
            date_time = row.css('td:nth-child(7) span:nth-child(2) ::text').get()

            yield {      
                'name' : name, 
                'high_low': high_low,
                'date_time' : date_time                
            }

Результат 

{"high_low": "\r\n146.44", "name": "3M", "date_time": "05/26/2020 04:15:11 PM UTC-0400"},
{"high_low": "\r\n42.22", "name": "AO Smith", "date_time": "05/26/2020 04:15:11 PM UTC-0400"},
{"high_low": "\r\n91.47", "name": "Abbott Laboratories", "date_time": "05/26/2020 04:15:11 PM UTC-0400"},
{"high_low": "\r\n92.10", "name": "AbbVie", "date_time": "05/26/2020 04:15:11 PM UTC-0400"},
{"high_low": "\r\n193.71", "name": "Accenture", "date_time": "05/26/2020 04:15:11 PM UTC-0400"},
{"high_low": "\r\n73.08", "name": "Activision Blizzard", "date_time": "05/25/2020 08:00:00 PM UTC-0400"},
{"high_low": "\r\n385.26", "name": "Adobe", "date_time": "05/25/2020 08:00:00 PM UTC-0400"},
{"high_low": "\r\n133.48", "name": "Advance Auto Parts", "date_time": "05/26/2020 04:15:11 PM UTC-0400"},
1 голос
/ 26 мая 2020

Вы пропустили "./" в начале выражения xpath. Я упростил ваши xpaths: 

def parse(self, response):
    stocks = response.xpath('//table[@class="table table-small"]/tr')

    for stock in stocks[1:]:
        item = InvestmentItem()
        item['name'] = stock.xpath('./td[1]/a/text()').get()
        item['px'] = stock.xpath('./td[2]/text()[1]').get().strip()
        item['pct'] = stock.xpath('./td[5]/span[2]/text()').get()

        yield item
...