Как сохранить «данные в формате CSV с помощью scrapy crawl -o items.csv» - PullRequest
Новая
       20

Как сохранить «данные в формате CSV с помощью scrapy crawl -o items.csv»

0 голосов
/ 26 июня 2019

Я извлекаю данные для тестирования с веб-сайта действительно.com.Информация, которая мне нужна, я могу почистить ее, например, заголовок, местоположение, дату, описание и т. Д. На оболочке Scrapy, но когда я создал файл Python и попытался сохранить его .csv, он не работает для меня .csv файл пуст.

Я прилагаю код этого файла и результат сканирования -o items.csv из cmd.

Вот мой код 

    # -*- coding: utf-8 -*-
        from scrapy import Spider
        from scrapy.http import Request

        class JobsSpider(Spider):
            name = 'jobs' 
            allowed_domains = ['indeed.com.pk']
            start_urls = ['http://indeed.com.pk']
            page=10
            job_url=""
            def parse(self, response):
                jobs=response.xpath('//*[@class="title"]/a/@href').extract()

                for job in jobs:
                    absolute_url=response.urljoin(job)
                    JobsSpider.job_url=absolute_url
                    yield Request(absolute_url, callback=self.parse_job)
            #process Next Page URL
            next_page_url="&start={page}".format(page=JobsSpider.page)
            JobsSpider.page+=10
            absolute_next_page_url=response.urljoin(next_page_url)
            yield Request(absolute_next_page_url)

            def parse_job(self, response):
                job_name= response.xpath('//*[@class="icl-u-xs-mb--xs icl-u- 
                xs-mt- 
                -none jobsearch-JobInfoHeader- 
                title"]/text()').extract_first()
                job_location=response.xpath('//*[@class="jobsearch- 
                InlineCompanyRating icl-u-xs-mt--xs  jobsearch- 
                DesktopStickyContainer- 
                companyrating"]/div[3]/text()').extract_first()
                posted_on_date= response.xpath('//*[@class="jobsearch- 
                JobMetadataFooter"]/text()').extract_first()
                job_descriptions=response.xpath('//*[@class="jobsearch- 
                jobDescriptionText"]/ul').extract_first()

                yield{
                   'job_name' : job_name,
                   'job_location' : job_location,
                   'posted_on_date' : posted_on_date,
                   'job_descriptions' : job_descriptions,
                   'JobsSpider.job_url' : JobsSpider.job_url
               }


C:\Users\Linus\Desktop\indeedemployerpk>scrapy crawl jobs -o items.csv
2019-06-26 06:40:27 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: indeedemployerpk)
2019-06-26 06:40:27 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 19.2.0, Python 2.7.15 |Anaconda, Inc.| (default, Dec 10 2018, 21:57:18) [MSC v.1500 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.0.2p  14 Aug 2018), cryptography 2.4.2, Platform Windows-10-10.0.17134
2019-06-26 06:40:27 [scrapy.crawler] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'indeedemployerpk.spiders', 'FEED_FORMAT': 'csv', 'SPIDER_MODULES': ['indeedemployerpk.spiders'], 'FEED_URI': 'items.csv', 'BOT_NAME': 'indeedemployerpk'}
2019-06-26 06:40:27 [scrapy.extensions.telnet] INFO: Telnet Password: 8858a3712aa75912
2019-06-26 06:40:27 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2019-06-26 06:40:27 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-06-26 06:40:27 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-06-26 06:40:27 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-06-26 06:40:27 [scrapy.core.engine] INFO: Spider opened
2019-06-26 06:40:27 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-06-26 06:40:27 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-06-26 06:40:28 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET http://www.indeed.com.pk/> from <GET http://indeed.com.pk>
2019-06-26 06:40:29 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.indeed.com.pk/> from <GET http://www.indeed.com.pk/>
2019-06-26 06:40:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.indeed.com.pk/> (referer: None)
2019-06-26 06:40:30 [scrapy.core.engine] INFO: Closing spider (finished)
2019-06-26 06:40:30 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 818,
 'downloader/request_count': 3,
 'downloader/request_method_count/GET': 3,
 'downloader/response_bytes': 8661,
 'downloader/response_count': 3,
 'downloader/response_status_count/200': 1,
 'downloader/response_status_count/301': 1,
 'downloader/response_status_count/302': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 6, 26, 1, 40, 30, 619000),
 'log_count/DEBUG': 3,
 'log_count/INFO': 9,
 'response_received_count': 1,
 'scheduler/dequeued': 3,
 'scheduler/dequeued/memory': 3,
 'scheduler/enqueued': 3,
 'scheduler/enqueued/memory': 3,
 'start_time': datetime.datetime(2019, 6, 26, 1, 40, 27, 759000)}
2019-06-26 06:40:30 [scrapy.core.engine] INFO: Spider closed (finished)
...