Пожалуйста, помогите мне с этим!
Моя цель - запускать одного паука из сценария ежедневно (и впоследствии развертывать его в Heroku).
Я могу запустить Scrapy из сценария с помощью следующий код.
from scraper_app.spiders.booking import BookingSpider
from scrapy.utils.project import get_project_settings
from scrapy.crawler import CrawlerProcess
from apscheduler.schedulers.twisted import TwistedScheduler
process = CrawlerProcess(get_project_settings())
process.crawl(BookingSpider)
process.start()
Если я добавлю TwistedScheduler для ежедневного запуска паука (как предложено { ссылка }), паук будет работать, но у меня нет вывода.
process = CrawlerProcess(get_project_settings())
scheduler = TwistedScheduler()
scheduler.add_job(process.crawl, 'cron', args=[BookingSpider], hour=11, minute=37)
scheduler.start()
process.start(False)
Журналы
(booking_scraper) MacBook-Air-Oduvan:booking_scraper oduvan$ python run.py
2020-04-05 11:56:01 [scrapy.utils.log] INFO: Scrapy 2.0.1 started (bot: scraper_app)
2020-04-05 11:56:01 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28) - [Clang 6.0 (clang-600.0.57)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1f 31 Mar 2020), cryptography 2.9, Platform Darwin-19.3.0-x86_64-i386-64bit
2020-04-05 11:56:01 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-04-05 11:56:01 [apscheduler.scheduler] INFO: Adding job tentatively -- it will be properly scheduled when the scheduler starts
2020-04-05 11:56:01 [apscheduler.scheduler] INFO: Added job "CrawlerRunner.crawl" to job store "default"
2020-04-05 11:56:01 [apscheduler.scheduler] INFO: Scheduler started
2020-04-05 11:56:01 [apscheduler.scheduler] DEBUG: Looking for jobs to run
2020-04-05 11:56:01 [apscheduler.scheduler] DEBUG: Next wakeup is due at 2020-04-05 11:57:00+03:00 (in 58.755398 seconds)
2020-04-05 11:57:00 [apscheduler.scheduler] DEBUG: Looking for jobs to run
2020-04-05 11:57:00 [apscheduler.scheduler] DEBUG: Next wakeup is due at 2020-04-06 11:57:00+03:00 (in 86399.996359 seconds)
2020-04-05 11:57:00 [apscheduler.executors.default] INFO: Running job "CrawlerRunner.crawl (trigger: cron[hour='11', minute='57'], next run at: 2020-04-06 11:57:00 +03)" (scheduled at 2020-04-05 11:57:00+03:00)
2020-04-05 11:57:00 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'scraper_app',
'CONCURRENT_REQUESTS_PER_DOMAIN': 2,
'DOWNLOAD_DELAY': 6,
'FEED_EXPORT_ENCODING': 'utf-8',
'NEWSPIDER_MODULE': 'scraper_app.spiders',
'SPIDER_MODULES': ['scraper_app.spiders'],
'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) '
'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 '
'Safari/537.36'}
2020-04-05 11:57:00 [scrapy.extensions.telnet] INFO: Telnet Password: 7d8e28a58a2b0c6d
2020-04-05 11:57:00 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2020-04-05 11:57:00 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-04-05 11:57:00 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-04-05 11:57:02 [scrapy.middleware] INFO: Enabled item pipelines:
['scraper_app.pipelines.BookingPipeline']
2020-04-05 11:57:02 [scrapy.core.engine] INFO: Spider opened
2020-04-05 11:57:02 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-04-05 11:57:02 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-04-05 11:57:02 [apscheduler.executors.default] INFO: Job "CrawlerRunner.crawl (trigger: cron[hour='11', minute='57'], next run at: 2020-04-06 11:57:00 +03)" executed successfully