Я использовал scrapy для создания поискового робота на этом сайте: https://www.forexfactory.com. Все было хорошо, до сегодняшнего дня я не получил ответа от сайта. Я пытался с командой Scrapy Shell. Это все еще не работает, см. Ошибку ниже. Я даже безуспешно пытался использовать почтальон для выполнения http GET-запроса. Я думаю, что на этом сайте есть что-то, чтобы предотвратить сканирование. Кто-нибудь может помочь? Спасибо.
> > (base) C:\Users\<username>>scrapy shell https://www.forexfactory.com/calendar?day=today 2020-04-01 14:46:39
> [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: scrapybot)
> 2020-04-01 14:46:39 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0,
> libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted
> 19.10.0, Python 3.7.7 (default, Mar 23 2020, 23:19:08) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1f 31 Mar 2020),
> cryptography 2.8, Platform Windows-10-10.0.18362-SP0 2020-04-01
> 14:46:39 [scrapy.crawler] INFO: Overridden settings:
> {'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter',
> 'LOGSTATS_INTERVAL': 0} 2020-04-01 14:46:39 [scrapy.extensions.telnet]
> INFO: Telnet Password: 212f9fd6b7a7f764 2020-04-01 14:46:39
> [scrapy.middleware] INFO: Enabled extensions:
> ['scrapy.extensions.corestats.CoreStats',
> 'scrapy.extensions.telnet.TelnetConsole'] 2020-04-01 14:46:39
> [scrapy.middleware] INFO: Enabled downloader middlewares:
> ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
> 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
> 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
> 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
> 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
> 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
> 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
> 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
> 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
> 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
> 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2020-04-01
> 14:46:39 [scrapy.middleware] INFO: Enabled spider middlewares:
> ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
> 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
> 'scrapy.spidermiddlewares.referer.RefererMiddleware',
> 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
> 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2020-04-01 14:46:39
> [scrapy.middleware] INFO: Enabled item pipelines: [] 2020-04-01
> 14:46:39 [scrapy.extensions.telnet] INFO: Telnet console listening on
> 127.0.0.1:6023 2020-04-01 14:46:39 [scrapy.core.engine] INFO: Spider opened 2020-04-01 14:46:40 [scrapy.downloadermiddlewares.redirect]
> DEBUG: Redirecting (302) to <GET
> https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:40
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:41
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:41
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:41
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:41
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:42
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:42
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:42
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:42
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:43
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:43
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:43
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:43
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:43
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:44
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:44
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:44
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:44
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:45
> [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to
> <GET https://www.forexfactory.com/calendar?day=today> from <GET
> https://www.forexfactory.com/calendar?day=today> 2020-04-01 14:46:45
> [scrapy.downloadermiddlewares.redirect] DEBUG: Discarding <GET
> https://www.forexfactory.com/calendar?day=today>: max redirections
> reached 2020-04-01 14:46:46 [asyncio] DEBUG: Using selector:
> SelectSelector [s] Available Scrapy objects: [s] scrapy scrapy
> module (contains scrapy.Request, scrapy.Selector, etc) [s] crawler
> <scrapy.crawler.Crawler object at 0x000001456195B608> [s] item
> {} [s] request <GET
> https://www.forexfactory.com/calendar?day=today> [s] settings
> <scrapy.settings.Settings object at 0x000001456195BA08> [s] Useful
> shortcuts: [s] fetch(url[, redirect=True]) Fetch URL and update
> local objects (by default, redirects are followed) [s] fetch(req)
> Fetch a scrapy.Request and update local objects [s] shelp()
> Shell help (print this help) [s] view(response) View response in
> a browser 2020-04-01 14:46:46 [asyncio] DEBUG: Using selector:
> SelectSelector In [1]: