Ошибка Scrapy 404 - проблема с перенаправлением FormRequest на веб-сайте BrickSeek - PullRequest
Новая
       16

Ошибка Scrapy 404 - проблема с перенаправлением FormRequest на веб-сайте BrickSeek

0 голосов
/ 07 февраля 2020

Я сейчас пытаюсь зайти на сайт brickseek методом FormRequest, но не могу войти в систему. Я продолжаю получать ошибку 404 при использовании команды scrapy crawl. Мне кажется, что scrapy неправильно перенаправляет страницу. Я также заметил, что мой логин и пароль вводятся на перенаправленной веб-странице, что странно.

Первая страница должна быть https://brickseek.com/login с информацией для входа, добавленной на эту страницу. Затем, после входа на вторую страницу, должно быть https://brickseek.com/account

Он продолжает перенаправлять меня на https://brickseek.com/products/?search=&log=12345678&pwd=12345678

Вот код: 

import scrapy
from scrapy.http import FormRequest
from scrapy.utils.response import open_in_browser
from ..items import QuotetutorialItem


class QuoteSpider(scrapy.Spider):
    """Enter Login information located in Inspect Element, find Network, then search for login."""
    name = 'login'
    start_urls = ['https://www.brickseek.com/login/']

    def parse(self, response):
        """Enter Login information located in Inspect Element, find Network, then search for login."""
        return FormRequest.from_response(response, formdata={'log': '12345678', 'pwd': '12345678'},
                                         callback=self.after_login)

    def after_login(self, response):
        """Enter what you want scrapy to web scrape."""
        open_in_browser(response)

        items = QuotetutorialItem()

        div_title = response.css('div.banner_content')
        title = div_title.css('span.banner__title-text::text').extract()
        items['title']: title

        yield items

Когда я ввожу команду scrapy crawl login, я получаю это: 

(venv) C:\Users\Trebor\PycharmProjects\NewScraper\quotetutorial\quotetutorial>scrapy crawl login
2020-02-07 00:19:27 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: quotetutorial)
2020-02-07 00:19:27 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.8.0 (tags/v3.8.0:fa919
fd, Oct 14 2019, 19:21:23) [MSC v.1916 32 bit (Intel)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Windows-10-10.0.18362-SP0
2020-02-07 00:19:28 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'quotetutorial', 'NEWSPIDER_MODULE': 'quotetutorial.spiders', 'SPIDER_MODULES': ['quotetutorial.spid
ers'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36'}
2020-02-07 00:19:28 [scrapy.extensions.telnet] INFO: Telnet Password: e8e844c7cbefd148
2020-02-07 00:19:28 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2020-02-07 00:19:28 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-02-07 00:19:28 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-02-07 00:19:28 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-02-07 00:19:28 [scrapy.core.engine] INFO: Spider opened
2020-02-07 00:19:28 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-02-07 00:19:28 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-02-07 00:19:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.brickseek.com/login/> (referer: None)
2020-02-07 00:19:29 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://brickseek.com/products/?search=&user_login=12345678&user_pass=12345678>
from <GET https://www.brickseek.com/products/?search=&user_login=12345678&user_pass=12345678>
2020-02-07 00:19:31 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://brickseek.com/products/?search=&user_login=12345678&user_pass=12345678> (referer: https://www.bric
kseek.com/login/)
2020-02-07 00:19:31 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://brickseek.com/products/?search=&user_login=12345678&user_pass=12345678>: HTTP s
tatus code is not handled or not allowed
2020-02-07 00:19:31 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-07 00:19:31 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1329,
 'downloader/request_count': 3,
 'downloader/request_method_count/GET': 3,
 'downloader/response_bytes': 41132,
 'downloader/response_count': 3,
 'downloader/response_status_count/200': 1,
 'downloader/response_status_count/301': 1,
 'downloader/response_status_count/404': 1,
 'elapsed_time_seconds': 2.460197,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 2, 7, 6, 19, 31, 193946),
 'httperror/response_ignored_count': 1,
 'httperror/response_ignored_status_count/404': 1,
 'log_count/DEBUG': 3,
 'log_count/INFO': 11,
 'request_depth_max': 1,
 'response_received_count': 2,
 'scheduler/dequeued': 3,
 'scheduler/dequeued/memory': 3,
 'scheduler/enqueued': 3,
 'scheduler/enqueued/memory': 3,
 'start_time': datetime.datetime(2020, 2, 7, 6, 19, 28, 733749)}
2020-02-07 00:19:31 [scrapy.core.engine] INFO: Spider closed (finished)

код cURL: 

import requests

cookies = {
    '__cfduid': 'da5ab5ff9b55ed8d9432e500501554604963',
    '__stripe_mid': '3689131-200-d1-9906-c3022a5de64',
    'wordpress_test_cookie': 'WP+Cookie+check',
    'ac_enable_tracking': '1',
    '_ga': 'GA1.2.114763001.15586575',
    '_hjid': '72b2d2b9-feef-ee-827a-2af3fe09ed',
    '__zlcmid': 'tZiQfA1Ce4mixx',
    '_form_1_': 'Sun Aug 04 2019 05:45:27 GMT-0500 (Central Daylight Time)',
    '_hjIncludedInSample': '1',
    'WRMCembrGyQs': '1L.OZR%40s7hny',
    'CpwgInmYKPQHM': 'mqJCzckWg7YOTl',
    'de-EnfISbN': '%5Dj_2Wqrp8a',
    'pfICHvSKW_lZr': 'ykbU6BaGwi0cS',
    '_gid': 'GA1.2.83233215.150787155',
    'bs_zip': '*****',
    'ABTasty': 'uid%3D190840545159480%26fst%3D15491552621%26pst%3D1579541792%26cs%3D1581042402480%26ns%3D8%26pvt%3D8%26pvs%3D1%26th%3D',
    'cf_clearance': '7bc4c20678aa9d74cbd8517348a72f0451dcf8-1581043791-0-250',
    '_gat_UA-55274071-1': '1',
    'wordpress_logged_in_03f9e3712464d83801b8eecd7edae': '12345678%7C151232373%7C1WXUNJmFaDGPkxwbJiPCmftQZEWYuXsRUe0qFM8kT%7C041b932ee01c4e7f5864d918a678084f4f543de2a532c64322a72ca5d39c40',
}

headers = {
    'authority': 'brickseek.zendesk.com',
    'cache-control': 'max-age=0',
    'origin': 'https://brickseek.com',
    'upgrade-insecure-requests': '1',
    'content-type': 'application/json; charset=UTF-8',
    'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Mobile Safari/537.36',
    'sec-fetch-user': '?1',
    'accept': '*/*',
    'sec-fetch-site': 'cross-site',
    'sec-fetch-mode': 'cors',
    'referer': 'https://brickseek.com/',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'en-US,en;q=0.9',
    'Referer': 'https://brickseek.com/',
    'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Mobile Safari/537.36',
    'cookie': '__cfduid=db073c10d776e79a06f2c8849e128a601554604964; TS011baee6=0130aff232f157d75c3e3c4e190122d294ad62f857be4c386f8fa205f622df7166e3284eeec932768fd9cf24cefd766096438d32; TS01a8b35a=01da6997013403b00a91c12774d32da66071ff49f68f15b5910144b1419a5eebec6ac1031cd95090f3216192aed2407a71e9a0cca',
    'if-none-match': 'W/"abd82b40b936e162ba2fd12ebca21"',
    'if-modified-since': 'Thu, 03 Oct 2019 19:25:17 GMT',
    'Origin': 'https://brickseek.com',
    'authorization': 'd62c460eb96a27a6dc5664ae30a091',
    'Upgrade-Insecure-Requests': '1',
    'Pragma': 'no-cache',
    'Accept-Encoding': 'identity;q=1, *;q=0',
    'Accept-Language': 'en-US,en;q=0.9',
    'Sec-WebSocket-Key': 'AtleA0fubdJ/pCYYR+0lQ==',
    'Upgrade': 'websocket',
    'Sec-WebSocket-Extensions': 'permessage-deflate; client_max_window_bits',
    'Cache-Control': 'no-cache',
    'Connection': 'Upgrade',
    'Sec-WebSocket-Version': '13',
    'Range': 'bytes=0-',
}

data = [
  ('log', '12345678'),
  ('pwd', '12345678'),
  ('wp-submit', 'Log In'),
  ('redirect_to', 'https://brickseek.com/'),
  ('mepr_process_login_form', 'true'),
  ('mepr_is_login_page', 'true'),
  ('_LkNiouJZhOFzHR', 'H4I_a9ZkfdCDPoj'),
  ('_LkNiouJZhOFzHR', 'H4I_a9ZkfdCDPoj'),
  ('YKJmh-NQXRuD', 'O[HDXTUysdu9'),
  ('YKJmh-NQXRuD', 'O[HDXTUysdu9'),
  ('EmWpdPiAGnc', 'ybBQluhGrT5'),
  ('EmWpdPiAGnc', 'ybBQluhGrT5'),
  ('DsdaYLSwTt', 'bWTpV]1ym3'),
  ('DsdaYLSwTt', 'bWTpV]1ym3'),
]

response = requests.post('https://brickseek.com/login/', headers=headers, cookies=cookies, data=data)

1 Ответ

0 голосов
/ 07 февраля 2020 
# -*- coding: utf-8 -*-
import scrapy
from scrapy.utils.response import open_in_browser
data={'log': 'YOUR_USER', 'pwd': 'YOUR_PASSWORD', 'wp-submit': 'Log In', 
'redirect_to': 'https://brickseek.com/', 'mepr_process_login_form': 'true', 
'mepr_is_login_page': 'true', '_LkNiouJZhOFzHR': 'H4I_a9ZkfdCDPoj', 
'YKJmh-NQXRuD': 'O[HDXTUysdu9', 'EmWpdPiAGnc': 'ybBQluhGrT5', 'DsdaYLSwTt': 'bWTpV]1ym3'}

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Content-Type': 'application/x-www-form-urlencoded',
    'Origin': 'https://brickseek.com',
    'Connection': 'keep-alive',
    'Referer': 'https://brickseek.com/login/',
    'Upgrade-Insecure-Requests': '1',
    'TE': 'Trailers',
}

class BrickseekSpider(scrapy.Spider):
    name = 'brickseek'
    allowed_domains = ['brickseek.com']

    def start_requests(self):
        yield scrapy.http.FormRequest(url='https://brickseek.com/login/',headers=headers,formdata=data,
        callback=self.parse)

    def parse(self,response):
        open_in_browser(response)
        #your code

Полный код для этого

Добро пожаловать на сайт PullRequest, где вы можете задавать вопросы и получать ответы от других членов сообщества.

Нет похожих вопросов

...