Scrapy: AttributeError: у объекта 'str' нет атрибута 'setdefault' - PullRequest
Новая
       19

Scrapy: AttributeError: у объекта 'str' нет атрибута 'setdefault'

0 голосов
/ 23 июня 2019

Я проверяю скрап и брызги. У меня есть паук с: 

class MySpider(scrapy.Spider):
    # start_urls = ["http://yahoo.com"]
    name = 'mytest'

    def __init__(self, state='CA', city='San_Francisco', *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.state = state
        self.city = city
        self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=state, city=city)]
        # self.le = LinkExtractor(allow=r'^https://www.trulia.com/property')


    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse, endpoint='render.html', args={'wait': 7.5},)

Паук работает, если я избавлюсь от функции init и раскомментирую строку strt-urls. при прохождении через него возникает проблема: 

self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=state, city=city)]

отслеживающий: 

2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider opened
2019-06-23 09:15:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-06-23 09:15:00 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method SplashMiddleware.spider_opened of <scrapy_splash.middleware.SplashMiddleware object at 0x
00000292FF8106A0>>
Traceback (most recent call last):
  File "  \lib\site-packages\twisted\internet\defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "....\lib\site-packages\pydispatch\robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 245, in spider_opened
    spider.state.setdefault(self.remote_keys_key, {})
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-06-23 09:15:00 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
  File "....\lib\site-packages\scrapy\core\engine.py", line 127, in _next_request
    request = next(slot.start_requests)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 165, in process_start_requests
    spider.state.setdefault(self.local_values_key, {})  # fingerprint => value dict

2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider opened
2019-06-23 09:15:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-06-23 09:15:00 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method SplashMiddleware.spider_opened of <scrapy_splash.middleware.SplashMiddleware object at 0x
00000292FF8106A0>>
Traceback (most recent call last):
  File "  \lib\site-packages\twisted\internet\defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "....\lib\site-packages\pydispatch\robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 245, in spider_opened
    spider.state.setdefault(self.remote_keys_key, {})
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-06-23 09:15:00 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
  File "....\lib\site-packages\scrapy\core\engine.py", line 127, in _next_request
    request = next(slot.start_requests)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 165, in process_start_requests
    spider.state.setdefault(self.local_values_key, {})  # fingerprint => value dict
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Closing spider (finished)
2019-06-23 09:15:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 838859),
 'log_count/ERROR': 2,
 'log_count/INFO': 9,
 'start_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 822180)}
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider closed (finished)

2019-06-23 09:15:00 [scrapy.core.engine] INFO: Closing spider (finished)
2019-06-23 09:15:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 838859),
 'log_count/ERROR': 2,
 'log_count/INFO': 9,
 'start_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 822180)}
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider closed (finished)

1 Ответ

1 голос
/ 23 июня 2019

Проблема в том, что ваше имя аргумента state, когда вы смотрите на код промежуточного программного обеспечения scrapy

https://github.com/scrapy-plugins/scrapy-splash/blob/e40ca4f3b367ab463273bee1357d3edfe0601f0d/scrapy_splash/middleware.py#L245

state имеет для него особое значение и для него используется dict. Так что вам нужно просто переименовать параметр 

class MySpider(scrapy.Spider):
    # start_urls = ["http://yahoo.com"]
    name = 'mytest'

    def __init__(self, STATE='CA', CITY='San_Francisco', *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.state = state
        self.city = city
        self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=STATE, city=CITY)]
        # self.le = LinkExtractor(allow=r'^https://www.trulia.com/property')


    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse, endpoint='render.html', args={'wait': 7.5},)
...