Python beautifulsoup AttributeError

Question

Я пытаюсь получить URL-адрес изображения, используя python beautifulsoup из html содержимого.

My HTML Содержимое:

<div id="photos" class="tab rel-photos multiple-photos">
   <span id="watch-this" class="classified-detail-buttons">
   <span id="c_id_10832265:c_type_202:watch_this">
   <a href="/watchlist/classified/baby-items/10832265/1/" id="watch_this_logged" data-require-auth="favoriteAd" data-tr-event-name="dpv-add-to-favourites">
   <i class="fa fa-fw fa-star-o"></i></a></span>
   </span>
   <span id="thumb1" class=" image">
      <a href="https://images.dubizzle.com/v1/files/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmbiI6ImYzYWdrZm8xcDBlai1EVUJJWlpMRSIsInciOlt7ImZuIjoiNWpldWk3cWZ6aWU2MS1EVUJJWlpMRSIsInMiOjUwLCJwIjoiY2VudGVyLGNlbnRlciIsImEiOjgwfV19.s1GmifnZr0_Bx4HG8RTR4puYcxN0asqAmnBvSpIExEI/image;p=main"
         id="a-photo-modal-view:263986810"
         rel="photos-modal"
         target="_new"
         onClick="return dbzglobal_event_adapter(this);">
         <div style="background-image:url(https://images.dubizzle.com/v1/files/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmbiI6ImYzYWdrZm8xcDBlai1EVUJJWlpMRSIsInciOlt7ImZuIjoiNWpldWk3cWZ6aWU2MS1EVUJJWlpMRSIsInMiOjUwLCJwIjoiY2VudGVyLGNlbnRlciIsImEiOjgwfV19.s1GmifnZr0_Bx4HG8RTR4puYcxN0asqAmnBvSpIExEI/image;p=main);"></div>
      </a>
   </span>
   <ul id="thumbs-list">
      <li>
         <span id="thumb2" class="image2">
            <a href="https://images.dubizzle.com/v1/files/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmbiI6Imtmc3cxMWgzNTB2cTMtRFVCSVpaTEUiLCJ3IjpbeyJmbiI6IjVqZXVpN3FmemllNjEtRFVCSVpaTEUiLCJzIjo1MCwicCI6ImNlbnRlcixjZW50ZXIiLCJhIjo4MH1dfQ.Wo2YqPdWav8shtmyVO2AdisHmLX-ZLDAiskLPAmTSPU/image;p=main" id="a-photo-modal-view:263986811" rel="photos-modal" target="_new" onClick="return dbzglobal_event_adapter(this);" >
               <div style="background-image:url(https://images.dubizzle.com/v1/files/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmbiI6Imtmc3cxMWgzNTB2cTMtRFVCSVpaTEUiLCJ3IjpbeyJmbiI6IjVqZXVpN3FmemllNjEtRFVCSVpaTEUiLCJzIjo1MCwicCI6ImNlbnRlcixjZW50ZXIiLCJhIjo4MH1dfQ.Wo2YqPdWav8shtmyVO2AdisHmLX-ZLDAiskLPAmTSPU/image;p=thumb_retina);"></div>
            </a>
         </span>
      </li>
      <li id="thumbnails-info">
         4 Photos
      </li>
   </ul>
   <div id="photo-count">
      4 Photos - Click to enlarge
   </div>
</div>

Мой python код:

images = soup.find("div", {"id": ["photos"]}).find_all("a")
for image in images:

    sk = image.get("href").replace("p=main","p=thumb_retina",1)
    print(sk)

Но я получаю сообщение об ошибке:

Traceback (most recent call last):
  File "/Users/evilslab/Documents/Websites/www.futurepoint.dev.cc/dobuyme/SCRAP/boats.py", line 47, in <module>
    images = soup.find("div", {"id": ["photos"]}).find_all("a")
AttributeError: 'NoneType' object has no attribute 'find_all'

Как я могу получить только URL-адрес из тега href?

Answer 1

Ваш код работает для меня, более полно (учитывая ваш HTML как html_doc):

from bs4 import BeautifulSoup

soup = BeautifulSoup(html_doc)
images = soup.find("div", {"id": ["photos"]}).find_all("a")
for image in images:
    print(image['href'].replace("p=main","p=thumb_retina",1)) 

Однако ваша проблема в том, что текст, возвращаемый requests с URL-адреса, равен не так же, как образец HTML, который вы даете. Несмотря на вашу попытку предоставить случайный пользовательский агент, сервер возвращает:

<li>You\'re a power user moving through this website with super-human speed.</li>\n                        <li>You\'ve disabled JavaScript in your web browser.</li>\n                        <li>A third-party browser plugin, such as Ghostery or NoScript, is preventing JavaScript from running. Additional information is available in this <a title=\'Third party browser plugins that block javascript\' href=\'http://ds.tl/help-third-party-plugins\' target=\'_blank\'>support article</a>.</li>\n                    </ul>\n                </div>\n                <p class="we-could-be-wrong" >\n                    We could be wrong, and sorry about that! Please complete the CAPTCHA below and we’ll get you back on dubizzle right away.

Поскольку CAPTCHA предназначен для предотвращения очистки, я предлагаю уважать пожелания администратора и не очищать его. Может быть, есть API?

Answer 2

Попробуйте это:

for item in soup.find_all('span'):
    try:
        link = item.find_all('a', href=True)[0].attrs.get('href', None)
    except IndexError:
        continue
    else:
        print(link)

вывод

/watchlist/classified/baby-items/10832265/1/
/watchlist/classified/baby-items/10832265/1/
https://images.dubizzle.com/v1/files/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmbiI6ImYzYWdrZm8xcDBlai1EVUJJWlpMRSIsInciOlt7ImZuIjoiNWpldWk3cWZ6aWU2MS1EVUJJWlpMRSIsInMiOjUwLCJwIjoiY2VudGVyLGNlbnRlciIsImEiOjgwfV19.s1GmifnZr0_Bx4HG8RTR4puYcxN0asqAmnBvSpIExEI/image;p=main
https://images.dubizzle.com/v1/files/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmbiI6Imtmc3cxMWgzNTB2cTMtRFVCSVpaTEUiLCJ3IjpbeyJmbiI6IjVqZXVpN3FmemllNjEtRFVCSVpaTEUiLCJzIjo1MCwicCI6ImNlbnRlcixjZW50ZXIiLCJhIjo4MH1dfQ.Wo2YqPdWav8shtmyVO2AdisHmLX-ZLDAiskLPAmTSPU/image;p=main