один Div не найден при просмотре с использованием python, другие Div обнаружены

Question

Я собираю названия продуктов, Старая цена, Новая цена с веб-сайта, используя Python Beautifulsoup.

Когда я проверяю страницу, используя chrome, я могу видеть div, однако, когда я Пройдите через python, новый ценовой блок не найден.

Вот мой код

import requests
from bs4 import BeautifulSoup

for j in range(1,8):
    page_link = 'https://sturdysports.com.au/?p='+str(j)
    page_response = requests.get(page_link)
    page_content = BeautifulSoup(page_response.content, 'html.parser')

    #This is product wrapper, there are 12 on each page, so we want to find all info under this tag
    products = page_content.find_all('li', class_='item product product-item')

    for i in range(0,len(products)):
        name = products[i].find('a', class_='product-item-link').text.strip()
        #Tried traversing using 2-3 methods. None works.
        prices = products[i].find('div',class_='price-box price-final_price')

        rrp = products[i].find('span', attrs={"data-price-type":"oldPrice"})["data-price-amount"]
        #op = products[i].find('span', attrs={"data-price-type":"finalPrice"})["data-price-amount"]
        op = products[i].find('span',class_='special-price')
        print(op)

        data = '"' + name + '","' + rrp + '","' + op + '"\n'
        print("  Item#: "+ str(i) + ' ' + data)

Answer 1

Проблема в том, что html.parser не может правильно разобрать код. Используйте lxml или html5lib для правильного анализа документа.

Например:

import requests
from bs4 import BeautifulSoup

print('{:<80} {:<20} {:<20}'.format('Name', 'Old Price', 'Final Price'))
for j in range(1,8):
    page_link = 'https://atsport.com.au/cricket/cricket-bats/?p='+str(j)
    page_response = requests.get(page_link)
    page_content = BeautifulSoup(page_response.content, 'lxml') # <-- change to lxml or html5lib

    for tag in page_content.select('.products li.item'):
        name = tag.select_one('.product-item-name')
        price = tag.select_one('.old-price .price')
        special_price = tag.select_one('.special-price .price')

        print('{:<80} {:<20} {:<20}'.format(name.get_text(strip=True), price.get_text(strip=True), special_price.get_text(strip=True)))

Отпечатки:

Name                                                                             Old Price            Final Price         
BAS Boundary English Willow Cricket Bat                                          $435.00              $399.00             
BAS Bow 20/20 Player Edition English Willow Cricket Bat                          $1,000.00            $849.00             
BAS Commander English Willow Cricket Bat                                         $675.00              $549.00             
BAS Exploder English Willow Cricket Bat                                          $800.00              $749.00             
BAS King Hitter English Willow Cricket Bat                                       $485.00              $449.00             
BAS Players Edition English Willow Cricket Bat                                   $1,200.00            $949.00             
BAS Retro Vintage MS Dhoni Classic English Willow Cricket Bat                    $650.00              $599.00             

...and so on.