BeautifulSoup4 l oop не находит последующие элементы

Question

Я пытаюсь удалить некоторые элементы с сайта.

Вот мой код:

driver.get('https://www.planitbig.com/Categories/Office/Binders-and-Binding-Supplies/Binders.aspx')


soup=BeautifulSoup(driver.page_source,'html5lib')
soup=BeautifulSoup(driver.page_source,'html5lib')

table1 = soup.find( "table", {"id":"ctl00_MainContentHolder_ProductGridView1_dlProducts"} ).find_all(('div',{"class":"record"}))




for x in table1:
    table1=BeautifulSoup(str(table1),'html5lib')
    _link=table1.find('a',attrs={'class':'product_link'})
    print(_link)

По какой-то причине оба find_next / Find_all возвращают значение сохранения, первый элемент в список продуктов. Что я тут не так делаю

Answer 1

Создайте WebDriverWait() и дождитесь появления таблицы, используя селектор visibility_of_element_located() и css.

Код :

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver=webdriver.Chrome()
driver.get('https://www.planitbig.com/Categories/Office/Binders-and-Binding-Supplies/Binders.aspx')
WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"#ctl00_MainContentHolder_ProductGridView1_dlProducts")))
soup=BeautifulSoup(driver.page_source,'html5lib')
for link in soup.select("#ctl00_MainContentHolder_ProductGridView1_dlProducts .record .recordname>a[href]"):
    print(link['href'])

Выход :

/Products/Durable-View-Binder-with-DuraHinge-and-Slant-Rings--3-Rings--1-Capacity--11-x-85--White--4Pack__AVE17575.aspx?ix=1&fc=L3C&bb=SpotLight
/Products/Economy-View-Binder-with-Round-Rings---3-Rings--1-Capacity--11-x-85--White__AVE05711.aspx?ix=2&fc=L3C&bb=SpotLight
/Products/Economy-Round-Ring-View-Binder--3-Rings--1-Capacity--11-x-85--White__UNV20962.aspx?ix=3&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--2-Capacity--11-x-85--White__UNV20982.aspx?ix=4&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--15-Capacity--11-x-85--White__UNV20972.aspx?ix=5&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--1-Capacity--11-x-85--Black__UNV20961.aspx?ix=6&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--3-Capacity--11-x-85--White__UNV20992.aspx?ix=7&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--05-Capacity--11-x-85--White__UNV20952.aspx?ix=8&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--3-Capacity--11-x-85--Black__UNV20991.aspx?ix=9&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--2-Capacity--11-x-85--Black__UNV20981.aspx?ix=10&fc=L3C&bb=NA
/Products/Slant-Ring-View-Binder--3-Rings--4-Capacity--11-x-85--White__UNV20994.aspx?ix=11&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--15-Capacity--11-x-85--Black__UNV20971.aspx?ix=12&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--1-Capacity--11-x-85--White--12Carton__UNV20962CT.aspx?ix=13&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--05-Capacity--11-x-85--Black__UNV20951.aspx?ix=14&fc=L3C&bb=NA
/Products/Deluxe-Round-Ring-View-Binder--3-Rings--1-Capacity--11-x-85--White__UNV20712.aspx?ix=15&fc=L3C&bb=NA
/Products/Slant-Ring-View-Binder--3-Rings--3-Capacity--11-x-85--White__UNV20748.aspx?ix=16&fc=L3C&bb=NA
/Products/Durable-View-Binder-with-DuraHinge-and-Slant-Rings--3-Rings--2-Capacity--11-x-85--White__AVE17032.aspx?ix=17&fc=L3C&bb=NA
/Products/Flexi-View-Binder-with-Round-Rings--3-Rings--1-Capacity--11-x-85--Navy-Blue__AVE17685.aspx?ix=18&fc=L3C&bb=NA
/Products/Slant-Ring-View-Binder--3-Rings--4-Capacity--11-x-85--Black__UNV20995.aspx?ix=19&fc=L3C&bb=NA
/Products/Durable-View-Binder-with-DuraHinge-and-Slant-Rings--3-Rings--1-Capacity--11-x-85--White__AVE17012.aspx?ix=20&fc=L3C&bb=NA
/Products/Economy-View-Binder-with-Round-Rings---3-Rings--3-Capacity--11-x-85--White__AVE05741.aspx?ix=21&fc=L3C&bb=NA
/Products/Slant-Ring-View-Binder--3-Rings--1-Capacity--11-x-85--White__UNV20742.aspx?ix=22&fc=L3C&bb=NA
/Products/Durable-View-Binder-with-DuraHinge-and-Slant-Rings--3-Rings--05-Capacity--11-x-85--White__AVE17002.aspx?ix=23&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--15-Capacity--11-x-85--White--6Pack__UNV20972PK.aspx?ix=24&fc=L3C&bb=NA
/Products/Economy-View-Binder-with-Round-Rings---3-Rings--15-Capacity--11-x-85--White__AVE05726.aspx?ix=25&fc=L3C&bb=NA
/Products/Slant-Ring-View-Binder--3-Rings--5-Capacity--11-x-85--White__UNV20997.aspx?ix=26&fc=L3C&bb=NA
/Products/Slant-Ring-View-Binder--3-Rings--2-Capacity--11-x-85--White__UNV20746.aspx?ix=27&fc=L3C&bb=NA
/Products/Slant-Ring-View-Binder--3-Rings--3-Capacity--11-x-85--Black__UNV20747.aspx?ix=28&fc=L3C&bb=NA
/Products/Economy-Non-View-Round-Ring-Binder--3-Rings--1-Capacity--11-x-85--Black__UNV31401.aspx?ix=29&fc=L3C&bb=NA
/Products/Economy-Round-Ring-View-Binder--3-Rings--1-Capacity--11-x-85--White--6Pack__UNV20962PK.aspx?ix=30&fc=L3C&bb=NA