Проблема с использованием Selenium для загрузки страницы и сбора информации

Question

Я пытаюсь очистить 200 лучших художников из https://www.rollingstone.com/charts/artists/, но после успешной загрузки всех 200 художников через Selenium с помощью .click (), чтобы нажать кнопку «загрузить еще» и .implicitly_wait () и вставляя их в свою БД, я получаю только 50 лучших художников.

driver.find_element_by_css_selector (code) .click () и driver.implicitly_wait (10) code

Художник, занявший первое место, имеет другой селектор css, поэтому я собрал остальную часть исполнителя с помощью этого кода

the_rest = soup.select( '#site_wrap > div.l-page__content.l-page__charts.l-page__charts--single.l-page__charts--artists > div.l-section.l-section__charts--single > div.c-content.c-content--charts > section.l-section__charts--grid > section.l-section__charts.c-chart__table--single')

И затем я использовал его для l oop для вставки художников

for rest in the_rest:
    rest_ranking = rest.select_one(
        'div.c-chart__table--top > div.c-chart__table--rank').text
    # while rest_ranking <= '200':
    rest_img = rest.select_one('div.c-chart__table--top > img')['src']
    rest_name = rest.select_one('div.c-chart__table--top > div.c-chart__table--details > div').text
    doc = {
        'name': rest_name,
        'img_url': rest_img,
        'lyrics': [],
        'probability': 0
    }
    db.selen_top200.insert_one(doc)

Но он вставляет только 50 лучших художников в мою базу данных.

Есть ли проблема с тем, как я нажимаю кнопку «Загрузить еще» с помощью Selenium или это кстати я все "остальные" художники?

Answer 1

Я пытаюсь нажать кнопку «Загрузить больше» с помощью этого селектора div.load-more-link span с .execute_script(...), используя al oop.

Сначала после импорта:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Попробуйте следующий код, чтобы получить ранг, имя и источник изображения:

driver.get('https://www.rollingstone.com/charts/artists/')
 
target_artis = 200
list_artist = []
while True:
    if(len(driver.find_elements_by_css_selector('div.load-more-link span')) >0):
        load_more = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'div.load-more-link span')))
        driver.execute_script("arguments[0].click()", load_more)
 
    list_artist = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '.l-section__charts')))
    if(len(list_artist)>=target_artis):
        break
 
print(len(list_artist))
 
list_rest_ranking = []
list_rest_img = []
list_rest_name = []
 
for x in range(target_artis):
    list_rest_ranking.append(driver.find_elements_by_css_selector('div.c-chart__table--rank')[x].text)
    list_rest_img.append(driver.find_elements_by_css_selector('img.c-chart__table--cover')[x].get_attribute('src'))
    list_rest_name.append(driver.find_elements_by_css_selector('div.c-chart__table--title p')[x].text)

print(list_rest_ranking)
print(list_rest_img)
print(list_rest_name)

Вывод в консоль:

[u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11', u'12', u'13', u'14', u'15', u'16', u'17', u'18', u'19', u'20', u'21', u'22', u'23', u'24', u'25', u'26', u'27', u'28', u'29', u'30', u'31', u'32', u'33', u'34', u'35', u'36', u'37', u'38', u'39', u'40', u'41', u'42', u'43', u'44', u'45', u'46', u'47', u'48', u'49', u'50', u'51', u'52', u'53', u'54', u'55', u'56', u'57', u'58', u'59', u'60', u'61', u'62', u'63', u'64', u'65', u'66', u'67', u'68', u'69', u'70', u'71', u'72', u'73', u'74', u'75', u'76', u'77', u'78', u'79', u'80', u'81', u'82', u'83', u'84', u'85', u'86', u'87', u'88', u'89', u'90', u'91', u'92', u'93', u'94', u'95', u'96', u'97', u'98', u'99', u'100', u'101', u'102', u'103', u'104', u'105', u'106', u'107', u'108', u'109', u'110', u'111', u'112', u'113', u'114', u'115', u'116', u'117', u'118', u'119', u'120', u'121', u'122', u'123', u'124', u'125', u'126', u'127', u'128', u'129', u'130', u'131', u'132', u'133', u'134', u'135', u'136', u'137', u'138', u'139', u'140', u'141', u'142', u'143', u'144', u'145', u'146', u'147', u'148', u'149', u'150', u'151', u'152', u'153', u'154', u'155', u'156', u'157', u'158', u'159', u'160', u'161', u'162', u'163', u'164', u'165', u'166', u'167', u'168', u'169', u'170', u'171', u'172', u'173', u'174', u'175', u'176', u'177', u'178', u'179', u'180', u'181', u'182', u'183', u'184', u'185', u'186', u'187', u'188', u'189', u'190', u'191', u'192', u'193', u'194', u'195', u'196', u'197', u'198', u'199', u'200']
[u'https://api.buzzanglemusic.com/images/artists/1374', u'https://api.buzzanglemusic.com/images/artists/1075289', u'https://api.buzzanglemusic.com/images/artists/2543301', u'https://api.buzzanglemusic.com/images/artists/4384780', u'https://api.buzzanglemusic.com/images/artists/4609765', u'https://api.buzzanglemusic.com/images/artists/5562543', u'https://api.buzzanglemusic.com/images/artists/1347', u'https://api.buzzanglemusic.com/images/artists/1350', u'https://api.buzzanglemusic.com/images/artists/6762453', u'https://api.buzzanglemusic.com/images/artists/1151925', u'https://api.buzzanglemusic.com/images/artists/1598398', u'https://api.buzzanglemusic.com/images/artists/6273541', u'https://api.buzzanglemusic.com/images/artists/197', u'https://api.buzzanglemusic.com/images/artists/4', u'https://api.buzzanglemusic.com/images/artists/6240779', u'https://api.buzzanglemusic.com/images/artists/10996', u'https://api.buzzanglemusic.com/images/artists/5019556', u'https://api.buzzanglemusic.com/images/artists/5330407', u'https://api.buzzanglemusic.com/images/artists/4878653', u'https://api.buzzanglemusic.com/images/artists/6462708', u'https://api.buzzanglemusic.com/images/artists/7501280', u'https://api.buzzanglemusic.com/images/artists/1340', u'https://api.buzzanglemusic.com/images/artists/931', u'https://api.buzzanglemusic.com/images/artists/4034234', u'https://api.buzzanglemusic.com/images/artists/4893654', u'https://api.buzzanglemusic.com/images/artists/6218616', u'https://api.buzzanglemusic.com/images/artists/3105901', u'https://api.buzzanglemusic.com/images/artists/5764159', u'https://api.buzzanglemusic.com/images/artists/1344', u'https://api.buzzanglemusic.com/images/artists/1380', u'https://api.buzzanglemusic.com/images/artists/202', u'https://api.buzzanglemusic.com/images/artists/162', u'https://api.buzzanglemusic.com/images/artists/804921', u'https://api.buzzanglemusic.com/images/artists/1345', u'https://api.buzzanglemusic.com/images/artists/19621', u'https://api.buzzanglemusic.com/images/artists/808927', u'https://api.buzzanglemusic.com/images/artists/223', u'https://api.buzzanglemusic.com/images/artists/1514', u'https://api.buzzanglemusic.com/images/artists/7', u'https://api.buzzanglemusic.com/images/artists/21', u'https://api.buzzanglemusic.com/images/artists/590329', u'https://api.buzzanglemusic.com/images/artists/7466212', u'https://api.buzzanglemusic.com/images/artists/263', u'https://api.buzzanglemusic.com/images/artists/594558', u'https://api.buzzanglemusic.com/images/artists/7520899', u'https://api.buzzanglemusic.com/images/artists/4529544', u'https://api.buzzanglemusic.com/images/artists/4304221', u'https://api.buzzanglemusic.com/images/artists/26', u'https://api.buzzanglemusic.com/images/artists/208', u'https://api.buzzanglemusic.com/images/artists/33', u'https://api.buzzanglemusic.com/images/artists/1079180', u'https://api.buzzanglemusic.com/images/artists/3305532', u'https://api.buzzanglemusic.com/images/artists/580367', u'https://api.buzzanglemusic.com/images/artists/54', u'https://api.buzzanglemusic.com/images/artists/4694249', u'https://api.buzzanglemusic.com/images/artists/1545', u'https://api.buzzanglemusic.com/images/artists/3391309', u'https://api.buzzanglemusic.com/images/artists/489706', u'https://api.buzzanglemusic.com/images/artists/185', u'https://api.buzzanglemusic.com/images/artists/812522', u'https://api.buzzanglemusic.com/images/artists/5681847', u'https://api.buzzanglemusic.com/images/artists/3195987', u'https://api.buzzanglemusic.com/images/artists/4179629', u'https://api.buzzanglemusic.com/images/artists/5212303', u'https://api.buzzanglemusic.com/images/artists/142', u'https://api.buzzanglemusic.com/images/artists/6384381', u'https://api.buzzanglemusic.com/images/artists/812157', u'https://api.buzzanglemusic.com/images/artists/192', u'https://api.buzzanglemusic.com/images/artists/692', u'https://api.buzzanglemusic.com/images/artists/1256797', u'https://api.buzzanglemusic.com/images/artists/5984690', u'https://api.buzzanglemusic.com/images/artists/5344432', u'https://api.buzzanglemusic.com/images/artists/2381', u'https://api.buzzanglemusic.com/images/artists/1338', u'https://api.buzzanglemusic.com/images/artists/28', u'https://api.buzzanglemusic.com/images/artists/6500275', u'https://api.buzzanglemusic.com/images/artists/4754086', u'https://api.buzzanglemusic.com/images/artists/189', u'https://api.buzzanglemusic.com/images/artists/715', u'https://api.buzzanglemusic.com/images/artists/11', u'https://api.buzzanglemusic.com/images/artists/690440', u'https://api.buzzanglemusic.com/images/artists/722962', u'https://api.buzzanglemusic.com/images/artists/6478745', u'https://api.buzzanglemusic.com/images/artists/1626', u'https://api.buzzanglemusic.com/images/artists/150', u'https://api.buzzanglemusic.com/images/artists/1356', u'https://api.buzzanglemusic.com/images/artists/221', u'https://api.buzzanglemusic.com/images/artists/6216355', u'https://api.buzzanglemusic.com/images/artists/5366988', u'https://api.buzzanglemusic.com/images/artists/771', u'https://api.buzzanglemusic.com/images/artists/610595', u'https://api.buzzanglemusic.com/images/artists/247467', u'https://api.buzzanglemusic.com/images/artists/5215781', u'https://api.buzzanglemusic.com/images/artists/7640', u'https://api.buzzanglemusic.com/images/artists/621107', u'https://api.buzzanglemusic.com/images/artists/4547105', u'https://api.buzzanglemusic.com/images/artists/143', u'https://api.buzzanglemusic.com/images/artists/4827213', u'https://api.buzzanglemusic.com/images/artists/769', u'https://api.buzzanglemusic.com/images/artists/1337', u'https://api.buzzanglemusic.com/images/artists/487', u'https://api.buzzanglemusic.com/images/artists/2643', u'https://api.buzzanglemusic.com/images/artists/8957464', u'https://api.buzzanglemusic.com/images/artists/807069', u'https://api.buzzanglemusic.com/images/artists/517460', u'https://api.buzzanglemusic.com/images/artists/662', u'https://api.buzzanglemusic.com/images/artists/6227591', u'https://api.buzzanglemusic.com/images/artists/330', u'https://api.buzzanglemusic.com/images/artists/2419669', u'https://api.buzzanglemusic.com/images/artists/7299117', u'https://api.buzzanglemusic.com/images/artists/630', u'https://api.buzzanglemusic.com/images/artists/38', u'https://api.buzzanglemusic.com/images/artists/743', u'https://api.buzzanglemusic.com/images/artists/295165', u'https://api.buzzanglemusic.com/images/artists/474', u'https://api.buzzanglemusic.com/images/artists/5085902', u'https://api.buzzanglemusic.com/images/artists/5433623', u'https://api.buzzanglemusic.com/images/artists/15510', u'https://api.buzzanglemusic.com/images/artists/1390', u'https://api.buzzanglemusic.com/images/artists/596038', u'https://api.buzzanglemusic.com/images/artists/465', u'https://api.buzzanglemusic.com/images/artists/503490', u'https://api.buzzanglemusic.com/images/artists/82', u'https://api.buzzanglemusic.com/images/artists/6427308', u'https://api.buzzanglemusic.com/images/artists/199', u'https://api.buzzanglemusic.com/images/artists/7144', u'https://api.buzzanglemusic.com/images/artists/44528', u'https://api.buzzanglemusic.com/images/artists/490', u'https://api.buzzanglemusic.com/images/artists/830802', u'https://api.buzzanglemusic.com/images/artists/178', u'https://api.buzzanglemusic.com/images/artists/20931', u'https://api.buzzanglemusic.com/images/artists/78', u'https://api.buzzanglemusic.com/images/artists/2599', u'https://api.buzzanglemusic.com/images/artists/31865', u'https://api.buzzanglemusic.com/images/artists/755031', u'https://api.buzzanglemusic.com/images/artists/5009248', u'https://api.buzzanglemusic.com/images/artists/7177348', u'https://api.buzzanglemusic.com/images/artists/9046744', u'https://api.buzzanglemusic.com/images/artists/472', u'https://api.buzzanglemusic.com/images/artists/6816051', u'https://api.buzzanglemusic.com/images/artists/603486', u'https://api.buzzanglemusic.com/images/artists/9518002', u'https://api.buzzanglemusic.com/images/artists/856682', u'https://api.buzzanglemusic.com/images/artists/3883529', u'https://api.buzzanglemusic.com/images/artists/173', u'https://api.buzzanglemusic.com/images/artists/236', u'https://api.buzzanglemusic.com/images/artists/225', u'https://api.buzzanglemusic.com/images/artists/4779775', u'https://api.buzzanglemusic.com/images/artists/723917', u'https://api.buzzanglemusic.com/images/artists/1823', u'https://api.buzzanglemusic.com/images/artists/881', u'https://api.buzzanglemusic.com/images/artists/7296610', u'https://api.buzzanglemusic.com/images/artists/3587186', u'https://api.buzzanglemusic.com/images/artists/678', u'https://api.buzzanglemusic.com/images/artists/866831', u'https://api.buzzanglemusic.com/images/artists/4231', u'https://api.buzzanglemusic.com/images/artists/218', u'https://api.buzzanglemusic.com/images/artists/22', u'https://api.buzzanglemusic.com/images/artists/1339', u'https://api.buzzanglemusic.com/images/artists/1352', u'https://api.buzzanglemusic.com/images/artists/181', u'https://api.buzzanglemusic.com/images/artists/806032', u'https://api.buzzanglemusic.com/images/artists/1364', u'https://api.buzzanglemusic.com/images/artists/2611', u'https://api.buzzanglemusic.com/images/artists/828', u'https://api.buzzanglemusic.com/images/artists/885', u'https://api.buzzanglemusic.com/images/artists/36569', u'https://api.buzzanglemusic.com/images/artists/4466817', u'https://api.buzzanglemusic.com/images/artists/13', u'https://api.buzzanglemusic.com/images/artists/227', u'https://api.buzzanglemusic.com/images/artists/2506', u'https://api.buzzanglemusic.com/images/artists/730102', u'https://api.buzzanglemusic.com/images/artists/9284021', u'https://api.buzzanglemusic.com/images/artists/9371339', u'https://api.buzzanglemusic.com/images/artists/164', u'https://api.buzzanglemusic.com/images/artists/1336', u'https://api.buzzanglemusic.com/images/artists/5722286', u'https://api.buzzanglemusic.com/images/artists/4351620', u'https://api.buzzanglemusic.com/images/artists/166', u'https://api.buzzanglemusic.com/images/artists/148', u'https://api.buzzanglemusic.com/images/artists/680', u'https://api.buzzanglemusic.com/images/artists/5880', u'https://api.buzzanglemusic.com/images/artists/335779', u'https://api.buzzanglemusic.com/images/artists/184', u'https://api.buzzanglemusic.com/images/artists/4375959', u'https://api.buzzanglemusic.com/images/artists/646', u'https://api.buzzanglemusic.com/images/artists/289', u'https://api.buzzanglemusic.com/images/artists/502', u'https://api.buzzanglemusic.com/images/artists/596334', u'https://api.buzzanglemusic.com/images/artists/4786404', u'https://api.buzzanglemusic.com/images/artists/224', u'https://api.buzzanglemusic.com/images/artists/379', u'https://api.buzzanglemusic.com/images/artists/29', u'https://api.buzzanglemusic.com/images/artists/9224269', u'https://api.buzzanglemusic.com/images/artists/6221', u'https://api.buzzanglemusic.com/images/artists/590503', u'https://api.buzzanglemusic.com/images/artists/804250', u'https://api.buzzanglemusic.com/images/artists/322', u'https://api.buzzanglemusic.com/images/artists/89', u'https://api.buzzanglemusic.com/images/artists/6944241']
[u'Drake', u'Lil Baby', u'DaBaby', u'Post Malone', u'Lil Uzi Vert', u'YoungBoy Never Broke Again', u'Future', u'The Weeknd', u'Juice Wrld', u'Gunna', u'Luke Combs', u'Polo G', u'Eminem', u'Taylor Swift', u'Rod Wave', u'Travis Scott', u'Xxxtentacion', u'Bad Bunny', u'Billie Eilish', u'Roddy Ricch', u'Pop Smoke', u'Kanye West', u'The Beatles', u'Harry Styles', u'A Boogie wit da Hoodie', u'Megan Thee Stallion', u'Morgan Wallen', u'Trippie Redd', u'Lil Wayne', u'J. Cole', u'Lady Gaga', u'Chris Brown', u'Kevin Gates', u'Kendrick Lamar', u'Ariana Grande', u'Khalid', u'Ed Sheeran', u'Mac Miller', u'Jason Aldean', u'Luke Bryan', u'Jhen\xe9 Aiko', u'NLE Choppa', u'Justin Bieber', u'Lil Durk', u'Iann Dior', u'Moneybagg Yo', u'Kodak Black', u'Florida Georgia Line', u'Beyonc\xe9', u'Kenny Chesney', u'Tory Lanez', u'Doja Cat', u'Young Thug', u'Thomas Rhett', u'Dua Lipa', u'Meek Mill', u'Kehlani', u'Sam Hunt', u'Rihanna', u'BTS', u'Lil Peep', u'NF', u'Kane Brown', u'Chloe x Halle', u'Maroon 5', u'YNW Melly', u'Migos', u'Imagine Dragons', u'Queen', u'Halsey', u'Lewis Capaldi', u'$uicideboy$', u'twenty one pilots', u'Frank Ocean', u'Blake Shelton', u'Lil Mosey', u'G Herbo', u'One Direction', u'Fleetwood Mac', u'Zac Brown Band', u'Maren Morris', u'Nav', u'Don Toliver', u'Panic! at the Disco', u'Bruno Mars', u'2Pac', u'Nicki Minaj', u'6IX9INE', u'SAINt JHN', u'Eagles', u'George Strait', u'Chris Stapleton', u'Russ', u'Tyler, The Creator', u'Eric Church', u'Anuel AA', u'Katy Perry', u'Jack Harlow', u'Elton John', u'Wiz Khalifa', u'AC/DC', u'Jonas Brothers', u'Lil Baby, 42 Dugg', u'Blackbear', u'H.E.R', u'Michael Jackson', u'Lil Skies', u'Red Hot Chili Peppers', u'Bazzi', u'Lil Tjay', u'The Rolling Stones', u'Tim McGraw', u'Tom Petty And The Heartbreakers', u'Summer Walker', u'Led Zeppelin', u'Playboi Carti', u'Surfaces', u'Childish Gambino', u'Gucci Mane', u'Melanie Martinez', u'Five Finger Death Punch', u'Old Dominion', u'Miranda Lambert', u'JayDaYoungan', u'Linkin Park', u'Nipsey Hussle', u'Coldplay', u'Metallica', u'Dan + Shay', u'John Mayer', u'Sam Smith', u'Jon Pardi', u'Kid Cudi', u'Logic', u'5 Seconds of Summer', u'Lil Yachty', u'Natanael Cano', u'StaySolidRocky', u'Pink Floyd', u'Lil Tecca', u'Brett Young', u'6Ix9Ine & Nicki Minaj', u'Tyler Childers', u'Ozuna', u'Adele', u'Jack Johnson', u'Usher', u'21 Savage', u'Lizzo', u'Machine Gun Kelly', u'Bob Marley & The Wailers', u'Sleepy hallow', u'Shawn Mendes', u'Billy Joel', u'Run The Jewels', u'YG', u'Green Day', u'Carrie Underwood', u'Trey Songz', u'A$AP Rocky', u'JAY-Z', u'G-Eazy', u'Notorious B.I.G.', u'Selena Gomez', u'Fall Out Boy', u'Creedence Clearwater Revival', u'Camila Cabello', u'Joji', u'Keith Urban', u'Demi Lovato', u'Frank Sinatra', u'Chance the Rapper', u'Chris Brown & Young Thug', u'Lady Gaga, Ariana Grande', u'Lana Del Rey', u'DJ Khaled', u'Key Glock', u'Brent Faiyaz', u'The Lumineers', u'Pink', u'Journey', u'Tame Impala', u'Jordan Davis', u'OneRepublic', u'YFN Lucci', u'Lynyrd Skynyrd', u'Sublime', u'Nirvana', u'The Chainsmokers', u'Cardi B', u'Calvin Harris', u'Johnny Cash', u'Lee Brice', u'Travis Scott, The Scotts, Kid Cudi', u'Alan Jackson', u'J. Balvin', u'PARTYNEXTDOOR', u'50 Cent', u'Garth Brooks', u'Powfu']