Красивый суп не может извлечь ссылки

Question

Я пытаюсь извлечь ссылки с этой веб-страницы: https://search.cisco.com/search?query=iot

Используя этот код, я ничего не получаю:

# Get Html Data from webpage
html = urllib.request.urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, 'html5lib')
# Retrieve all of the anchor tags    
tags = soup('a') for tag in tags:
    print(tag.get('href'))

Я попробовал метод find_all()но была такая же проблема.

Answer 1

Похоже, что Java-скрипт рендерит на страницы. Вы можете использовать селен и красивый суп для извлечения ссылок.

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Chrome()
driver.get("https://search.cisco.com/search?query=iot&locale=enUS")
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.quit()

for a in soup.find_all('a', href=True):
    print(a['href'])

Вывод:

https://onesearch.cloudapps.cisco.com/searchpage?queryFilter=iot
/login?query=iot&locale=enUS
/login?query=iot&locale=enUS
https://secure.opinionlab.com/ccc01/o.asp?id=pGuoWfLm&static=1&custom_var=undefined%7CS%7CenUS%7Ciot%7Cundefined%7CNA
https://www.cisco.com/c/en/us/support/index.html
//www.cisco.com/en/US/support/tsd_most_requested_tools.html
https://apps.cisco.com/WOC/WOConfigUI/pages/configset/configset.jsp?flow=nextgen&createNewConfigSet=Y
http://www.cisco-servicefinder.com/ServiceFinder.aspx
http://www.cisco-servicefinder.com/WarrantyFinder.aspx
//www.cisco.com/web/siteassets/sitemap/index.html    
https://www.cisco.com/c/dam/en/us/products/collateral/se/internet-of-things/at-a-glance-c45-731471.pdf?dtid=osscdc000283
https://www.cisco.com/c/en/us/solutions/internet-of-things/overview.html?dtid=osscdc000283
https://www.cisco.com/c/en/us/solutions/internet-of-things/iot-kinetic.html?dtid=osscdc000283
https://www.cisco.com/c/m/en_us/solutions/internet-of-things/iot-system.html?dtid=osscdc000283
https://learningnetworkstore.cisco.com/internet-of-things?dtid=osscdc000283
https://connectedfutures.cisco.com/tag/internet-of-things/?dtid=osscdc000283
https://blogs.cisco.com/internet-of-things?dtid=osscdc000283
https://learningnetwork.cisco.com/community/internet_of_things?dtid=osscdc000283
https://learningnetwork.cisco.com/community/learning_center/training-catalog/internet-of-things?dtid=osscdc000283
https://blogs.cisco.com/digital/internet-of-things-at-mwc?dtid=osscdc000283
https://cwr.cisco.com/
https://engage2demand.cisco.com/LP=4213?dtid=osscdc000283
https://engage2demand.cisco.com/LP=15823?dtid=osscdc000283
https://video.cisco.com/detail/video/4121788948001/internet-of-things:-empowering-the-enterprise?dtid=osscdc000283
https://video.cisco.com/detail/video/4121788948001/internet-of-things:-empowering-the-enterprise?dtid=osscdc000283
https://video.cisco.com/detail/video/3740968721001/protecting-the-internet-of-things?dtid=osscdc000283
https://video.cisco.com/detail/video/3740968721001/protecting-the-internet-of-things?dtid=osscdc000283
https://video.cisco.com/detail/video/4657296333001/the-internet-of-things:-the-vision-and-new-directions-ahead?dtid=osscdc000283
https://video.cisco.com/detail/video/4657296333001/the-internet-of-things:-the-vision-and-new-directions-ahead?dtid=osscdc000283
/search/videos?locale=enUS&query=iot
/search/videos?locale=enUS&query=iot
https://secure.opinionlab.com/ccc01/o.asp?id=pGuoWfLm&static=1&custom_var=undefined%7CS%7CenUS%7Ciot%7Cundefined%7CNA

Answer 2

Вам не нужен селен. Лучше использовать запросы. Страница использует API, поэтому запрос от этого

import requests

body = {"query":"iot","startIndex":0,"count":10,"searchType":"CISCO","tabName":"Cisco","debugScoreExplain":"false","facets":[],"localeStr":"enUS","advSearchFields":{"allwords":"","phrase":"","words":"","noOfWords":"","occurAt":""},"sortType":"RELEVANCY","isAdvanced":"false","dynamicRelevancyId":"","accessLevel":"","breakpoint":"XS","searchProfile":"","ui":"one","searchCat":"","searchMode":"text","callId":"j5JwndwQZZ","requestId":1558540148392,"bizCtxt":"","qnaTopic":[],"appName":"CDCSearhFE","social":"false"}
r = requests.post('https://search.cisco.com/api/search', json = body).json()

for item in r['items']:
    print(item['url'])

Изменить параметры, чтобы получить больше результатов и т. Д.

Answer 3

Попробуйте следовать шаблону, указанному в документации :

for link in soup.find_all('a'):
    print(link.get('href'))