невозможно l oop сверх значения href при просмотре веб-страниц

Question

Я очень плохо знаком с веб-скребком. Мне нужно удалить ссылки тегов привязки определенного раздела на веб-странице, но, к сожалению, мне не хватает чего-то, что я не смог найти. печатает только одну ссылку.

import requests
from bs4 import BeautifulSoup

page  = requests.get('https://www.privacy.gov.ph/memorandum-circulars/')
prefix = 'https://www.privacy.gov.ph' 
soup = BeautifulSoup(page.content,'html.parser')

container = soup.findAll("section", {"class": "news_content"})
for circulars in container:

     pdf =  prefix + circulars.div.a['href'].replace("..", "")
     print(pdf)

Answer 1

Попробуйте код ниже

import requests
from bs4 import BeautifulSoup

page  = requests.get('https://www.privacy.gov.ph/memorandum-circulars/')
prefix = 'https://www.privacy.gov.ph' 
soup = BeautifulSoup(page.content,'html.parser')

container = soup.findAll("section", {"class": "news_content"})
for circulars in container:
    for a in circulars.findAll('a', href=True):
         pdf =  prefix + a['href'].replace("..", "") if prefix not in a['href'] else a['href']
         print(pdf)

Выход

https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-01-security-of-personal-data-in-government-agencies/
https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-02-data-sharing-agreements-involving-government-agencies/
https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-03-personal-data-breach-management/
https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-04-rules-of-procedure/
https://www.privacy.gov.ph/npc-circular-17-01-registration-data-processing-notifications-regarding-automated-decision-making/
https://www.privacy.gov.ph/wp-content/uploads/2017/08/NPC17-01_Appendix-1.pdf
https://www.privacy.gov.ph/npc-circular-no-18-01-rules-of-procedure-on-requests-for-advisory-opinions/
https://www.privacy.gov.ph/npc-circular-no-18-02-guidelines-on-compliance-checks/
https://www.privacy.gov.ph/npc-circular-no-18-03-rules-on-mediation-before-the-national-privacy-commission

Answer 2

Это потому, что вы сделали find_all в разделе, но есть только один раздел.

Вы можете сделать это:

КОД:

import requests
from bs4 import BeautifulSoup

page  = requests.get('https://www.privacy.gov.ph/memorandum-circulars/')
prefix = 'https://www.privacy.gov.ph' 
soup = BeautifulSoup(page.content,'html.parser')

section = soup.find("section", {"class": "news_content"})
for link in section.find_all("a"):
     pdf =  prefix + link['href'].replace(prefix,"").replace("..", "")
     print(pdf)

РЕЗУЛЬТАТЫ:

https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-01-security-of-personal-data-in-government-agencies/
https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-02-data-sharing-agreements-involving-government-agencies/
https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-03-personal-data-breach-management/
https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-04-rules-of-procedure/
https://www.privacy.gov.ph/npc-circular-17-01-registration-data-processing-notifications-regarding-automated-decision-making/
https://www.privacy.gov.ph/wp-content/uploads/2017/08/NPC17-01_Appendix-1.pdf
https://www.privacy.gov.ph/npc-circular-no-18-01-rules-of-procedure-on-requests-for-advisory-opinions/
https://www.privacy.gov.ph/npc-circular-no-18-02-guidelines-on-compliance-checks/
https://www.privacy.gov.ph/npc-circular-no-18-03-rules-on-mediation-before-the-national-privacy-commission

Answer 3

Попробуйте решение simpleified_do c, которое поможет вам превратить относительные пути в полные пути.

import requests
from simplified_scrapy.simplified_doc import SimplifiedDoc
url = 'https://www.privacy.gov.ph/memorandum-circulars/'
page  = requests.get(url)
doc = SimplifiedDoc(page.text)
container = doc.getElement('section',attr='class',value='news_content')
lstA = container.listA(url=url)
print ([a.url for a in lstA])

Результат:

['https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-01-security-of-personal-data-in-government-agencies/', 'https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-02-data-sharing-agreements-involving-government-agencies/', 'https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-03-personal-data-breach-management/', 'https://www.privacy.gov.ph/memorandum-circulars/npc-circular-16-04-rules-of-procedure/', 'https://www.privacy.gov.ph/npc-circular-17-01-registration-data-processing-notifications-regarding-automated-decision-making/', 'https://www.privacy.gov.ph/wp-content/uploads/2017/08/npc17-01_appendix-1.pdf', 'https://www.privacy.gov.ph/npc-circular-no-18-01-rules-of-procedure-on-requests-for-advisory-opinions/', 'https://www.privacy.gov.ph/npc-circular-no-18-02-guidelines-on-compliance-checks/', 'https://www.privacy.gov.ph/npc-circular-no-18-03-rules-on-mediation-before-the-national-privacy-commission']

Вы можете получить примеры SimplifiedDo c здесь