Получение заголовков и ссылок на заголовки с NPR.org с помощью Beautiful Soup

Question

У меня проблемы с этим, и я не могу найти хороший способ выбрать их все. Мне нужно получать живые обновления, а также обычные заголовки. Практически каждый заголовок на сайте, выделенный жирным шрифтом. Кроме того, мне нужно получить встроенную ссылку, которая активируется при нажатии на них. У меня есть базовые c знания о HTML, и я уже кое-что пропустил, но по какой-то причине борюсь с этим. Может ли кто-нибудь провести меня через это?

При дальнейшем осмотре мне кажется, что я могу найти дочерние элементы всех классов статей?

Answer 1

Для получения правильной страницы необходимо, чтобы были определены правильные cookies=. Затем мы выбираем все <a> <h3 class="title">, чтобы получить ссылки и заголовки.

Например:

import requests
from bs4 import BeautifulSoup


url = 'https://www.npr.org/?refresh=true'

cookies = {'choiceVersion': "1", 'dateOfChoice': "1584369909889", 'trackingChoice': "true"}
soup = BeautifulSoup(requests.get(url, cookies=cookies).content, 'html.parser')

for a in soup.select('a[href]:has(h3.title)'):
    print('{:<90}{}'.format(a.h3.text, a['href']))

Печать:

Trump Told China To 'Go Ahead' With Prison Camps, Bolton Alleges In New Book              https://www.npr.org/2020/06/17/875876905/trump-told-china-to-go-ahead-with-concentration-camps-bolton-alleges-in-new-book
John Bolton Unloads On Former Boss Trump, Even If It's A Little Late Now                  https://www.npr.org/2020/06/17/879609378/john-bolton-unloads-on-former-boss-trump-even-if-its-a-little-late-now
Pompeo And China's Top Diplomat Meet In Hawaii As Relations Worsen                        https://www.npr.org/2020/06/18/879854568/pompeo-and-chinas-top-diplomat-meet-in-hawaii-as-relations-worsen
Former Atlanta Police Officer Who Shot Rayshard Brooks Charged With Felony Murder         https://www.npr.org/sections/live-updates-protests-for-racial-justice/2020/06/17/879509659/former-atlanta-police-officer-who-shot-rayshard-brooks-charged-with-felony-murde
Oakland Mayor Launches Hate Crime Investigation Into Nooses Found At Park                 https://www.npr.org/sections/live-updates-protests-for-racial-justice/2020/06/17/879758336/oakland-mayor-launches-hate-crime-investigation-into-nooses-found-at-park
Arbery Family Lawyer On Trump Meeting: 'He Doesn't Feel Like There's Systemic Racism'     https://www.npr.org/sections/live-updates-protests-for-racial-justice/2020/06/17/879682712/civil-rights-attorney-comments-on-his-meeting-with-president-trump
Tim Scott Says Dick Durbin's 'Token' Comment 'Hurts My Soul'                              https://www.npr.org/2020/06/17/879717148/tim-scott-says-dick-durbins-token-comment-hurts-my-soul
'From Here to Equality' Author Makes A Case, And A Plan, For Reparations                  https://www.npr.org/2020/06/17/879041052/william-darity-jr-discusses-reparations-racial-equality-in-his-new-book
'Hampton' No More: Man Sheds Family Name With Ties To Confederate General                 https://www.npr.org/sections/live-updates-protests-for-racial-justice/2020/06/17/879662628/hampton-no-more-man-sheds-family-name-with-ties-to-confederate-general
'Interrupt The Systems': Robin DiAngelo On 'White Fragility' And Anti-Racism              https://www.npr.org/2020/06/17/879136931/interrupt-the-systems-robin-diangelo-on-white-fragility-and-anti-racism
2020 Electoral Map Ratings: Biden Has An Edge Over Trump, With 5 Months To Go             https://www.npr.org/2020/06/17/877951588/2020-electoral-map-ratings-biden-has-an-edge-over-trump-with-5-months-to-go
Scientists Find The Biggest Soft-Shelled Egg Ever, Nicknamed 'The Thing'                  https://www.npr.org/2020/06/17/877679868/scientists-find-the-biggest-soft-shelled-egg-ever-nicknamed-the-thing
Justice Department Proposes Rolling Back Legal Protections For Online Platforms           https://www.npr.org/2020/06/17/879150136/doj-proposes-rolling-back-legal-protections-for-online-platforms
The Cameras Are Rolling On The Bold And The Beautiful                                     https://www.npr.org/sections/coronavirus-live-updates/2020/06/17/879773843/the-cameras-are-rolling-on-the-bold-and-the-beautiful
Why Now, White People?                                                                    https://www.npr.org/2020/06/16/878963732/why-now-white-people
Aunt Jemima Will Change Name, Image As Brands Confront Racial Stereotypes                 https://www.npr.org/sections/live-updates-protests-for-racial-justice/2020/06/17/879104818/acknowledging-racial-stereotype-aunt-jemima-will-change-brand-name-and-image
Northeast: Coronavirus-Related Restrictions By State                                      https://www.npr.org/2020/05/01/847331283/northeast-coronavirus-related-restrictions-by-state
South: Coronavirus-Related Restrictions By State                                          https://www.npr.org/2020/05/01/847415273/south-coronavirus-related-restrictions-by-state
West: Coronavirus-Related Restrictions By State                                           https://www.npr.org/2020/05/01/847416108/west-coronavirus-related-restrictions-by-state
Midwest: Coronavirus-Related Restrictions By State                                        https://www.npr.org/2020/06/11/847413697/midwest-coronavirus-related-restrictions-by-state
Amid Confusion About Reopening, An Expert Explains How To Assess COVID-19 Risk            https://www.npr.org/2020/06/17/879255417/amid-confusion-about-reopening-an-expert-explains-how-to-assess-covid-risk
TDC video carousel                                                                        https://www.npr.org/series/589466438/planet-money-shorts
5 Years After Charleston Church Massacre, What Have We Learned?                           https://www.npr.org/2020/06/17/878828088/5-years-after-charleston-church-massacre-what-have-we-learned
Ancient Bones Offer Clues To How Long Ago Humans Cared For The Vulnerable                 https://www.npr.org/sections/goatsandsoda/2020/06/17/878896381/ancient-bones-offer-clues-to-how-long-ago-humans-cared-for-the-vulnerable
Rita Indiana: La Monstra Returns With 'Black Sabbath Dembow'                              https://www.npr.org/2020/06/17/879316231/rita-indiana-la-monstra-returns-with-black-sabbath-dembow
Tracking The Pandemic: Are Coronavirus Cases Rising Or Falling In Your State?             https://www.npr.org/sections/health-shots/2020/03/16/816707182/map-tracking-the-spread-of-the-coronavirus-in-the-u-s
Which States Are Reopening? A State-By-State Guide                                        https://www.npr.org/2020/03/12/815200313/what-governors-are-doing-to-tackle-spreading-coronavirus

Answer 2

Вы можете использовать модуль requests для загрузки кода HTML страницы, а затем проанализировать его по тегам h3, которые, как я заметил, используются для заголовков.

Затем вы можете использовать .find(string) метод, чтобы найти такой тег HTML, и когда вы это сделаете, найдите следующий экземпляр </h3> из этого индекса кода HTML и далее.

Я не мог понять, какой и как много заголовков, которые вы хотите проанализировать, но вы можете использовать while l oop, чтобы продолжать синтаксический анализ для каждого тега h3 на странице, пока вы не сможете найти новый (метод .find() должен вернуть - 1, если строка не найдена).