Веб-статьи из WSJ, использующие Beautifulsoup в Python 3.7?

Question

Я пытаюсь удалить статьи из Wall Street Journal, используя Beautifulsoup в Python.Однако код, который я выполняю, выполняется без каких-либо ошибок (код выхода 0), но без результатов.Я не понимаю, что происходит?Почему этот код не дает ожидаемых результатов.

Я даже оплатил подписку.

Я знаю, что что-то не так, но не могу найти проблему.

import time

import requests

from bs4 import BeautifulSoup

url = 'https://www.wsj.com/search/term.html?KEYWORDS=cybersecurity&min-date=2018/04/01&max-date=2019/03/31' \
  '&isAdvanced=true&daysback=90d&andor=AND&sort=date-desc&source=wsjarticle,wsjpro&page={}'

pages = 32
for page in range(1, pages+1):
    res = requests.get(url.format(page))
    soup = BeautifulSoup(res.text,"lxml")
    for item in soup.select(".items.hedSumm li > a"):
        resp = requests.get(item.get("href"))
        _href = item.get("href")

        try:
            resp = requests.get(_href)
        except Exception as e:
            try:
            resp = requests.get("https://www.wsj.com" + _href)
        except Exception as e:
            continue
    sauce = BeautifulSoup(resp.text,"lxml")
    date = sauce.select("time.timestamp.article__timestamp.flexbox__flex--1")
    date = date[0].text
    tag = sauce.select("li.article-breadCrumb span").text
    title = sauce.select_one("h1.wsj-article-headline").text
    content = [elem.text for elem in sauce.select("p.article-content")]
    print(f'{date}\n {tag}\n {title}\n {content}\n')

    time.sleep(3)

Как я уже писал в коде, я пытаюсь удалить дату, заголовок, тег и содержание всех статей.Было бы полезно, если бы я мог получить предложения о своих ошибках, что я должен сделать, чтобы получить желаемые результаты.

Answer 1

Замените ваш код:

resp = requests.get(item.get("href"))

Кому:

_href = item.get("href")

try:
    resp = requests.get(_href)
except Exception as e:
    try:
        resp = requests.get("https://www.wsj.com"+_href)
    except Exception as e:
        continue

Поскольку большая часть item.get("href") не предоставляет правильную ссылку на веб-сайт, например, вы получаете такую ​​ссылку.

/news/types/national-security
/public/page/news-financial-markets-stock.html
https://www.wsj.com/news/world

Только https://www.wsj.com/news/world является действительным URL сайта. поэтому вам нужно объединить base URL с _href.

Обновление

import time
import requests
from bs4 import BeautifulSoup
from bs4.element import Tag

url = 'https://www.wsj.com/search/term.html?KEYWORDS=cybersecurity&min-date=2018/04/01&max-date=2019/03/31' \
  '&isAdvanced=true&daysback=90d&andor=AND&sort=date-desc&source=wsjarticle,wsjpro&page={}'

pages = 32

for page in range(1, pages+1):
    res = requests.get(url.format(page))
    soup = BeautifulSoup(res.text,"lxml")

    for item in soup.find_all("a",{"class":"headline-image"},href=True):
        _href = item.get("href")
        try:
            resp = requests.get(_href)
        except Exception as e:
            try:
                resp = requests.get("https://www.wsj.com"+_href)
            except Exception as e:
                continue

        sauce = BeautifulSoup(resp.text,"lxml")
        dateTag = sauce.find("time",{"class":"timestamp article__timestamp flexbox__flex--1"})
        tag = sauce.find("li",{"class":"article-breadCrumb"})
        titleTag = sauce.find("h1",{"class":"wsj-article-headline"})
        contentTag = sauce.find("div",{"class":"wsj-snippet-body"})

        date = None
        tagName = None
        title = None
        content = None

        if isinstance(dateTag,Tag):
            date = dateTag.get_text().strip()

        if isinstance(tag,Tag):
            tagName = tag.get_text().strip()

        if isinstance(titleTag,Tag):
            title = titleTag.get_text().strip()

        if isinstance(contentTag,Tag):
            content = contentTag.get_text().strip()

        print(f'{date}\n {tagName}\n {title}\n {content}\n')
        time.sleep(3)

O / P: * * тысяча двадцать-одна

March 31, 2019 10:00 a.m. ET
 Tech
 Care.com Removes Tens of Thousands of Unverified Listings
 The online child-care marketplace Care.com scrubbed its site of tens of thousands of unverified day-care center listings just before a Wall Street Journal investigation published March 8, an analysis shows. Care.com, the largest site in the U.S. for finding caregivers, removed about 72% of day-care centers, or about 46,594 businesses, listed on its site, a Journal review of the website shows. Those businesses were listed on the site as recently as March 1....

Updated March 29, 2019 6:08 p.m. ET
 Politics
 FBI, Retooling Once Again, Sets Sights on Expanding Cyber Threats
 The FBI has launched its biggest transformation since the 2001 terror attacks to retrain and refocus special agents to combat cyber criminals, whose threats to lives, property and critical infrastructure have outstripped U.S. efforts to thwart them. The push comes as federal investigators grapple with an expanding range of cyber attacks sponsored by foreign adversaries against businesses or national interests, including Russian election interference and Chinese cyber thefts from American companies, senior bureau executives...