Как найти определенное слово в нескольких веб-страницах или URL-адресах и посчитать его, используя Python

Question

Ниже мой код. Пожалуйста, проверьте и исправьте меня.

import requests

from bs4 import BeautifulSoup

url = ["https://www.tensorflow.org/","https://www.tomordonez.com/"]

the_word = input()

r = requests.get(url, allow_redirects=False)

soup = BeautifulSoup(r.content, 'lxml')

words = soup.find(text=lambda text: text and the_word in text)

print(words)

count = len(words)

print('\nUrl: {}\ncontains {} of word: {}'.format(url, count, the_word))

Как я могу изменить свой код для анализа нескольких URL-адресов и подсчета, сколько раз встречается определенное слово?

Answer 1

import requests
from bs4 import BeautifulSoup

url_list = ["https://www.tensorflow.org/","https://www.tomordonez.com/"]

#the_word = input()
the_word = 'Python'

total_words = []
for url in url_list:
    r = requests.get(url, allow_redirects=False)
    soup = BeautifulSoup(r.content.lower(), 'lxml')
    words = soup.find_all(text=lambda text: text and the_word.lower() in text)
    count = len(words)
    words_list = [ ele.strip() for ele in words ]
    for word in words:
        total_words.append(word.strip())

    print('\nUrl: {}\ncontains {} of word: {}'.format(url, count, the_word))
    print(words_list)


#print(total_words)
total_count = len(total_words)

Выход:

Url: https://www.tensorflow.org/
contains 0 of word: Python
[]

Url: https://www.tomordonez.com/
contains 8 of word: Python
['web scraping with python', 'this is a tutorial on web scraping with python. learn to scrape websites with python and beautifulsoup.', 'python unit testing tutorial', 'this is a tutorial about unit testing in python.', 'pip install ssl module in python is not available', 'troubleshooting ssl module in python is not available', 'python context manager', 'a short tutorial about python context manager: "with" statement.']

Answer 2

Вы можете использовать модуль re для поиска определенного текста.

import requests
import re
from bs4 import BeautifulSoup

urls = ["https://www.tensorflow.org/","https://www.tomordonez.com/"]

the_word ='Tableau'

for url in urls:
 print(url)
 r = requests.get(url, allow_redirects=False)
 soup = BeautifulSoup(r.text, 'html.parser')
 words = soup.find_all(text=re.compile(the_word))
 print(len(words))