создание нескольких страниц с использованием Python

Question

У меня проблема с очисткой веб-страницы

URL-адрес увеличивается на 30, начиная с 1. Он содержит много страниц со списками средних школ в Кении.На каждой странице есть список из 30 школ.Я хочу очистить все данные с помощью следующего кода, но он дает содержание только одной страницы, которая составляет 30 школ.У меня есть строка в формате URL, но по-прежнему возвращаются данные для одной страницы.Мой код:

#IMPORTING RELEVANT PACKAGES FOR THE WORK
import requests
from bs4 import BeautifulSoup
import time

#DEFINING THE FIRST WEBPAGE
num = 1
#STRING FORMATTING THE URL TO CAPTURE DIFFRENT PAGES
url = 'https://www.kenyaplex.com/schools/?start={}&SchoolType=private-secondary-schools'.format(num)
#DEIFING THE BROWSER HEADERS SO THAT CAN WORK ON IT WITHOUT ERRORS
headers = {'User-Agent':'Mozilla'}
#GOING THROUGH ALL THE PAGES AND THE LINKS
while num < 452:
    url = 'https://www.kenyaplex.com/schools/?start={}&SchoolType=private-secondary-schools'.format(num)
    time.sleep(1)
    num += 30
    response = requests.get(url,headers)
    soup = BeautifulSoup(response.text,'html.parser')
    school_info = soup.find_all('div', attrs={'class':'c-detail'})
#EXTRACTING SPECIFIC RECORDS    
records = []
for name in school_info:
    Name_of_The_School = name.find('a').text
    Location_of_The_School = name.contents[2][2:]
    Contact_of_The_School = name.contents[4]
    Information_Link = name.find('a')['href']
#converting the records to a tuple
       records.append((Name_of_The_School,
                       Location_of_The_School,
                       Contact_of_The_School,
                       Information_Link))
#EXPORTING TO A PANDAS FILE    
import pandas as pd
df = pd.DataFrame(records, columns = ['Name of The School',
                                      'Location of The School',
                                      'Contact of The School',
                                      'Information_Link'])
df.to_csv('PRIVATE_SECONDARY.csv', index = False, encoding = 'utf-8')

Answer 1

Просто плохая логика, в каждой итерации цикла while она перезаписывает локальную переменную school_info, таким образом, то, что у вас осталось в следующем цикле for, будет последним значением в while цикл.

Я позволил себе реструктурировать его:

import requests
from bs4 import BeautifulSoup
import time
import pandas as pd

headers = {'User-Agent':'Mozilla'}

def get_url(batch):
    return 'https://www.kenyaplex.com/schools/?start={}&SchoolType=private-secondary-schools'.format(batch)

school_data = []
records = []

for batch in range(1, 453, 30):  # the scrapper saves the results per iteration
    response  = requests.get(get_url(batch), headers)
    soup = BeautifulSoup(response.text,'html.parser')
    school_info = soup.find_all('div', attrs={'class':'c-detail'})
    school_data.extend(school_info)

for name in school_data:  # further parsing and records collection
    Name_of_The_School = name.find('a').text
    Location_of_The_School = name.contents[2][2:]
    Contact_of_The_School = name.contents[4]
    Information_Link = name.find('a')['href']
    records.append((Name_of_The_School,Location_of_The_School,Contact_of_The_School,Information_Link))

    time.sleep(1)

df = pd.DataFrame(records, columns = ['Name of The School','Location of The School','Contact of The School','Information_Link'])
df.to_csv('PRIVATE_SECONDARY.csv', index = False, encoding = 'utf-8')

Answer 2

Перемещение records = [] за пределами while петля:

records = []
while num < 452:
    url = 'https://www.kenyaplex.com/schools/?start={}&SchoolType=private-secondary-schools'.format(num)
    time.sleep(1)
    num += 30
    response = requests.get(url,headers)
    soup = BeautifulSoup(response.text,'html.parser')
    school_info = soup.find_all('div', attrs={'class':'c-detail'})
    #EXTRACTING SPECIFIC RECORDS    
    for name in school_info:
        Name_of_The_School = name.find('a').text
        Location_of_The_School = name.contents[2][2:]
        Contact_of_The_School = name.contents[4]
        Information_Link = name.find('a')['href']
        #converting the records to a tuple
        records.append((Name_of_The_School,
                        Location_of_The_School,
                        Contact_of_The_School,
                        Information_Link))