Получение списка из результатов поиска с веб-сайта с использованием Beautifulsoup и скрыть в dataframe - PullRequest
Новая
       5

Получение списка из результатов поиска с веб-сайта с использованием Beautifulsoup и скрыть в dataframe

1 голос
/ 27 марта 2020

Я хочу получить результат с сайта и преобразовать его в датафрейм. Проблема в том, что на каждой странице 10 результатов, а остальные можно получить с помощью нумерации страниц, но я борюсь там. Я хочу получить всю информацию, доступную в 1 результате, например: имя университета, URL-ссылка, стоимость, изображение, язык и дата и т. Д. c. 

    import requests
    from bs4 import BeautifulSoup


    getpage= requests.get(url)

    getpage_soup= BeautifulSoup(getpage.text, 'html.parser')
    url =
    all_links= getpage_soup.findAll('a')

    for link in all_links:
        link = link.get('href')
        print (link)

как желаемый вывод, я хотел бы есть датафрейм 

URL      University    Cost    Date    Image    Language

1 Ответ

1 голос
/ 27 марта 2020

Все есть в ответе json через API. Просто нужно нормализовать, чтобы преобразовать из json в фрейм данных: 

import requests
from pandas.io.json import json_normalize

url = 'https://www2.daad.de/deutschland/studienangebote/international-programmes/api/solr/en/search.json'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'}
payload = {
'cert':'', 
'admReq': '',
'scholarshipLC':'', 
'scholarshipSC': '',
'fos': '',
'langDeAvailable':'', 
'langEnAvailable': '',
'sort': '4',
'q': '',
'limit': '9999',
'offset': '',
'display': 'list',
'isSep': ''}

jsonData = requests.get(url, headers=headers, params=payload).json()

df = json_normalize(jsonData['courses'])

cols = ['link', 'academy', 'costString', 'tuitionFees', 'date', 'image', 'languages' ]
df_subset = df[cols]

Вывод:

Вот все: 

print (df.head().to_string())
                                     academy applicationDeadline beginning                    city costString                                         courseName                                    courseNameShort  courseType                                               date                                   dateString financialSupport    id                                              image  isCompleteOnlinePossible  isElearning languageLevelEnglish       languageLevelGerman  languages                                               link preparationForDegree programmeDuration requestLanguage structuredResearch                                            subject supportInternationalStudents tuitionFees typeOfElearning
0                      University of Potsdam                None      None                 Potsdam       None  22nd International Summer Academy Sans Souci 2020  22nd International Summer Academy Sans Souci 2020           5  [{'start': '2020-08-03', 'end': '2020-08-28', ...  The date of the course has not yet been set             None  4993  /app/ip/media/cache/resolve/16_9_300px/uploads...                     False        False                 None          [B1, B2, C1, C2]   [German]  /deutschland/studienangebote/international-pro...                 None              None              en               None  German Language and Literature (German, German...                         None        None              []
1   Osnabrück University of Applied Sciences                None      None               Osnabrück       None          22nd International Winter Language School               International Winter Language School           5  [{'start': '2020-02-06', 'end': '2020-02-26', ...  The date of the course has not yet been set             None  5003  /app/ip/media/cache/resolve/16_9_300px/uploads...                     False        False                 None  [A1, A2, B1, B2, C1, C2]   [German]  /deutschland/studienangebote/international-pro...                 None              None              en               None  German Language Course (including Literature a...                         None        None              []
2  Martin Luther University Halle-Wittenberg                None      None  Lutherstadt Wittenberg       None                  24th International Summer Academy                  24th International Summer Academy           5  [{'start': '2020-08-09', 'end': '2020-09-03', ...  The date of the course has not yet been set             None  4920                                               None                     False        False                 None              [B1, B2, C1]   [German]  /deutschland/studienangebote/international-pro...                 None              None              en               None  German Language and Literature (German, German...                         None        None              []
3   Osnabrück University of Applied Sciences                None      None               Osnabrück       None          28th International Summer Language School               International Summer Language School           5  [{'start': '2020-08-17', 'end': '2020-09-14', ...  The date of the course has not yet been set             None  4940  /app/ip/media/cache/resolve/16_9_300px/uploads...                     False        False                 None  [A1, A2, B1, B2, C1, C2]   [German]  /deutschland/studienangebote/international-pro...                 None              None              en               None  German Language Course (including Literature a...                         None        None              []
4              Technische Universität Berlin                None      None                  Berlin       None                           3D Scanning and Printing                           3D Scanning and Printing           6                                  [{'costs': 2300}]  The date of the course has not yet been set             None  5119                                               None                     False        False                 [B2]                      None  [English]  /deutschland/studienangebote/international-pro...                 None              None              en               None                             Engineering in general                         None        None              []
....

Или просто Speci c столбцы: 

print (df_subset.head(20).to_string())
                                                 link                                    academy costString tuitionFees                                               date                                              image          languages
0   /deutschland/studienangebote/international-pro...                      University of Potsdam       None        None  [{'start': '2020-08-03', 'end': '2020-08-28', ...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
1   /deutschland/studienangebote/international-pro...   Osnabrück University of Applied Sciences       None        None  [{'start': '2020-02-06', 'end': '2020-02-26', ...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
2   /deutschland/studienangebote/international-pro...  Martin Luther University Halle-Wittenberg       None        None  [{'start': '2020-08-09', 'end': '2020-09-03', ...                                               None           [German]
3   /deutschland/studienangebote/international-pro...   Osnabrück University of Applied Sciences       None        None  [{'start': '2020-08-17', 'end': '2020-09-14', ...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
4   /deutschland/studienangebote/international-pro...              Technische Universität Berlin       None        None                                  [{'costs': 2300}]                                               None          [English]
5   /deutschland/studienangebote/international-pro...     Otto von Guericke University Magdeburg       None        none  [{'start': '2020-08-03', 'end': '2020-08-21', ...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
6   /deutschland/studienangebote/international-pro...             Technische Universität Ilmenau       None        None  [{'start': '2020-08-03', 'end': '2020-08-26', ...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
7   /deutschland/studienangebote/international-pro...             Technische Universität Ilmenau       None        None  [{'start': '2020-08-03', 'end': '2020-08-26', ...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
8   /deutschland/studienangebote/international-pro...             Technische Universität Ilmenau       None        None  [{'start': '2020-09-07', 'end': '2020-09-30', ...                                               None           [German]
9   /deutschland/studienangebote/international-pro...                           Trier University       None        None  [{'start': '2020-08-03', 'end': '2020-08-28', ...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
10  /deutschland/studienangebote/international-pro...  SRH Berlin University of Applied Sciences       None        None  [{'start': '2020-02-03', 'end': '2020-02-16', ...                                               None          [English]
11  /deutschland/studienangebote/international-pro...                            Kiel University       None        None  [{'start': '2020-07-20', 'end': '2020-08-14', ...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
12  /deutschland/studienangebote/international-pro...    Julius-Maximilians-Universität Würzburg       None        None  [{'start': '2020-09-21', 'end': '2020-09-25', ...                                               None          [English]
13  /deutschland/studienangebote/international-pro...                       University of Bremen       None        None  [{'start': '2020-09', 'end': '2021-08', 'regis...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
14  /deutschland/studienangebote/international-pro...  Martin Luther University Halle-Wittenberg       None        None  [{'start': '2020-07-05', 'end': '2020-07-30', ...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
15  /deutschland/studienangebote/international-pro...                      University of Cologne       None        none                                                 []  /app/ip/media/cache/resolve/16_9_300px/uploads...  [German, English]
16  /deutschland/studienangebote/international-pro...   FH Aachen University of Applied Sciences       None        None  [{'start': '2020-01-06', 'end': '2020-02-14', ...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
17  /deutschland/studienangebote/international-pro...         Friedrich Schiller University Jena       None        none                                                 []                                               None  [German, English]
18  /deutschland/studienangebote/international-pro...                         Leipzig University       None        None  [{'start': '2020-08-31', 'end': '2020-09-25', ...  /app/ip/media/cache/resolve/16_9_300px/uploads...           [German]
19  /deutschland/studienangebote/international-pro...   Heilbronn University of Applied Sciences       None        None           [{'start': '2020-09', 'end': '2021-06'}]                                               None  [German, English]
....
Добро пожаловать на сайт PullRequest, где вы можете задавать вопросы и получать ответы от других членов сообщества.

Нет похожих вопросов

...