Python Таблицы BeautifulSoup не обрабатываются - PullRequest
0 голосов
/ 05 мая 2020

На сайте Basketball-reference.com есть веб-страницы с данными об отдельных игроках, каждая из которых содержит несколько таблиц. Я хочу извлечь таблицу «Стрельба» (6-я таблица) для определенных c игроков. Я пробовал несколько вариантов составных find_alls, но не могу выполнить синтаксический анализ после первой таблицы. Вот пример с использованием Леброна Джеймса.

from urllib.request import urlopen
from bs4 import BeautifulSoup

url = r'https://www.basketball-reference.com/players/j/jamesle01.html'
html = urlopen(url)
soup = BeautifulSoup(html, features='lxml')

print(soup.find_all('table')[1].encode('utf-8'))
print(pd.read_html(url))

Ответы [ 2 ]

1 голос
/ 05 мая 2020

Вы должны использовать селен вместо urlopen для получения страницы, потому что таблица отображается с помощью js.

from bs4 import BeautifulSoup
from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument("headless")
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://www.basketball-reference.com/players/j/jamesle01.html')
soup = BeautifulSoup(driver.page_source,'html.parser')

s1 = soup.select('#shooting')
print(s1)
0 голосов
/ 05 мая 2020

Нет необходимости использовать Selenium. Таблица находится в комментариях. Быстрее просто вытащить комментарии и получить таблицы. Затем проанализируйте его с помощью Pandas 'read_html(), который использует beautifulsoup под капотом

import pandas as pd
import requests
from bs4 import BeautifulSoup, Comment

url = 'https://www.basketball-reference.com/players/j/jamesle01.html'

response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
comments = soup.find_all(string=lambda text: isinstance(text, Comment))

tables = ''
for c in comments:
    if '<table' in c:
        tables += c

soup = BeautifulSoup(tables, 'html.parser')
shootingTable = soup.find('table', {'id':'shooting'})
df = pd.read_html(str(shootingTable))[0]

Вывод:

print (df.to_string())
   Unnamed: 0_level_0 Unnamed: 1_level_0 Unnamed: 2_level_0 Unnamed: 3_level_0 Unnamed: 4_level_0 Unnamed: 5_level_0 Unnamed: 6_level_0 Unnamed: 7_level_0 Unnamed: 8_level_0   Unnamed: 9_level_0  Unnamed: 10_level_0  Unnamed: 11_level_0  Unnamed: 12_level_0  Unnamed: 13_level_0  Unnamed: 14_level_0 Unnamed: 15_level_0 Unnamed: 16_level_0 Unnamed: 17_level_0 Unnamed: 18_level_0 Unnamed: 19_level_0 Unnamed: 20_level_0    2-Pt Field Goals                   3-Pt Field Goals                          
   Unnamed: 0_level_1 Unnamed: 1_level_1 Unnamed: 2_level_1 Unnamed: 3_level_1 Unnamed: 4_level_1 Unnamed: 5_level_1 Unnamed: 6_level_1 Unnamed: 7_level_1 Unnamed: 8_level_1 % of FGA by Distance % of FGA by Distance % of FGA by Distance % of FGA by Distance % of FGA by Distance % of FGA by Distance     FG% by Distance     FG% by Distance     FG% by Distance     FG% by Distance     FG% by Distance     FG% by Distance Unnamed: 21_level_1  Dunks         Unnamed: 24_level_1 Corner        Heaves     
               Season                Age                 Tm                 Lg                Pos                  G                 MP                FG%              Dist.                   2P                  0-3                 3-10                10-16               16-3pt                   3P                  2P                 0-3                3-10               10-16              16-3pt                  3P              %Ast'd   %FGA     Md.              %Ast'd   %3PA    3P%   Att.  Md.
0             2003-04               19.0                CLE                NBA                 SG               79.0             3122.0              0.417               11.2                0.855                0.315                0.168                0.161                0.211                0.145               0.438               0.604               0.356               0.313               0.352               0.290               0.442  0.064    91.0               0.746  0.286  0.323    3.0  0.0
1             2004-05               20.0                CLE                NBA                 SF               80.0             3388.0              0.472               12.1                0.817                0.311                0.137                0.121                0.248                0.183               0.499               0.725               0.370               0.324               0.373               0.351               0.445  0.076   126.0               0.500  0.185  0.491    0.0  0.0
2             2005-06               21.0                CLE                NBA                 SF               79.0             3361.0              0.480               12.4                0.792                0.353                0.092                0.094                0.253                0.208               0.518               0.689               0.411               0.304               0.398               0.335               0.303  0.065   114.0               0.417  0.185  0.343    5.0  0.0
3             2006-07               22.0                CLE                NBA                 SF               78.0             3190.0              0.476               12.2                0.809                0.333                0.125                0.111                0.241                0.191               0.513               0.725               0.406               0.361               0.346               0.319               0.321  0.068   108.0               0.495  0.145  0.444    5.0  1.0
4             2007-08               23.0                CLE                NBA                 SF               75.0             3027.0              0.484               12.2                0.781                0.373                0.090                0.080                0.238                0.219               0.531               0.706               0.426               0.303               0.372               0.315               0.333  0.069   109.0               0.398  0.095  0.353    2.0  0.0
5             2008-09               24.0                CLE                NBA                 SF               81.0             3054.0              0.489               13.4                0.762                0.333                0.092                0.074                0.263                0.238               0.535               0.728               0.453               0.286               0.388               0.344               0.318  0.083   128.0               0.424  0.107  0.341    7.0  0.0
6             2009-10               25.0                CLE                NBA                 SF               76.0             2966.0              0.503               13.4                0.747                0.342                0.082                0.061                0.262                0.253               0.560               0.745               0.468               0.344               0.398               0.333               0.369  0.080   117.0               0.341  0.111  0.395    4.0  0.0
7             2010-11               26.0                MIA                NBA                 SF               79.0             3063.0              0.510               12.7                0.812                0.284                0.172                0.094                0.262                0.188               0.552               0.744               0.459               0.443               0.445               0.330               0.284  0.068    98.0               0.609  0.211  0.322    5.0  1.0
8             2011-12               27.0                MIA                NBA                 SF               62.0             2326.0              0.531               11.5                0.873                0.329                0.136                0.135                0.272                0.127               0.556               0.758               0.484               0.481               0.384               0.362               0.360  0.092   104.0               0.519  0.121  0.333    0.0  0.0
9             2012-13               28.0                MIA                NBA                 PF               76.0             2877.0              0.565               11.2                0.812                0.373                0.133                0.105                0.202                0.188               0.602               0.776               0.494               0.415               0.447               0.406               0.378  0.109   144.0               0.553  0.161  0.488    0.0  0.0
10            2013-14               29.0                MIA                NBA                 PF               77.0             2902.0              0.567               11.4                0.774                0.399                0.123                0.084                0.168                0.226               0.622               0.796               0.554               0.404               0.366               0.379               0.395  0.101   134.0               0.534  0.144  0.568    0.0  0.0
11            2014-15               30.0                CLE                NBA                 SF               69.0             2493.0              0.488               12.6                0.735                0.332                0.127                0.116                0.159                0.265               0.536               0.722               0.414               0.356               0.377               0.354               0.315  0.072    88.0               0.467  0.115  0.385    1.0  0.0
12            2015-16               31.0                CLE                NBA                 SF               76.0             2709.0              0.520                9.6                0.801                0.459                0.119                0.093                0.129                0.199               0.573               0.722               0.361               0.348               0.404               0.309               0.378  0.083   111.0               0.598  0.117  0.273    0.0  0.0
13            2016-17               32.0                CLE                NBA                 SF               74.0             2794.0              0.548               11.0                0.746                0.432                0.106                0.105                0.102                0.254               0.611               0.780               0.385               0.440               0.307               0.363               0.355  0.112   145.0               0.492  0.102  0.371    0.0  0.0
14            2017-18               33.0                CLE                NBA                 PF               82.0             3026.0              0.542               10.9                0.743                0.423                0.114                0.089                0.118                0.257               0.603               0.766               0.378               0.421               0.371               0.367               0.301  0.080   120.0               0.356  0.084  0.382    1.0  0.0
15            2018-19               34.0                LAL                NBA                 SF               55.0             1937.0              0.510               12.3                0.701                0.380                0.132                0.077                0.112                0.299               0.582               0.755               0.297               0.417               0.447               0.339               0.340  0.081    85.0               0.342  0.073  0.292    0.0  0.0
16            2019-20               35.0                LAL                NBA                 PG               60.0             2094.0              0.498               12.8                0.676                0.366                0.139                0.072                0.099                0.324               0.570               0.728               0.423               0.341               0.359               0.349               0.247  0.066    76.0               0.459  0.055  0.429    1.0  0.0
17             Career                NaN                NaN                NBA                NaN             1258.0            48329.0              0.504               12.0                0.781                0.360                0.121                0.098                0.201                0.219               0.549               0.734               0.418               0.368               0.386               0.344               0.347  0.080  1898.0               0.469  0.129  0.387   34.0  2.0
18                NaN                NaN                NaN                NaN                NaN                NaN                NaN                NaN                NaN                  NaN                  NaN                  NaN                  NaN                  NaN                  NaN                 NaN                 NaN                 NaN                 NaN                 NaN                 NaN                 NaN    NaN     NaN                 NaN    NaN    NaN    NaN  NaN
19         11 seasons                NaN                CLE                NBA                NaN              849.0            33130.0              0.492               11.9                0.782                0.363                0.113                0.100                0.206                0.218               0.535               0.721               0.397               0.343               0.375               0.337               0.352  0.077  1257.0               0.456  0.133  0.375   28.0  1.0
20          4 seasons                NaN                MIA                NBA                NaN              294.0            11168.0              0.543               11.8                0.816                0.345                0.142                0.103                0.225                0.184               0.582               0.771               0.493               0.439               0.414               0.369               0.353  0.092   480.0               0.556  0.164  0.432    5.0  1.0
21          2 seasons                NaN                LAL                NBA                NaN              115.0             4031.0              0.504               12.6                0.688                0.373                0.136                0.074                0.106                0.312               0.576               0.741               0.364               0.379               0.404               0.345               0.293  0.074   161.0               0.406  0.064  0.356    1.0  0.0
...