красивая таблица списков, пропущено имя ссылки

Question

Попытка удалить таблицу с BeautifulSoup, которая выглядит следующим образом:

https://www.basketball-reference.com/boxscores/202001110HOU.html

Я использовал следующий код:

import requests
response=requests.get(url,headers=headers)
soup=BeautifulSoup(response.content, 'html.parser')
columns = ['Pname','MP','FG','FGA','FG%','3P','3PA',"3P%",
           'FT','FTA','FT%','ORB','DRB','TRB','AST','STL','BLK','TOV','PF','PTS',"+/-"]
stat_table=soup.find_all('table',class_ = "sortable stats_table") 
stat_table=stat_table[0]
body=[]
for row in stat_table.find_all("tr"):
    for cell in row.find_all('td'):
        body.append(cell.text.split(' '))

stat_table:

[<table class="sortable stats_table" data-cols-to-freeze="1" id="box-MIN-game-basic"><caption>Minnesota Timberwolves (15-23) Table</caption>
 <colgroup><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/><col/></colgroup>
 <thead>
 <tr class="over_header"><th></th>
 <th aria-label="" class="over_header center" colspan="20" data-stat="header_tmp">Basic Box Score Stats</th>
 </tr>
 <tr>
 <th aria-label="Starters" class="poptip sort_default_asc center" data-stat="player" scope="col">Starters</th>
 <th aria-label="Minutes Played" class="poptip center" data-over-header="Basic Box Score Stats" data-stat="mp" data-tip="Minutes Played" scope="col">MP</th>
 <th aria-label="Field Goals" class="poptip center" data-over-header="Basic Box Score Stats" data-stat="fg" data-tip="Field Goals" scope="col">FG</th>
 <th aria-label="Field Goal Attempts" class="poptip center" data-over-header="Basic Box Score Stats" data-stat="fga" data-tip="Field Goal Attempts" scope="col">FGA</th>
 <th aria-label="Field Goal Percentage" class="poptip center" data-over-header="Basic Box Score Stats" data-stat="fg_pct" data-tip="Field Goal Percentage" scope="col">FG%</th>
 <th aria-label="3-Point Field Goals" class="poptip center" data-over-header="Basic Box Score Stats" data-stat="fg3" data-tip="3-Point Field Goals" scope="col">3P</th>

вывод начинается только с MP и все имена, я полагаю, поскольку они являются ссылками, теряются. Как я могу это исправить?

Answer 1

Проблема в том, что имена находятся внутри <th> тегов, а цифры c данных внутри <td> тегов. Одним из решений является использование селектора CSS для выбора тегов <th> и <td>:

import requests
from bs4 import BeautifulSoup

url = "https://www.basketball-reference.com/boxscores/202001110HOU.html"

soup = BeautifulSoup(requests.get(url).content, 'html.parser')

data = []
for tr in soup.select('#box-MIN-game-basic tr'):
    if tr.find('td'):
        row = [td.get_text(strip=True) for td in tr.select('th, td')] # <-- select both names and numeric data
        data.append(row)

# print it:
print(('{:<20} ' + ' '.join(['{:<5}'] * 20)).format('Starters', 'MP', 'FG', 'FGA', 'FG%', '3P', '3PA', '3P%', 'FT', 'FTA', 'FT%', 'ORB', 'DRB', 'TRB', 'AST', 'STL', 'BLK', 'TOV', 'PF', 'PTS', '+/-'))
for row in data:
    print(('{:<20} ' + ' '.join(['{:<5}'] * 20)).format(*row))

Печать:

Starters             MP    FG    FGA   FG%   3P    3PA   3P%   FT    FTA   FT%   ORB   DRB   TRB   AST   STL   BLK   TOV   PF    PTS   +/-  
Jarrett Culver       33:08 6     14    .429  3     6     .500  0     0           2     6     8     5     0     0     4     0     15    -9   
Robert Covington     26:57 4     10    .400  1     4     .250  2     5     .400  2     5     7     2     6     2     0     1     11    -23  
Andrew Wiggins       24:37 3     13    .231  1     7     .143  4     4     1.000 1     2     3     1     0     0     2     1     11    -25  
Shabazz Napier       20:27 1     7     .143  0     4     .000  1     2     .500  0     2     2     3     0     0     4     0     3     -17  
Gorgui Dieng         18:41 5     7     .714  4     5     .800  0     0           1     5     6     3     1     0     2     4     14    -12  
Josh Okogie          30:23 5     11    .455  2     7     .286  4     5     .800  2     3     5     5     4     1     1     2     16    -2   
Jeff Teague          21:38 2     9     .222  0     3     .000  1     2     .500  2     0     2     4     0     0     4     3     5     -26  
Keita Bates-Diop     19:03 2     5     .400  1     3     .333  6     8     .750  1     1     2     1     0     0     0     1     11    -9   
Naz Reid             17:34 5     14    .357  1     8     .125  3     4     .750  1     1     2     1     0     1     0     2     14    -8   
Treveon Graham       13:47 1     3     .333  0     1     .000  2     2     1.000 0     1     1     2     1     0     1     3     4     -11  
Jordan Bell          7:50  1     1     1.000 0     0           1     4     .250  0     2     2     1     0     0     0     0     3     +8   
Noah Vonleh          5:55  1     1     1.000 0     0           0     0           1     3     4     0     0     0     0     1     2     -16  
Team Totals          240   36    95    .379  13    48    .271  24    36    .667  13    31    44    28    12    4     18    18    109        

Answer 2

import pandas as pd

df = pd.read_html(
    "https://www.basketball-reference.com/boxscores/202001110HOU.html")[0]

df.to_csv("out.csv", index=False, header=False)

Вывод: Просмотр онлайн

Для второй таблицы:

import pandas as pd

df = pd.read_html(
    "https://widgets.sports-reference.com/wg.fcgi?css=1&site=bbr&url=%2Fboxscores%2F202001110HOU.html&div=div_box-MIN-game-advanced")[0]

df.to_csv("out.csv", index=False, header=False)

Вывод: просмотр онлайн

Answer 3

Вы можете ошибаться. Пожалуйста, используйте ID, чтобы найти. Вот решение, использующее упрощенный c.

import requests
from simplified_scrapy.simplified_doc import SimplifiedDoc
url = 'https://www.basketball-reference.com/boxscores/202001110HOU.html'
response=requests.get(url)
columns = ['Pname','MP','FG','FGA','FG%','3P','3PA',"3P%",
           'FT','FTA','FT%','ORB','DRB','TRB','AST','STL','BLK','TOV','PF','PTS',"+/-"]

doc = SimplifiedDoc(response.text)
stat_table = doc.getElement('table',attr='id',value='box-MIN-game-advanced')
stat_table = stat_table.tbody
trs = stat_table.trs.notContains('thead',attr='class')
for tr in trs:
  tds = tr.children
  colValues = [td.text for td in tds]
  print (colValues)
  # If you need the link address, 
  href = tds[0].a.href
  # If you want the full path, 
  href = doc.absoluteUrl(url,tds[0].a.href)

Результат:

['Jarrett Culver', '33:08', '.536', '.536', '.429', '.000', '4.9', '20.2', '11.4', '26.5', '0.0', '0.0', '22.2', '20.2', '91', '129']
['Robert Covington', '26:57', '.451', '.450', '.400', '.500', '6.0', '20.7', '12.2', '12.3', '9.5', '7.1', '0.0', '16.9', '103', '103']
['Andrew Wiggins', '24:37', '.373', '.269', '.538', '.308', '3.3', '9.1', '5.7', '6.5', '0.0', '0.0', '11.9', '25.4', '75', '132']
['Shabazz Napier', '20:27', '.190', '.143', '.571', '.286', '0.0', '10.9', '4.6', '20.9', '0.0', '0.0', '33.7', '21.6', '39', '131']
['Gorgui Dieng', '18:41', '1.000', '1.000', '.714', '.000', '4.4', '29.9', '15.1', '33.3', '2.3', '0.0', '22.2', '17.9', '134', '121']
['Josh Okogie', '30:23', '.606', '.545', '.636', '.455', '5.4', '11.0', '7.7', '28.1', '5.6', '3.2', '7.0', '17.4', '128', '116']
['Jeff Teague', '21:38', '.253', '.222', '.333', '.222', '7.5', '0.0', '4.4', '28.1', '0.0', '0.0', '28.8', '23.9', '58', '133']
['Keita Bates-Diop', '19:03', '.646', '.500', '.600', '1.600', '4.3', '5.9', '4.9', '8.1', '0.0', '0.0', '0.0', '16.7', '143', '132']
['Naz Reid', '17:34', '.444', '.393', '.571', '.286', '4.6', '6.4', '5.4', '12.2', '0.0', '5.5', '0.0', '33.4', '97', '129']
['Treveon Graham', '13:47', '.515', '.333', '.333', '.667', '0.0', '8.1', '3.4', '21.4', '3.1', '0.0', '20.5', '13.2', '100', '124']
['Jordan Bell', '7:50', '.543', '1.000', '.000', '4.000', '0.0', '28.5', '12.0', '20.5', '0.0', '0.0', '0.0', '13.1', '110', '127']
['Noah Vonleh', '5:55', '1.000', '1.000', '.000', '.000', '13.8', '56.6', '31.8', '0.0', '0.0', '0.0', '0.0', '6.3', '208', '120']