Как пройти через всю таблицу HTML и преобразовать в данные JSON?

Question

Я работаю над python3, я уже преобразовал html-таблицу в объект json, но она не перебирает всю таблицу, а просто выводит первую строку.это мой код:

html_source= """<div><table cellspacing="0" cellpadding="4" 
rules="all" border="2" id="ctl00_ContentPlaceHolder1_GridView1" 
style="background-color:White;border-color:#3366CC;border- 
width:2px;border-style:Solid;font-size:Medium;font-weight:bold;border- 
collapse:collapse;">
<tr style="color:#CCCCFF;background-color:#003399;font-weight:bold;">
<th scope="col">AC NO</th><th scope="col">PART NO</th><th 
 scope="col">SR NO</th><th scope="col">Voter Name</th><th 
scope="col">ID CARD NO</th><th scope="col">GENDER</th><th 
scope="col">AGE</th><th scope="col">&nbsp;</th><th scope="col">&nbsp; 
</th>
</tr><tr style="color:#003399;background-color:White;">
<td>211</td><td>396</td><td>294</td><td>name 1</td><td>UVP7645302</td> 
<td>M</td><td>28</td><td><input type="button" value="Polling Station 
Address"onclick="javascript:__doPostBack(&#39;ctl00$ContentPlaceHolder1$GridView1&#39;,&#39;View Details$0&#39;)" style="width:150px;" /></td><td><input type="button" value="Family" onclick="javascript:__doPostBack(&#39;ctl00$ContentPlaceHolder1$GridView1& 
#39;,&#39;Family$0&#39;)" /></td>
</tr><th scope="col">AC NO</th><th scope="col">PART NO</th><th 
scope="col">SR NO</th><th scope="col">Voter Name</th><th 
 scope="col">ID CARD NO</th><th scope="col">GENDER</th><th 
 scope="col">AGE</th><th scope="col">&nbsp;</th><th scope="col">&nbsp; 
</th>
</tr><tr style="color:#003399;background-color:White;">
<td>211</td><td>396</td><td>295</td><td>name 2</td><td>UVP7645302</td> 
<td>M</td><td>28</td><td><input type="button" value="Polling Station>"""

soup = BeautifulSoup(html_source,'html.parser')


for table in soup.find_all('table'):
    keys = [th.get_text(strip=True)for th in table.find_all('th')]
    values = [td.get_text(strip=True)for td in table.find_all('td')]
    d = dict(zip(keys,values))
    #print(d)
    mydict =  (json.dumps(d))

empty = {k: v for k, v in d.items() if not v}
for k in empty:
del d[k]
print(json.dumps(d,ensure_ascii=False))

мой ожидаемый результат:

{"AC NO": "211", "PART NO": "396", "SR NO": "294", "Имя избирателя": "Имя 1", "ID CARD NO": "UVP7645302", "GENDER": "M", "AGE": "28"}, {"AC NO": "211"," PART NO ":" 396 "," SR NO ":" 294 "," Имя избирателя ":" name 2 "," ID CARD NO ":" UVP7645302 "," GENDER ":" M "," AGE":" 28 "}

фактический выход:

{" AC NO ":" 211 "," PART NO ":" 396 "," SR NO ": «294», «Имя избирателя»: «Имя 1», «ID CARD NO»: «UVP7645302», «GENDER»: «M», «AGE»: «28»}

Answer 1

Использование pandas библиотеки:

from bs4 import BeautifulSoup
import pandas as pd 

html_source= """<div><table cellspacing="0" cellpadding="4" 
rules="all" border="2" id="ctl00_ContentPlaceHolder1_GridView1" 
style="background-color:White;border-color:#3366CC;border- 
width:2px;border-style:Solid;font-size:Medium;font-weight:bold;border- 
collapse:collapse;">
<tr style="color:#CCCCFF;background-color:#003399;font-weight:bold;">
<th scope="col">AC NO</th><th scope="col">PART NO</th><th 
 scope="col">SR NO</th><th scope="col">Voter Name</th><th 
scope="col">ID CARD NO</th><th scope="col">GENDER</th><th 
scope="col">AGE</th><th scope="col">&nbsp;</th><th scope="col">&nbsp; 
</th>
</tr><tr style="color:#003399;background-color:White;">
<td>211</td><td>396</td><td>294</td><td>name 1</td><td>UVP7645302</td> 
<td>M</td><td>28</td><td><input type="button" value="Polling Station 
Address"onclick="javascript:__doPostBack(&#39;ctl00$ContentPlaceHolder1$GridView1&#39;,&#39;View Details$0&#39;)" style="width:150px;" /></td><td><input type="button" value="Family" onclick="javascript:__doPostBack(&#39;ctl00$ContentPlaceHolder1$GridView1& 
#39;,&#39;Family$0&#39;)" /></td>
</tr><th scope="col">AC NO</th><th scope="col">PART NO</th><th 
scope="col">SR NO</th><th scope="col">Voter Name</th><th 
 scope="col">ID CARD NO</th><th scope="col">GENDER</th><th 
 scope="col">AGE</th><th scope="col">&nbsp;</th><th scope="col">&nbsp; 
</th>
</tr><tr style="color:#003399;background-color:White;">
<td>211</td><td>396</td><td>295</td><td>name 2</td><td>UVP7645302</td> 
<td>M</td><td>28</td><td><input type="button" value="Polling Station>"""

table = pd.read_html(html_source)[0]
print(table.to_dict('records'))

O / P:

[{'AC NO': 211, 'PART NO': 396, 'SR NO': 294, 'Voter Name': 'name 1', 'ID CARD NO': 'UVP7645302', 'GENDER': 'M', 'AGE': 28, 'Unnamed: 7': nan, 'Unnamed: 8': nan}, {'AC NO': 211, 'PART NO': 396, 'SR NO': 295, 'Voter Name': 'name 2', 'ID CARD NO': 'UVP7645302', 'GENDER': 'M', 'AGE': 28, 'Unnamed: 7': nan, 'Unnamed: 8': nan}]

Если вы хотите удалить Unnamed из словаря, добавьте эту строку перед print(table.to_dict('records'))оператор

table = table.loc[:,~table.columns.str.startswith('Unnamed')]

O / P:

[{'AC NO': 211, 'PART NO': 396, 'SR NO': 294, 'Voter Name': 'name 1', 'ID CARD NO': 'UVP7645302', 'GENDER': 'M', 'AGE': 28}, {'AC NO': 211, 'PART NO': 396, 'SR NO': 295, 'Voter Name': 'name 2', 'ID CARD NO': 'UVP7645302', 'GENDER': 'M', 'AGE': 28}]