Как получить значения из вложенных таблиц, используя Beautifulsoup

Question

Мне нужно получить имя и цену каждой строки в примере html ниже, однако, когда я использую beatifulsoup для find_all('tr'), он возвращает все tr основной таблицы и вложенных таблиц. Каков наилучший способ извлечения только значения и цены каждой строки?

soup = BeautifulSoup(f, 'html.parser')
priceTable = soup.find('table', attrs={"class":"table table-hover table-responsive"})

Выше есть то, что у меня есть, и оно возвращает "все" tr, включая вложенные таблицы.

Мне нужно получить все названия и цену каждого предмета перед ним и, наконец, сохранить их в CSV-файле

 <table class="table table-hover table-responsive">
                                        <tbody><tr>
                    <td style="vertical-align: middle; width: 20%;" class="hidden-xs">
                                                    <img class="retailer-logo" data-placement="right" src="/images/20180813125BhYNMEK8lgOpXj3zxze53WmqeRWov7h.jpg" alt="Contact Energy" style="width:150px;" title="" data-original-title="" />
                                            </td>
                    <td style="vertical-align: middle; width: 75px;" class="hidden-xs">
                        <img src="/images/result-arrow.png" />
                    </td>
                    <td>
                        <table style="width: 100%;">
                            <tbody><tr class="visible-xs">
                                <td class="text-center" colspan="2">
  <img class="retailer-logo" data-placement="right" src="/images/20180813125BhYNMEK8lgOpXj3zxze53WmqeRWov7h.jpg" alt="Contact Energy" style="width:150px;" title="" data-original-title="" />
                                  </td>
                            </tr>
                            <tr>
                                <td colspan="3"><h4>Contact Energy Saver Plus</h4></td>
                            </tr>
                            <tr style="text-transform: uppercase">
                                <td width="150px">Electricity:</td>
                                <td>$242.85                                    <a class="plan-breakdown" data-placement="right" title="" data-original-title="&lt;table&gt;&lt;tr&gt;&lt;td&gt;Anytime&lt;/td&gt;&lt;td&gt;$0.334&lt;/td&gt;&lt;td&gt;per kWh&lt;/td&gt;&lt;tr&gt;&lt;td&gt;Daily&lt;/td&gt;&lt;td&gt;$0.333&lt;/td&gt;&lt;td&gt;per day&lt;/td&gt;&lt;tr&gt;&lt;td&gt;EA Levy&lt;/td&gt;&lt;td&gt;$0.0013&lt;/td&gt;&lt;td&gt;per kWh&lt;/td&gt;&lt;/table&gt;"><i class="glyphicon glyphicon-info-sign"> </i></a>
                                </td>
                            </tr>
                                                        <tr style="text-transform: uppercase">
                                <td>Discount:</td>
                                <td>$63.14                                    (26%)
                                                                    </td>
                            </tr>
                                                            <tr>
                                    <td colspan="3">
                                        <a class="plan-detail" data-placement="right" title="" data-original-title="&lt;ul&gt;&lt;li&gt;Provides fixed pricing until 31 June 2021 unless there are changes to taxes and levies.&lt;/li&gt;&lt;li&gt;24% Prompt Payment Discount when you pay on time. additional 1% discount for paying by direct debit (excl. credit card), and 1% discount for getting bills and correspondence by email. Up to 26% PPD available.&lt;/li&gt;&lt;li&gt;An early termination fee of $150 per contracted ICP if you terminate the contract before the end date�(31/06/2021). Fee may be waived if you are moving house and take Contact Energy to the new property.&lt;/li&gt;&lt;li&gt;Not available to prepay customers.&lt;/li&gt;&lt;/ul&gt;"><i class="glyphicon glyphicon-info-sign"> </i> What you need to know</a>
                                    </td>
                                </tr>
                                                        <tr class="visible-xs">
                                <td colspan="2">
                                    <h3 class="total">$179.71</h3>
                                                                            <div class="incentive">
                                            <b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
                                            Special PPD &amp; Fixed rates<br />
                                                                                            <a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="Receive�a special Prompt Payment Discount and fixed rates until 31 June 2021 unless there are changes to taxes and levies">More Info</a>
                                                                                    </div>
                                                                    </td>
                            </tr>
                            <tr class="visible-xs">
                                <td colspan="2">
                                                                            <form id="w0" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" />                                        <input type="hidden" name="query_id" value="409884" />                                        <input type="hidden" name="plan_group_id" value="54" />                                        <input type="hidden" name="plan_stage_id" value="367" />                                        <button type="submit" class="btn btn-block btn-switch" style="max-width: 100%; margin-top: 10px">Switch Now!</button>                                        </form>                                        <div class="wannatalk" style="max-width: 100%">
                                            Want to talk?<br />
                                            Call our friendly team on<br />
                                            <b>0800 179 482</b>
                                        </div>
                                                                    </td>
                            </tr>
                        </tbody></table>
                    </td>
                    <td style="text-align: center" class="hidden-xs">
                        <h3 class="total">$179.71</h3>
                                                    <div class="incentive">
                                <b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
                                Special PPD &amp; Fixed rates<br />
                                                                    <a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="Receive�a special Prompt Payment Discount and fixed rates until 31 June 2021 unless there are changes to taxes and levies">More Info</a>
                                                            </div>
                                            </td>
                    <td class="hidden-xs">
                                                    <form id="w1" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" />                            <input type="hidden" name="query_id" value="409884" />                            <input type="hidden" name="plan_group_id" value="54" />                            <input type="hidden" name="plan_stage_id" value="367" />                            <button type="submit" class="btn btn-block btn-switch">Switch Now!</button>                            </form>                            <div class="wannatalk">
                                Want to talk?<br />
                                Call our friendly team on<br />
                                <b>0800 179 482</b>
                            </div>
                                            </td>
                </tr>
                            <tr>
                    <td style="vertical-align: middle; width: 20%;" class="hidden-xs">
                                                    <img class="retailer-logo" data-placement="right" src="/images/20171013102LzWd_kdtQOk4yxxyZuCZBG6q7xIuClx.jpg" alt="Powershop" style="width:150px;" title="" data-original-title="" />
                                            </td>
                    <td style="vertical-align: middle; width: 75px;" class="hidden-xs">
                        <img src="/images/result-arrow.png" />
                    </td>
                    <td>
                        <table style="width: 100%;">
                            <tbody><tr class="visible-xs">
                                <td class="text-center" colspan="2">
                                                                            <img class="retailer-logo" data-placement="right" src="/images/20171013102LzWd_kdtQOk4yxxyZuCZBG6q7xIuClx.jpg" alt="Powershop" style="width:150px;" title="" data-original-title="" />
                                                                    </td>
                            </tr>
                            <tr>
                                <td colspan="3"><h4>Powershop Saver</h4></td>
                            </tr>
                            <tr style="text-transform: uppercase">
                                <td width="150px">Electricity:</td>
                                <td>$183.40                                    <a class="plan-breakdown" data-placement="right" title="" data-original-title="&lt;table&gt;&lt;tr&gt;&lt;td&gt;Anytime&lt;/td&gt;&lt;td&gt;$0.2508&lt;/td&gt;&lt;td&gt;per kWh&lt;/td&gt;&lt;tr&gt;&lt;td&gt;Daily&lt;/td&gt;&lt;td&gt;$0.30&lt;/td&gt;&lt;td&gt;per day&lt;/td&gt;&lt;tr&gt;&lt;td&gt;EA Levy&lt;/td&gt;&lt;td&gt;$0.00&lt;/td&gt;&lt;td&gt;per kWh&lt;/td&gt;&lt;/table&gt;"><i class="glyphicon glyphicon-info-sign"> </i></a>
                                </td>
                            </tr>
                                                        <tr style="text-transform: uppercase">
                                <td>Discount:</td>
                                <td>$0.00                                    (0%)
                                                                    </td>
                            </tr>
                                                            <tr>
                                    <td colspan="3">
                                        <a class="plan-detail" data-placement="right" title="" data-original-title="&lt;ul&gt;&lt;li&gt;The price estimate is based on forecast charges from Powershop for the next 12 months.&lt;/li&gt;&lt;li&gt;It assumes you purchase the Powershop Simple Saver powerpack once a month and special powerpacks that are made available from time to time.&lt;/li&gt;&lt;li&gt;This offer does not require a contract or a minimum supply period.&lt;/li&gt;&lt;li&gt;New customers will get a $150 power credit applied over their first 12 months ($25 straight away, $10 on the next 10�monthly account&#10;review periods, and a final credit of $25 in the final account review period of&#10;your first year as a Powershop customer).�&lt;/li&gt;&lt;/ul&gt;"><i class="glyphicon glyphicon-info-sign"> </i> What you need to know</a>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
<h3 class="total">$183.40</h3>
<div class="incentive">
<b style="text-transform: uppercase">SPECIAL SwitchMe OFFER</b><br />
Get $150 off your bill over 12 months!<br />                                                                                            <a style="font-size: 0.9em;" class="incentive-info" title="" data-original-title="&lt;div&gt;&lt;div&gt;New customers will get a $150 power credit applied over their first 12 months ($25 straight away, then $10 for the next 10�monthly account&#10;review periods, and a final credit of $25 in the final account review period of&#10;your first year as a Powershop customer).&lt;/div&gt;&lt;div&gt;�&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;br&gt;&lt;/div&gt;&lt;div&gt;&lt;br&gt;&lt;/div&gt;">More Info</a>                                                                                    </div>
</td>
</tr>
<tr class="visible-xs">
<td colspan="2">
                                                                            <form id="w2" action="/switch/" method="post">
<input type="hidden" name="_csrf" value="Hi21xBvkP6NpUl0UcaFwxn4U5-94Jj8KqEeprOfuG9tMfP2gStRY6RFrBGdF6gGvT0uM3CAQaVvOPpnq1IddtQ==" />                                        <input type="hidden" name="query_id" value="409884" />                                        <input type="hidden" name="plan_group_id" value="53" />                                        <input type="hidden" name="plan_stage_id" value="273" />                                        <button type="submit" class="btn btn-block btn-switch" style="max-width: 100%; margin-top: 10px">Switch Now!</button>                                        </form><div class="wannatalk" style="max-width: 100%">
Want to talk?<br />
Call our friendly team on<br />
<b>0800 179 482</b>
</div>
</td>
</tr>
</tbody></table>
</td>

---

поэтому вывод должен быть: из тд [3] и тд [4] в первом ряду:

Свяжитесь с Energy Saver Plus $ 179,71

, а затем следующий ряд:

Powershop Saver $ 183,40

и так до последней строки (основной таблицы).

Answer 1

Мне действительно удалось решить это с помощью регулярных выражений. Мне нравится подход, приведенный в ответе выше, гораздо лучше, особенно с использованием zip (), но я добавил сюда свое решение на тот случай, если оно станет удобным для некоторых других читателей.

deals=[]
prices=[]    
results={}
with open("prices.html", "r") as f:
    soup = BeautifulSoup(f, 'html.parser')
    priceTable = soup.find('table', attrs={"class":"table table-hover table-responsive"})
    tbody = priceTable.find('tbody')
    pplanPattern = '<td\ colspan="3"><h4>([^<]+)<\/h4><\/td>'
    pricePatterns = '<h3 class="total">([^<]+)<\/h3>'

    for rw in tbody:
        plan = re.search(pplanPattern, rw)
        price = re.search(pricePatterns, rw)
        if plan:
            deals.append(plan.group(1))
        if price:
            deals.append(price.group(1))
            results[plan.group(1)] = price.group(1)

Answer 2

Процесс, аналогичный описанному в комментариях, но разные селекторы

from bs4 import BeautifulSoup as bs

html = '''yourhtml'''

soup = bs(html, 'lxml')
names = [item.text for item in soup.select('.table h4 ')]
prices = [item.text for item in soup.select('[colspan="2"] > .total')]
results = list(zip(names, prices))
print(results)