получение таблицы селена html в pandas массив данных с BeautifulSoup

Question

Я успешно использовал селен для очистки таблицы html, которая требовала нажатия кнопки перед очисткой.

Таким образом, селен работает и успешно сохраняет таблицу html как переменную 'r'.

Но я спотыкаюсь, но получаю это в pandas.

Как и URL, в фрейме данных должно быть 5 столбцов и около 30 строк.

Кто-нибудь может увидеть, в чем дело?

import pandas as pd
from selenium import webdriver
from bs4 import BeautifulSoup

browser = webdriver.Firefox(executable_path=r'/Users/computer_name/Documents/python/web_drivers/geckodriver')
browser.get('https://www.investing.com/equities/exxon-mobil-income-statement')
linkElem = browser.find_element_by_link_text('Annual')
linkElem.click()
r = browser.find_element_by_css_selector("#rrtable > table").get_attribute('innerHTML')
browser.quit()

soup = BeautifulSoup(r, 'html.parser')

df = pd.DataFrame(soup)
print(df)

Большое спасибо

Answer 1

Как только вы получите soup element, затем используйте pd.read_html() Вам нужно использовать outerHTML вместо innerHTML

r = browser.find_element_by_css_selector("#rrtable > table").get_attribute('outerHTML')
browser.quit()
soup = BeautifulSoup(r, 'html.parser')
df = pd.read_html(str(soup))[0]
print(df)

Выход :

                                     Period Ending:  ...                                          201631/12
0                                       Total Revenue  ...                                             200628
1   Revenue 255583 279332 237162 200628  Other Rev...  ...  Revenue 255583 279332 237162 200628  Other Rev...
2                                             Revenue  ...                                             200628
3                                Other Revenue, Total  ...                                                  -
4                              Cost of Revenue, Total  ...                                             136098
5                                        Gross Profit  ...                                              64530
6                            Total Operating Expenses  ...                                             199692
7   Selling/General/Admin. Expenses, Total 41923 4...  ...  Selling/General/Admin. Expenses, Total 41923 4...
8              Selling/General/Admin. Expenses, Total  ...                                              39819
9                              Research & Development  ...                                               1467
10                        Depreciation / Amortization  ...                                              22308
11          Interest Expense (Income) - Net Operating  ...                                                  -
12                           Unusual Expense (Income)  ...                                                  -
13                    Other Operating Expenses, Total  ...                                                  -
14                                   Operating Income  ...                                                936
15       Interest Income (Expense), Net Non-Operating  ...                                               4353
16                      Gain (Loss) on Sale of Assets  ...                                                  -
17                                         Other, Net  ...                                               2680
18                            Net Income Before Taxes  ...                                               7969
19                         Provision for Income Taxes  ...                                               -406
20                             Net Income After Taxes  ...                                               8375
21                                  Minority Interest  ...                                               -535
22                               Equity In Affiliates  ...                                                  -
23                                U.S GAAP Adjustment  ...                                                  -
24              Net Income Before Extraordinary Items  ...                                               7840
25                          Total Extraordinary Items  ...                                                  -
26                                         Net Income  ...                                               7840
27                    Total Adjustments to Net Income  ...                                                  -
28  Income Available to Common Excluding Extraordi...  ...                                               7840
29                                Dilution Adjustment  ...                                                  -
30                                 Diluted Net Income  ...                                               7840
31                    Diluted Weighted Average Shares  ...                                               4177
32          Diluted EPS Excluding Extraordinary Items  ...                                               1.88
33                   DPS - Common Stock Primary Issue  ...                                               2.98
34                             Diluted Normalized EPS  ...                                               1.88