Скребок загружен через javascript - PullRequest
Новая
       4

Скребок загружен через javascript

0 голосов
/ 20 января 2020

Я пытаюсь очистить таблицу, которая генерируется через javascript, но я борюсь. Мой код: 

from selenium import webdriver
driver = webdriver.Firefox(executable_path=r'path\to\geckodriver')


# driver = webdriver.Firefox()
url = 'https://af.ktnlandscapes.com/'
driver.get(url)

table = driver.find_element_by_xpath('//*[@id="list-view"]')
table_html = table.get_attribute('innerHTML')

df = read_html(table_html)
print (df)

driver.close()

Я получаю ошибку: 

ValueError: No tables found

1 Ответ

2 голосов
/ 20 января 2020

Наиболее вероятный сценарий - таблица загружена не полностью, прежде чем вы попытаетесь прочитать HTML из таблицы. Чтобы убедиться, что таблица загружена, вы можете использовать следующий код: 

# get table -- first wait for table and all rows to fully load
WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, "//*[@id='list-view']/tbody/tr")))

table = driver.find_element_by_xpath("//*[@id='list-view']")

Вы также можете просто использовать обычный Selenium для очистки таблицы - здесь нет необходимости использовать read_html, если это не ваша любимая стратегия. Вот некоторые примеры значений таблицы очистки кода: 

driver = webdriver.Chrome();

driver.get("https://af.ktnlandscapes.com/")

# get table -- first wait for table to fully load
WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, "//*[@id='list-view']/tbody/tr")))
table = driver.find_element_by_xpath("//*[@id='list-view']")

# get rows
rows = table.find_elements_by_xpath("tbody/tr")

# iterate rows and get cells
for row in rows:

    # get cells
    cells = row.find_elements_by_xpath("td")

    # print cell contents
    for cell in cells:
        print(cell.text)

    # print newline to separate rows
    print('\n')

driver.close()
driver.quit()

Приведенный выше пример кода извлекает все строки и ячейки таблицы, просматривает их во вложенном for и печатает значение каждой ячейки. Вы можете изменить это в соответствии с потребностями вашей программы.

Вывод: 

Networks
> Industry and research networks (networks)
A Network of Integrated Technolog...
Food and Drink, Plants...
 South West


Research Capabilities
> Livestock and Aquaculture Research Capabili...
Aberystwyth University (Institute...
Livestock and Aquaculture
 Wales


Research & Training Programmes
> Advanced Training Partnerships
Advanced Training Partnership for...
Livestock and Aquaculture
 Wales


Research Capabilities
> Livestock and Aquaculture Research Capabili...
Agri-Food and Biosciences Institu...
Livestock and Aquaculture
 Northern Ireland


Research Capabilities
> Plants and Crops Research Capabilities
Agri-Food and Biosciences Institu...
Plants and Crops
 Northern Ireland


Networks
> Industry Networks
Agri-Tech East
Livestock and Aquacult...
 East of England


Underpinning organisations
> Quality assurance bodies
Agricultural Industries Confedera...
Livestock and Aquacult...
 East of England


Underpinning organisations
> Trade associations
Agricultural Industries Confedera...
Livestock and Aquacult...
 East of England


Funders
> Levy bodies (funders)
Agriculture and Horticulture Deve...
Livestock and Aquacult...
 West Midlands


Underpinning organisations
> Levy bodies (underpinning)
Agriculture and Horticulture Deve...
Livestock and Aquacult...
 West Midlands


Innovation centres
> Centres for Agricultural Innovation
Agriculutral Engineering and Prec...
Livestock and Aquacult...
 Scotland


Research & Training Programmes
> Advanced Training Partnerships
AgriFood Advanced Training Partne...
AgriFood ATP, Plants a...
 East Midlands


Innovation centres
> Centres for Agricultural Innovation
Agrimetrics
Livestock and Aquacult...
 East of England


Networks
> Industry and research networks (networks)
Agrisearch
Livestock and Aquacult...
 Northern Ireland


Funders
> Research funding bodies (funders)
Agrisearch
Livestock and Aquacult...
 Northern Ireland


Networks
> Industry and research networks (networks)
Anaerobic Digestion Network
Livestock and Aquacult...
 South East


Key sector organisations
> UK Government organisations (key sector)
Animal and Plant Health Agency
Livestock and Aquacult...
 Nationwide


Research & Training Programmes
> Research and Technology Clubs
Animal Health Research Club
ARC, Livestock and Aqu...
 Nationwide


Underpinning organisations
> Professional organisations
Association of Applied Biologists
Livestock and Aquacult...
 West Midlands


Research Capabilities
> Livestock and Aquaculture Research Capabili...
Bangor University (College of Nat...
Livestock and Aquaculture
 Wales
...