Как загружать динамический контент с помощью Selenium + Python

Question

На странице профиля пользователя Quora я хочу показать весь текст для ответа каждого пользователя, нажав «more», но я не смог получить список «more» элементов, вместо этого я получаю пустой список,как это сделать?

Вот мой код:

more_buttons = browser.find_elements_by_name('more')
for more_button in more_buttons:
        more_button.click()

Вот пример профиля пользователя: https://www.quora.com/profile/Ashley-Isabella-3

Answer 1

По сути, вы переходите по ссылкам (когда вы нажимаете), поэтому вы можете просто собрать ссылки и затем использовать цикл с .get.Использовать return [...document.querySelectorAll ] синтаксис

from selenium import webdriver

d = webdriver.Chrome()
d.get("https://www.quora.com/profile/Ashley-Isabella-3")
links = d.execute_script("return [...document.querySelectorAll('.ui_qtext_more_link')].map(link => link['href'])")
info = []
for link in links:
    d.get(link)
    pageInfo = d.execute_script("return [...document.querySelectorAll('.ui_qtext_para')].map(link => link.innerText)")
    info.append(pageInfo)
print(info)

Answer 2

На странице профиля пользователя Quora, чтобы перейти по ссылке с текстом (больше) и получить ответы пользователя, вы можете использовать следующее решение:

• Блок кода:

  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  
  driver=webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
  driver.get("https://www.quora.com/profile/Ashley-Isabella-3")
  more_buttons = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.ui_qtext_more_link")))
  for more_button in more_buttons:
      more_button.click()
  my_text_elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.ui_qtext_expanded>span.ui_qtext_rendered_qtext>p>b")))
  for my_text_element in my_text_elements:
      print(my_text_element.text)
  

• Консольный вывод:

  11 mistakes you make when trying to lose belly fat
  You could be doing the completely wrong exercise.
  1. You're not getting enough sleep
  2. You’re doing the wrong type of workout
  3. You’re consuming too much sugar
  4. You're not eating enough protein
  5. You’re feeling stressed or anxious
  6. You’re expecting a quick fix
  7. You’re not tracking your progress
  8. You’re crash-dieting
  9. You’re doing too much
  10. You haven't got the right exercise balance
  11. You’ve lost motivation
  
  9 Ways To Burn Fat Fast
  Implement these 9 fat-burning tips that use exercise and diet and watch the body fat melt like the butter you're no longer using.
  1. Stay Off The Scale
  2. Reduce Your Calories Gradually
  3. Vary Your Caloric Intake
  4. Train With Weights
  5. Do High-Intensity Intervals (HIIT)
  6. Eat More Fat
  7. Cut Carbs
  8. Increase Your Protein
  9. Eat 6 Smaller Meals Per Day, Not 2-3 Feasts
  Bonus:
  
  Researchers Explain 6 Reasons Why A Man Falls In Love
  1. HE’S CAPTIVATED BY THE WHOLE PACKAGE
  2. HE FEELS HE CAN MAKE HER HAPPY
  3. WHEN SHE’S OPEN TO LOVE
  4. THE WAY HE FEELS AROUND HER
  5. SHE SAYS “YES” TO LIFE
  6. HE FEELS PURPOSE AND PASSION
  Must Watch: Secret
  For You in Under 20 Minutes...
  These are the 7 weirdest things men find attractive in women
  Who knew anyone would ever find stubbornness attractive in a woman?
  1. Weakness
  2. Applying makeup
  3. Stubbornness
  4. Stretch marks
  5. Wearing glasses
  6. No make up
  7. Clumsiness
  Secret
  For You in Under 20 Minutes...
  5 surprising ways to get great skin
  There are so many bad ideas out there about how to achieve great skin. Here are five ways to get on the right track, regardless of 'the rules.'
  1. You don't need to wash your face with hot water to get it clean.
  2. Oil is good for your skin, not bad.
  3. You don't need to scrub to exfoliate.
  4. What matters most for healthy skin is not what you put on it, but what you eat.
  5. Chocolate doesn't cause acne, but bread and pasta might
  Make Your Skin Naturally Glow
  1. Skip the long, steamy showers and opt for shorter, cooler sprays.
  2. Check the dryness of your skin by scratching a small area on your arm or leg with your fingernail.
  3. Treat your neck and chest like an extension of your face.
  4. Run a humidifier every night in the winter to moisturize the air in your bedroom.
  
  12 easy tips for glowing skin
  1: Discover these glowing skin home remedies:
  2: Sweat it out for healthy skin
  3: Practice Yoga for Face Glow
  4: Know who you are!
  5: Follow a natural diet for glowing skin
  6: Indulge yourself…weekly
  7: Make Sudarshan Kriya your beauty mantra
  8: Meditate everyday to get Glowing Skin
  9: Silence is truly golden
  10: Save your mind at any cost
  11: 18 till I …why not?
  12: Flex those facial muscles
  15 Reasons Yoga Is Better Than The Gym
  1. Yoga benefits the mind, body and spirit.
  2. Yoga benefits your full body—externally and internally.
  3. Yoga teaches acceptance.
  4. Yoga is about focusing on yourself.
  5. Yoga will make you lean.
  6. Yoga is more efficient.
  7. You can do yoga anywhere.
  8. Yoga is kinder to the body.
  9. Yoga eases your aches and pains.
  10. Yoga helps you breathe easier.
  11. Yoga is calming.
  12. Yoga reduces stress.
  13. Everyone can practice yoga.
  14. Yoga improves concentration.
  15. Yogis rock.
  
  9 Ways to Tell If You’re Exercising Enough
  The sign: You never feel sore or tired the day after a workout
  The sign: You're not losing any weight
  The sign: You barely break a sweat
  The sign: The last rep feels hard, but you could probably do another
  The sign: You can chat with your friend throughout your workout
  The sign: You're not getting any more flexible in your yoga classes
  The sign: You run several days a week but aren't getting faster
  The sign: You're making strides in boot camp but are always stiff and un-flexible
  The sign: You take the same class regularly but aren't getting better at the exercises
  
  How To Build Muscle Mass After 50
  Exercise Plan
  Strength-Training Exercises
  Weight, Reps and Sets
  Aerobics Workouts
  
  Savannah Guthrie Says The Keto Diet Is Making Her Mind Sharp—And Her Body Sluggish
  She has zero desire to work out.
  
  How to Lose Weight in 10 Days at Home
  Flight of Stairs / Treadmill
  Lots of Water Fresh –
  Fruits and Vegetables –
  Low-Fat Foods Fish / Meat –
  Avoid Junk Food –
  Moderate Exercise -
  Drink lots of water -
  Day 1
  Day 2 and 3
  Day 4 and 5
  Day 6 and 7
  Day 8 and 9
  Day 10
  
  What are natural ways to make my face look younger?
  Smoothing Exfoliator
  Tightening Toner
  Plumping Moisturizing Facial Mask
  
  10 Ways to Lose 25 Pounds
  Follow these steps and weight loss won’t seem impossible.
  1. Do some detective work.
  2. Get moving.
  3. Plan ahead.
  4. Fill up on fruits and veggies.
  5. Don’t drink your calories.
  6. Don’t do anything drastic.
  7. Set smart calorie targets.
  8. Keep track.
  9. Do not make weight the only measure of success.
  10. Just practice; don’t try to be perfect.
  

Answer 3

Это потому, что вы используете неправильный селектор CSS.Попробуйте:

more_buttons = browser.find_elements_by_class_name('ui_qtext_more_link')

Вы можете найти нужные свойства, щелкнув правой кнопкой мыши по элементу в обычном браузере и нажав «Проверить».