Зацикливание Python и очистка веб-страниц |BeautifulSoup

Question

В настоящее время пытаюсь зациклить следующий веб-запрос ...

Моя текущая проблема заключается в том, что я могу получить только первого футболиста из таблицы (у меня есть таблица HTML внизу) а не полные 10 игроков, мои непосредственные мысли о том, что цикл не работает, и я не уверен, где я иду не так.Я использую метод BeautifulSoup для сбора данных.

TD; DR Моя ошибка заключается в том, что в моем CSV-файле отображается только 1 проигрыватель вместо 10 проигрывателей, доступных в HTML

Код Python

 from urllib.request import urlopen as uReq
 from urllib.request import Request
 from bs4 import BeautifulSoup as soup

 my_url = "https://www.fctables.com/teams/stoke-194901/"

 #opening up connection , grabbing page
 uClient = uReq(my_url)
 page_html = uClient.read()
 uClient.close()

 #html parsing
 page_soup = soup(page_html, "html.parser")

 topScorers = page_soup.findAll("table",{"class":"table table-striped table-bordered table-hover stage-table table-condensed top_scores"})


 filename = "stokeGoals.csv"
 f = open(filename, "w")

 headers = "player, goal_scored, average_goal"

 f.write(headers)

 for topScorer in topScorers:
 #top 10 players who scored
 player = topScorer.a["title"]

#top 10 goalscorers for the team
goalpp = topScorer.findAll("div", {"class": "progress"})

#average goal per game
avg = topScorer.findAll("div", {"class": "label label-primary"})
avgpp = avg[0].text.strip()


print("player: " + player)
print("goal_scored: " + goalpp)
print("AVG: "+ avgpp)

f.write(player + "," +goalpp.replace("," , "|")+ "," + avgpp +"\n")

f.close()   

Код HTML для таблицы / веб-сайта, который я удаляю данные с

 <table class="table table-striped table-bordered table-hover stage-table table-condensed top_scores">
<thead>
    <tr>
        <th>#</th>
        <th class="tl">Player</th>
        <th data-toggle="tooltip" title="Goals scores by player / Goals scores by his team">goals</th>
        <th data-toggle="tooltip" title="Average goals">
            Avg
        </th>
    </tr>
</thead>
<tbody>
    <tr>
        <td>1</td>
        <td class="tl psh" data-id="212996">
            <img alt="Benik Afobe" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/s4/s4glg58a2350823d58/benik-afobe.png" width="20" /> <a href="/players/benik_afobe-212996/" title="Benik Afobe">Afobe</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 100%;">
                    <span class="goal_p">6</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.4</div>
        </td>
    </tr>
    <tr>
        <td>2</td>
        <td class="tl psh" data-id="320050">
            <img alt="Thomas Ince" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/t5/t5ni157c703a92110b/thomas-ince.jpg" width="20" /> <a href="/players/thomas_ince-320050/" title="Thomas Ince">Ince</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 83.333333333333%;">
                    <span class="goal_p">5</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.6</div>
        </td>
    </tr>
    <tr>
        <td>3</td>
        <td class="tl psh" data-id="308648">
            <img alt="Saido Berahino" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/po/poyhu58a234e0da106/saido-berahino.png" width="20" /> <a href="/players/saido_berahino-308648/" title="Saido Berahino">Berahino</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 66.666666666667%;">
                    <span class="goal_p">4</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.3</div>
        </td>
    </tr>
    <tr>
        <td>4</td>
        <td class="tl psh" data-id="257340">
            <img alt="Joe Allen" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/6w/6w45558a234deae78e/joe-allen.png" width="20" /> <a href="/players/joe_allen-257340/" title="Joe Allen">Allen</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 50%;">
                    <span class="goal_p">3</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.4</div>
        </td>
    </tr>
    <tr>
        <td>5</td>
        <td class="tl psh" data-id="234407">
            <img alt="Erik Pieters" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/et/et08558a234dd63b68/erik-pieters.png" width="20" /> <a href="/players/erik_pieters-234407/" title="Erik Pieters">Pieters</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 50%;">
                    <span class="goal_p">3</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.4</div>
        </td>
    </tr>
    <tr>
        <td>6</td>
        <td class="tl psh" data-id="299368">
            <img alt="Peter Crouch" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/qp/qptn558a234df86f1f/peter-crouch.png" width="20" /> <a href="/players/peter_crouch-299368/" title="Peter Crouch">Crouch</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 33.333333333333%;">
                    <span class="goal_p">2</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.3</div>
        </td>
    </tr>
    <tr>
        <td>7</td>
        <td class="tl psh" data-id="214479">
            <img alt="Bojan Krkic" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/pl/pleyv57eaedf0afeac/bojan-krkic.jpg" width="20" /> <a href="/players/bojan_krkic-214479/" title="Bojan Krkic">Krkic</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 33.333333333333%;">
                    <span class="goal_p">2</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.4</div>
        </td>
    </tr>
    <tr>
        <td>8</td>
        <td class="tl psh" data-id="253114">
            <img alt="James McClean" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/gb/gbjmm58a234f55a560/james-mcclean.png" width="20" /> <a href="/players/james_mcclean-253114/" title="James McClean">McClean</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
                    <span class="goal_p">1</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.1</div>
        </td>
    </tr>
    <tr>
        <td>9</td>
        <td class="tl psh" data-id="309022">
            <img alt="Sam Clucas" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/g7/g7dig58a234cb144a3/sam-clucas.png" width="20" /> <a href="/players/sam_clucas-309022/" title="Sam Clucas">Clucas</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
                    <span class="goal_p">1</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.3</div>
        </td>
    </tr>
    <tr>
        <td>10</td>
        <td class="tl psh" data-id="215724">
            <img alt="Bruno Martins Indi" class="img-circle" height="20" src="https://static.fctables.com/upload/images/20x20/hk/hkung58a234de0dfaa/bruno-martins-indi.png" width="20" /> <a href="/players/bruno_martins_indi-215724/" title="Bruno Martins Indi">Indi</a>
            <div class="slider">
                <div class="inner"></div>
            </div>
        </td>
        <td width="30%">
            <div class="progress">
                <div aria-valuemax="100" aria-valuemin="0" aria-valuenow="55" class="progress-bar progress-bar-primary" role="progressbar" style="width: 16.666666666667%;">
                    <span class="goal_p">1</span>
                </div>
            </div>
        </td>
        <td>
            <div class="label label-primary">0.2</div>
        </td>
    </tr>
</tbody>

Answer 1

Сохраняя URL-адрес как есть, вы можете попробовать следующее, чтобы получить требуемые результаты:

import requests
from bs4 import BeautifulSoup

url = "https://www.fctables.com/teams/stoke-194901/"

res = requests.get(url)
soup = BeautifulSoup(res.text,"html.parser")
for items in soup.select(".top_scores tbody tr"):
    name = items.select_one("td a[href^='/players/']").get("title")
    goal = items.select_one("td .goal_p").text
    avrg = items.select_one("td .label-primary").text
    print(name, goal, avrg)

Вывод, который вы должны получить:

Benik Afobe 6 0.4
Thomas Ince 5 0.6
Saido Berahino 4 0.3
Joe Allen 3 0.4
Erik Pieters 3 0.4
Peter Crouch 2 0.3
Bojan Krkic 2 0.4
James McClean 1 0.1
Sam Clucas 1 0.3
Bruno Martins Indi 1 0.2

Answer 2

Веб-страница, которую вы указали, загружает данные через XMLHttpRequest

Вы можете получить html непосредственно из:

https://www.fctables.com/xml/table_participant/?template_id=&season_id=52%2C38%2C88&type_home=overall&type=top_score&lang_id=2&team_id=194901&limit=10

Через указанный выше URL вы можетеполучить всю необходимую информацию без дополнительного html шума , то есть:

my_url = "https://www.fctables.com/xml/table_participant/?template_id=&season_id=52%2C38%2C88&type_home=overall&type=top_score&lang_id=2&team_id=194901&limit=10"

uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()

page_soup = soup(page_html, "html.parser")

_names = page_soup.findAll("img",{"class":"img-circle"})
_goals = page_soup.findAll("span",{"class":"goal_p"})
_avg = page_soup.findAll("div",{"class":"label label-primary"})
x = 0
for name in _names:
    name = name['alt']
    avg = _avg[x].get_text()
    goals = _goals[x].get_text()
    print(name, avg, goals)
    x+=1

---

Benik Afobe 0.4 6
Thomas Ince 0.6 5
Saido Berahino 0.3 4
Joe Allen 0.4 3
Erik Pieters 0.4 3
Peter Crouch 0.3 2
Bojan Krkic 0.4 2
James McClean 0.1 1
Sam Clucas 0.3 1
Bruno Martins Indi 0.2 1

---

Примечание:

Настройтеurl значения по мере необходимости, вы можете изменить top_score, type, team_id, limit, и т. Д ...