Как очистить PDF-ссылки с веб-страниц с неизменными URL-адресами?

Question

Я работаю над проектом по очистке веб-страниц, и меня просят очистить все ссылки в формате PDF с веб-сайта: https://www.sebi.gov.in/sebiweb/home/HomeAction.do?doListing=yes&sid=3&s. На веб-сайте 397 страниц, но все страницы имеют одинаковый URL. Я попробовал инструмент проверки элементов и обнаружил, что код javascript помогает переходить на разные страницы. Но все же я не могу понять, как запустить мой скрипт для всех страниц.

Ниже мой код.

from bs4 import BeautifulSoup
import lxml
url = 'https://www.sebi.gov.in/sebiweb/home/HomeAction.do?doListing=yes&sid=3&s'

conn = urllib2.urlopen(url)
html = conn.read()

soup = BeautifulSoup(html)
links = soup.find_all('a')
urls=[]
for tag in links:
    link = tag.get('href',None)
    if link is not None and link.endswith('html'):
        #urls.append(link)
        purl=link
        new=urllib2.urlopen(purl)
        htm=new.read()
        sp=BeautifulSoup(htm)
        nl=sp.find_all('a')
        nm=sp.find_all('iframe')
        for i in nl:
            q=i.get('href',None)
            title=i.get('title',None)
            if q is not None and q.endswith('pdf'):
                print(q)
                urls.append(q)
        for j in nm:
            z=j.get('src',None)
            title=j.get('title',None)
            if z is not None and z.endswith('pdf')and title is not None:
                print(z)
                print(title)
                urls.append(z)
        

print(len(urls))

Answer 1

Вы можете использовать их API, расположенный на https://www.sebi.gov.in/sebiweb/ajax/home/getnewslistinfo.jsp, для загрузки данных.

Например:

from bs4 import BeautifulSoup
from requests import get


api_url = 'https://www.sebi.gov.in/sebiweb/ajax/home/getnewslistinfo.jsp'
payload = {
    'nextValue': "1",
    'next': "n",
    'search': "",
    'fromDate': "",
    'toDate': "",
    'fromYear': "",
    'toYear': "",
    'deptId': "",
    'sid': "3",
    'ssid': "-1",
    'smid': "0",
    'intmid': "-1",
    'sText': "Filings",
    'ssText': "-- All Sub Section --",
    'smText': "",
    'doDirect':  "1",
}

page = 0
while True:
    print('Page {}...'.format(page))
    payload['doDirect'] = page
    soup = BeautifulSoup(requests.post(api_url, data=payload).content, 'html.parser')
    rows = soup.select('tr:has(td)')
    if not rows:
        break
    for tr in rows:
        row = [td.get_text(strip=True) for td in tr.select('td')] + [tr.a['href']]
        print(*row, sep='\t')
    page += 1

Печать:

...

Page 1...
Jun 25, 2020    Mindspace Business Parks REIT – Addendum to Draft Prospectus    https://www.sebi.gov.in/filings/reit-issues/jun-2020/mindspace-business-parks-reit-addendum-to-draft-prospectus_46928.html
Jun 25, 2020    Amrit Corp. Ltd. - Public Announcement  https://www.sebi.gov.in/filings/buybacks/jun-2020/amrit-corp-ltd-public-announcement_46927.html
Jun 24, 2020    NIIT Technologies Buyback - Post Buyback - Public Advertisement https://www.sebi.gov.in/filings/buybacks/jun-2020/niit-technologies-buyback-post-buyback-public-advertisement_46923.html
Jun 23, 2020    Addendum to Letter of Offer of Arvind Fashions Limited  https://www.sebi.gov.in/filings/rights-issues/jun-2020/addendum-to-letter-of-offer-of-arvind-fashions-limited_46941.html
Jun 23, 2020    Genesis Exports Limited - Draft letter of Offer https://www.sebi.gov.in/filings/buybacks/jun-2020/genesis-exports-limited-draft-letter-of-offer_46911.html
Jun 23, 2020    Genesis Exports Limited - Public Announcement   https://www.sebi.gov.in/filings/buybacks/jun-2020/genesis-exports-limited-public-announcement_46909.html
Jun 19, 2020    Coral India Finance and Housing Limited – Post Buy-back Public Announcement https://www.sebi.gov.in/filings/buybacks/jun-2020/coral-india-finance-and-housing-limited-post-buy-back-public-announcement_46900.html
Jun 19, 2020    Network Limited https://www.sebi.gov.in/filings/takeovers/jun-2020/network-limited_46890.html
Jun 17, 2020    KSOLVES INDIA LIMITED   https://www.sebi.gov.in/filings/public-issues/jun-2020/ksolves-india-limited_46996.html
Jun 10, 2020    Happiest Minds Technologies Limited https://www.sebi.gov.in/filings/public-issues/jun-2020/happiest-minds-technologies-limited_46843.html
Jun 08, 2020    IM+ Capitals Limited    https://www.sebi.gov.in/filings/takeovers/jun-2020/im-capitals-limited_46786.html
Jun 05, 2020    HealthCare Global Enterprises Limited   https://www.sebi.gov.in/filings/takeovers/jun-2020/healthcare-global-enterprises-limited_46773.html
Jun 02, 2020    Jaikumar Constructions Ltd. - DRHP  https://www.sebi.gov.in/filings/public-issues/jun-2020/jaikumar-constructions-ltd-drhp_46774.html
Jun 02, 2020    Mahindra Focused Equity Yojana  https://www.sebi.gov.in/filings/mutual-funds/jun-2020/mahindra-focused-equity-yojana_46767.html
Jun 02, 2020    GRANULES INDIA LIMITED - Dispatch advertisement https://www.sebi.gov.in/filings/buybacks/jun-2020/granules-india-limited-dispatch-advertisement_46765.html
Jun 02, 2020    GRANULES INDIA LIMITED  - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/jun-2020/granules-india-limited-letter-of-offer_46764.html
Jun 02, 2020    Motilal Oswal Multi Asset Fund  https://www.sebi.gov.in/filings/mutual-funds/jun-2020/motilal-oswal-multi-asset-fund_46762.html
Jun 02, 2020    Principal Large Cap Fund    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/principal-large-cap-fund_46761.html
Jun 02, 2020    Mahindra Arbitrage Yojana   https://www.sebi.gov.in/filings/mutual-funds/jun-2020/mahindra-arbitrage-yojana_46760.html
Jun 02, 2020    HSBC Mid Cap Equity Fund    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/hsbc-mid-cap-equity-fund_46759.html
Jun 01, 2020    Tanla Solutions Limited - DLOF  https://www.sebi.gov.in/filings/buybacks/jun-2020/tanla-solutions-limited-dlof_46750.html
Jun 01, 2020    Axis Banking ETF    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/axis-banking-etf_46748.html
Jun 01, 2020    Kalpataru Power Transmission Limited - Public Announcement  https://www.sebi.gov.in/filings/buybacks/jun-2020/kalpataru-power-transmission-limited-public-announcement_46746.html
Jun 01, 2020    Reliance Industries Limited - Addendum dated May 22, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-22-2020_46745.html
Jun 01, 2020    Reliance Industries Limited - Addendum dated May 19, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-19-2020_46744.html
Page 2...
Jun 01, 2020    Reliance Industries Limited - Addendum dated May 18, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-18-2020_46743.html
May 29, 2020    Muthoottu Mini Financiers Limited- Prospectus   https://www.sebi.gov.in/filings/debt-offer-document/may-2020/muthoottu-mini-financiers-limited-prospectus_46769.html
May 29, 2020    Coral India Housing and Finance Limited - Advertisement https://www.sebi.gov.in/filings/buybacks/may-2020/coral-india-housing-and-finance-limited-advertisement_46732.html
May 29, 2020    TANLA SOLUTIONS LIMITED - Public Announcement   https://www.sebi.gov.in/filings/buybacks/may-2020/tanla-solutions-limited-public-announcement_46731.html
May 28, 2020    Tips Industries Limited - Dispatch Advertisement    https://www.sebi.gov.in/filings/buybacks/may-2020/tips-industries-limited-dispatch-advertisement_46723.html
May 27, 2020    KLM Axiva Finvest Limited - Prospectus  https://www.sebi.gov.in/filings/debt-offer-document/may-2020/klm-axiva-finvest-limited-prospectus_46755.html
May 26, 2020    Tips Industries Limited - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/may-2020/tips-industries-limited-letter-of-offer_46708.html
May 26, 2020    Axis Capital Protection Oriented Fund - Series 7-10 https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-capital-protection-oriented-fund-series-7-10_46707.html
May 26, 2020    ICICI Prudential Alpha Low Vol 30 ETF   https://www.sebi.gov.in/filings/mutual-funds/may-2020/icici-prudential-alpha-low-vol-30-etf_46706.html
May 22, 2020    NIIT Technologies Ltd. - Letter of Offer    https://www.sebi.gov.in/filings/buybacks/may-2020/niit-technologies-ltd-letter-of-offer_46700.html
May 22, 2020    NIIT Technologies Ltd. - Dispatch Advertisement https://www.sebi.gov.in/filings/buybacks/may-2020/niit-technologies-ltd-dispatch-advertisement_46699.html
May 22, 2020    Coral India Finance and Housing Limited - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/may-2020/coral-india-finance-and-housing-limited-letter-of-offer_46698.html
May 22, 2020    Jay Ushin Limited   https://www.sebi.gov.in/filings/takeovers/may-2020/jay-ushin-limited_46697.html
May 22, 2020    Pennar Industries - Post Buyback Public Announcement    https://www.sebi.gov.in/filings/buybacks/may-2020/pennar-industries-post-buyback-public-announcement_46696.html
May 22, 2020    Axis Global Equity Alpha Fund of Fund.  https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-global-equity-alpha-fund-of-fund-_46695.html
May 21, 2020    Axis Global Disruption Fund of Fund https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-global-disruption-fund-of-fund_46694.html
May 18, 2020    Reliance Industries Limited https://www.sebi.gov.in/filings/rights-issues/may-2020/reliance-industries-limited_46675.html
May 14, 2020    Public Advertisement of Spencer's Retail Limited    https://www.sebi.gov.in/filings/rights-issues/may-2020/public-advertisement-of-spencer-s-retail-limited_46693.html
May 12, 2020    Spencer's Retail Limited    https://www.sebi.gov.in/filings/rights-issues/may-2020/spencer-s-retail-limited_46692.html
May 12, 2020    Sequent Scientific Limited  https://www.sebi.gov.in/filings/takeovers/may-2020/sequent-scientific-limited_46662.html
May 11, 2020    Arvind Fashions Limited https://www.sebi.gov.in/filings/rights-issues/may-2020/arvind-fashions-limited_46659.html
May 05, 2020    JK Paper Limited - Public Announcement  https://www.sebi.gov.in/filings/buybacks/may-2020/jk-paper-limited-public-announcement_46647.html
May 05, 2020    Aurionpro Solutions Limited - Post BuyBack Advertisement    https://www.sebi.gov.in/filings/buybacks/may-2020/aurionpro-solutions-limited-post-buyback-advertisement_46646.html
May 04, 2020    KSOLVES INDIA LIMITED   https://www.sebi.gov.in/filings/public-issues/may-2020/ksolves-india-limited_46644.html
May 04, 2020    SBI ETF Consumption https://www.sebi.gov.in/filings/mutual-funds/may-2020/sbi-etf-consumption_46639.html
Page 3...

... and so on.

Answer 2

Похоже, что веб-сайт отправляет запрос POST на getnewslistinfo.jsp и возвращает новое содержимое таблицы как html. Вы можете открыть свою сеть (Ctrl + Shift + E на Firefox), затем перейти на следующую страницу и увидеть сделанный запрос и его параметры.

Вы можете имитировать этот запрос POST и изменить соответствующие параметры для следующей страницы (из того, что я видел, это должно быть nextValue и doDirect), используя urllib2 (или предпочтительно запросы ). После того, как вы получите контент, вы можете просто проанализировать его с помощью BeautifulSoup и извлечь теги a, как вы это уже сделали. * который с заданным номером страницы возвращает содержимое html, getLinks(html) который с учетом html страницы получает все ссылки из таблицы и возвращает их в виде списка. Таким образом, ваш код будет более читаемым, и его будет проще отлаживать и использовать.