Веб-сайт загружен событием JavaScript, которое динамически отображает данные после загрузки страницы. Библиотека
requests не сможет отображать JavaScript на лету. так что вы можете использовать selenium или requests_html. и действительно, есть много модулей, которые могут сделать это.
Теперь у нас есть еще одна опция в таблице, чтобы отследить, откуда данные отображаются. Мне удалось найти запрос XHR , который используется для извлечения данных из back-end API и их рендеринга на стороне пользователя.
Вы можете получить XHR запрос открытым Developer-Tools и проверка Сеть и проверка XHR/JS запросов, сделанных в зависимости от типа вызова, таких как fetch
Ниже приведен простой вызов, где предел составляет maximum из 600, настроенный на parameter size
Таким образом, вам нужно будет loop увеличивать с шагом 600 каждый раз, пока вы не достигнете всего 4803, что является максимальным результатом для sap. Ниже у вас есть действительный диктант JSON, доступ к которому вы можете получить с помощью keys.
import requests
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0",
}
data = "{\"returnCount\":{},\"repository\":\"partnerfinder\",\"type\":\"content\",\"filters\":[{\"field\":\"AUTHOR\",\"values\":[\"PWP\"],\"type\":\"exact\"}],\"returnResults\":{\"sort\":[{\"field\":\"SHORTNAME\",\"order\":\"asc\"}],\"page\":{\"size\":600},\"outputFields\":[\"TITLE\",\"EMAIL\",\"PARTNERID\",\"DESCRIPTION\",\"PHONE\",\"AWARDID\",\"FOCUSAREA\",\"LEVEL\",\"URL\",\"SHORTNAME\"]}}"
def main(url):
with requests.Session() as req:
r = req.post(url, data=data, headers=headers).json()
# print(r.keys())
# humanview = json.dumps(r, indent=4)
for item in r["result"]["results"]["results"]:
print(item["TITLE"])
main("https://partneredge.sap.com/bin/fiji/es/search-results")
Пример вывода:
IBM World Trade Corporation - Egypt Branch
Origo hf.
2BM A/S
Accenture (China) Co. Limited
Accenture (UK) Ltd.
ACCENTURE - CONSULTORES DE GESTÃO, S.A.
Accenture AB
Accenture AG
Accenture AS
Accenture Australia Pty Ltd
Accenture B.V.
Accenture Chile Asesorías y Serv. Ltda.
Accenture Company Limited
Accenture do Brasil Ltda.
Accenture GmbH
Accenture GmbH
Accenture Inc.
Accenture Japan Ltd.
Accenture Limited LAGOS
Accenture LLC
Accenture Middle East BV
ACCENTURE OUTSOURCING SERVICES, S.A.
Accenture Pte Ltd
Accenture S.A.
Accenture S.p.A.
Accenture S.R.L.
ACCENTURE SAS
Accenture Saudi Arabia Limited
Accenture Sdn Bhd
Accenture Solutions Private Limited
Accenture Solutions Sdn Bhd
Accenture Sp. z o.o.
Accenture Technology Solutions Oy
Accenture Technology Ventures B.V.branch
Accenture, Inc.
Accenture, S.L.
Accenture
Accenture
addIT Dienstleistungen GmbH & Co KG
advanced business Consulting GmbH
Ageless LLC
All for One Group AG
All for One Steeb GmbH
All Tax Platform - Soluções Tributárias S.A.
AMAZON WEB SERVICES INC
AO Deloitte and Touche CIS
Apigee Corporation
Atlantconsult LLC
Atos (Taiwan) Ltd
Atos AG
Atos Belgium SA
Atos Bilisim Danismanlik ve Müsteri Hizmetleri San. A.S.
Atos Bulgaria Competency Center EOOD
Atos Inc.
Atos India Pvt. Ltd.
Atos Information Technology (S) Pte Ltd
Atos Information Technology GmbH
Atos Information Technology (China) Co., Ltd.
Atos IT Services UK Limited
Atos It Services
Atos IT Solutions and Services A/S
Atos IT Solutions and Services AB
Atos IT Solutions and Services GmbH
ATOS IT SOLUTIONS AND SERVICES IBERIA SL
Atos IT Solutions and Services LLC
Atos IT Solutions and Services Oy
Atos IT Solutions and Services, Inc.
Atos IT Solutions and Services, s.r.o.
Atos IT Solutions and Services s.r.o.
ATOS IT SOLUTIONS AND SERVICES D.O.O. BEOGRAD
Atos IT Solutions and Services
Atos IT Solutions Romania SRL
Atos IT Solutions and Services Limited
Atos Italia S.p.A.
Atos Nederland B.V.
Atos Origin FZ LLC
Atos Polska S.A.
Atos Polska S.A.
Atos Saudi Arabia
Atos Serviços de Tecnologia da Informaçao do Brasil Ltda.
AUGUSTA REEVES
Axxis Consulting (S) Pte Ltd
B4B Solutions GmbH
B4B Solutions GmbH
Bluekey Seidor (Pty) Ltd.
BSG Partners Co., Ltd.
BULL SAS
CAPGEMINI (CHINA) CO., LTD
Capgemini AB
Capgemini Argentina S.A.
Capgemini Australia Pty Limited
Capgemini Belgium N.V.
Capgemini Brasil S.A.
CAPGEMINI ESPAÑA, S.L.
Capgemini Italia S.p.A.
Capgemini Mexico, S. de R.L. de C.V.
Capgemini Nederland B.V.
Capgemini Norge AS
CAPGEMINI PORTUGAL - SERVIÇOS DE CONSULTORIA E INFORMÁTICA, S.A.
Capgemini Services Malaysia Sdn Bhd
Capgemini Singapore Pte Ltd
Capgemini Technologies LLC
Capgemini Technologies LLC
Capgemini Technology Services India Ltd. Block 2&3,Plot no IT3,IT4, 2nd &
CAPGEMINI TECHNOLOGY SERVICES
Capgemini UK Plc
Celonis SE
CenturyLink CenturyTel LLC
CIDEON Software & Services GmbH & Co.KG
Cisco Systems, Inc.
COMPAGNIE IBM FRANCE SAS
CPM BRAXIS TECNOLOGIA LTDA.
Crystal Solutions S.A.
CSC Computer Sciences Brasil S.A.
CTAC België NV
Ctac N.V.
DC Extended Delivery S. de R.L. de C.V.
Deloitte & Co. S.A.
Deloitte & Touche Consulting Group ICS Pte Ltd.
Deloitte & Touche S.R.L.
Deloitte AB
Deloitte Asesores y Consultores Ltda
Deloitte Assessoria e Consultoria Ltda
Deloitte Business Solutions S.A
Deloitte Consulting & Advisory Deloitte Entity
Deloitte Consulting (Pty) Ltd
Deloitte Consulting (SEA) Sdn Bhd
Deloitte Consulting B.V.
DELOITTE CONSULTING CR, S.A.
Deloitte Consulting ehf.
Deloitte Consulting Ltd.
Deloitte Consulting Product Services LLC
Deloitte Consulting Pte Ltd
Deloitte Consulting S.A.
Deloitte Consulting S.r.l
DELOITTE CONSULTING VIETNAM CO., LTD
DELOITTE CONSULTORES, S.A.
Deloitte Inc.
DELOITTE LIMITED
Deloitte LLP
Deloitte Servicios Profesionales Ltda
Deloitte Tax & Consulting
Deloitte Technology Limited
Deloitte Tohmatsu Consulting LLC
Deloitte Touche Tohmatsu
Deloitte
Detay Danismanlik Bilgisayar Hizmetleri San. ve Dis Tic.A.S.
Deutsche Telekom Healthcare and Security Solutions GmbH
DSC Software AG
Dunn Solutions Group
DXC Technologies
DXC Technology (Middle East) FZ LLC
DXC Technology Austria GmbH
DXC Technology Czech Republic s.r.o.
DXC Technology Deutschland GmbH
DXC TECHNOLOGY FRANCE
DXC Technology Japan, Ltd.
DXC Technology Services LLC
DXC Technology Services Singapore Pte. Ltd.
DXC Technology Services Vietnam Company Limited
DXC Technology Spain, S.A.
DXD APPLICATIONS AND IT SOLUTIONS, S.L.
DynaSys Solutions Ltd.
Edenhouse Solutions Limited
Edraky LLC Edraky
EIT Services India Private Limited
Enterprise IT Services MiddleEast FZ LLC
ENTERPRISE SERVICES BRASIL SERVICOS DE TECNOLOGIA LTDA
Enterprise Services d.o.o. Beograd
EntServ Enterprise Services Mexico S. de R.L. de C.V.
ENTSERV PHILIPPINES,INC
EPAM Systems, Inc.
Everis Chile S.A.
Everis México, S. de R.L. de C.V.
Everis Peru Sociedad Anonima Cerrada
Excellence Delivered ExD Pvt. Ltd
EY Brightree Sdn. Bhd.
Fasttrack ERP Solutions Inc.
FastTrack Solutions, Inc.
FUJITSU LIMITED
GE Digital
Hewlett Packard Colombia LTDA.
Hewlett Packard Enterprise Company
Hewlett Packard Enterprise India Private Limited
Hewlett Packard Enterprise Polska sp. z o.o.
HEWLETT PACKARD FRANCE SAS
Hewlett-Packard (M) Sdn Bhd
Hewlett-Packard (Schweiz) GmbH
Hewlett-Packard (Thailand) Ltd.
Hewlett-Packard HK SAR Limited
Hitachi Vantara Corporation
Huawei Technologies Co., Ltd
IBM (China) Company Limited
IBM ARGENTINA SRL
IBM Australia Limited
IBM Belgium B.V.B.A.
IBM Brasil - Indústria, Máquinas e Serviços Limitada.
IBM BULGARIA EOOD
IBM Canada Limited
IBM Ceska republika, spol. s r.o.
IBM China / Hong Kong Limited (Partner)
IBM Corporation International Technical
IBM Danmark ApS
IBM de Chile S.A.C.
IBM De Colombia & Cia. S.C.A.
IBM del Peru S.A.C.
IBM Deutschland GmbH
IBM Eastern Europa/Asia Ltd.
IBM Eesti OÜ
IBM Engineering Technology (Shanghai) Co Ltd.
IBM GLOBAL SERVICES ESPAÑA, S.A.
IBM Global Services ve Teknoloji Hizmetleri
IBM Hellas S.A.
IBM HRVATSKA D.O.O.
IBM India Pvt Ltd
IBM Int. Business Machines AS
IBM International Business Machines doo
IBM Ireland Ltd
IBM Italia S.p.A.
IBM Italia S.p.A
IBM Italia SpA (Pakistan)
IBM Japan Services Company Ltd.
IBM Japan, Ltd.
IBM Korea, INC.
IBM Malaysia Sdn Bhd
IBM Middle East FZ LLC
IBM Middle East FZ-LLC SaudiArabiaBranch
IBM Polska Sp. z o.o.
IBM Qatar SSC
IBM Romania SRL
IBM Schweiz AG
IBM Singapore Pte Ltd
IBM Slovenija d.o.o.
IBM South Africa (Pty) Ltd
IBM Svenska AB
IBM Taiwan Corporation
IBM Thailand Co., Ltd.
IBM United Kingdom Limited
IBM World Trade Corporation
ICM.S S.r.l.
Illumiti Corp.
Illumiti Inc
Illumiti One Inc
Implema AB
In Cloud Solutions Ltd
Intelligroup Saudi Arabia Company Ltd
IPS Co., Ltd.
ISS Consulting (Thailand) Ltd.
Itelis d.o.o.
itelligence a/s
Itelligence AB
Itelligence AG, Niederlassung Wien
itelligence AG
itelligence AG
Itelligence AS
Itelligence Benelux Holding BV
Itelligence Bilgi Sistemleri A.S.
Itelligence Business Solutions (UK) Limited
Itelligence Business Solutions
itelligence Business Solutions Canada, Inc.
itelligence Consulting (Shanghai) Ltd
ITELLIGENCE FRANCE
itelligence Hungary Informatika Kft.
Itelligence India Software Solutions Private Limited
Itelligence LLC
itelligence Outsourcing MSC Sdn Bhd
itelligence Slovakia, s.r.o.
Itelligence Software Solutions
itelligence Sp. z o.o.
itelligence, a.s.
Itelligence, Inc.
itelligence
KBJ S.A.
KWP Austria GmbH
KWP INSIDE HR GmbH
Lenovo Global Technology HK Limited
MIBCON a.s.
MSS Seidor Colombia S.A.S.
MSS Seidor Peru SAC
MSS Seidor, S.L.
Myanmar Information Technology Pte Ltd.
NetApp, Inc.
NTT DATA Business Solutions Malaysia Sdn Bhd
NTT DATA Business Solutions Australia Pty Ltd
NTT DATA Business Solutions Singapore Pte Ltd
NTT DATA Romania SA
NTT DATA VIETNAM CO. LTD
Nutanix Inc
OOO T-Systems CIS
Open Text Corporation
ORBIT Gesellschaft für Applikations- und Informationssysteme mbH
OSC Smart Integration GmbH
OSIsoft, LLC
Oxygen Business Solutions Ltd
Oxygen Business Solutions Pty Ltd
Pearl Norge AS
Price Waterhouse & Co Asesores de Empresas S.R.L
PricewaterhouseCoopers Advisory N.V.
PricewaterhouseCoopers Advisory S.p.A.
PricewaterhouseCoopers Business Solutions SA
PricewaterhouseCoopers Consulting (Australia) Pty Limited
PricewaterhouseCoopers Consulting (Singapore) PTE LTD
PricewaterhouseCoopers Consulting (Thailand) Ltd.
PriceWaterhouseCoopers Consultores Auditores SpA
PricewaterhouseCoopers GmbH Wirtschaftsprüfungsgesellschaft
PricewaterhouseCoopers Inc
PricewaterhouseCoopers LLP
PriceWaterhouseCoopers LLP
PricewaterhouseCoopers LLP
PricewaterhouseCoopers Management Consulting (Shanghai) Limited
Pricewaterhousecoopers Management Consultants SRL
PricewaterhouseCoopers Private Limited
Pricewaterhousecoopers, S.C.
PricewaterhouseCoopers Česká republika s.r.o.
Process Partner AG
Proquire LLC
PT Accenture
PT Deloitte Consulting
PT IBM Indonesia
PT Soltius Indonesia
Pure Storage Inc.
PwC Consulting LLC
PWC Consulting Services (M) Sdn Bhd.
PwC Enterprise Advisory cvba/scrl
PwC Société coopérative
Questionmark Computing Limited
QZing Technology (Beijing) Company Limited
Real Consulting IT Business Solutions SA
Red Hat, Inc.
RED POINT SOFTWARE SOLUTIONS SRL
Redwood Alliances B.V.
SAS Institute Inc.
SAVIC Technologies Private Limited
Seidor Chile S.A.
Seidor Colombia SAS
Seidor Consulting Peru SAC
Seidor Consulting SA
SEIDOR CONSULTING, S.L.
Seidor Crystalis - Tecnologia da Informação S.A.
Seidor Crystalis Costa Rica
Seidor Maroc, SARL
SEIDOR MEXICO SAPI de CV
Seidor Middle East & North Africa FZ-LLC
Seidor Portugal, Lda
Seidor S.A.
Seidor UK Ltd
Seidor Uruguay Informatica S.A
Seidor USA Corp
Servicios Informáticos Itelligence, S.A
Shanghai Acloudear Info. Tech. Co., Ltd.
SOA PEOPLE SA/NV
SOA PEOPLE SAS
Société Conseil Groupe LGS.
Sofigate Business Technologies Oy
SOFTINSA - ENGENHARIA DE SOFTWARE AVANÇADO, LDA
SUSE Software Solutions Germany GmbH
Syniti
Systex Corporation
T-Systems Austria GesmbH
T-Systems do Brasil Ltda
T-Systems International GmbH
T-SYSTEMS ITC IBERIA, S.A.U.
T-Systems Malaysia Sdn. Bhd.
T-Systems Multimedia Solutions GmbH
T-Systems México, S.A. de C.V.
T-Systems Nederland B.V.
T-Systems North America, Inc.
T-Systems P.R.China Ltd.
T-Systems Polska Spolka z o.o.
T-Systems Schweiz AG
T-Systems Singapore Pte Ltd
T-Systems South Africa (Pty) Ltd
TalentChamp Consulting GmbH
TEAMIDEA GROUP LLC
TECNILÓGICA ECOSISTEMAS, S.A.U.
Telekom Deutschland GmbH
Terralink Technologies LLC
The Hackett Group d/b/a Answerthink, Inc.
unit-IT Dienstleistungs GmbH & Co KG
Utopia Global, Inc.
VIEWNEXT S.A.
Vistex, Inc.
Westrocon Seidor (PTY) Ltd
Öhrlings PricewaterhouseCoopers AB
3Hold Technologies, SL
4process AG
Aareon Deutschland GmbH
AB Consulting Group Zrt.
Abacus Cambridge Partners (Middle East) Ltd.
Abacus Cambridge Partners Saudia LLC
Abacus Consulting Technology Pvt Ltd
AbacusConsulting
abat AG
abc Consulting S.A.
ABeam Consulting (Malaysia) Sdn Bhd
ABeam Consulting (Shanghai) Co., Ltd.
ABeam Consulting (Thailand) Limited
ABeam Consulting Ltd.
Abeo Vietnam Co., Ltd
Abide Consult AG
abilis GmbH
ABM Global Solutions, Inc.
Abracon GmbH
Absoft Limited
Acando Consulting AB
Accely Consulting India Private Limited
Acclimation Pty Ltd
ACEteK Software Limited
AchieveIT Solutions, Inc.
ACI Holdings
ACJ Consulting Co., Ltd.
Acorel VAR B.V.
ACRON Bilisim A.S.
Actualisap Consultores Bolivia S.A.
Acuma Solutions Ltd
Adesso SE
ADP Consultores S.R.L.
AdS Consulting, Beratung für angewandte Informationsverarbeitung GmbH
Advanced Applications GmbH
Advanced Business Software
Advanced Business Solutions
Advanced Electronics Company
Advantech Technologies Ltd
Advantic Consultores de sistemas de información S.L.
AEKANSA S.A. SYPSOFT360
AFON Technologies Pte Ltd
AG Consultancy & Apps Ltd
Agentil SA
AGILITA AG
Agion
AICOMP Cloud GmbH
AKT Solutions Ltd
AL BILAD ARABIA CO. LTD.
ALEXANDER MOORE S.A.
Alfa Sistemas de Gestao LTDA
Alfacloud LTD
all4cloud GmbH & Co. KG
Allgeier ES France
Allos S.r.l.
alogis AG
Altab S.A.
Altaflux Corporation
ALTEA UP S.r.l.
Altevie Technologies S.r.l.
ALTIM TECNOLOGIAS DE LA INFORMACIÓN,S.L.
AMS Advanced Management Solutions Ltd
AMS Solutions Limited
AMT - CONSULTING, S.A.
Anda Srl
ANSI Information Systems, Inc.
anthesis GmbH
AO "ECOPSY"
AODYS
APPCENTRIC Solutions Inc.
applied international informatics GmbH
APPTechnology Experts, Inc.
APSIA
apsolut GmbH
Arago Consulting Iberia
ARAGO CONSULTING SAS
Arete Bilgisayar, Otomasyon Egitim ve Danismanlik Hizm. San. Tic. Ltd. Sti.
Arete Global Company Branch FZ-LLC
Arineo GmbH
Arinso Iberica S.A.U.
Arithnea GmbH
Arvato Systems GmbH
AS Emergn
ASAP soft
ASAR AMERICA INC.
Ascarii Ltd
Ascentium Corporation DBA SMITH
AScorpi GmbH
Asecom B.V.
ASG Group Limited
Aspire HR Inc.
ASPN CO.,LTD.
Attune Consulting USA, Inc.
Aubay Italia S.p.A.
avantum Consult AG
Avectris AG
Avtenta, napredne poslovne rešitve, d.o.o.
Axians ICT Austria GmbH
Axians NEO Solutions & Technology GmbH
Axianseu II Digital CONSULTING SA
Axxiom Soluções Tecnológicas S.A
Ayesa Advanced Technologies, S.A.
b1 consulting GmbH
Babiel GmbH
Backoffice Associates
BAITCON S.A.
Baraka IT Solutions (Pty) Ltd
BC SKILLS CONSULTING
BCI Consulting SRL
BCS Business Consulting Services Kft.
BDO Unicon Business Solutions AO
BDO Ziv Haft Consulting @ Management Ltd
be one solutions Americas Inc.
be one solutions Deutschland GmbH
be one solutions Japan K.K.
Be1Eye GmbH
BearingPoint GmbH
Beijing AVA Technology Inc.
Beijing Ether Electronics Group Co., Ltd.
Beijing Faujor Technology Co., Ltd.
Beijing Pactera Services Limited
Beijing Shunshiheng Technology Development Co,. Ltd.
BENOY LLC
Bestcom Infotech Corporation
Beyond Technologies Consulting Inc.
BEYOND TECHNOLOGIES
BGP Management Consulting S.p.A.
BH Consulting Co., Ltd.
Bilot Oyj
Birchman Solutions Ltd
BizTech Partners Co., Ltd
Blend IT Consultoria e Serviços em Informática Ltda
Blue Pencil Consulting Pty Ltd
Bluekey Seidor (K) Limited
BluePrint Technologies Private Limited
Bluetree Solutions Pty Ltd
BluLeader Pty Ltd
BMS Global Services LLC
bneXt Inc.
BosCloud Jiangsu Science and Technology Co., Ltd.
Boyum IT A/S
Bramasol, Inc.
Brave New World Consulting Pty Ltd t/as BNW Consulting
bridgX
Bright Business Partners
Britehouse a Division of Dimension Data (pty) Ltd
BS&C
BSGOne Co., Ltd. BSGOne
BTC Bilisim Hizmetleri A.S.
BTC Business Technology Consulting AG
BUSINESS AT WORK
Business Process Solutions SA de CV Xamai
Business Service Center LLC
BXT Solution Co., Ltd
BXTI Soluciones en Tecnología de Información S.A. de C.V.
C2E Teknoloji Servisleri Ticaret Anonim Sirketi
Caleo Consulting GmbH
Camelot ITLab GmbH
CANCOM Managed Services GmbH
CAPLAN Corporation
Castaliaz Technologies Private Limited
CBL CONSULTING
CCelera s.r.l.
CEGB Corporation
Celeritech Mexico SAPI de CV
CEO Consultoría, S.R.L
CEREALOG
CGI Suomi Oy
CGI Sverige AB
Chain Services TI S.A.C. Csti S.A.C.
Chengdu Biz-United Information Technology Co., Ltd.
China National Software and Service Co., Ltd.
Chinasoft Technology (Shenzhen) Corporation Limited
cHReative Consultoria LTDA
Cibernetica, S.A.
Citek Technology Joint Stock Company
Clarex Srl
Clariba Consulting S.L.U.
Clientis AG
Clients First Business Solutions LLC
Clockwork Business Solutions Private Limited
Clockwork Inc.
Cloudera Limited
Cloudway Consulting Private Limited
CM CONSULTING
CNBM Technology Co., Ltd
CNT Management Consulting AG
Codestone Solutions Ltd
CODILOG - ELIANCE
Cogniscient Business Solutions Private Limited
Cognitus Consulting LLC
Columbus Systems GmbH
COMLINE Computer + Softwarelösungen SE
COMMON MANAGEMENT SOLUTIONS, S.L.
Complete Business Solutions
COMPTA - EQUIPAMENTOS E SERVIÇOS DE INFORMÁTICA, S.A.
Compuage Infocom Limited
CompuNet, S.A.
CompuTec S.A.
Computer Systems Pvt Ltd
ComSol AG Commercial Solutions
CON.SE s.r.l.
ConCorn LLC.
CONPLUS Mittelstandslösungen GmbH
CONSEILS PLUS
Conseils-Plus
Consensus International, LLC
Consensus S.A.S.
Consilio GmbH
consolut.gmbh
Consulting 4U, s.r.o.
Consultoría Organizacional S.A.S.
ConVista Consulting AG
Cormeta AG
Corponet Implements, S.A. de C.V.
Corporacion Saratella, S.A. Vivo Consulting