Так что некоторые думают, что не работают в вашем фрагменте.Во-первых, чтобы получить ссылку, нужно сфокусировать BeautifulSoup на теге a.
Затем вам не нужно пробовать все тезисы, кроме.
Вот фрагмент кода,включая второй вызов для получения сводного текста:
import requests
from bs4 import BeautifulSoup
pages = [10, 20, 30, 40, 50]
for page in pages:
source = requests.get('https://www.indeed.co.in/jobs?q=data+scientist&start='.format()).text
soup = BeautifulSoup(source, 'lxml')
results = []
for jobs in soup.findAll(class_='result'):
result = {
'job_title': '',
'company': '',
'summary': '',
'link': '',
'summary_text': ''
}
job_title = jobs.find('h2', {'class': 'jobtitle'})
if job_title and job_title.find('a'):
result['job_title'] = job_title.find('a').get('title')
result['link'] = "https://www.indeed.co.in{0}".format(job_title.find('a').get('href'))
#else:
# print("no job title for ", jobs)
company_span = jobs.find('span', {'class': 'company'})
if company_span:
result['company'] = company_span.get_text()
summary = jobs.find('span', class_='summary')
if summary:
result['summary'] = summary.get_text()
results.append(result)
for r in results:
#print(r['link'])
source = requests.get(r['link'])
soup = BeautifulSoup(source.text, 'lxml')
description = soup.find('div', {'class' : 'jobsearch-JobComponent-description'})
if description:
r['summary_text'] = description.get_text()
print(results)
ВЫХОД:
[{'company': '\n DataMetica',
'job_title': 'Big-Data, Analytics Opportunities - Tech Savvy Talented '
'Freshers',
'link': 'https://www.indeed.co.in/rc/clk?jk=72e59a4376e3c7f1&fccid=f753310165e7a862&vjs=3',
'summary': '\n'
' Datametica supports the fresh minds to engage with '
'evolving tools and technologies working on Big data, Data '
'Science, Information Analytics and related...',
'summary_text': 'Pune, MaharashtraFresherJob Description\n'
'\n'
'Experience - 0 to 1 Years\n'
'\n'
'Selected candidates would get training and opportunity to '
'work on live projects in Big-Data, Analytics & Data '
'Science\n'
'\n'
'Candidates from Top Ranked Colleges or Premier Institutes '
'like IIT, NIT, REC, IIIT are preferred.\n'
'\n'
'Do you have knowledge on RDBMS Systems like Oracle, MY SQL, '
'Teradata and experience in solving analytical problems? Did '
'you use Java, C and C++ for your projects?\n'
'\n'
'If yes, then just apply with us.\n'
'\n'
'Datametica supports the fresh minds to engage with evolving '
'tools and technologies working on Big data, Data Science, '
'Information Analytics and related technologies like Hadoop, '
'Java, NoSQL.\n'
'\n'
'Added Advantage if you possess:\n'
'B.E/ B. Tech in Computer Science (graduated in 2016 & '
'2017)\n'
'Minimum 60% in Graduation\n'
'Good Communication Skills\n'
'0 to 1 Year experience'},
...
...
{'company': '\n\n Barclays',
'job_title': 'Junior Data Scientist',
'link': 'https://www.indeed.co.in/rc/clk?jk=2473a92840979437&fccid=057abf3fd357e717&vjs=3',
'summary': '\n'
' Junior Data Scientist. Junior Data Scientist - '
'90227028. Experience with the Python Data Science/Machine '
'learning stack....',
'summary_text': 'Pune, MaharashtraJunior Data Scientist - 90227028\n'
'Primary Location:IN-Maharashtra-Pune\n'
'Job Type:Permanent/Regular\n'
'Posting Range:3 Apr 2019 - 11 Apr 2019\n'
'Description\n'
'\n'
'Job Title: Junior Data Scientist\n'
'Location: Pune\n'
'\n'
'The Technology Chief Data Office exists to support and '
'enhance Barclays’ Technology function by leveraging its '
'most important asset: data. Within this, the mission '
'statement of the Data Science team is to enable Barclays to '
'react to things before they happen: to drive predictive '
'decision making by leveraging data on Technology, People, '
'and Process. We employ machine learning and artificial '
'intelligence models to discover the hidden patterns in the '
'data which describes Barclays, and use these to make '
'measured predictions. By understanding the rules which '
'govern the future evolution of any given resource, we can '
'make the right decisions in the present, driving matters '
'towards the business’ desired end goals.\n'
'\n'
'What will you be doing?\n'
'Develop machine learning and artificial intelligence '
'solutions as part of the project roadmap of the team\n'
'Support the team in balancing strategic project work with '
'incoming needs for data-driven methods.\n'
'Be agile, quick-thinking, and practical.\n'
'Evangelise for solving problems through Data across the '
'bank – contribute to the presence of our team in horizontal '
'bank-wide forums.\n'
'Contribute a creative and analytical/technical viewpoint of '
'problems\n'
'Support the team in supplying stakeholders with whatever '
'supplementary material they may require in order to get our '
'output into large-scale production.\n'
'Apply technical and analytical expertise to exploring and '
'examining data with the goal of discovering patterns and '
'previously hidden insights, which in turn can provide a '
'competitive advantage or address a pressing business '
'problem.\n'
'Implement model output within infrastructure, business '
'tools and workflow processes: turn data into something that '
'drives action within the business.\n'
'Leverage knowledge of mathematical and statistical '
'concepts, to bridge the gap between technologists and '
'mathematicians, ensuring software solutions meet business '
'goals.\n'
'What we’re looking for:\n'
'Experience solving real-world problems and creating value '
'through the end-to-end, productionised application of Data '
'Science, Machine Learning, and Artificial Intelligence '
'methods.\n'
'Experience with the Python Data Science/Machine learning '
'stack.\n'
'Master’s level degree in Science, Technology, Engineering, '
'Mathematics, or other relevant field, and associated '
'mathematical/analytical skills\n'
'Excellent interpersonal, written and verbal communication '
'skills is a must\n'
'Good presentation skills with ability to explain '
'sophisticated solution in layman terms\n'
'Skills that will help you in the role:\n'
'Experience using cloud solutions such as AWS/GCP\n'
'Experience using parallelised data storage and computation '
'solutions such as Hadoop\n'
'Experience with TensorFlow, neural networks, xgboost, nltk\n'
'Where will you be working?\n'
'PuneBarclays recently announced the creation of a new '
'world-class campus at Gera Commerzone located in Kharadi. '
'All Pune based roles will eventually start to move to this '
'new campus starting September 2019. In the run up to that, '
'during the course of 2018, there may be transitory '
'movements of some roles to other temporary sites. Please '
'speak with your recruiter about the specific location plans '
'for your role.\n'
'\n'
'For further information on EVP, please click on the link '
'below\n'
'https://now.barclays.com/WCP/content/intranet/en/functions/operations-and-technology/global-service-centre/EVP.html\n'
'\n'
'Be More at Barclays\n'
'At Barclays, each day is about being more – as a '
'professional, and as a person. ‘Be More @ Barclays’ '
'represents our core promise to all current and future '
'employees. It’s the characteristic that we want to be '
'associated with as an employer, and at the heart of every '
'employee experience. We empower our colleagues to Be More '
'Globally Connected, working on international projects that '
'improve the way millions of customers handle their '
'finances. Be More Inspired by working alongside the most '
'talented people in the industry, and delivering imaginative '
'new solutions that are redefining the future of finance. Be '
'More Impactful by having the opportunity to work on '
'cutting-edge projects, and Be More Valued for who you are.\n'
'\n'
'Interested and want to know more about Barclays? Visit '
'home.barclays/who-we-are/ for more details.\n'
'\n'
'Our Values\n'
'Everything we do is shaped by the five values of Respect, '
'Integrity, Service, Excellence and Stewardship. Our values '
'inform the foundations of our relationships with customers '
'and clients, but they also shape how we measure and reward '
'the performance of our colleagues. Simply put, success is '
'not just about what you achieve, but about how you achieve '
'it.\n'
'\n'
'Our Diversity\n'
'We aim to foster a culture where individuals of all '
'backgrounds feel confident in bringing their whole selves '
'to work, feel included and their talents are nurtured, '
'empowering them to contribute fully to our vision and '
'goals.\n'
'\n'
'Our Benefits\n'
'Our customers are unique. The same goes for our colleagues. '
"That's why at Barclays we offer a range of benefits, "
'allowing every colleague to choose the best options for '
'their personal circumstances. These include a competitive '
'salary and pension, health care and all the tools, '
'technology and support to help you become the very best you '
'can be. We are proud of our dynamic working options for '
'colleagues. If you have a need for flexibility, then please '
'discuss this with us.'}]