无法提取每项工作的所有要求

Question

enter image description here您好，我正在尝试使用 python 进行网络抓取，我想提取职位、公司名称、薪水和所需技能。除了需要技能的一件事之外，我设法获得了我想要的所有信息。 Nir Elbaz 帮助我部分解决了这个问题，但出于某种原因，我只获得了每个职位描述所需的第一项技能。这是我的代码以及我得到的结果的屏幕截图。

import pandas as pd 
import requests 
from bs4 import BeautifulSoup
import csv 
from itertools import zip_longest 
job_title = []
company_name = []
location = []
salary = []
links = []
skills = []
page_number = 0

while True: 
    result = requests.get(f"https://www.reed.co.uk/jobs/data-analyst-jobs?pageno={page_number}")
    src = result.content
    soup = BeautifulSoup(src, 'lxml')
    page_limit = soup.find('div', {"class":"page-counter"})
    Job_titles = soup.find_all('h3', {"class":"title"})
    Company_names = soup.find_all('a', {'class':"gtmJobListingPostedBy"})
    Locations = soup.find_all('li', {'class':"location"})
    Salary = soup.find_all('li',{"class": "salary"})
    #Job_skills = soup.find('strong').text
    #Job_skills = soup.find_all('div',{"class":"description"})
    

    for i in range(len(Job_titles)):
        
        job_title.append(Job_titles[i].text)
        links.append(Job_titles[i].find('a').attrs['href'])
        company_name.append(Company_names[i].text)
        location.append(Locations[i].text)
        salary.append(Salary[i].text)
    
    page_number += 1
    if(page_number >= 2):
        print("pages ended here")
        break 

for link in links: 
    url = "https://www.reed.co.uk/" + link
    result = requests.get(url)
    src = result.content 
    soup = BeautifulSoup(src, 'lxml')
    requirements = soup.find('span', {'itemprop':"description"}).find('ul')
    #print(requirements)
    skills_text = ""
    if not(requirements==None):
        for li in requirements.text.split(';'):
            #print(str(li))
            skills_text += str(li)
    #skills_text = skills_text[0:3]
    skills.append(skills_text.strip())

file_list = [job_title, company_name, location, salary, skills]
exported = zip_longest(*file_list)

Answer 1

尝试将代码中的这一部分替换为：

if not(requirements==None):
    for li in requirements.find('li'):
        print(str(li))
        skills_text += str(li) + ', '
skills_text = skills_text[:-2]
skills.append(skills_text)

无法提取每项工作的所有要求

Not being able to extract all the requirements of each job

python

web-scraping

beautifulsoup

python-requests

screen-scraping