从 HTML 不同结构和工资符号的工作规范中提取工资

Question

我希望尽可能以通用的方式从工作规格中提取薪水信息（考虑到可以指定薪水的多种方式（前面有和没有 'Salary' 一词，我们没有尾随零、范围等）

采用三个不同的工作规范，我用 urllib2 获取 HTML，然后对单词 'salary in each' 执行不区分大小写的初始 grep。结果差异很大（请原谅来自 Jupyter 的不太漂亮的粘贴）：

In [52]:

urllib2
Out[52]:
<module 'urllib2' from '/Users/Evan/anaconda/lib/python2.7/urllib2.pyc'>
In [82]:

案例 #1

reponse = urllib2_urlopen('http://apply.ovoenergycareers.co.uk/vacancies/453/cro-manager/london/')
In [83]:

content = reponse.read()
In [84]:

save_html('salarygrep1', content)
In [59]:

!grep -i salary salarygrep1.html
!grep -i salary salarygrep1.html
  <dt class="field_salary">Salary</dt>
  <dd class="value_salary">
In [86]:

with open('salarygrep1.html') as s:
    for line in s:
        if 'salary' in line.lower():
            print line
  <dt class="field_salary">Salary</dt>

  <dd class="value_salary">

In [79]:

案例 #2

reponse = urllib2_urlopen('http://apply.ovoenergycareers.co.uk/vacancies/475/ovo-telesales-agent/bristol/')
In [80]:

content = reponse.read()
In [81]:

save_html('salarygrep2', content)
In [63]:

!grep -i salary salarygrep2.html
  <dt class="field_salary">Salary</dt>
  <dd class="value_salary">
    Salary: �18,000 + benefits & competitive commission scheme; OTE range: �20,500 - �30,000
In [87]:

with open('salarygrep2.html') as s:
    for line in s:
        if 'salary' in line.lower():
            print line

  <dt class="field_salary">Salary</dt>

  <dd class="value_salary">

    Salary: �18,000 + benefits & competitive commission scheme; OTE range: �20,500 - �30,000

In [88]:

案例 #3

reponse = urllib2_urlopen('https://gs7.globalsuccessor.com/centrica02/tpl_centrica02.asp?s=4A515F4E5A565B1A&jobid=48490,2356610248&key=21798303&c=028859657862&pagestamp=dbykvxmmwfnblykbqc')
In [89]:

content = reponse.read()
In [90]:

save_html('salarygrep3', content)
In [67]:

!grep -i salary salarygrep3.html
!grep -i salary salarygrep3.html
<p id="igSoundBite"><em><div>Salary: &#163;28-&#163;38K depending on experience</div></em></p><h3 id="igJobDesc0">Job Description</h3><p><div>Assistant Product Development Manager </div>
In [95]:

with open('salarygrep3.html') as s:
    for line in s:
        if 'salary' in line.lower():
            print line
<p id="igSoundBite"><em><div>Salary: &#163;28-&#163;38K depending on experience</div></em></p><h3 id="igJobDesc0">Job Description</h3><p><div>Assistant Product Development Manager </div>

In [70]:

案例 #4

reponse = urllib2_urlopen('http://jobs.emounlimited.com/senior-digital-project-manager/')
In [71]:

content = reponse.read()
In [72]:

save_html('salarygrep4', content)
In [94]:

!grep -i salary salarygrep4.html
In [92]:

with open('salarygrep4.html') as s:
    for line in s:
        if 'salary' in line.lower():
            print line
In [ ]:

在案例 #1 中，实际工资是不同的 <div>，并且没有被捡起来。
在案例 #2 中，工资确实被检测到，但 (i) 它是一个需要处理的范围，并且 (ii) 存在非 ASCII 货币字符（英国英镑）。
在案例 #3 中，薪水也被检测到，但 (i) 它是一个需要处理的范围，并且 (ii) 存在非 ASCII 货币字符（英国英镑）。
以防万一，因为指定了薪水但没有提到 'Salary' 这个词，所以没有检测到任何东西。

考虑到页面设计和薪资规格的广泛差异，一种通用（或通用）正则表达式或正则表达式组合可能会做到这一点，这是否不合理？诡计？如果没有，我将如何构建 it/them？或者实际上，是否有一种 Python 方法可以减少对正则表达式的依赖？

Answer 1

这是一个想法：

通过BeautifulSoup
获取body元素的文本（我们对其余部分不感兴趣）
使用正则表达式，select 数量

代码：

# -*- coding: utf-8 -*-
import re

import requests
from bs4 import BeautifulSoup

urls = [
    "http://apply.ovoenergycareers.co.uk/vacancies/453/cro-manager/london/",
    "http://apply.ovoenergycareers.co.uk/vacancies/475/ovo-telesales-agent/bristol/",
    "https://gs7.globalsuccessor.com/centrica02/tpl_centrica02.asp?s=4A515F4E5A565B1A&jobid=48490,2356610248&key=21798303&c=028859657862&pagestamp=dbykvxmmwfnblykbqc",
    "http://jobs.emounlimited.com/senior-digital-project-manager/"
]

money_pattern = re.compile(ur"($|£)([0-9.,]+K?)(?:\s*-\s*(?:$|£)*([0-9.,]+K?)*)*")
for url in urls:
    soup = BeautifulSoup(requests.get(url).text, "html.parser")
    text = soup.body.text

    print("URL: " + url)
    for currency, amount1, amount2 in money_pattern.findall(text):
        if not amount1 and not amount2:
            continue

        if not amount2:
            print("Single amount found: %s, currency: %s" % (amount1, currency))
        else:
            print("Range found: %s - %s, currency: %s" % (amount1, amount2, currency) )
    print("------")

输出：

URL: http://apply.ovoenergycareers.co.uk/vacancies/453/cro-manager/london/
Range found: 40,000 - 50,000, currency: £
------
URL: http://apply.ovoenergycareers.co.uk/vacancies/475/ovo-telesales-agent/bristol/
Single amount found: 18,000, currency: £
Range found: 20,500 - 30,000, currency: £
------
URL: https://gs7.globalsuccessor.com/centrica02/tpl_centrica02.asp?s=4A515F4E5A565B1A&jobid=48490,2356610248&key=21798303&c=028859657862&pagestamp=dbykvxmmwfnblykbqc
Range found: 28 - 38K, currency: £
------
URL: http://jobs.emounlimited.com/senior-digital-project-manager/
Range found: 36 - 40,000, currency: £
------

希望这至少能让你入门。

从 HTML 不同结构和工资符号的工作规范中提取工资

Pulling salary from HTML job specs of differing structure and salary notation

python

regex

xpath

grep

lxml