解析 Python 中的 url

Question

我想从一个网站上抓取一些关于几家公司的信息，下面你可以看到两家公司 url 的信息相同：

url 公司 "A1" :

url = 'http://www.example.com/wps/portal/!ut/p/c1/lYuxDoIwGAYf6f8oVMIoDLVIGrTQ0C6mAxqiFAejr2_dXNSYG--OHEWCv08nf5uW4C80kFsdFCqzlxmDSIoEcqt3Rc_WDHUevX3z0ByQWWO6qhEpgL9u0Rr-ujXTbZ1Csh-3HgOpzTKPZMnlH8uSU0e2jJH9Gl3nfnio4_kJwpcKOw!!/dl2/d1/L3dJMjIyQSEhL3dQRUJGUUJnTlFCaGpRQmhyUUJoSFFCZ1BRQmdBQSEhL1lJNXcvN19OMENWUkk0MjBHMTkxMElLU1E5VTJBMjBCNQ!!/?CompanyName=A1&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/'

url 公司 "A2":

url = 'http://www.example.com/wps/portal/!ut/p/c1/lYuxDoIwGAYf6f8oVMKomNQiadBCQ7uYDmiIUhyMvr51c1Fjbrw7chQJ_j6e_G2cg79QT25xUCjNXmYMIikSyK3eFR1bMlR59PbNQ3NAZrVpy1qkAP66RWP469ZMN1UKyX7cegikNvM0kCWXfyxXnFqy6xjZr9F16vqHOp6fJIF9Sw!!/dl2/d1/L3dJMjIyQSEhL3dQRUJGUUJnTlFCaGpRQmhyUUJoSFFCZ1BRQmdBQSEhL1lJNXcvN19OMENWUkk0MjBHMTkxMElLU1E5VTJBMjBCNQ!!/?CompanyName=A2&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/'

如您所见，url 在这部分之前不断变化：

/?CompanyName=A2&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/

这部分之后：

http://www.example.com/wps/portal/!ut/p/c1/

我想要的是忽略 url 中的任何更改部分，只传递以下公司名称：

Companies = ['A1', 'A2', 'A3']

as exapmle , 当然会带for循环:

url = 'http://www.example.com/wps/portal/!ut/p/c1/04_SB8K8xLLM9MSSzPy8xBz9CP0os3g_A-ewIE8TIwN3Q0tDA0_v4EDLUCNHIwMvc6B8JJK8QbCpgYGniU9YiLOPu7GBgQFJut0DwkxBuoONggO8jA08jQjo9vPIz03Vj9SPMsepyslUP0Q_0hWoKBKvooLc0IhyQ91AAHb2Eas!/dl2/d1/L0lDUmlTUSEhL3dHa0FKRnNBL1lCUlp3QSEhL2Vu/?typeofsearch=1&company=&CompanyName=' + companies +'A1&tabOrder=1&x=0&y=0'

我该怎么做？

Answer 1

不用正则表达式也可以，

url = 'http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=A1&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/'
url = url.replace('Name=A1', 'Name=')
compaines = ['a1','a2','a3']
for ele in compaines:
    start = url.find("CompanyName=") + 12
    newUrl1 = url[:start] + ele
    newUrl2 = newUrl1 + url[start:]
    print newUrl2

注意：出于测试目的，我已经简化了 URL。

如果你运行上面的程序，你会得到下面的输出，

    http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=a1&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/
http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=a2&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/
http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=a3&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/

Answer 2

l = ['http://www.example.com/wps/portal/!ut/p/c1/lYuxDoIwGAYf6f8oVMKomNQiadBCQ7uYDmiIUhyMvr51c1Fjbrw7chQJ_j6e_G2cg79QT25xUCjNXmYMIikSyK3eFR1bMlR59PbNQ3NAZrVpy1qkAP66RWP469ZMN1UKyX7cegikNvM0kCWXfyxXnFqy6xjZr9F16vqHOp6fJIF9Sw!!/dl2/d1/L3dJMjIyQSEhL3dQRUJGUUJnTlFCaGpRQmhyUUJoSFFCZ1BRQmdBQSEhL1lJNXcvN19OMENWUkk0MjBHMTkxMElLU1E5VTJBMjBCNQ!!/?CompanyName=A2&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/','http://www.example.com/wps/portal/!ut/p/c1/lYuxDoIwGAYf6f8oVMIoDLVIGrTQ0C6mAxqiFAejr2_dXNSYG--OHEWCv08nf5uW4C80kFsdFCqzlxmDSIoEcqt3Rc_WDHUevX3z0ByQWWO6qhEpgL9u0Rr-ujXTbZ1Csh-3HgOpzTKPZMnlH8uSU0e2jJH9Gl3nfnio4_kJwpcKOw!!/dl2/d1/L3dJMjIyQSEhL3dQRUJGUUJnTlFCaGpRQmhyUUJoSFFCZ1BRQmdBQSEhL1lJNXcvN19OMENWUkk0MjBHMTkxMElLU1E5VTJBMjBCNQ!!/?CompanyName=A1&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/']

import re
for el in l:
    mo = re.search(r'CompanyName=(.*?)&', el)
    if mo:
        print(mo.group(1))

A2
A1

解析 Python 中的 url

Parsing a url in Python

python

url

parsing

screen-scraping