urllib没有return请求的内容

Question

我有两个要废弃的页面： url_1 and url_2

它们之间的唯一区别是 url_1 是第一页，而 url_2 是同一域的第三页。

我正在使用 urrlib 读取网址：

from urllib.request import urlopen
html_1 = urlopen(url_1).read()
html_2 = urlopen(url_2).read()

遗憾的是html_2与html_1内容相同。阅读周围，我发现这可能是因为服务器将我视为机器人。出于这个原因，我使用 request 模块 Beautiful Soup 来解析页面：

import requests
from bs4 import BeautifulSoup
session = requests.Session()
headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5)AppleWebKit 537.36 (KHTML, like Gecko) Chrome", "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"}

req_1 = session.get(url_1, headers=headers)
bsObj_1 = BeautifulSoup(req_1.text)
req_2 = session.get(url_2, headers=headers)
bsObj_2 = BeautifulSoup(req_2.text)

内容还是一样。我该如何解决？

Answer 1

试试这个：

import requests
from bs4 import BeautifulSoup
import time

url_1 = 'https://www.zoekscholen.onderwijsinspectie.nl/zoek-en-vergelijk?searchtype=generic&zoekterm=&pagina=&filterSectoren=BVE'
url_2 = 'https://www.zoekscholen.onderwijsinspectie.nl/zoek-en-vergelijk?searchtype=generic&zoekterm=&pagina=3&filterSectoren=BVE'

headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5)AppleWebKit 537.36 (KHTML, like Gecko) Chrome",
            "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"}

with requests.Session() as s:
    s.headers.update(headers)
    s.get('https://www.zoekscholen.onderwijsinspectie.nl/')
    req_1 = s.get(url_1)
    soup1 = BeautifulSoup(req_1.text, "lxml")
    print(soup1.find("div", {"id": "mainResults"}).find_all("h2")[0].text)
    time.sleep(1)
    req_2 = s.get(url_2)
    soup2 = BeautifulSoup(req_2.text, "lxml")
    print(soup2.find("div", {"id": "mainResults"}).find_all("h2")[0].text)

输出：

Resultaten 1 - 20 van 165

Resultaten 41 - 60 van 165

urllib没有return请求的内容

Urllib does not return the requested content

python

urllib2

beautifulsoup

web-scraping

python-requests