python 网页抓取请求错误（mod 安全）

Question

我是新手，我尝试获取 tutorial.I 的网页源代码得到 beautifulsoup 安装，请求安装。起初我想抓取 source.I 正在从“https://pythonhow.com/example.html”做这个抓取工作。我没有做任何非法的事情，我认为这个网站也是为这个 purposes.Here 我的代码建立的:

import requests
from bs4 import BeautifulSoup

r=requests.get("http://pythonhow.com/example.html")
c=r.content
c

我遇到了 mod 安全错误：

b'<head><title>Not Acceptable!</title></head><body><h1>Not Acceptable!</h1><p>An appropriate representation of the requested resource could not be found on this server. This error was generated by Mod_Security.</p></body></html>'

感谢所有与我们打交道的人 me.Respectly

Answer 1

您可以通过为请求提供用户代理轻松解决此问题。通过这样做，该网站会认为有人实际上是在使用网络浏览器访问该网站。

这是您要使用的代码：

import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:55.0) Gecko/20100101 Firefox/55.0',
}

r = requests.get("http://pythonhow.com/example.html", headers=headers)
c = r.content

print(c)

这给了你预期的输出

b'<!DOCTYPE html>\n<html>\n<head>\n<style>\ndiv.cities {\n    background-color:black;\n    color:white;\n    margin:20px;\n    padding:20px;\n} \n</style>\n</head>\n<body>\n<h1 align="center"> Here are three big cities </h1>\n<div class="cities">\n<h2>London</h2>\n<p>London is the capital of England and it\'s been a British settlement since 2000 years ago. </p>\n</div>\n<div class="cities">\n<h2>Paris</h2>\n<p>Paris is the capital city of France. It was declared capital since 508.</p>\n</div>\n<div class="cities">\n<h2>Tokyo</h2>\n<p>Tokyo is the capital of Japan and one of the most populated cities in the world.</p>\n</div>\n</body>\n</html>'

Answer 2

尝试使用 url module

而不是请求

from urllib.request import urlopen

page = urlopen("http://pythonhow.com/example.html")
page.read()

Answer 3

我们只需要传递一个名为 headers 的参数...

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
}
page = requests.get(url, headers=headers)
soup = BeautifulSoup(page.content, 'html.parser')

python 网页抓取请求错误（mod 安全）

python web scraping request error(mod security)

python

python-requests

mod-security