Python 3.4 urllib.request 错误 (http 403)
Python 3.4 urllib.request error (http 403)
我正在尝试打开并解析 html 页面。在python 2.7.8我没问题:
import urllib
url = "https://ipdb.at/ip/66.196.116.112"
html = urllib.urlopen(url).read()
一切都很好。但是我想移动到 python 3.4，然后我收到 HTTP 错误 403（禁止访问）。我的代码：
import urllib.request
html = urllib.request.urlopen(url) # same URL as before
File "C:\Python34\lib\urllib\request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 461, in open
response = meth(req, response)
File "C:\Python34\lib\urllib\request.py", line 574, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python34\lib\urllib\request.py", line 499, in error
return self._call_chain(*args)
File "C:\Python34\lib\urllib\request.py", line 433, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 582, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
它适用于其他不使用 https 的 URL。
url = 'http://www.stopforumspam.com/ipcheck/212.91.188.166'
还可以。
该站点似乎不喜欢 Python 3.x 的用户代理。
指定 User-Agent 将解决您的问题：
import urllib.request
req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
html = urllib.request.urlopen(req).read()
注意 Python 2.x urllib 版本也收到 403 状态，但不同于 Python 2.x urllib2 和 Python 3.x urllib，它不会引发异常。
您可以通过以下代码确认：
print(urllib.urlopen(url).getcode()) # => 403
这是我在学习python-3时在urllib上收集的一些笔记：
我保留了它们，以防它们可能派上用场或帮助其他人。
如何导入 urllib.request 和 urllib.parse:
import urllib.request as urlRequest
import urllib.parse as urlParse
如何发出 GET 请求：
url = "http://www.example.net"
# open the url
x = urlRequest.urlopen(url)
# get the source code
sourceCode = x.read()
如何发出 POST 请求：
url = "https://www.example.com"
values = {"q": "python if"}
# encode values for the url
values = urlParse.urlencode(values)
# encode the values in UTF-8 format
values = values.encode("UTF-8")
# create the url
targetUrl = urlRequest.Request(url, values)
# open the url
x = urlRequest.urlopen(targetUrl)
# get the source code
sourceCode = x.read()
如何发出 POST 请求（403 forbidden 响应）：
url = "https://www.example.com"
values = {"q": "python urllib"}
# pretend to be a chrome 47 browser on a windows 10 machine
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"}
# encode values for the url
values = urlParse.urlencode(values)
# encode the values in UTF-8 format
values = values.encode("UTF-8")
# create the url
targetUrl = urlRequest.Request(url = url, data = values, headers = headers)
# open the url
x = urlRequest.urlopen(targetUrl)
# get the source code
sourceCode = x.read()
如何发出 GET 请求（403 forbidden 响应）：
url = "https://www.example.com"
# pretend to be a chrome 47 browser on a windows 10 machine
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"}
req = urlRequest.Request(url, headers = headers)
# open the url
x = urlRequest.urlopen(req)
# get the source code
sourceCode = x.read()
我正在尝试打开并解析 html 页面。在python 2.7.8我没问题:
import urllib
url = "https://ipdb.at/ip/66.196.116.112"
html = urllib.urlopen(url).read()
一切都很好。但是我想移动到 python 3.4，然后我收到 HTTP 错误 403（禁止访问）。我的代码：
import urllib.request
html = urllib.request.urlopen(url) # same URL as before
File "C:\Python34\lib\urllib\request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 461, in open
response = meth(req, response)
File "C:\Python34\lib\urllib\request.py", line 574, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python34\lib\urllib\request.py", line 499, in error
return self._call_chain(*args)
File "C:\Python34\lib\urllib\request.py", line 433, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 582, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
它适用于其他不使用 https 的 URL。
url = 'http://www.stopforumspam.com/ipcheck/212.91.188.166'
还可以。
该站点似乎不喜欢 Python 3.x 的用户代理。
指定 User-Agent 将解决您的问题：
import urllib.request
req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
html = urllib.request.urlopen(req).read()
注意 Python 2.x urllib 版本也收到 403 状态，但不同于 Python 2.x urllib2 和 Python 3.x urllib，它不会引发异常。
您可以通过以下代码确认：
print(urllib.urlopen(url).getcode()) # => 403
这是我在学习python-3时在urllib上收集的一些笔记：
我保留了它们，以防它们可能派上用场或帮助其他人。
如何导入 urllib.request 和 urllib.parse:
import urllib.request as urlRequest
import urllib.parse as urlParse
如何发出 GET 请求：
url = "http://www.example.net"
# open the url
x = urlRequest.urlopen(url)
# get the source code
sourceCode = x.read()
如何发出 POST 请求：
url = "https://www.example.com"
values = {"q": "python if"}
# encode values for the url
values = urlParse.urlencode(values)
# encode the values in UTF-8 format
values = values.encode("UTF-8")
# create the url
targetUrl = urlRequest.Request(url, values)
# open the url
x = urlRequest.urlopen(targetUrl)
# get the source code
sourceCode = x.read()
如何发出 POST 请求（403 forbidden 响应）：
url = "https://www.example.com"
values = {"q": "python urllib"}
# pretend to be a chrome 47 browser on a windows 10 machine
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"}
# encode values for the url
values = urlParse.urlencode(values)
# encode the values in UTF-8 format
values = values.encode("UTF-8")
# create the url
targetUrl = urlRequest.Request(url = url, data = values, headers = headers)
# open the url
x = urlRequest.urlopen(targetUrl)
# get the source code
sourceCode = x.read()
如何发出 GET 请求（403 forbidden 响应）：
url = "https://www.example.com"
# pretend to be a chrome 47 browser on a windows 10 machine
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"}
req = urlRequest.Request(url, headers = headers)
# open the url
x = urlRequest.urlopen(req)
# get the source code
sourceCode = x.read()