Python requests.get 和 urllib.urlopen 向浏览器返回不同的 HTML

Question

我确定这是一个非常常见的问题，但我只是想了解正在发生的事情，以便我可以学习。

问题: 运行 urllib.urlopen 和 requests.get return 与 [=42] 不同 HTML =] 在浏览器中编辑。我已经禁用 Javascript 以防万一它通过 AJAX 或其他方式进行，但问题仍然存在。我还检查了 XHR 调用的开发人员工具或其他但没有发现任何东西。

Python代码：

headers = {'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate, sdch',
'Accept-Language':'en-GB,en-US;q=0.8,en;q=0.6',
'Cache-Control':'max-age=0',
'Connection':'keep-alive',
'Cookie':'tgaLiveAssistantVisible=false; ASP.NET_SessionId=c5wxbmboygytpn20gubpbh0o; .ASPXANONYMOUS=lLuVr11J1Tbq9zbQPk-ZpOFMQQ4gOlePIJpfGJwAnsFNV-1gUyvJpfb2ubnIPGZg8dQnlygyrKDj1KGf14vUehOx6Iwolslm0NZZkVbfdRvv9nBZhrmAiC-2MQijnBlmZPwinqBVVVrmBmVQavyOqUVzKWr8qapeBUO4nHWgzEy_1MqPx2njtCs0DvZ3IUdz0; __utmt=1; __utma=185625580.34928295.1483869478.1484065473.1484068673.4; __utmb=185625580.17.10.1484068673; __utmc=185625580; __utmz=185625580.1484065473.3.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)',
'Host':'training.gov.au',
'Upgrade-Insecure-Requests':'1',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
}
# Load the RTO page
r = urllib.urlopen('https://training.gov.au/Organisation/Details/45001')
# r = requests.get('https://training.gov.au/Organisation/Details/45001', headers=headers)
soup = BeautifulSoup(r.read(),"html.parser")
print(soup)

Returns底部的HTML:

<div class="display-row">
<div class="display-label">ABN:</div>
<div class="display-field-no-width">
</div>
</div>
</div>

而在浏览器中：

Request URL:http://training.gov.au/Organisation/Details/45001
Request Method:GET
Status Code:200 OK
Remote Address:117.53.170.212:80

Returns HTML:

<div class="display-row">
<div class="display-label">ABN:</div>
<div class="display-field-no-width">
<a href="http://www.abr.business.gov.au/search.aspx?SearchText=90608470113"title="View organisation on ABN Lookup"rel="external">90 608 470 113
<span class="accessibilityOnly">(external link)</span>
<img src="/Content/images/openNewWindow.png" alt=""/>
</a>
</div>
</div>

有人可以向我解释为什么会这样吗？

Answer 1

服务器可能会根据您的浏览器类型给您不同的响应。如果您希望获得与浏览器相同的响应，您可以查看 selenium python 绑定。

Answer 2

使用请求：

import requests

url = 'https://training.gov.au/Organisation/Details/45001'
r = requests.get(url).text

输出：

<div class="display-row">
                <div class="display-label">ABN:</div>
                <div class="display-field-no-width">
                <a href="http://www.abr.business.gov.au/search.aspx?SearchText=90608470113"title="View organisation on ABN Lookup"rel="external">90 608 470 113
<span class="accessibilityOnly">(external link)</span>
<img src="/Content/images/openNewWindow.png" alt=""/>
</a>

                </div>
            </div>

Python requests.get 和 urllib.urlopen 向浏览器返回不同的 HTML

Python requests.get and urllib.urlopen returning different HTML to browser

python

web-scraping

python-requests

screen-scraping

urllib