提交 HTTP POST 请求时出现问题

Question

我在提交 HTTP Post 请求时遇到问题。我这个程序的目的是从网站上抓取歌词，然后在文本摘要器中使用该字符串。我在汇总器网站上提交 POST 请求时遇到问题。目前使用下面的代码，它不提交请求。它只是 returns 页面。我觉得可能是content-type不同的原因，但我不确定。

我的代码：

def summarize(lyrics):
    url = 'http://www.freesummarizer.com'
    values = {'text' : lyrics,
              'maxsentences' : '1',
              'maxtopwords' : '40',
              'email' : 'your@email.com' }

    headers = {'User-Agent' : 'Mozilla/5.0'}

    cookies = {'_jsuid': '777245265', '_ga':'GA1.2.164138903.1423973625', '__smToken':'elPdHJINsP5LvAYhia6OAA68', '__smListBuilderShown':'true', '_first_pageview':'1', '_gat':'1', '_eventqueue':'%7B%22heatmap%22%3A%5B%7B%22type%22%3A%22heatmap%22%2C%22href%22%3A%22%252F%22%2C%22x%22%3A324%2C%22y%22%3A1800%2C%22w%22%3A640%7D%5D%2C%22events%22%3A%5B%5D%7D', 'PHPSESSID':'28b0843d49700e134530fbe32ea62923', '__smSmartbarShown':'true'}
    r = requests.post(url, data=values, headers=headers)
    print(r.text)

我的回复：

'transfer-encoding': 'chunked'
'set-cookie': 'PHPSESSID=1f10ec11e6f9040cbb5a81e16bfcdf7f; path=/', 
'expires': 'Thu, 19 Nov 1981 08:52:00 GMT'
'keep-alive': 'timeout=5, max=100'
'server': 'Apache'
'connection': 'Keep-Alive'
'pragma': 'no-cache'
'cache-control': 'no-store, no-cache, must-revalidate, post-check=0, pre-check=0'
'date': 'Fri, 27 Feb 2015 18:38:41 GMT'
'content-type': 'text/html'

本站回复成功：

Host: freesummarizer.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:35.0) Gecko/20100101 Firefox/35.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://freesummarizer.com/
Cookie: _jsuid=777245265; _ga=GA1.2.164138903.1423973625; __smToken=elPdHJINsP5LvAYhia6OAA68; __smListBuilderShown=true; _first_pageview=1; _gat=1; _eventqueue=%7B%22heatmap%22%3A%5B%7B%22type%22%3A%22heatmap%22%2C%22href%22%3A%22%252F%22%2C%22x%22%3A324%2C%22y%22%3A1800%2C%22w%22%3A640%7D%5D%2C%22events%22%3A%5B%5D%7D; PHPSESSID=28b0843d49700e134530fbe32ea62923; __smSmartbarShown=true
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 6044

Answer 1

requests 似乎一切正常。

但我认为这里的问题是您使用了错误的工具来完成这项工作。

我相信您正在寻找的工具是 Selenium。

Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.

你绝对应该看看这个工具。

Selenium docs

提交 HTTP POST 请求时出现问题

Issue with submitting an HTTP POST request

python

urllib2

http-post

web-scraping