脚本在抓取多个值时抛出错误

Question

我的脚本试图从一个网站上抓取存储在一个文本文件中的 80000 个 ID。当我运行我的代码只有一个 o/p 时，它运行没问题，但是当我将所有输入放在一个循环中时，我得到一个错误。

我的代码：

from bs4 import BeautifulSoup
import mechanize

br = mechanize.Browser()
response = br.open("https://www.matsugov.us/myproperty")

for form in br.forms():
    if form.attrs.get('name') == 'frmSearch':
        br.form = form
        break

br.form['ddlType']=["taxid"]

with open("names.txt") as ins:
    tx = ins.read().splitlines()

    for x in tx:
        br['txtParm'] = x
        req = br.submit().read()
        soup = BeautifulSoup(req, 'html.parser')
        table = soup.find('td', {'class': 'Grid_5'})

        for row in table:
            print row

错误：

AttributeError: mechanize._mechanize.Browser instance has no attribute __setitem__ (perhaps you forgot to .select_form()?)

Answer 1

你放错了循环！把它放在 mechanize 浏览器下面，然后在最后放一个 try catch。它会起作用。一旦你尝试过就发表评论，它有效。

脚本在抓取多个值时抛出错误

Script throws error while scraping multiple values

python

mechanize

beautifulsoup

web-scraping

python-2.7