Python：html 抓取器从具有相同基数的多个页面中提取特定单词之间的信息 url。这是我到目前为止所拥有的

Question

import csv
import requests 
from bs4 import BeautifulSoup
from itertools import izip

grant_number = ['0901289','0901282','0901260']
#IMPORTANT NOTE: PLACE GRANT NUMBERS BETWEEN STRINGS WITH NO SPACES

start = 'this site'
end = 'Please report errors'
#start and end show the words that come right before the publication data
my_string = []
#my_string is an empty list for the publication data


for x in grant_number:      # Number of pages plus one 
    url = "http://nsf.gov/awardsearch/showAward?AWD_ID={}".format(x)
    r = requests.get(url)
    soup = BeautifulSoup(r.content, "html.parser")
    soup_string = str(soup)
    my_string[int(x)] = soup_string[(soup_string.index(start)+len(start)):soup_string.index(end)]
with open('NSF.csv', 'wb') as f:
    #Default Filename is NSF.csv ; This can be changed by editing the first field after 'open('
    writer = csv.writer(f)
    writer.writerows(izip(grant_number, my_string))
#this imports the lists into a csv file with two columns, grant number on left, publication data on right

Python 告诉我

line 26, in my_string[int(x)] = soup_string[(soup_string.index(start)+len(start)):soup_string.index(end)] IndexError: list assignment index out of range

我该如何解决这个问题？

Answer 1

问题是 my_string[x] 正在尝试获取 my_string 的 x 列表索引，但根据您对 grant_number 列表的定义，x 是一个字符串。

您可能想改为附加到最初的空字符串。

for x in grant_number:      # Number of pages plus one 
    url = "http://nsf.gov/awardsearch/showAward?AWD_ID={}".format(x)
    r = requests.get(url)
    soup = BeautifulSoup(r.content, "html.parser")
    soup_string = str(soup)
    my_string.append(soup_string[(soup_string.index(start)+len(start)):soup_string.index(end)])

Python：html 抓取器从具有相同基数的多个页面中提取特定单词之间的信息 url。这是我到目前为止所拥有的

Python: html scraper to extract information between certain words from multiple pages with the same base url. Here is what I have so far

html

python

screen-scraping

bots