如何抓取多个页面并将数据写入 excel

Question

如何在excel中拼多页。例如我想抓取“http://econpy.pythonanywhere.com/ex/001.html” 考虑到页数未知，如何抓取下一页

另外我写了一个代码，它在 excel 中打印 nonetype 而不是数据

from bs4 import BeautifulSoup
from urllib.request import urlopen

page_url = "http://econpy.pythonanywhere.com/ex/001.html"
new_file = "Mynew.csv"
f = open(new_file, "w")
Headers = "Header1, Header2\n"
f.write(Headers)


html = urlopen(page_url)
soup = BeautifulSoup(html, "html.parser")
buyer_info = soup.find_all("div", {"title":"buyer-info"})
for i in buyer_info:
    Header1 = i.find("div", {"title":"buyer-name"})
    Header2 = i.find("span", {"class":"item-price"})
    salmon = print(Header1.get_text())
    salam = print(Header2.get_text())
    f.write("{}".format(salmon)+ "{}".format(salam))
f.close()

我做错了什么？

Answer 1

直到第一页我都解决了...这是代码

from bs4 import BeautifulSoup
from urllib.request import urlopen

page_url = "http://econpy.pythonanywhere.com/ex/001.html"
new_file = "Mynew.csv"
f = open(new_file, "w")
Headers = "Header1,Header2\n"
f.write(Headers)

html = urlopen(page_url)
soup = BeautifulSoup(html, "html.parser") 
buyer_info = soup.find_all("div", {"title":"buyer-info"})
for i in buyer_info:
    Header1 = i.find("div", {"title":"buyer-name"})
    Header2 = i.find("span", {"class":"item-price"})
    f.write('{},{}\n'.format(Header1.text, Header2.text))
f.close()

现在麻烦来了如何为多个页面分配空间，意味着如何抓取下一页？

Answer 2

试一试，如果您有任何问题，请告诉我。我使用 "css selector" 和 "requests" 来完成操作。

import csv ; import requests
from bs4 import BeautifulSoup

outfile = open('Mynew.csv', 'w',newline='')
writer = csv.writer(outfile)
writer.writerow(["Name","Price"])

for page in range(1,6):
    html = requests.get("http://econpy.pythonanywhere.com/ex/00{0}.html".format(page))
    soup = BeautifulSoup(html.text, "html.parser")
    for item in soup.select("div[title=buyer-info]"):
        Header1 = item.select_one("div[title=buyer-name]").get_text()
        Header2 = item.select_one("span.item-price").get_text()
        writer.writerow([Header1, Header2])
        print(Header1,Header2)
outfile.close()

如何抓取多个页面并将数据写入 excel

How Do I scrape Multiple pages and write data into excel

web-scraping

python-3.6