使用 BeautifulSoup returns 空结果抓取 google 新闻

Question

我正在尝试使用以下代码抓取 google 新闻：

from bs4 import BeautifulSoup
import requests
import time
from random import randint


def scrape_news_summaries(s):
    time.sleep(randint(0, 2))  # relax and don't let google be angry
    r = requests.get("http://www.google.co.uk/search?q="+s+"&tbm=nws")
    content = r.text
    news_summaries = []
    soup = BeautifulSoup(content, "html.parser")
    st_divs = soup.findAll("div", {"class": "st"})
    for st_div in st_divs:
        news_summaries.append(st_div.text)
    return news_summaries


l = scrape_news_summaries("T-Notes")
#l = scrape_news_summaries("""T-Notes""")
for n in l:
    print(n)

尽管这段代码以前可以工作，但我现在不明白为什么它不再工作了。有没有可能是我被 google 封禁了，因为我只运行代码 3 或 4 次？（我也尝试使用 Bing 新闻，但不幸的是结果为空...）

谢谢。

Answer 1

我试过运行代码，它在我的电脑上运行良好。

您可以尝试打印请求的状态代码，看看它是否不是 200。

from bs4 import BeautifulSoup
import requests
import time
from random import randint


def scrape_news_summaries(s):
    time.sleep(randint(0, 2))  # relax and don't let google be angry
    r = requests.get("http://www.google.co.uk/search?q="+s+"&tbm=nws")
    print(r.status_code)  # Print the status code
    content = r.text
    news_summaries = []
    soup = BeautifulSoup(content, "html.parser")
    st_divs = soup.findAll("div", {"class": "st"})
    for st_div in st_divs:
        news_summaries.append(st_div.text)
    return news_summaries


l = scrape_news_summaries("T-Notes")
#l = scrape_news_summaries("""T-Notes""")
for n in l:
    print(n)

https://www.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/ 获取表明您已被禁止的状态代码列表。

使用 BeautifulSoup returns 空结果抓取 google 新闻

Scraping google news with BeautifulSoup returns empty results

python

web-scraping

beautifulsoup

google-news