使用 BeautifulSoup 抓取网页

Question

我正在尝试抓取此站点：https://www.senate.gov/general/contact_information/senators_cfm.cfm

我的代码：

import requests
from bs4 import BeautifulSoup

URL = 'https://www.senate.gov/general/contact_information/senators_cfm.cfm'
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'html.parser')

print(soup)

问题是它实际上并没有访问该站点。我在 soup var 中得到的 HTML 根本不是正确网页中的 HTML。

我不确定从这里到哪里去！任何和所有帮助将不胜感激。

Answer 1

这对我有用

headers = {
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
    }
r = requests.get(URL,headers=headers)

在此处找到信息 - https://towardsdatascience.com/5-strategies-to-write-unblock-able-web-scrapers-in-python-5e40c147bdaf

Answer 2

重复

试试看：

import requests
from bs4 import BeautifulSoup

URL = 'https://www.senate.gov/general/contact_information/senators_cfm.cfm'

page = requests.post(URL, headers=headers)

soup = BeautifulSoup(page.content, 'html.parser')
print(soup)

使用 BeautifulSoup 抓取网页

Scraping webpage using BeautifulSoup

python

web-scraping

beautifulsoup

html-parsing

web-scraping-language