网页抓取 - Table 没有 Class 或 ID

Question

我正在做一个项目，需要从以下网站抓取 table - https://www.cmegroup.com/markets/energy/crude-oil/brent-crude-oil.settlements.html#tradeDate=03%2F31%2F2022

我面临的问题是，当我检查元素时，table 似乎没有类。

我是 web-scraping/BeautifulSoup 的新手，如果能得到任何帮助，我将不胜感激。

url: https://www.cmegroup.com/markets/energy/crude-oil/brent-crude-oil.settlements.html#tradeDate=03%2F31%2F2022

Answer 1

实际上，url 又名 table 是动态的，您可以借助 selenium、bs4 和 pandas 获取它，如下所示：

脚本：

import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import pandas as pd
from bs4 import BeautifulSoup
driver = webdriver.Chrome(ChromeDriverManager().install())

url ="https://www.cmegroup.com/markets/energy/crude-oil/brent-crude-oil.settlements.html#tradeDate=03%2F31%2F2022"
    
driver.maximize_window()
driver.get(url)
time.sleep(5)

soup = BeautifulSoup(driver.page_source, 'lxml')

df = pd.read_html(str(soup))[0]
print(df)

输出：

   Month  Open  High  Low  Last  Change  Settle  Est. Volume  Prior day OI
1   MAY 22     -     -    -     -    UNCH  113.45            0             0
2   JUN 22     -     -    -     -   -6.73  104.71            0             0
3   JLY 22     -     -    -     -   -5.91  102.73            0             0
4   AUG 22     -     -    -     -   -5.09  100.98            0             0
5   SEP 22     -     -    -     -   -4.34   99.43            0             0
6   OCT 22     -     -    -     -   -3.74   98.04            0             0
7   NOV 22     -     -    -     -   -3.29   96.75            0             0
8   DEC 22     -     -    -     -   -2.93   95.51            0             0
9   JAN 23     -     -    -     -   -2.62   94.35            0             0
10  FEB 23     -     -    -     -   -2.33   93.33            0             0
11  MAR 23     -     -    -     -   -2.04   92.53            0             0
12  APR 23     -     -    -     -   -1.78   91.78            0             0

网页抓取 - Table 没有 Class 或 ID

Web Scraping - Table Without Class or ID

beautifulsoup

scrapy

python-3.x