使用 Python Selenium 进行 AmCharts 抓取

Question

我正在尝试从此 URL

中抓取 JSON 对象中的 AmCharts 图形值

使用以下命令

driver.execute_script("AmCharts.charts[0].dataProvider")

在脚本中，它 returns None 而浏览器控制台 returns JSON 对象与 AmCharts 数据

网页界面是这样的

如何正确检索此 dataProvider 数组。提前致谢。

Answer 1

图表不在 Selenium 视口中，因此首先我们必须处理垂直滚动到所需图表的问题，然后我看到 //*[name()='tspan'] xpath 包含 [ 中存在的水平和垂直值=28=] 还有。

代码：

driver_path = r'D:\chromedriver.exe'

driver = webdriver.Chrome(driver_path)
driver.maximize_window()
wait = WebDriverWait(driver, 30)

driver.get("https://eg.pricena.com/en/product/oppo-reno-5g-price-in-egypt")

driver.execute_script("window.scrollTo(0, 1000)")

ele = driver.find_element_by_xpath("//*[name()='svg']")
driver.execute_script("arguments[0].scrollIntoView(true);", ele)

char_val = []
for elem in driver.find_elements(By.XPATH, "//*[name()='tspan']"):
    print(elem.text)

进口：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

输出：

Mar
May
Jul
Sep
Nov
Mar
May
Jul
Sep
Nov
5,000
5,500
6,000
6,500
7,000
7,500
8,000

Process finished with exit code 0

Answer 2

您需要将 return 添加到 execute_script 调用中才能访问脚本中的值，例如driver.execute_script("return AmCharts.charts[0].dataProvider")

完整代码如下：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait

driver_path = r"path/to/chromedriver"

driver = webdriver.Chrome(driver_path)
driver.maximize_window()
wait = WebDriverWait(driver, 30)

driver.get("https://eg.pricena.com/en/product/oppo-reno-5g-price-in-egypt")

# scroll into the div so that the chart will render
driver.execute_script("document.getElementById('product_pricechart').scrollIntoView()")

# wait until the chart div has been rendered before accessing the data provider
wait.until(lambda x: x.find_element_by_class_name("amcharts-chart-div").is_displayed())

# display chart data
print(driver.execute_script("return AmCharts.charts[0].dataProvider"))

driver.close()

使用 Python Selenium 进行 AmCharts 抓取

AmCharts Scraping Using Python Selenium

python

selenium

amcharts

web-scraping