我正在尝试通过单击下一步按钮获取更多链接来提取报纸数据

Question

我正在尝试通过单击链接来提取商业标准报纸经济版块数据，但我没有成功。

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# Here Chrome is used

# URL of website
url = "https://www.business-standard.com/search?q=economy"

# Opening the website
driver.get(url)

# #button = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='#']"))).click()
button = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[contains(., 'Next')]]"))).click()

Answer 1

这里有几个问题：

1. 您需要关闭浮动横幅
2. 您使用了错误的定位器。
3. 点击立即返回的元素时，无需在左侧定义button元素

这应该会更好：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# Here Chrome is used

# URL of website
url = "https://www.business-standard.com/search?q=economy"

# Opening the website
driver.get(url)

wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.btnno"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[@class='next-colum']"))).click()