Python, selenium webdriver chrome, 从众多网页元素中获取页面源

Question

你好，我在 python 中为 chrome 使用 selenium webdriver，并试图从 https://www.google.com/travel/things-to-do

中抓取一些数据

我在这里专注于可以在这里看到的地点描述：

因此，为了获得每个单独的描述，我必须按下景点并将 html 保存到列表中，以便将来使用 BeautifulSoup 进行解析。

每次点击都会刷新页面，所以我在考虑以某种方式计算所有显示的景点，然后循环点击每个景点并保存描述。

有人知道如何处理吗？继承人简单的代码，让你到我被困的地方

chrome_options = webdriver.ChromeOptions()
#chrome_options.headless = True
chrome_options.add_argument('--incognito')
#chrome_options.add_argument('--headless')
s=Service(ChromeDriverManager().install())
driver = webdriver.Chrome(executable_path=r"\chromedriver.exe", options=chrome_options, service=s)
driver.get("https://www.google.com/travel/things-to-do/see-all?dest_mid=%2Fm%2F081m_&dest_state_type=sattd&dest_src=yts&q=Warszawa#ttdm=52.227486_21.004941_13&ttdmf=%252Fm%252F0862m")

# If you are not running webdriver in incognito mode you might skip the below button since it goes through accepting cookies
button = driver.find_element_by_xpath("/html/body/c-wiz/div/div/div/div[2]/div[1]/div[4]/form/div[1]/div/button")

button.click()
time.sleep(1)

objects = driver.find_elements_by_class_name('f4hh3d')

for k in objects:
    k.click()
    time.sleep(5)

Answer 1

对于每个景点索引，您可以单击它来打开详细信息，获取详细信息，关闭详细信息，再次获取景点列表并前往下一个景点。
这样的事情应该有效：

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver.get("https://www.google.com/travel/things-to-do/see-all?dest_mid=%2Fm%2F081m_&dest_state_type=sattd&dest_src=yts&q=Warszawa#ttdm=52.227486_21.004941_13&ttdmf=%252Fm%252F0862m")
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.f4hh3d")))
time.sleep(1)
attractions = driver.find_elements_by_css_selector('div.f4hh3d')
for i in range(len(attractions)):
    attractions = driver.find_elements_by_css_selector('div.f4hh3d')
    attractions[i].click()
    description = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div[jsname="BEZjkb"] div[jsname="bN97Pc"]'))).text
    #do with the description what you want
    #close the attraction by clicking the button
    driver.find_element_by_css_selector('div.reh1ld button')

Python, selenium webdriver chrome, 从众多网页元素中获取页面源

Python, selenium webdriver chrome, obtain page source from inside of many web elements

python

selenium

selenium-chromedriver

selenium-webdriver