遍历 Selenium 中的列表并将结果保存到数据框中

Question

我正在尝试遍历列表，通过 selenium 在网页上搜索并将结果存储在 df 中。如何将每个列表项的循环结果存储到 df 中？

from selenium.webdriver.common.keys import Keys
import pandas as pd
import numpy as np


url  = 'https://au.finance.yahoo.com/australia/'
driver_path = 'chromedriver.exe'
browser = Chrome(executable_path= driver_path)
loop_search = browser.find_element_by_id('yfin-usr-qry')

search_companies = ['Commonwealth Bank','Rio Tinto','Wesfarmers']


for i in search_companies:
    loop_search.send_keys(i)
    browser.find_element_by_id('search-button').click()
    comp = browser.find_element_by_id('quote-header-info').text
    df3 = [comp]```

Still fairly new to Python! Thank you!

Answer 1

如果您只是运行您的代码并执行 print(comp)

您会看到以下错误：

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=99.0.4844.74)

所以在保存到DF之前，我们需要解决这个问题：

可以通过在循环中像这样重新定义 web 元素来解决这个问题：

loop_search = wait.until(EC.visibility_of_element_located((By.ID, "yfin-usr-qry")))

保存到DF的完整代码：

driver_path = 'chromedriver.exe'
browser = Chrome(executable_path= driver_path)
wait = WebDriverWait(driver, 20)
url  = 'https://au.finance.yahoo.com/australia/'
driver.get(url)

search_companies = ['Commonwealth Bank','Rio Tinto','Wesfarmers']

company_details_lst = []
for i in search_companies:
    time.sleep(2)
    loop_search = wait.until(EC.visibility_of_element_located((By.ID, "yfin-usr-qry")))
    loop_search.send_keys(i)
    time.sleep(2)
    wait.until(EC.element_to_be_clickable((By.ID, "search-button"))).click()
    time.sleep(2)
    comp = wait.until(EC.element_to_be_clickable((By.ID, "quote-header-info"))).text
    company_details_lst.append(comp)
    #print(comp)

data = {
         'Details': company_details_lst
        }

df = pd.DataFrame.from_dict(data)
df.to_csv('output.csv', index = 0

进口：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

在运行编写代码后，您应该会在项目文件夹中看到一个名为 output.csv

的 csv 文件

内部内容为：

遍历 Selenium 中的列表并将结果保存到数据框中

iterate over a list in Selenium and saving the results into a dataframe

python

pandas

selenium

jupyter