Selenium For Loop 卡在弹出消息上

Question

我有一个几乎可以按预期工作的应用程序。在第 5 个实例的循环运行秒后出现问题。搜索状态有两个结果导致相同的最终结果。发生这种情况时，我想 select 两者中的第一个。

弹出消息如下所示：

我正在使用以下代码创建列表然后循环：

from selenium import webdriver
import pandas as pd
import random
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.proxy import Proxy, ProxyType
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time




#service = Service('C:\Program Files\Chrome Driver\chromedriver.exe')
URL = "https://mor.nlm.nih.gov/RxClass/search?query=ALIMENTARY TRACT AND METABOLISM"
driver = webdriver.Chrome('C:\Program Files\Chrome Driver\chromedriver.exe')
driver.get(URL)

category = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.drug_class img+a")))]
classid = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.propText")))]
dfObj = pd.DataFrame(category) 
dfObj.columns =['Category']
dfObj.dropna(inplace = True)
new = dfObj["Category"].str.split("(", n = 1, expand = True)
dfObj["New Category"]= new[0]
dfObj["Count"]= new[1]
dfObj.drop(columns =["Category"], inplace = True)
dfObj['Count'] = dfObj['Count'].str.rstrip(')')
dfObj['IsNumber'] = dfObj['Count'].str.isnumeric()
dfObj = dfObj[(dfObj['IsNumber'] == True)]
searchcat = dfObj['New Category'].tolist()
print(searchcat)
dfObj.to_csv('tabledf.csv',index=False)
time.sleep(8)
driver.quit()

for search in searchcat:
    page = f"https://mor.nlm.nih.gov/RxClass/search?query={search}"
    driver = webdriver.Chrome('C:\Program Files\Chrome Driver\chromedriver.exe')
    driver.get(page)
    time.sleep(4)
    table = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'tr.dbsearch')))
    time.sleep(4)
    filename = search[0:30]+'table.csv'
    pd.read_html(driver.page_source)[1].iloc[:,:-1].to_csv(filename,index=False)        
    time.sleep(4)
    driver.quit()

如果我手动点击每个搜索结果，循环将继续运行。但是，我希望 selenium 总是 select 第一个选项。我该怎么做？

更新代码：

from selenium import webdriver
import pandas as pd
import random
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.proxy import Proxy, ProxyType
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait, TimeoutException
import time

with webdriver.Chrome('C:\Program Files\Chrome Driver\chromedriver.exe') as driver:

    URL = "https://mor.nlm.nih.gov/RxClass/search?query=ALIMENTARY TRACT AND METABOLISM"
    driver.get(URL)
    category = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.drug_class img+a")))]
    dfObj = pd.DataFrame(category) 
    dfObj.columns =["Category"]
    dfObj.dropna(inplace = True)
    new = dfObj["Category"].str.split("(", n = 1, expand = True)
    dfObj["New Category"]= new[0]
    dfObj["Count"]= new[1]
    dfObj.drop(columns =["Category"], inplace = True)
    dfObj["Count"] = dfObj["Count"].str.rstrip(')')
    dfObj["IsNumber"] = dfObj["Count"].str.isnumeric()
    dfObj = dfObj[(dfObj["IsNumber"] == True)]
    searchcat = dfObj["New Category"].tolist()
    dfObj.to_csv('tabledf.csv',index=False)
    time.sleep(3)

    for search in searchcat:
        page = f"https://mor.nlm.nih.gov/RxClass/search?query={search}"
        driver = webdriver.Chrome('C:\Program Files\Chrome Driver\chromedriver.exe')
        driver.get(page)
        table = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'tr.dbsearch')))
        modal_wait = WebDriverWait(driver, 1)
        try:
            modal_el = modal_wait.until(EC.visibility_of_element_located((By.ID, 'optionModal')))
            modal_el.find_element(By.CSS_SELECTOR, '.uloption').click()
        except TimeoutException:
            pass
        filename = search[0:30]+'table.csv'
        classid = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.table-responsive div.propText strong:nth-child(2)")))]
        classname = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.table-responsive div.propText strong:nth-child(1)")))]
        classtype = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.table-responsive div.propText strong:nth-child(3)")))]
        df = pd.read_html(driver.page_source)[1].iloc[:,:-1]
        df["ClassID"] = pd.Series(classid)
        df["ClassName"] = pd.Series(classname)
        df["ClassType"] = pd.Series(classtype)
        df.to_csv(filename,index=False)        
        time.sleep(4)
        driver.quit()

Answer 1

首先，我建议您使用 with 上下文管理器。它将自行处理 opening/closing 驱动程序 (Chrome)。这确保如果出现任何异常，它仍将被关闭。

为此，请使用：

with webdriver.Chrome() as driver:
   ...

在你的代码中，我看到你 close/open 每个 URL 一个新的浏览器。这不是必需的，不这样做会加速您的脚本。只需使用 driver.get() 更改 URL.

对于您的主要问题，只需添加一部分代码即可检测模态并选择第一个选项。类似的东西

modal_wait = WebDriverWait(driver, 1)
try:
    modal_el = modal_wait.until(EC.element_to_be_clickable((By.ID, 'optionModal')))
    modal_el.find_element(By.CSS_SELECTOR, '.uloption').click()
except TimeoutException:
    pass

您必须包括以下导入：

from selenium.webdriver.support.wait import WebDriverWait, TimeoutException
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

Selenium For Loop 卡在弹出消息上

Selenium For Loop Stuck on a popup message

python

selenium

web-scraping

selenium-chromedriver