Selenium with Firefox：FirefoxProfile 在第二次使用时停止工作

Question

我正在尝试将 selenium 与 firefox 结合使用来创建我想重复使用的浏览器的配置文件。我首先用我想要的扩展加载它，然后调用一个函数，将该配置文件作为参数用于抓取。

profile = webdriver.FirefoxProfile()
profile.add_extension(extension='extension/1/path')
profile.add_extension(extension='extension/2/path')


def sele_scrape(profile,url):

  options = Options()
  options.headless = True

  driver = webdriver.Firefox(firefox_profile=profile,options=options,executable_path='path/to/geckodriver')

  driver.get(url)
  source=driver.page_source
  driver.quit()


sele_scrape(profile,'url1')
sele_scrape(profile,'url2')

URL 的顺序或我使用的 URL 并不重要。第二个总是导致这个错误：

File "(the python file for this project)", line 37, in <module>
    sele_scrape(profile,'url2')
  File "(the python file for this project)"
    driver = webdriver.Firefox(firefox_profile=profile,options=options,executable_path='path/to/geckodriver')

File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/firefox/webdriver.py", line 166, in __init__
    capabilities.update(options.to_capabilities())
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/firefox/options.py", line 180, in to_capabilities
    opts["profile"] = self._profile.encoded
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 173, in encoded
    self.update_preferences()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 102, in update_preferences
    self._write_user_prefs(self.default_preferences)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 226, in _write_user_prefs
    with open(self.userPrefs, "w") as f:

FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/8j/lrckmwqs5bs9b4srrjbj6r5c0000gn/T/tmpiluv90lz/user.js'

这一行

FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/8j/lrckmwqs5bs9b4srrjbj6r5c0000gn/T/tmpiluv90lz/user.js'

对我来说特别奇怪，因为我不知道它试图指的是什么，也不知道为什么。

任何帮助，甚至只是我可以寻找的方向，将不胜感激！

Answer 1

可能当您 quit() 然后它会删除个人资料 - 您将不得不重新创建个人资料。

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# --- functions ---

def sele_scrape(url):

    profile = webdriver.FirefoxProfile()
    #profile.add_extension(extension='extension/1/path')
    #profile.add_extension(extension='extension/2/path')
    
    options = Options()
    #options.headless = True
    
    driver = webdriver.Firefox(firefox_profile=profile, options=options)#,executable_path='path/to/geckodriver')
    
    driver.get(url)
    source = driver.page_source
    driver.quit()

# --- main ---

sele_scrape('https://whosebug.com')
sele_scrape('https://whosebug.com')

如果我使用 close() 而不是 quit()

，代码可以正常工作

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# --- functions ---

def sele_scrape(url, profile, options):
   
    driver = webdriver.Firefox(firefox_profile=profile, options=options)#,executable_path='path/to/geckodriver')
    
    driver.get(url)
    source = driver.page_source
    driver.close()

# --- main ---

profile = webdriver.FirefoxProfile()
#profile.add_extension(extension='extension/1/path')
#profile.add_extension(extension='extension/2/path')

options = Options()
#options.headless = True

sele_scrape('https://whosebug.com', profile, options)
sele_scrape('https://whosebug.com', profile, options)

坦率地说，如果应该重用配置文件，那么我将只创建一次 driver，然后跳过 close()。

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# --- functions ---

def sele_scrape(url, driver):
   
    driver.get(url)
    source = driver.page_source

# --- main ---

profile = webdriver.FirefoxProfile()
#profile.add_extension(extension='extension/1/path')
#profile.add_extension(extension='extension/2/path')

options = Options()
#options.headless = True

driver = webdriver.Firefox(firefox_profile=profile, options=options)#,executable_path='path/to/geckodriver')

sele_scrape('https://whosebug.com', driver)
sele_scrape('https://whosebug.com', driver)

driver.close()

Selenium with Firefox：FirefoxProfile 在第二次使用时停止工作

Selenium with Firefox: FirefoxProfile stops working when used a second time

python

selenium

web-scraping

selenium-firefoxdriver

selenium-webdriver