Selenium returns 与在浏览器中查看的来源不同 html

Question

我正在尝试使用 Selenium 通过单击 site 中的“加载更多”按钮来加载下一页结果。然而，selenium 加载的 html 页面的源代码并没有显示（加载）浏览时可以看到的实际产品。这是我的代码：

from selenium import webdriver      
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import os
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

#browser = webdriver.Firefox()#Chrome('./chromedriver.exe')
URL = "https://thekrazycouponlady.com/coupons-for/costco"
PATIENCE_TIME = 60
LOAD_MORE_BUTTON_XPATH = '//button[@class = "kcl-btn ng-scope"]/span' 
caps = DesiredCapabilities.PHANTOMJS
# driver = webdriver.Chrome(r'C:\Python3\selenium\webdriver\chromedriver_win32\chromedriver.exe')
caps["phantomjs.page.settings.userAgent"] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"
driver = webdriver.PhantomJS(r'C:\Python3\selenium\webdriver\phantomjs-2.1.1-windows\bin\phantomjs.exe',service_log_path=os.path.devnull,desired_capabilities=caps)
driver.get(URL)

while True:
    try:
        time.sleep(20)
        html = driver.page_source.encode('utf-8')
        print(html)
        loadMoreButton = driver.find_element_by_xpath(LOAD_MORE_BUTTON_XPATH)


        loadMoreButton.click()

    except Exception as e:
        print (e)
        break
print ("Complete")

driver.quit()

不确定我是否可以在此处附上示例 html 文件以供参考。无论如何，问题是什么？如何使用 selenium 加载与通过浏览器完全相同的页面？

Answer 1

这可能是由于使用了 PhantomJS，它不再维护并从 Selenium 3.8.1 开始弃用。使用 Chrome 无头代替。

options = Options()
options.headless = True
driver = webdriver.Chrome(CHROMEDRIVER_PATH, chrome_options=options)

Selenium returns 与在浏览器中查看的来源不同 html

Selenium returns different html source than viewed in browser

selenium

webdriver

phantomjs