Error: "Element is not currently visible and so may not be interacted with" with selenuim

Question

我想抓取 google playstore 搜索结果的完全呈现的网页。

完全呈现的页面包含所有搜索项目，而非呈现的页面只有 20 个项目。（请参阅https://play.google.com/store/search?q=best&c=apps&hl=en）

我尝试使用 selenium 抓取页面，但收到以下错误消息。

Traceback (most recent call last):
File "play_test_2.py", line 25, in test_play_test2
driver.find_element_by_id("show-more-button").click()
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 65, in click
self._execute(Command.CLICK_ELEMENT)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 385, in _execute
return self._parent.execute(command, params)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 173, in execute
self.error_handler.check_response(response)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response
raise exception_class(message, screen, stacktrace)
ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with
Stacktrace:
at fxdriver.preconditions.visible (file:///var/folders/8_/n90htn1d0_j4h7l9yt04chl80000gn/T/tmpErWdUz/extensions/fxdriver@googlecode.com/components/command-processor.js:8959:5)
at DelayedCommand.prototype.checkPreconditions_ (file:///var/folders/8_/n90htn1d0_j4h7l9yt04chl80000gn/T/tmpErWdUz/extensions/fxdriver@googlecode.com/components/command-processor.js:11618:1)
at DelayedCommand.prototype.executeInternal_/h (file:///var/folders/8_/n90htn1d0_j4h7l9yt04chl80000gn/T/tmpErWdUz/extensions/fxdriver@googlecode.com/components/command-processor.js:11635:11)
at fxdriver.Timer.prototype.setTimeout/<.notify          (file:///var/folders/8_/n90htn1d0_j4h7l9yt04chl80000gn/T/tmpErWdUz/extensions/fxdriver@googlecode.com/components/command-processor.js:548:5)

以下代码由 Selenuim IDE.

# -*- coding: utf-8 -*-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
import unittest, time, re

class PlayTest2(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Firefox()
        self.driver.implicitly_wait(30)
        self.base_url = "https://play.google.com/"
        self.verificationErrors = []
        self.accept_next_alert = True

    def test_play_test2(self):
        driver = self.driver
        driver.get(self.base_url + "/store/search?q=best&c=apps")
        driver.find_element_by_id("gbqfb").click()
        driver.find_element_by_id("show-more-button").click()
        driver.find_element_by_id("show-more-button").click()

    def is_element_present(self, how, what):
        try: self.driver.find_element(by=how, value=what)
        except NoSuchElementException, e: return False
        return True

    def is_alert_present(self):
        try: self.driver.switch_to_alert()
        except NoAlertPresentException, e: return False
        return True

    def close_alert_and_get_its_text(self):
        try:
            alert = self.driver.switch_to_alert()
            alert_text = alert.text
            if self.accept_next_alert:
                alert.accept()
            else:
                alert.dismiss()
            return alert_text
        finally: self.accept_next_alert = True

    def tearDown(self):
        self.driver.quit()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

我认为发生错误是因为 google 商店搜索结果页面有 "show more" 按钮，该按钮在满足某些特定条件时显示，例如向下和向上滚动然后向下滚动再次.

如何解决此问题并抓取 google 个搜索结果页面？

Answer 1

test_play_test2 调用

driver.find_element_by_id("show-more-button").click()

两次.
我想在第一次点击 selenium 后找不到相同的元素并且失败了。只需从此方法中删除最新的一行。

更新：你说的对，按钮的问题只有在页面滚动后才会出现。因此我们必须滚动 window。以下 javascript 命令很有用

((JavascriptExecutor) driver).executeScript("window.scrollTo(0,document.body.scrollHeight);");

一个命令是不够的，我们做了好几次：

for (int i=0; i<5; i++ ){
    ((JavascriptExecutor) driver).executeScript("window.scrollTo(0,document.body.scrollHeight);");
    try{
        (new WebDriverWait(driver, 5/*sec*/))
                        .until(ExpectedConditions.visibilityOf(element));
        break;
    }
    catch (org.openqa.selenium.TimeoutException e){
    }
}
if(element.isDisplayed()){
    element.click();
}

抱歉，我在这个例子中使用了 Java 代码，但我希望这会让你明白。

Error: "Element is not currently visible and so may not be interacted with" with selenuim

Error: "Element is not currently visible and so may not be interacted with" with selenuim

javascript

jquery

selenium