如何使用Selenium获取房屋数据

Question

我正在尝试从网页 http://www.har.com/4311-Childress-St/sale_40763013 获取数据。它有房屋地址、价格和其他信息。我试图获取所有数据，但只成功检索到地址、城市和邮编。下面是我的代码。我如何获取其他信息，例如县、故事等？

def getHarData(driver):
driver.get("http://www.har.com/4311-Childress-St/sale_40763013")
try:
    address = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "heading_22")))
    cityzip = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "sub_heading")))
    #price = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "heading_22 pb15"))) 
    print (address.text + ", " + cityzip.text+ ", " +price.text)
except TimeoutException:
    print("data not found")

Answer 1

如果你只需要一些特定的字段，我会做一个很好的可重用函数来通过字段name/label:[=13=获取字段值]

def get_field_value(driver, field):
    field = field.capitalize() + ":"
    return driver.find_element_by_xpath("//div[@class = 'dc_label' and . = '%s']/following-sibling::div[@class = 'dc_value']" % field).text

用法：

county = get_field_value(driver, "county")
print(county)  # prints "Harris County"

完整的工作示例：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait


def get_field_value(driver, field):
    field = field.capitalize() + ":"
    return driver.find_element_by_xpath("//div[@class = 'dc_label' and . = '%s']/following-sibling::div[@class = 'dc_value']" % field).text

driver = webdriver.Firefox()
driver.get("http://www.har.com/4311-Childress-St/sale_40763013")

# wait for the page to load
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CLASS_NAME, "dc_title")))

county = get_field_value(driver, "county")
print(county)

如何使用Selenium获取房屋数据

How to use Selenium to get the housing data

python

selenium

web

web-scraping

screen-scraping