使用 Python selenium 查找 href link

Question

我正在做一个项目，用 selenium 导航和抓取网站，python。

hemnet_path = "https://www.hemnet.se/"
driver = webdriver.Chrome(path, chrome_options=options)
driver.get(hemnet_path)

脚本导航到 this site 并遍历所有列表以获取价格、位置、hyperlink 等，方法是像这样加载结果元素：

result = driver.find_elements(By.CLASS_NAME,"sold-property-listing")

然后它会读取如下信息：

for r in result:

    try:
        location = r.find_element(By.CLASS_NAME,"sold-property-listing__location")
    except:
        location = 'Missing'

我遇到的问题是 hyperlink-string（屏幕截图）不像其他元素那样“表现”。

我似乎无法通过 class 或 CSS select 或 href select，因为我收到以下错误：

我已经成功地使用 XPATH 得到了 link 像这样：

但问题是，当我遍历结果时，XPATH 始终引用同一个广告，在本例中为 2。我考虑过在 XPATH 中循环遍历整数，但事实证明这是非常不可靠的。

我不确定我在这里遗漏了什么以及为什么我会收到这个奇怪的错误。

如有任何帮助，我们将不胜感激。 HTML 下面提供了...元素。

<li class="sold-results__normal-hit">
        <a class="hcl-card" data-target-blank="true" href="https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-26-1492088" rel="noopener" target="_blank">
  <div class="sold-property-listing">
    <div class="sold-property-listing__location">
      <h2 class="sold-property-listing__heading">
        <span class="property-icon property-icon--result"><svg width="14" height="16" viewBox="0 0 14 16" xmlns="http://www.w3.org/2000/svg"><title>Lägenhet</title><desc><span class="svg-icon__fallback-text">Lägenhet</span></desc><path class="svg-icon__shape" d="M0 1.333v13.334C0 15.403.597 16 1.333 16h4v-5.333H8V16h4c.737 0 1.333-.597 1.333-1.333V1.333C13.333.597 12.737 0 12 0H1.333C.597 0 0 .597 0 1.333" fill="#C1569D" fill-rule="evenodd"></path></svg>
</span>
        <span class="item-result-meta-attribute-is-bold item-link qa-selling-price-title">Karlslundsvägen 26</span>
      </h2>

      <div>
        <span class="hide-element">
          Lägenhet
        </span>
          <span class="item-link">
            Barkarbystaden,
</span>        Järfälla kommun
      </div>
    </div>

    <div class="sold-property-listing__size">
        <div class="clear-children">
          <div class="sold-property-listing__subheading sold-property-listing--left">
              78&nbsp;m²
            &nbsp;
              3&nbsp;rum
          </div>

            <div class="sold-property-listing__fee">
              4&nbsp;233&nbsp;kr/mån
            </div>
        </div>


    </div>

    <div class="sold-property-listing__price">
      <div class="clear-children">
        <span class="sold-property-listing__subheading sold-property-listing--left">
            Slutpris 3&nbsp;295&nbsp;000&nbsp;kr
</span>      </div>

      <div class="clear-children">
          <div class="sold-property-listing__sold-date sold-property-listing--left">
            Såld 5 november 2021
          </div>

          <div class="sold-property-listing__price-per-m2 sold-property-listing--left">
            42&nbsp;244&nbsp;kr/m²
          </div>
      </div>
    </div>

      <div class="sold-property-listing__price-change">
        ±0&nbsp;%
      </div>

    <div class="sold-property-listing__broker">
      Länsförsäkringar Fastighetsförmedling Järfälla
    </div>

      <div class="sold-property-listing__labels">
        <div class="hcl-labels-list hcl-labels-list--row-direction">
            <span class="hcl-labels-list__label-item">
              <span class="hcl-label hcl-label--feature hcl-label--on-white-background">
                Uteplats
              </span>
            </span>
            <span class="hcl-labels-list__label-item">
              <span class="hcl-label hcl-label--feature hcl-label--on-white-background">
                Hiss
              </span>
            </span>
        </div>
      </div>
  </div>

</a>
    </li>

Answer 1

有一个接受 cookie 按钮，您必须先点击该按钮。

其次，你需要定位所有锚标签，使用下面的 CSS 然后滚动到每个标签让 selenium 查看它。

代码：

driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)

driver.get("https://www.hemnet.se/salda/bostader?location_ids%5B%5D=473241&item_types%5B%5D=bostadsratt")
try:
    cookies = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.consent__button-wrapper .hcl-button--primary")))
    driver.execute_script("arguments[0].scrollIntoView(true);", cookies)
    driver.execute_script("arguments[0].click();", cookies)
except:
    print("Could not click")
    driver.quit()
    pass

all_hrefs = driver.find_elements(By.CSS_SELECTOR, "a.hcl-card")

for href in all_hrefs:
    driver.execute_script("arguments[0].scrollIntoView(true);", href)
    link = href.get_attribute('href')
    print(link)

进口：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

输出：

https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-26-1492088
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-fanriksvagen-41c-1490860
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-flygfaltsvagen-19-1490628
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-gripengatan-11,-van-2-1492409
https://www.hemnet.se/salda/lagenhet-2rum-barkarby-jarfalla-kommun-karlslundsvagen-24-1488804
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-kalvshallavagen-42-1488444
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-drakengatan-5-1488052
https://www.hemnet.se/salda/lagenhet-4rum-jarfalla-barkarbystaden-jarfalla-kommun-kalvshallavagen-36-1488008
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-barkarbyvagen-34-1487737
https://www.hemnet.se/salda/lagenhet-3rum-barkarby-jarfalla-kommun-flygarvagen-31-1487734
https://www.hemnet.se/salda/lagenhet-1rum-barkarby-jarfalla-kommun-jaktvagen-2-1487278
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-viggengatan-8-1487099
https://www.hemnet.se/salda/lagenhet-5rum-jarfalla-barkarbystaden-jarfalla-kommun-karlslundsvagen-3,-4-tr-1485991
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-jarfalla-kommun-barkarbyvagen-69-1485845
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-flygfaltsvagen-23-1485339
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-barkarbyvagen-36-1485338
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-majorsvagen-16-1484118
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-parkvagen-8,-2tr,-1482755
https://www.hemnet.se/salda/lagenhet-3rum-barkarby-jarfalla-kommun-flygarvagen-43-1482418
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-kalvshallavagen-32-1482219
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-barkarbyvagen-66-1481399
https://www.hemnet.se/salda/lagenhet-4rum-barkarbystaden-jarfalla-kommun-barkarbyvagen-34-1480551
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-parkvagen-8-1479755
https://www.hemnet.se/salda/lagenhet-3,5rum-jarfalla-barkarbystaden-jarfalla-kommun-flygfaltsvagen-19-1478682
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-kalvshallavagen-30-1477736
https://www.hemnet.se/salda/lagenhet-4rum-barkarby-jarfalla-kommun-skalbyvagen-17-1477210
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-jarfalla-kommun-viggengatan-11-1477547
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-barkarbystaden-jarfalla-kommun-majorsvagen-18-1478670
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-viggengatan-11-1476409
https://www.hemnet.se/salda/lagenhet-1rum-barkarbystaden-jarfalla-kommun-drakengatan-10-1475228
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-gripengatan-20-1474602
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-barkarbystaden-jarfalla-kommun-drakengatan-3,-2tr-1473715
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-viggengatan-10-1472189
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-12-1472092
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-kalvshallavagen-44-van-4-4-1471582
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-17-1471427
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-parkvagen-8,-3tr,-1471250
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-26-utan-balkong-1471234
https://www.hemnet.se/salda/lagenhet-3rum-barkarby-jarfalla-kommun-flygarvagen-33-1471126
https://www.hemnet.se/salda/lagenhet-3rum-jarfalla-barkarbystaden-jarfalla-kommun-parkvagen-5-1471961
https://www.hemnet.se/salda/lagenhet-4rum-barkarbystaden-jarfalla-kommun-kalvshallavagen-36-1468708
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-majorsvagen-18-1467460
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-majorsvagen-16-1467443
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-barkarbystaden-jarfalla-kommun-parkvagen-8-1467324
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-barkarby-jarfalla-kommun-pilotvagen-11-1465674
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-viggengatan-12-1465518
https://www.hemnet.se/salda/lagenhet-2rum-barkarby-jarfalla-kommun-attackvagen-4-1465147
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-viggengatan-8-1463893
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-18-1461358
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-viggengatan-10,-van-4-1460508

使用 Python selenium 查找 href link

Finding href link with Python selenium

html

python

selenium

href