为什么Selenium有时找不到href而不会报错

Question

我是爬行新手，正在尝试爬行。 https://www.stradivarius.com/tr/en/woman/clothing/shop-by-product/sweatshirts-c1390587.html 网页，有时我可以获得 href，但通常代码给了我空列表？你有什么建议吗？

这是包裹：

import requests
from tqdm import tqdm
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import *
from selenium.webdriver.support import expected_conditions as EC
import time
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import json
import pandas as pd
import warnings
warnings.filterwarnings(action='ignore')
from unidecode import unidecode
import re
import time
from webdriver_manager.chrome import ChromeDriverManager
browser = webdriver.Chrome(ChromeDriverManager().install())

urlist = []
browser.get('https://www.stradivarius.com/tr/kad%C4%B1n/giyim/%C3%BCr%C3%BCne-g%C3%B6re-al%C4%B1%C5%9Fveri%C5%9F/sweatshi%CC%87rt-c1390587.html')
html = browser.page_source
soup = BeautifulSoup(html)
browser.implicitly_wait(90)
product_links=soup.find_all('a', {'id':'hrefRedirectProduct'})
for a in product_links:
    urlist.append(product_links["href"])

Answer 1

可能数据还没有呈现。你有 .implicitly_wait(90) 但它是在你已经拉出 html 之后。因此，您需要在代码中将其上移。

urlist = []
browser.get('https://www.stradivarius.com/tr/kad%C4%B1n/giyim/%C3%BCr%C3%BCne-g%C3%B6re-al%C4%B1%C5%9Fveri%C5%9F/sweatshi%CC%87rt-c1390587.html')
browser.implicitly_wait(90)  #<--- wait for the page to render BEFORE...
html = browser.page_source  # ...grabing the html source
soup = BeautifulSoup(html)
product_links=soup.find_all('a', {'id':'hrefRedirectProduct'})
for a in product_links:
    urlist.append(product_links["href"])

更好的解决方案可能是从源头获取数据。

这是否包含您想要的 href？

import requests
import pandas as pd

url = 'https://www.stradivarius.com/itxrest/2/catalog/store/54009571/50331068/category/1390587/product?languageId=-43&appId=1'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36'}
jsonData = requests.get(url, headers=headers).json()

df = pd.DataFrame(jsonData['products'])

输出：

print(df['productUrl'])
0                            kolej-sweatshirt-l06710711
1     oversize-hard-rock-cafe-baskl-sweatshirt-l0670...
2     oversize-hard-rock-cafe-baskl-sweatshirt-l0670...
3     oversize-hard-rock-cafe-kapusonlu-sweatshirt-l...
4                         fermuarl-sweatshirt-l06521718
                       
60     fermuarl-oversize-kapusonlu-sweatshirt-l06765643
61                   dikisli-basic-sweatshirt-l06519703
62    jogging-fit-pantolon-ve-sweatshirt-seti-l01174780
63                          naylon-sweatshirt-l08221191
64                   dikisli-basic-sweatshirt-l06519703
Name: productUrl, Length: 65, dtype: object

为什么Selenium有时找不到href而不会报错

Why Selenium sometimes can't find href witout error

selenium

beautifulsoup

web-crawler