如何使用 Selenium 提取 NewEgg 上显卡发布的所有标题文本 Python

How to extract all the title text for Graphics Card Posting on NewEgg using Selenium Python

我目前正在开发一个 Python 脚本来提取 RTX 3080 显卡的信息,我正在 运行 遇到问题让我的脚本为每个帖子获取每个显卡标题页。我的脚本如下

from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import time


options = Options()

PATH = 'C://Program Files (x86)//chromedriver.exe'

driver = webdriver.Chrome(PATH)

url = 'https://www.newegg.com/p/pl?d=RTX+3080'
actions = ActionChains(driver)

driver.maximize_window()
driver.get(url)

card_path = '/html/body/div[8]/div[3]/section/div/div/div[1]/div/dl[1]/dd/ul[2]/li/a'
desktop_graphics_cards = driver.find_element(By.XPATH, card_path)
desktop_graphics_cards.click()
time.sleep(5)

graphics_card = []
shipping_cost = []
price = []
total_cost = []


card_data = driver.find_elements(By.XPATH, '//div[@class = "item-cell"]')
print(card_data)
for card in card_data:
    graphics_card0 = card.find_element(By.XPATH, '//a[@title = "View Details"]').get_attribute('innerText')
    print(graphics_card0)

每当我 运行 我的脚本时,它都会给我第一张显卡的标题,但会为其余显卡标题的每个帖子重复相同的标题。比如我做print(len(card_data))时,它returns46个贴子(页面上每个“item-cell”的显卡数量),但是我的print( graphics_card0) returns 类似“MSI Gaming GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 ATX 显卡 RTX 3080 GAMING Z TRIO 10G LHR”之类的东西出现了 46 次。关于如何进行的任何建议?我还将在下面附上 html 的图片:

截图一:

截图二:

我对您的部分代码进行了更改:

card_data = driver.find_elements(By.XPATH, '//a[@title = "View Details"]')
for card in card_data:
    graphics_card0 = card.get_attribute('innerText')
    print(graphics_card0)

然后输出变成:

ASUS TUF Gaming NVIDIA GeForce RTX 3080 V2 OC Edition Graphics Card (PCIe 4.0, 10GB GDDR6X, LHR, HDMI 2.1, DisplayPort 1.4a, Dual Ball Fan Bearings, Military-grade Certification, GPU Tweak II)
MSI Gaming GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 ATX Video Card RTX 3080 GAMING Z TRIO 10G LHR
GIGABYTE AORUS GeForce RTX 3080 MASTER 10GB GDDR6X PCI Express 4.0 ATX Video Card GV-N3080AORUS M-10GD (rev. 3.0) (LHR)
GIGABYTE Gaming OC GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 ATX Video Card GV-N3080GAMING OC-10GD (rev. 2.0) (LHR)
MSI Suprim GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 Video Card RTX 3080 SUPRIM X 10G LHR
MSI Ventus GeForce RTX 3080 PCI Express 4.0 Video Card RTX 3080 VENTUS 3X PLUS 10G OC LHR
GIGABYTE Gaming GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card GV-N308TGAMING OC-12GD
ASUS TUF Gaming GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 Video Card TUF-RTX3080TI-O12G-GAMING
MSI Gaming GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 Video Card RTX 3080 Ti Gaming X Trio 12G
ASUS ROG Strix NVIDIA GeForce RTX 3080 Ti OC Edition Gaming Graphics Card (PCIe 4.0, 12GB GDDR6X, HDMI 2.1, Axial-tech Fan Design, 2.9-Slot, Super Alloy Power II, ASUS Auto-Extreme Technology)
GIGABYTE Geforce RTX 2060 6G Graphics Card, 2 x WINDFORCE Fans, 6GB 192-Bit GDDR, GV-N2060D6-6GD Video Card
Colorful AXE RTX 3060Ti 12G LHR Video Card PCI Express 4.0 Nvidia Graphics Card GPU Video Cards for Desktop Graphics Cards
LISM GTX 650ti 2GB (Computer display is 1050ti4G) GDDR5 128 Bit for PC Low-Noise and High-Definition Desktop Discrete Video Card for PC
UNIKA Radeon RX 560 million Drawing 4GD5 V2 RX 560 4G D5 Video Card for Gaming
NVIDIA GeForce RTX 3080 Ti Founders Edition 12GB GDDR6X
ZOTAC AMP Holo GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 Video Card ZT-A30810F-10P
ASUS ROG STRIX GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 x16 ATX Video Card ROG-STRIX-RTX3080-O10G-WHITE-V2 (LHR)
GIGABYTE AORUS GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card GV-N308TAORUS M-12GD
Graphics HD7450 2GB GDDR3 64 Bit PCIE 2.0 HDMI-Compatible+VGA+DVI Interface Video Card with Cooling Fan
MSI Ventus GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 Video Card RTX 3080 VENTUS 3X 10G OC LHR
GIGABYTE AORUS GeForce RTX 3080 XTREME WATERFORCE WB 10G (rev. 2.0) Graphics Card, WATERFORCE Water Block Cooling System, 10GB 320-bit GDDR6X, GV-N3080AORUSX WB-10GD Rev2.0 Video Card (LHR)
GIGABYTE AORUS GeForce RTX 3080 MASTER 10G (rev. 2.0) Graphics Card, Max Covered Cooling, 10GB 320-bit GDDR6X, GV-N3080AORUS M-10GD Rev2.0 Video Card
GIGABYTE Vision GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card GV-N308TVISION OC-12GD
GIGABYTE AORUS GeForce RTX 3080 XTREME 10GB GDDR6X PCI Express 4.0 ATX Video Card GV-N3080AORUS X-10GD (rev. 2.0) (LHR)
ASUS ROG STRIX GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 x16 Video Card ROG-STRIX-LC-RTX3080TI-O12G-GAMING
ZOTAC Trinity OC GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card ZT-A30810J-10P
EVGA GeForce RTX 3080 Ti 12GB XC3 ULTRA 12G-P5-3955-KR Video Conference Bundle (w/ Speaker Mic, Monitor, Webcam)
GIGABYTE Vision OC GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 ATX Video Card GV-N3080VISION OC-10GD (rev. 2.0) (LHR)
GIGABYTE AORUS GeForce RTX 3080 XTREME WATERFORCE 10GB GDDR6X PCI Express 4.0 ATX Video Card GV-N3080AORUSX W-10GD (rev. 2.0) (LHR)
MSI Gaming GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 ATX Video Card RTX 3080 GAMING Z TRIO 10G LHR
GIGABYTE AORUS GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card GV-N308TAORUSX W-12GD
MSI Ventus GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card RTX 3080 Ti Ventus 3X 12G OC
ASUS ROG Strix GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 Video Card ROG-STRIX-RTX3080-O10G-GAMING
ZOTAC GAMING GeForce RTX 3080 Trinity OC White Edition ZT-A30800K-10P
MSI Ventus GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card RTX 3080 Ti VENTUS 3X 12G
MSI Suprim GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card RTX 3080 Ti SUPRIM X 12G
GIGABYTE GeForce RTX 3080 Gaming OC V2 D6X 10GB
ASUS ROG STRIX GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 x16 ATX Video Card ROG-STRIX-RTX3080-O10G-WHITE-V2 (LHR)
MSI Gaming GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 ATX Video Card RTX 3080 GAMING Z TRIO 10G LHR
GIGABYTE GeForce RTX 3080 VISION OC 10GB Video Card, GV-N3080VISION OC-10GD
GIGABYTE AORUS GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card GV-N308TAORUSX WB-12GD

提取所有标题,例如MSI Gaming GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 ATX 显卡 RTX 3080 GAMING Z TRIO 10G LHR,等使用 Selenium you can use List Comprehension and you can use either of the following Locator Strategies

  • 使用 css_selectorget_attribute("innerHTML"):

    driver.get("https://www.newegg.com/p/pl?d=RTX+3080")
    print([my_elem.get_attribute("innerHTML") for my_elem in driver.find_elements(By.CSS_SELECTOR, "a[title = 'View Details']")])
    
  • 使用 xpathtext 属性:

    driver.get("https://www.newegg.com/p/pl?d=RTX+3080")
    print([my_elem.text for my_elem in driver.find_elements(By.XPATH, "//a[@title = 'View Details']")])
    

理想情况下你需要诱导 WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

  • 使用 CSS_SELECTORtext 属性:

    driver.get("https://www.newegg.com/p/pl?d=RTX+3080")
    print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[title = 'View Details']")))])
    
  • 使用 XPATHget_attribute("innerHTML"):

    driver.get("https://www.newegg.com/p/pl?d=RTX+3080")
    print([my_elem.get_attribute("innerText") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[@title = 'View Details']")))])
    
  • 控制台输出:

    ['GIGABYTE A520M S2H mATX AM4 4+3 Phases Digital PWM, GIGABYTE Gaming GbE LAN, NVMe PCIe 3.0 x4 M.2, 3 Display Interfaces, Q-Flash Plus, RGB Fusion 2.0, Motherboard', 'MSI Gaming GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 ATX Video Card RTX 3080 GAMING Z TRIO 10G LHR', 'Goldshell LT5 PRO 2455MH/S ASIC LTC Scrypt algorithm mining Dogecoin + Litecoin miner power consumption of 3100W', 'Goldshell Mini-DOGE 185MH/S Ordinary version (without psu)DOGE& LTC Mining Machine Low noise Small&simple Home Mining Home Riching', 'Goldshell LT5 PRO 2455MH/S ASIC LTC Scrypt algorithm mining Dogecoin + Litecoin miner power consumption of 3100W', 'Goldshell Mini-DOGE 185MH/S Ordinary version (without psu)DOGE& LTC Mining Machine Low noise Small&simple Home Mining Home Riching', 'ASUS TUF Gaming NVIDIA GeForce RTX 3080 V2 OC Edition Graphics Card (PCIe 4.0, 10GB GDDR6X, LHR, HDMI 2.1, DisplayPort 1.4a, Dual Ball Fan Bearings, Military-grade Certification, GPU Tweak II)', 'ABS Gladiator Gaming PC - Windows 10 Home - Intel i7 11700KF - GeForce RTX 3080 - G.Skill TridentZ RGB 16GB DDR4 3200MHz - 1TB Intel M.2 NVMe SSD', 'ABS ROG Gundam Limited Edition Gaming PC - Windows 10 Home - Intel i7 11700K - STRIX Gundam GeForce RTX 3080 - G.Skill 32GB 3200MHz - 1TB Intel M.2 - STRIX 360mm AIO', 'ABS Gladiator Gaming PC - Windows 10 Home - Intel i9 11900KF - GeForce RTX 3080 - G.Skill TridentZ RGB 16GB DDR4 3200 MHz - 1TB Intel M.2 NVMe SSD', 'Goldshell HS-BOX 235GH/S(without PSU)BOX & HNSB Mining Machine Low noise Small&simple Home Mining Home Riching', 'Goldshell CK-BOX 1050GH/S(without psu) CKB Mining Machine Low noise Small&simple Home Mining Home Riching', 'Adamant Custom 32-Core Video Editing Rendering Modelling Workstation Computer AMD Threadripper 3970X 3.7Ghz TRX40 2 ZENITH 256Gb DDR4 2x2TB NVMe 1800MBs SSD 10TB HDD 1000W Geforce RTX 3080', 'Adamant Custom 14-Core Rendering Modelling Workstation Computer Intel i9 10940X 3.3Ghz X299 AORUS 128Gb DDR4 2x2TB NVMe SSD 10TB HDD 850W Toughpower PSU Geforce RTX 3080 10Gb', 'GIGABYTE AORUS GeForce RTX 3080 MASTER 10GB GDDR6X PCI Express 4.0 ATX Video Card GV-N3080AORUS M-10GD (rev. 3.0) (LHR)', 'ABS Legend Gaming PC - Windows 10 Home - Intel i9 11900KF - GeForce RTX 3080 Ti - G.Skill TridentZ RGB 16GB DDR4 3200 MHz - 1TB Intel M.2 NVMe SSD', 'GIGABYTE Gaming OC GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 ATX Video Card GV-N3080GAMING OC-10GD (rev. 2.0) (LHR)', 'MSI Suprim GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 Video Card RTX 3080 SUPRIM X 10G LHR', 'MSI Ventus GeForce RTX 3080 PCI Express 4.0 Video Card RTX 3080 VENTUS 3X PLUS 10G OC LHR', 'ASUS ROG Strix NVIDIA GeForce RTX 3080 Ti OC Edition Gaming Graphics Card (PCIe 4.0, 12GB GDDR6X, HDMI 2.1, Axial-tech Fan Design, 2.9-Slot, Super Alloy Power II, ASUS Auto-Extreme Technology)', 'ASUS TUF Gaming GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 Video Card TUF-RTX3080TI-O12G-GAMING', 'MSI Gaming GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 Video Card RTX 3080 Ti Gaming X Trio 12G', 'NVIDIA GeForce RTX 3080 Ti Founders Edition 12GB GDDR6X', 'ZOTAC AMP Holo GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 Video Card ZT-A30810F-10P', 'GIGABYTE AORUS GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card GV-N308TAORUS M-12GD', 'ASUS ROG STRIX GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 x16 ATX Video Card ROG-STRIX-RTX3080-O10G-WHITE-V2 (LHR)', 'MSI Gaming Desktop Aegis RS 11TF-223US Intel Core i7 11th Gen 11700K (3.60 GHz) 32 GB DDR4 2 TB HDD 2 TB PCIe Gen3 SSD NVIDIA GeForce RTX 3080 Ti Windows 10 Home 64-bit', 'MSI Gaming Desktop Aegis R 11TE-097US Intel Core i7 11th Gen 11700 (2.50GHz) 16GB DDR4 3TB HDD 1 TB PCIe SSD NVIDIA GeForce RTX 3080 Windows 10 Home 64-bit', 'Skytech PRISM II Gaming PC Desktop - AMD Ryzen 7 5800X, RTX 3080 10GB GDDR6X, 16GB DDR4 3200, 1TB Gen4 SSD, RGB Fans, AC Wi-Fi, Windows 10 Home 64-bit, Black', 'MSI Ventus GeForce RTX 3080 10GB GDDR6X PCI Express 4.0 Video Card RTX 3080 VENTUS 3X 10G OC LHR', 'GIGABYTE AORUS GeForce RTX 3080 XTREME WATERFORCE WB 10G (rev. 2.0) Graphics Card, WATERFORCE Water Block Cooling System, 10GB 320-bit GDDR6X, GV-N3080AORUSX WB-10GD Rev2.0 Video Card (LHR)', 'Skytech Shiva Gaming PC Desktop - AMD Ryzen 5 5600X 3.7 GHz, RTX 3080 10 GB GDDR6X, 16 GB DDR4 3200, 1 TB NVMe SSD, B550 Motherboard, 750W Gold PSU, AC WiFi, Windows 10 Home 64-bit', 'CLX SET VR-Ready Gaming Desktop - Liquid Cooled Intel Core i9 10900KF 3.7Ghz 10-Core Processor, 32GB DDR4 Memory, GeForce RTX 3080 10GB GDDR6X Graphics, 960GB SSD, 4TB HDD, WiFi, Win 10 Home 64-bit', 'AORUS MODEL X Gaming PC Computer Desktop (Intel i9-11900K, NVIDIA GeForce RTX 3080 10GB GDDR6X, 16GB DDR4 RAM, 3TB M.2 SSD) - GB-AMXI9N8A-2051', 'GIGABYTE AORUS GeForce RTX 3080 MASTER 10G (rev. 2.0) Graphics Card, Max Covered Cooling, 10GB 320-bit GDDR6X, GV-N3080AORUS M-10GD Rev2.0 Video Card', 'iBUYPOWER Gaming Desktop Revolt 3 i7BG Intel Core i7 11th Gen 11700KF (3.60GHz) 16GB DDR4 1 TB NVMe SSD NVIDIA GeForce RTX 3080 Windows 10 Home 64-bit', 'ASUS ROG Strix G15CE Gaming & Entertainment Desktop PC (Intel i7-11700KF 8-Core, 32GB RAM, 1TB m.2 SATA SSD + 3TB HDD (3.5), NVIDIA GeForce RTX 3080 Ti, Wifi, Bluetooth, 6xUSB 3.1, Win 10 Pro)', 'iBUYPOWER Gaming Desktop Trace5MR1003Ti Intel Core i7 11th Gen 11700KF (3.60GHz) 16GB DDR4 1 TB PCIe SSD NVIDIA GeForce RTX 3080 Ti Windows 10 Home 64-bit', 'GIGABYTE Vision GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card GV-N308TVISION OC-12GD', 'GIGABYTE AORUS GeForce RTX 3080 XTREME 10GB GDDR6X PCI Express 4.0 ATX Video Card GV-N3080AORUS X-10GD (rev. 2.0) (LHR)', 'ZOTAC Trinity OC GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card ZT-A30810J-10P', 'CLX SET VR-Ready Gaming Desktop - Liquid Cooled AMD Ryzen 9 5900X 3.7Ghz 12-Core Processor, 64GB DDR4 Memory, GeForce RTX 3080 10GB GDDR6X Graphics, 1TB SSD, 6TB HDD, WiFi, Windows 10 Home 64-bit', 'ASUS ROG STRIX GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 x16 Video Card ROG-STRIX-LC-RTX3080TI-O12G-GAMING', 'GIGABYTE Gaming GeForce RTX 3080 Ti 12GB GDDR6X PCI Express 4.0 ATX Video Card GV-N308TGAMING OC-12GD']
    
  • 注意:您必须添加以下导入:

    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    

结尾

Link 到有用的文档: