BeautifulSoup findAll returns 空列表
BeautifulSoup findAll returns empty list
我想编写一个代码来收集 Steam 配置文件中的游戏名称，但无论我在代码中使用什么div，都会返回一个空列表。
import requests
from bs4 import BeautifulSoup
r = requests.get('https://steamcommunity.com/profiles/76561198287287846/games/?tab=all')
soup = BeautifulSoup(r.content, 'lxml')
games = soup.findAll("div",{"class":"gameListRow"})
print(games)
如前所述，内容由 JavaScript 动态提供，一种方法是使用 selenium。在新代码中请使用 find_all() 而不是旧语法 findAll().
例子
import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
url = f'https://steamcommunity.com/profiles/76561198287287846/games/?tab=all'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.get(url)
time.sleep(5)
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.close()
games = soup.find_all("div",{"class":"gameListRow"})
print(games)
一个工作示例 selenium BeautifulSoup。请 运行 代码。
import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
url = 'https://steamcommunity.com/profiles/76561198287287846/games/?tab=all'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
time.sleep(2)
driver.get(url)
time.sleep(5)
soup = BeautifulSoup(driver.page_source, 'lxml')
games = soup.findAll("div",{"class":"gameListRow"})
for game in games:
name= game.find("div",class_="gameListRowItemName ellipsis").text
print(name)
输出：
Dota 2
Counter-Strike: Global Offensive
Sea of Thieves
SCUM
Rocket League
Arma 3
ARK: Survival Evolved
DayZ
Rust
Garry's Mod
Unturned
Grand Theft Auto IV: The Complete Edition
Grand Theft Auto V
Euro Truck Simulator 2
Insurgency
PAYDAY 2
Mount & Blade: Warband
Valheim
War Thunder
Mount & Blade II: Bannerlord
RimWorld
Reign Of Kings
Robocraft
American Truck Simulator
theHunter: Call of the Wild™
Phasmophobia
Scrap Mechanic
Team Fortress 2
Cities: Skylines
Stellaris
UBOAT
Tiger Knight: Empire War
Warframe
Don't Starve Together
Holdfast: Nations At War
Fishing Planet
Fistful of Frags
Half-Life
Beat Cop
Of Guards And Thieves
Chivalry: Medieval Warfare
Terraria
911 Operator
We Need To Go Deeper
The Escapists 2
Business Tour - Online Multiplayer Board Game
Deceit
Crossout
Pummel Party
Paladins
Hand Simulator
Heroes & Generals
Ranch Simulator
Brawlhalla
Medal of Honor(TM) Multiplayer
Gang Beasts
Guns of Icarus Online
Tower Unite
Jalopy
College Kings
Stick Fight: The Game
We Were Here
Contraband Police: Prologue
Battlefield: Bad Company™ 2
Satellite Reign
Ylands
Prison Simulator: Prologue
Shadows of War
Battlerite
The Mean Greens - Plastic Warfare
War Trigger 2
Scribble It!
Radical Heights
Block N Load
Rubber Bandits: Summer Prologue
SinVR
Dead Maze
DC Universe Online
DiRT 3 Complete Edition
ATLAS
Paunch
Apex Legends
Crusader Kings II
Trophy Fishing 2
Amnesia: A Machine for Pigs
Amnesia: The Dark Descent
Argo
ARK: Survival Of The Fittest
Arma 3 Helicopters
Arma 3 Karts
Arma 3 Malden
Arma 3 Marksmen
Arma 3 Zeus
Armor Of Heroes
Artifact Foundry
Battlefield™ 2042 Open Beta
Brütal Legend
Cities: Skylines - Carols, Candles and Candy
Cities: Skylines - Match Day
College Bound
Company of Heroes 2
DCS World Steam Edition
Dungeons 2
Eador. Masters of the Broken World
F1 2015
Fallout Shelter
Friday the 13th: The Game
Galactic Civilizations II: Ultimate Edition
GROUND BRANCH
GROUND BRANCH CTE (Community Test Environment)
Guacamelee! Super Turbo Championship Edition
Guns of Icarus Alliance
Hacknet
Hacknet Official Soundtrack
Hide and Seek
HITMAN™
HITMAN™ 2
Human: Fall Flat
Kopanito All-Stars Soccer
Last Man Standing
Layers of Fear
Medal of Honor(TM) Single Player
Medal of Honor Pre-Order
Metro 2033
Mirage: Arcane Warfare
Mount & Blade: Warband - Napoleonic Wars
Of Guards and Thieves - Firefight
Of Guards and Thieves - Map Workshop
Outlast
Outlast: Whistleblower DLC
Paladins - Public Test
Penguins Arena: Sedna's World
Psychonauts
Quake Champions
RimWorld - Ideology
Rust - Staging Branch
Sanctum 2
Spacelords
Spec Ops: The Line
Survarium
Sven Co-op
The Darkness II
The Flame in the Flood
The Red Solstice
The Walking Dead
Tiger Knight
Totally Accurate Battlegrounds
Turbo Pug DX
Unreal Gold
Unturned - Early Access
World of Tanks Blitz
我想编写一个代码来收集 Steam 配置文件中的游戏名称，但无论我在代码中使用什么div，都会返回一个空列表。
import requests
from bs4 import BeautifulSoup
r = requests.get('https://steamcommunity.com/profiles/76561198287287846/games/?tab=all')
soup = BeautifulSoup(r.content, 'lxml')
games = soup.findAll("div",{"class":"gameListRow"})
print(games)
如前所述，内容由 JavaScript 动态提供，一种方法是使用 selenium。在新代码中请使用 find_all() 而不是旧语法 findAll().
例子
import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
url = f'https://steamcommunity.com/profiles/76561198287287846/games/?tab=all'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.get(url)
time.sleep(5)
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.close()
games = soup.find_all("div",{"class":"gameListRow"})
print(games)
一个工作示例 selenium BeautifulSoup。请 运行 代码。
import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
url = 'https://steamcommunity.com/profiles/76561198287287846/games/?tab=all'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
time.sleep(2)
driver.get(url)
time.sleep(5)
soup = BeautifulSoup(driver.page_source, 'lxml')
games = soup.findAll("div",{"class":"gameListRow"})
for game in games:
name= game.find("div",class_="gameListRowItemName ellipsis").text
print(name)
输出：
Dota 2
Counter-Strike: Global Offensive
Sea of Thieves
SCUM
Rocket League
Arma 3
ARK: Survival Evolved
DayZ
Rust
Garry's Mod
Unturned
Grand Theft Auto IV: The Complete Edition
Grand Theft Auto V
Euro Truck Simulator 2
Insurgency
PAYDAY 2
Mount & Blade: Warband
Valheim
War Thunder
Mount & Blade II: Bannerlord
RimWorld
Reign Of Kings
Robocraft
American Truck Simulator
theHunter: Call of the Wild™
Phasmophobia
Scrap Mechanic
Team Fortress 2
Cities: Skylines
Stellaris
UBOAT
Tiger Knight: Empire War
Warframe
Don't Starve Together
Holdfast: Nations At War
Fishing Planet
Fistful of Frags
Half-Life
Beat Cop
Of Guards And Thieves
Chivalry: Medieval Warfare
Terraria
911 Operator
We Need To Go Deeper
The Escapists 2
Business Tour - Online Multiplayer Board Game
Deceit
Crossout
Pummel Party
Paladins
Hand Simulator
Heroes & Generals
Ranch Simulator
Brawlhalla
Medal of Honor(TM) Multiplayer
Gang Beasts
Guns of Icarus Online
Tower Unite
Jalopy
College Kings
Stick Fight: The Game
We Were Here
Contraband Police: Prologue
Battlefield: Bad Company™ 2
Satellite Reign
Ylands
Prison Simulator: Prologue
Shadows of War
Battlerite
The Mean Greens - Plastic Warfare
War Trigger 2
Scribble It!
Radical Heights
Block N Load
Rubber Bandits: Summer Prologue
SinVR
Dead Maze
DC Universe Online
DiRT 3 Complete Edition
ATLAS
Paunch
Apex Legends
Crusader Kings II
Trophy Fishing 2
Amnesia: A Machine for Pigs
Amnesia: The Dark Descent
Argo
ARK: Survival Of The Fittest
Arma 3 Helicopters
Arma 3 Karts
Arma 3 Malden
Arma 3 Marksmen
Arma 3 Zeus
Armor Of Heroes
Artifact Foundry
Battlefield™ 2042 Open Beta
Brütal Legend
Cities: Skylines - Carols, Candles and Candy
Cities: Skylines - Match Day
College Bound
Company of Heroes 2
DCS World Steam Edition
Dungeons 2
Eador. Masters of the Broken World
F1 2015
Fallout Shelter
Friday the 13th: The Game
Galactic Civilizations II: Ultimate Edition
GROUND BRANCH
GROUND BRANCH CTE (Community Test Environment)
Guacamelee! Super Turbo Championship Edition
Guns of Icarus Alliance
Hacknet
Hacknet Official Soundtrack
Hide and Seek
HITMAN™
HITMAN™ 2
Human: Fall Flat
Kopanito All-Stars Soccer
Last Man Standing
Layers of Fear
Medal of Honor(TM) Single Player
Medal of Honor Pre-Order
Metro 2033
Mirage: Arcane Warfare
Mount & Blade: Warband - Napoleonic Wars
Of Guards and Thieves - Firefight
Of Guards and Thieves - Map Workshop
Outlast
Outlast: Whistleblower DLC
Paladins - Public Test
Penguins Arena: Sedna's World
Psychonauts
Quake Champions
RimWorld - Ideology
Rust - Staging Branch
Sanctum 2
Spacelords
Spec Ops: The Line
Survarium
Sven Co-op
The Darkness II
The Flame in the Flood
The Red Solstice
The Walking Dead
Tiger Knight
Totally Accurate Battlegrounds
Turbo Pug DX
Unreal Gold
Unturned - Early Access
World of Tanks Blitz