无法使用 BS4 从 table 中仅提取可见文本
Cannot pull only visible text from table using BS4
我正在尝试从 this 网页上抓取位置和球员姓名等数据。我的代码如下。
#create url for the wikipedia data we are going to scrape
wikiURL = "https://en.wikipedia.org/wiki/2012_NFL_Draft"
#create array to store player info in
teams_players = []
# request and parse wikiURL
r = requests.get(wikiURL)
soup = BeautifulSoup(r.content, "html.parser")
#find table in wikipedia
playerData = soup.find('table', {"class": "wikitable sortable"})
for row in playerData.find_all('tr')[1:]:
cols = row.find_all(['td', 'th'])
if len(cols) < 6:
continue
teams_players.append((cols[5].text.strip(), cols[4].text.strip() ))
for team, player in teams_players:
print('{:35} {}'.format(team, player))
问题是有一个 "sortkey" 带有文本的 span 标签和名称字段中显示的文本，所以输出最终被加倍并显示符号。
QB Luck, AndrewAndrew Luck †
QB Griffin III, RobertRobert Griffin III †
我尝试搜索 {"class": "fn"} 但这只是 returns 一个空括号列表。
如何只拉出显示的文本而忽略符号？
如果您只想要名称和位置，您可以简化代码以在每个 td 中查找每个 span table 与 class fn，从中获取文本然后查找下一个 td 并从中提取文本td 的主播.
from bs4 import BeautifulSoup
import requests
soup = BeautifulSoup(requests.get("https://en.wikipedia.org/wiki/2012_NFL_Draft").content,"lxml")
table = soup.select_one("table.wikitable.sortable")
for name_tag in table.select("tr + tr td span.fn"):
print(name_tag.text, name_tag.find_next("td").a.text)
如果我们 运行 代码，您可以看到我们得到了我们想要的所有数据，并且没有任何符号：
In [1]: from bs4 import BeautifulSoup
...: import requests
...: soup = BeautifulSoup(requests.get("https://en.wikipedia.org/wiki/2012_NF
...: L_Draft").content,"lxml")
...: table = soup.select_one("table.wikitable.sortable")
...: for name_tag in table.select("tr + tr td span.fn"):
...: print(name_tag.text, name_tag.find_next("td").a.text)
...:
Andrew Luck QB
Robert Griffin III QB
Trent Richardson RB
Matt Kalil OT
Justin Blackmon WR
Morris Claiborne CB
Mark Barron S
Ryan Tannehill QB
Luke Kuechly LB
Stephon Gilmore CB
Dontari Poe NT
Fletcher Cox DT
Michael Floyd WR
Michael Brockers DT
Bruce Irvin DE
Quinton Coples DE
Dre Kirkpatrick CB
Melvin Ingram LB
Shea McClellin DE
Kendall Wright WR
Chandler Jones DE
Brandon Weeden QB
Riley Reiff OT
David DeCastro G
Dont'a Hightower LB
Whitney Mercilus DE
Kevin Zeitler G
Nick Perry LB
Harrison Smith S
A. J. Jenkins WR
Doug Martin RB
David Wilson RB
Brian Quick WR
Coby Fleener TE
Courtney Upshaw LB
Derek Wolfe DT
Mitchell Schwartz OT
Andre Branch DE
Janoris Jenkins CB
Amini Silatolu G
Cordy Glenn OT
Jonathan Martin OT
Stephen Hill WR
Jeff Allen G
Alshon Jeffery WR
Mychal Kendricks LB
Bobby Wagner LB
Tavon Wilson S
Kendall Reyes DT
Isaiah Pead RB
Jerel Worthy DT
Zach Brown LB
Devon Still DT
Ryan Broyles WR
Peter Konz C
Mike Adams OT
Brock Osweiler QB
Lavonte David LB
Vinny Curry DE
Kelechi Osemele G
LaMichael James RB
Casey Hayward CB
Rueben Randle WR
Dwayne Allen TE
Trumaine Johnson CB
Josh Robinson CB
Ronnie Hillman RB
DeVier Posey WR
T. J. Graham WR
Bryan Anger P
Josh LeRibeus G
Olivier Vernon DE
Brandon Taylor S
Donald Stephenson OT
Russell Wilson QB
Brandon Brooks G
Demario Davis LB
Michael Egnew TE
Brandon Hardin S
Jamell Fleming CB
Tyrone Crawford DE
Mike Martin DT
Mohamed Sanu WR
Bernard Pierce RB
Dwight Bentley CB
Sean Spence LB
John Hughes DT
Nick Foles QB
Akiem Hicks DT
Jake Bequette DE
Lamar Holmes OT
T. Y. Hilton WR
Brandon Thompson DT
Jayron Hosley CB
Tony Bergstrom G
Chris Givens WR
Lamar Miller RB
Gino Gradkowski G
Ben Jones C
Travis Benjamin WR
Omar Bolden CB
Kirk Cousins QB
Frank Alexander DE
Joe Adams WR
Nigel Bradham LB
Robert Turbin RB
Devon Wylie WR
Philip Blake C
Alameda Ta'amu DT
Ladarius Green TE
Evan Rodriguez TE
Bobby Massie OT
Kyle Wilber LB
Jaye Howard DT
Coty Sensabaugh CB
Orson Charles TE
Joe Looney G
Jarius Wright WR
Keenan Robinson LB
James-Michael Johnson LB
Keshawn Martin WR
Nick Toon WR
Brandon Boykin CB
Ron Brooks CB
Ronnell Lewis LB
Jared Crick DE
Adrien Robinson TE
Rhett Ellison FB
Miles Burris LB
Christian Thompson S
Brandon Mosley OT
Mike Daniels DT
Jerron McMillian S
Greg Childs WR
Matt Johnson S
Josh Chapman DT
Malik Jackson DE
Tahir Whitehead LB
Robert Blanton S
Najee Goode LB
Adam Gettis G
Brandon Marshall LB
Josh Norman CB
Zebrie Sanders OT
Taylor Thompson DE
DeQuan Menzie CB
Tank Carder LB
Chris Greenwood CB
Johnnie Troutman G
Rokevious Watkins G
Senio Kelemete G
Danny Coale WR
Dennis Kelly OT
Korey Toomer LB
Josh Kaddu LB
Shaun Prater CB
Bradie Ewing FB
Jack Crawford DE
Chris Rainey RB
Ryan Miller G
Randy Bullock K
Corey White S
Terrell Manning LB
Jonathan Massaquoi DE
Darius Fleming LB
Marvin Jones WR
George Iloka S
Juron Criner WR
Asa Jackson CB
Vick Ballard RB
Greg Zuerlein K
Jeremy Lane CB
Alfred Morris RB
Keith Tandy CB
Blair Walsh K
Mike Harris CB
Justin Bethel S
Mark Asper G
Andrew Tiller G
Trenton Robinson S
Winston Guy S
Cyrus Gray RB
B.J. Cunningham WR
Isaiah Frey CB
Ryan Lindley QB
James Hanna TE
Josh Bush S
Danny Trevathan LB
Christo Bilukidi DT
Markelle Martin S
Dan Herron RB
Charles Mitchell S
Tom Compton OT
Marvin McNutt WR
Nick Mondek OT
Jonte Green CB
Nate Ebner CB
Tommy Streeter WR
Jason Slowey OT
Brandon Washington G
Matt McCants OT
Terrance Ganaway RB
Robert Griffin G
Emmanuel Acho LB
Billy Winn DT
LaVon Brazill WR
Brad Nortman P
Justin Anderson G
Audie Cole LB
Scott Solomon DE
Michael Smith RB
Richard Crawford CB
Kheeston Randall DT
D. J. Campbell S
Jordan Bernstine CB
Jerome Long DT
Trevor Guyton DE
Greg McCoy CB
Nate Potter OT
Caleb McSurdy ILB
Travis Lewis OLB
Alfonzo Dennard CB
J. R. Sweezy G
David Molk C
Rishard Matthews WR
Jeris Pendleton DT
Bryce Brown RB
Nathan Stupar OLB
Toney Clemons WR
Greg Scruggs DE
Drake Dunsmore TE
Marcel Jones OT
Jeremy Ebert WR
DeAngelo Tyson DT
Cam Johnson DE
Junior Hemingway WR
Markus Kuhn DT
David Paulson TE
Andrew Datko OT
Antonio Allen S
B. J. Coleman QB
Jordan White WR
Trevin Wade CB
Terrence Frederick CB
Brad Smelley TE
Kelvin Beachum G
Travian Robertson DT
Edwin Baker RB
John Potter K
Daryl Richardson RB
Chandler Harnish QB
我正在尝试从 this 网页上抓取位置和球员姓名等数据。我的代码如下。
#create url for the wikipedia data we are going to scrape
wikiURL = "https://en.wikipedia.org/wiki/2012_NFL_Draft"
#create array to store player info in
teams_players = []
# request and parse wikiURL
r = requests.get(wikiURL)
soup = BeautifulSoup(r.content, "html.parser")
#find table in wikipedia
playerData = soup.find('table', {"class": "wikitable sortable"})
for row in playerData.find_all('tr')[1:]:
cols = row.find_all(['td', 'th'])
if len(cols) < 6:
continue
teams_players.append((cols[5].text.strip(), cols[4].text.strip() ))
for team, player in teams_players:
print('{:35} {}'.format(team, player))
问题是有一个 "sortkey" 带有文本的 span 标签和名称字段中显示的文本，所以输出最终被加倍并显示符号。
QB Luck, AndrewAndrew Luck †
QB Griffin III, RobertRobert Griffin III †
我尝试搜索 {"class": "fn"} 但这只是 returns 一个空括号列表。
如何只拉出显示的文本而忽略符号？
如果您只想要名称和位置，您可以简化代码以在每个 td 中查找每个 span table 与 class fn，从中获取文本然后查找下一个 td 并从中提取文本td 的主播.
from bs4 import BeautifulSoup
import requests
soup = BeautifulSoup(requests.get("https://en.wikipedia.org/wiki/2012_NFL_Draft").content,"lxml")
table = soup.select_one("table.wikitable.sortable")
for name_tag in table.select("tr + tr td span.fn"):
print(name_tag.text, name_tag.find_next("td").a.text)
如果我们 运行 代码，您可以看到我们得到了我们想要的所有数据，并且没有任何符号：
In [1]: from bs4 import BeautifulSoup
...: import requests
...: soup = BeautifulSoup(requests.get("https://en.wikipedia.org/wiki/2012_NF
...: L_Draft").content,"lxml")
...: table = soup.select_one("table.wikitable.sortable")
...: for name_tag in table.select("tr + tr td span.fn"):
...: print(name_tag.text, name_tag.find_next("td").a.text)
...:
Andrew Luck QB
Robert Griffin III QB
Trent Richardson RB
Matt Kalil OT
Justin Blackmon WR
Morris Claiborne CB
Mark Barron S
Ryan Tannehill QB
Luke Kuechly LB
Stephon Gilmore CB
Dontari Poe NT
Fletcher Cox DT
Michael Floyd WR
Michael Brockers DT
Bruce Irvin DE
Quinton Coples DE
Dre Kirkpatrick CB
Melvin Ingram LB
Shea McClellin DE
Kendall Wright WR
Chandler Jones DE
Brandon Weeden QB
Riley Reiff OT
David DeCastro G
Dont'a Hightower LB
Whitney Mercilus DE
Kevin Zeitler G
Nick Perry LB
Harrison Smith S
A. J. Jenkins WR
Doug Martin RB
David Wilson RB
Brian Quick WR
Coby Fleener TE
Courtney Upshaw LB
Derek Wolfe DT
Mitchell Schwartz OT
Andre Branch DE
Janoris Jenkins CB
Amini Silatolu G
Cordy Glenn OT
Jonathan Martin OT
Stephen Hill WR
Jeff Allen G
Alshon Jeffery WR
Mychal Kendricks LB
Bobby Wagner LB
Tavon Wilson S
Kendall Reyes DT
Isaiah Pead RB
Jerel Worthy DT
Zach Brown LB
Devon Still DT
Ryan Broyles WR
Peter Konz C
Mike Adams OT
Brock Osweiler QB
Lavonte David LB
Vinny Curry DE
Kelechi Osemele G
LaMichael James RB
Casey Hayward CB
Rueben Randle WR
Dwayne Allen TE
Trumaine Johnson CB
Josh Robinson CB
Ronnie Hillman RB
DeVier Posey WR
T. J. Graham WR
Bryan Anger P
Josh LeRibeus G
Olivier Vernon DE
Brandon Taylor S
Donald Stephenson OT
Russell Wilson QB
Brandon Brooks G
Demario Davis LB
Michael Egnew TE
Brandon Hardin S
Jamell Fleming CB
Tyrone Crawford DE
Mike Martin DT
Mohamed Sanu WR
Bernard Pierce RB
Dwight Bentley CB
Sean Spence LB
John Hughes DT
Nick Foles QB
Akiem Hicks DT
Jake Bequette DE
Lamar Holmes OT
T. Y. Hilton WR
Brandon Thompson DT
Jayron Hosley CB
Tony Bergstrom G
Chris Givens WR
Lamar Miller RB
Gino Gradkowski G
Ben Jones C
Travis Benjamin WR
Omar Bolden CB
Kirk Cousins QB
Frank Alexander DE
Joe Adams WR
Nigel Bradham LB
Robert Turbin RB
Devon Wylie WR
Philip Blake C
Alameda Ta'amu DT
Ladarius Green TE
Evan Rodriguez TE
Bobby Massie OT
Kyle Wilber LB
Jaye Howard DT
Coty Sensabaugh CB
Orson Charles TE
Joe Looney G
Jarius Wright WR
Keenan Robinson LB
James-Michael Johnson LB
Keshawn Martin WR
Nick Toon WR
Brandon Boykin CB
Ron Brooks CB
Ronnell Lewis LB
Jared Crick DE
Adrien Robinson TE
Rhett Ellison FB
Miles Burris LB
Christian Thompson S
Brandon Mosley OT
Mike Daniels DT
Jerron McMillian S
Greg Childs WR
Matt Johnson S
Josh Chapman DT
Malik Jackson DE
Tahir Whitehead LB
Robert Blanton S
Najee Goode LB
Adam Gettis G
Brandon Marshall LB
Josh Norman CB
Zebrie Sanders OT
Taylor Thompson DE
DeQuan Menzie CB
Tank Carder LB
Chris Greenwood CB
Johnnie Troutman G
Rokevious Watkins G
Senio Kelemete G
Danny Coale WR
Dennis Kelly OT
Korey Toomer LB
Josh Kaddu LB
Shaun Prater CB
Bradie Ewing FB
Jack Crawford DE
Chris Rainey RB
Ryan Miller G
Randy Bullock K
Corey White S
Terrell Manning LB
Jonathan Massaquoi DE
Darius Fleming LB
Marvin Jones WR
George Iloka S
Juron Criner WR
Asa Jackson CB
Vick Ballard RB
Greg Zuerlein K
Jeremy Lane CB
Alfred Morris RB
Keith Tandy CB
Blair Walsh K
Mike Harris CB
Justin Bethel S
Mark Asper G
Andrew Tiller G
Trenton Robinson S
Winston Guy S
Cyrus Gray RB
B.J. Cunningham WR
Isaiah Frey CB
Ryan Lindley QB
James Hanna TE
Josh Bush S
Danny Trevathan LB
Christo Bilukidi DT
Markelle Martin S
Dan Herron RB
Charles Mitchell S
Tom Compton OT
Marvin McNutt WR
Nick Mondek OT
Jonte Green CB
Nate Ebner CB
Tommy Streeter WR
Jason Slowey OT
Brandon Washington G
Matt McCants OT
Terrance Ganaway RB
Robert Griffin G
Emmanuel Acho LB
Billy Winn DT
LaVon Brazill WR
Brad Nortman P
Justin Anderson G
Audie Cole LB
Scott Solomon DE
Michael Smith RB
Richard Crawford CB
Kheeston Randall DT
D. J. Campbell S
Jordan Bernstine CB
Jerome Long DT
Trevor Guyton DE
Greg McCoy CB
Nate Potter OT
Caleb McSurdy ILB
Travis Lewis OLB
Alfonzo Dennard CB
J. R. Sweezy G
David Molk C
Rishard Matthews WR
Jeris Pendleton DT
Bryce Brown RB
Nathan Stupar OLB
Toney Clemons WR
Greg Scruggs DE
Drake Dunsmore TE
Marcel Jones OT
Jeremy Ebert WR
DeAngelo Tyson DT
Cam Johnson DE
Junior Hemingway WR
Markus Kuhn DT
David Paulson TE
Andrew Datko OT
Antonio Allen S
B. J. Coleman QB
Jordan White WR
Trevin Wade CB
Terrence Frederick CB
Brad Smelley TE
Kelvin Beachum G
Travian Robertson DT
Edwin Baker RB
John Potter K
Daryl Richardson RB
Chandler Harnish QB