Beautiful Soup Headers 数据 table

Question

我正在从 IMDB 中搜集演员表（IMDB API 没有全面的 cast/credits 数据）。我想要的最终产品是一个 table 三列，它从网页中的所有 table 中获取数据并像这样对它们进行排序：

Produced by | Gary Kurtz | producer 

Produced by | George Lucas | executive producer

Music by    | John Williams |

（以星球大战为例，http://www.imdb.com/title/tt0076759/fullcredits?ref_=tt_cl_sm#cast）

下面的代码差不多了，但是有一大堆不必要的空格，.parent函数肯定是用错了。找到 table 以上的 h4 值的最佳方法是什么？

这是代码。

 with open(fname, 'r') as f:
        soup = BeautifulSoup(f.read(),'html5lib')
        soup.prettify()


        with open(fname, 'r') as f:
        soup = BeautifulSoup(f.read(),'html5lib')
        soup.prettify()

        for child in soup.find_all('td',{'class':'name'}):
            print child.parent.text, child.parent.parent.parent.parent.parent.parent.text.encode('utf-8')

我正在尝试从这些 h4 headers

中获取 "Directed by" 等值

Answer 1

欢迎使用 Whosebug。似乎您可以同时找到 h4 和 table，因为它们在 html 中成对出现，因此您可以将它们压缩为循环遍历它们。之后，您只需获取文本并对其进行格式化。将您的代码更改为：

soup = BeautifulSoup(f.read(), 'html5lib')
for h4,table in zip(soup.find_all('h4'),soup.find_all('table')):
    header4 = " ".join(h4.text.strip().split())
    table_data = [" ".join(tr.text.strip().replace("\n", "").replace("...", "|").split())  for tr in table.find_all('tr')]
    print("%s | %s \n")%(header4,table_data)

这将打印：

Directed by | [u'George Lucas'] 

Writing Credits | [u'George Lucas | (written by)'] 

Cast (in credits order) verified as complete | ['', u'Mark Hamill | Luke Skywalker', u'Harrison Ford | Han Solo', u'Carrie Fisher | Princess Leia Organa', u'Peter Cushing | Grand Moff Tarkin',...]

Produced by | [u'Gary Kurtz | producer', u'George Lucas | executive producer', u'Rick McCallum | producer (1997 special version)'] 

Music by | [u'John Williams'] 

...

Answer 2

这将避免详尽使用父函数

from urllib.request import urlopen
from bs4 import BeautifulSoup

#this will find all headers eg produced by
def get_header(url):
    bsObj = BeautifulSoup(urlopen(url))
    headers = bsObj.find("div", {"id":"fullcredits_content"}).findAll("h4", {"class":"dataHeaderWithBorder"})
    return headers
#this will find all names eg gary kurtz
def get_table(url):
    bsObj = BeautifulSoup(urlopen(url))
    table = bsObj.findAll("td", {"class":"name"})
    return table

url = "http://www.imdb.com/title/tt0076759/fullcredits"
header= get_header(url)
table  = get_table(url)
#title  = get_title(url)
for h in header:
    for t in table:
        print(h.get_text())
        print(t.get_text())
        print("............")

Beautiful Soup Headers 数据 table

Beautiful Soup Headers with table data

python

beautifulsoup

imdb