如何将 HTML/URL 中的下拉选择菜单元素转换为 Pandas Dataframe？

Question

在创建用于匹配和提取 ID 和 SubID 及其名称的数据集时，从请求模块获取文件后，我在 HTML 中有以下代码 -

    <div class="feature">
            <h5>Network</h5>
            <div>
                <div class="row">
                    <ul class="tree network-tree">
                    
                        
    <li class="classification class-C  ">
        <span>
        <input type="checkbox" >
        <a href="/network/nt06410+N01032+N01031" target="_blank">nt06410</a> Calcium signaling

        </span>
        <ul>
    <li class="entry class-D network ">
        <span>
        <input type="checkbox" >
        <a href="/entry/N01032" data-entry="N01032" target="_blank">N01032</a>
        
        Mutation-inactivated PRKN to mGluR1 signaling pathway

        </span>
    </li>

    <li class="entry class-D network ">
        <span>
        <input type="checkbox" >
        <a href="/entry/N01031" data-entry="N01031" target="_blank">N01031</a>
        
        Mutation-caused aberrant SNCA to VGCC-Ca2+ -apoptotic pathway

        </span>
    </li>

        </ul>
    </li>

我想要做的是获取这个特定的下拉选择菜单，突出显示特定链接到 pandas 数据框 -

ID	Name	Subname	SubID	Network
nt06410	Calcium signaling	Mutation-inactivated PRKN to mGluR1 signaling pathway	N01032	nt06410+N01032+N01031

到目前为止我的代码是 -

data = soup.find_all("ul", {"class": "tree network-tree"})

# get all list elements
lis = data[0].find_all('li')

# add a helper lambda, just for readability
find_ul = lambda x: x.find_all('ul')
uls = [find_ul(elem) for elem in lis if find_ul(elem) != []]

# use a nested list comprehension to iterate over the <ul> tags
# and extract text from each <li> into sublists
text = [[li.text.encode('utf-8') for li in ul[0].find_all('li')] for ul in uls]

print(text[0][1])

Answer 1

您可以使用嵌套的 for 循环并将项目追加到列表中。您将在外循环中重复项目，例如内部循环中每个实例的 ID，例如子ID 将列表列表转换为末尾带有 pandas 的 DataFrame。

results = []

for network in soup.select('.row .classification'):
    a = network.select_one('a')
    _id = a.text
    _network = a['href'].split('network/')[-1]
    name = a.next_sibling.strip()

    for pathway in network.select('.network'):
        b = pathway.select_one('a')
        subname = b.next_sibling.strip()
        subid = b.text
        results.append([_id, name, subname, subid, _network])
df = pd.DataFrame(results, columns = ['ID', 'Name', 'Subname', 'SubID', 'Network'])

使用相关 link 测试：https://www.kegg.jp/pathway/hsa05022

N.B。 KEGG 提供免费的 JSON 层次结构下载。

如何将 HTML/URL 中的下拉选择菜单元素转换为 Pandas Dataframe？

How to convert a dropdown selection menu element in HTML/URL to Pandas Dataframe?

python

html

pandas

dataframe

screen-scraping