解析大型 XML 并使用 XML 中的数据（使用 python 或其他）生成数据帧的最佳方法是什么？

Question

我尝试根据 XML 文件的信息制作一个 table（或 csv，我正在使用 pandas 数据帧）。

文件在这里（.zip 是 14 MB，XML 是 ~370MB），https://nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.zip。它具有不同语言的包信息 - node.js、python、java 等。又名美国政府组织 NVD 的 CPE 2.3 列表。

这是前 30 行的样子：

<cpe-list xmlns:config="http://scap.nist.gov/schema/configuration/0.1" xmlns="http://cpe.mitre.org/dictionary/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:scap-core="http://scap.nist.gov/schema/scap-core/0.3" xmlns:cpe-23="http://scap.nist.gov/schema/cpe-extension/2.3" xmlns:ns6="http://scap.nist.gov/schema/scap-core/0.1" xmlns:meta="http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2" xsi:schemaLocation="http://scap.nist.gov/schema/cpe-extension/2.3 https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd http://cpe.mitre.org/dictionary/2.0 https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2 https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd http://scap.nist.gov/schema/scap-core/0.3 https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd http://scap.nist.gov/schema/configuration/0.1 https://scap.nist.gov/schema/nvd/configuration_0.1.xsd http://scap.nist.gov/schema/scap-core/0.1 https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd">
  <generator>
    <product_name>National Vulnerability Database (NVD)</product_name>
    <product_version>4.9</product_version>
    <schema_version>2.3</schema_version>
    <timestamp>2022-03-17T03:51:01.909Z</timestamp>
  </generator>
  <cpe-item name="cpe:/a:%240.99_kindle_books_project:%240.99_kindle_books:6::~~~android~~">
    <title xml:lang="en-US">[=13=].99 Kindle Books project [=13=].99 Kindle Books (aka com.kindle.books.for99) for android 6.0</title>
    <references>
      <reference href="https://play.google.com/store/apps/details?id=com.kindle.books.for99">Product information</reference>
      <reference href="https://docs.google.com/spreadsheets/d/1t5GXwjw82SyunALVJb2w0zi3FoLRIkfGPc7AMjRF0r4/edit?pli=1#gid=1053404143">Government Advisory</reference>
    </references>
    <cpe-23:cpe23-item name="cpe:2.3:a:$0.99_kindle_books_project:$0.99_kindle_books:6:*:*:*:*:android:*:*"/>
  </cpe-item>

XML文件的树结构很简单，根是'cpe-list'，子元素是'cpe-item'，孙元素是'title'， 'references' 和 'cpe23-item'.

来自'title'，我想要元素中的文本；来自 'cpe23-item'，我想要属性 'name'；来自 'references'，我想要它的曾孙 'reference'.

的属性 'href'

数据框应如下所示：

  | cpe23_name   | title_text            | ref1  | ref2  | ref3  | ref_other
0 | 'cpe23name 1'| 'this is a python pkg'| 'url1'| 'url2'| NaN   | NaN
1 | 'cpe23name 2'| 'this is a java pkg'  | 'url1'| 'url2'| NaN   | NaN
...

我的代码在这里，大约 100 秒完成：

import xml.etree.ElementTree as et

xtree = et.parse("official-cpe-dictionary_v2.3.xml")
xroot = xtree.getroot()

import time
start_time = time.time()

df_cols = ["cpe", "text", "vendor", "product", "version", "changelog", "advisory", 'others']

title      = '{http://cpe.mitre.org/dictionary/2.0}title'
ref        = '{http://cpe.mitre.org/dictionary/2.0}references'
cpe_item   = '{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item'

p_cpe = None
p_text = None
p_vend = None
p_prod = None
p_vers = None
p_chan = None
p_advi = None
p_othe = None
rows = []

i = 0

while i < len(xroot):
    for elm in xroot[i]:
        if elm.tag == title:
                p_text = elm.text
                #assign p_text
       
        elif elm.tag == ref:
            for nn in elm:
                s = nn.text.lower()
                #check the lower text in refs
                
                if 'version' in s:
                    p_vers = nn.attrib.get('href')
                    #assign p_vers
                elif 'advisor' in s:
                    p_advi = nn.attrib.get('href')
                    #assign p_advi
                elif 'product' in s:
                    p_prod = nn.attrib.get('href')
                    #assign p_prod
                elif 'vendor' in s:
                    p_vend = nn.attrib.get('href')
                    #assign p_vend
                elif 'change' in s:
                    p_chan = nn.attrib.get('href')
                    #assign p_vend
                else:
                    p_othe = nn.attrib.get('href')
            
        elif elm.tag == cpe_item:
            p_cpe = elm.attrib.get("name")
            #assign p_cpe
           
        else:
            print(elm.tag)
    row = [p_cpe, p_text, p_vend, p_prod, p_vers, p_chan, p_advi, p_othe]
    rows.append(row)        
    
    p_cpe = None
    p_text = None
    p_vend = None
    p_prod = None
    p_vers = None
    p_chan = None
    p_advi = None
    p_othe = None
    print(len(rows)) #this shows how far I got during the running time
    i+=1
    
out_df1 = pd.DataFrame(rows, columns = df_cols)# move this part outside the loop by removing the indent

print("---853k rows take %s seconds ---" % (time.time() - start_time))

更新：更快的方法是将倒数第二行移出循环。由于 'rows' 已经在每个循环中获取信息，因此无需每次都创建一个新的数据帧。运行时间现在是 136.0491042137146 秒。耶！

Answer 1

由于您的 XML 相当平坦，请考虑最近添加的 IO 模块，pandas.read_xml 在 v1.3 中引入。鉴于 XML 使用默认命名空间，要引用 xpath 中的元素，请使用 namespaces 参数：

url = "https://nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.zip"

df = pd.read_xml(
    url, xpath=".//doc:cpe-item", namespaces={'doc': 'http://cpe.mitre.org/dictionary/2.0'}
)

如果您没有安装默认解析器 lxml，请使用 etree 解析器：

df = pd.read_xml(
    url, xpath=".//doc:cpe-item", namespaces={'doc': 'http://cpe.mitre.org/dictionary/2.0'}, parser="etree"
)

解析大型 XML 并使用 XML 中的数据（使用 python 或其他）生成数据帧的最佳方法是什么？

What is the best way to parse large XML and genarate a dataframe with the data in the XML (with python or else)?

python

pandas

xml

dataframe

elementtree