如何将 XML NER 数据从 CRAFT 语料库转换为 spaCy 的 JSON 格式?

How to convert XML NER data from the CRAFT corpus to spaCy's JSON format?

如何在 CRAFT corpus 上使用 spaCy 为生物医学 NER 构建命名实体识别 (NER) 模型?

我很难将该语料库中给出的 xml 文件预处理为 spacy 使用的任何格式,任何一点帮助将不胜感激。 我首先将 xml 文件转换为 json 格式,但 spacy 不接受。 spacy 需要什么格式的训练数据? 我什至尝试构建自己的 NER 模型,但无法预处理 xml article.

中给出的文件

下面是一个使用spacy训练NER模型的例子,包括训练数据的预期格式(来自spacy's docs):

import random

import spacy


TRAIN_DATA = [
        ("Uber blew through  million a week", {"entities": [(0, 4, "ORG")]}),
        ("Google rebrands its business apps", {"entities": [(0, 6, "ORG")]})]

nlp = spacy.blank("en")
optimizer = nlp.begin_training()
for i in range(20):
    random.shuffle(TRAIN_DATA)
    for text, annotations in TRAIN_DATA:
        nlp.update([text], [annotations], sgd=optimizer)
nlp.to_disk("/model")

我正在使用的 XML 文件可在线获得 here。示例记录如下所示:

<passage>
<infon key="section_type">ABSTRACT</infon>
<infon key="type">abstract</infon>
<offset>141</offset>
<text>
Breast cancer is the most frequent tumor in women, and in nearly two-thirds of cases, the tumors express estrogen receptor alpha (ERalpha, encoded by ESR1). Here, we performed whole-exome sequencing of 16 breast cancer tissues classified according to ESR1 expression and 12 samples of whole blood, and detected 310 somatic mutations in cancer tissues with high levels of ESR1 expression. Of the somatic mutations validated by a different deep sequencer, a novel nonsense somatic mutation, c.2830 C>T; p.Gln944*, in transcriptional regulator switch-independent 3 family member A (SIN3A) was detected in breast cancer of a patient. Part of the mutant protein localized in the cytoplasm in contrast to the nuclear localization of ERalpha, and induced a significant increase in ESR1 mRNA. The SIN3A mutation obviously enhanced MCF7 cell proliferation. In tissue sections from the breast cancer patient with the SIN3A c.2830 C>T mutation, cytoplasmic SIN3A localization was detected within the tumor regions where nuclear enlargement was observed. The reduction in SIN3A mRNA correlates with the recurrence of ER-positive breast cancers on Kaplan-Meier plots. These observations reveal that the SIN3A mutation has lost its transcriptional repression function due to its cytoplasmic localization, and that this repression may contribute to the progression of breast cancer.
</text>
<annotation id="38">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="246" length="23"/>
<text>estrogen receptor alpha</text>
</annotation>
<annotation id="39">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="271" length="7"/>
<text>ERalpha</text>
</annotation>
<annotation id="40">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="291" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="41">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="392" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="42">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="512" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="43">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="720" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="44">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="868" length="7"/>
<text>ERalpha</text>
</annotation>
<annotation id="45">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="915" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="46">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="930" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="47">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1048" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="48">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1087" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="49">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1201" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="50">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1331" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="51">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="185" length="5"/>
<text>women</text>
</annotation>
<annotation id="52">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="762" length="7"/>
<text>patient</text>
</annotation>
<annotation id="53">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="1031" length="7"/>
<text>patient</text>
</annotation>
<annotation id="54">
<infon key="identifier">29278</infon>
<infon key="type">Species</infon>
<location offset="397" length="10"/>
<text>expression</text>
</annotation>
<annotation id="55">
<infon key="identifier">29278</infon>
<infon key="type">Species</infon>
<location offset="517" length="10"/>
<text>expression</text>
</annotation>
<annotation id="56">
<infon key="identifier">c.2830C>T</infon>
<infon key="type">DNAMutation</infon>
<location offset="1054" length="10"/>
<text>c.2830 C>T</text>
</annotation>
<annotation id="57">
<infon key="identifier">CVCL:0031</infon>
<infon key="type">CellLine</infon>
<location offset="964" length="4"/>
<text>MCF7</text>
</annotation>
<annotation id="58">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1494" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="59">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="346" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="60">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="743" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="61">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1017" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="62">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="477" length="6"/>
<text>cancer</text>
</annotation>
<annotation id="63">
<infon key="identifier">p.Q944*</infon>
<infon key="type">ProteinMutation</infon>
<location offset="642" length="9"/>
<text>p.Gln944*</text>
</annotation>
<annotation id="64">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="1130" length="5"/>
<text>tumor</text>
</annotation>
<annotation id="65">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="176" length="5"/>
<text>tumor</text>
</annotation>
<annotation id="66">
<infon key="identifier">c.2830C>T</infon>
<infon key="type">DNAMutation</infon>
<location offset="630" length="10"/>
<text>c.2830 C>T</text>
</annotation>
<annotation id="67">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1258" length="14"/>
<text>breast cancers</text>
</annotation>
<annotation id="68">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="231" length="6"/>
<text>tumors</text>
</annotation>
<annotation id="69">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="141" length="13"/>
<text>Breast cancer</text>
</annotation>
</passage>

这里有一些代码可以让你开始。这不是一个完整的解决方案,但是您提出的问题非常难,而且您没有任何起始代码。

它不跟踪 identifierNCBI Homologene 属性,但我认为这些可以单独存储在字典中。

import xml.etree.cElementTree as ET

import spacy

nlp = spacy.load('en_core_web_sm')

# this is one child of the XML doc
# https://www.ncbi.nlm.nih.gov/research/pubtator-api/publications/export/biocxml?pmcids=PMC6207735
passage_string = """
<passage>
<infon key="section_type">ABSTRACT</infon>
<infon key="type">abstract</infon>
<offset>141</offset>
<text>
Breast cancer is the most frequent tumor in women, and in nearly two-thirds of cases, the tumors express estrogen receptor alpha (ERalpha, encoded by ESR1). Here, we performed whole-exome sequencing of 16 breast cancer tissues classified according to ESR1 expression and 12 samples of whole blood, and detected 310 somatic mutations in cancer tissues with high levels of ESR1 expression. Of the somatic mutations validated by a different deep sequencer, a novel nonsense somatic mutation, c.2830 C>T; p.Gln944*, in transcriptional regulator switch-independent 3 family member A (SIN3A) was detected in breast cancer of a patient. Part of the mutant protein localized in the cytoplasm in contrast to the nuclear localization of ERalpha, and induced a significant increase in ESR1 mRNA. The SIN3A mutation obviously enhanced MCF7 cell proliferation. In tissue sections from the breast cancer patient with the SIN3A c.2830 C>T mutation, cytoplasmic SIN3A localization was detected within the tumor regions where nuclear enlargement was observed. The reduction in SIN3A mRNA correlates with the recurrence of ER-positive breast cancers on Kaplan-Meier plots. These observations reveal that the SIN3A mutation has lost its transcriptional repression function due to its cytoplasmic localization, and that this repression may contribute to the progression of breast cancer.
</text>
<annotation id="38">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="246" length="23"/>
<text>estrogen receptor alpha</text>
</annotation>
<annotation id="39">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="271" length="7"/>
<text>ERalpha</text>
</annotation>
<annotation id="40">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="291" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="41">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="392" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="42">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="512" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="43">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="720" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="44">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="868" length="7"/>
<text>ERalpha</text>
</annotation>
<annotation id="45">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="915" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="46">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="930" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="47">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1048" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="48">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1087" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="49">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1201" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="50">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1331" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="51">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="185" length="5"/>
<text>women</text>
</annotation>
<annotation id="52">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="762" length="7"/>
<text>patient</text>
</annotation>
<annotation id="53">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="1031" length="7"/>
<text>patient</text>
</annotation>
<annotation id="54">
<infon key="identifier">29278</infon>
<infon key="type">Species</infon>
<location offset="397" length="10"/>
<text>expression</text>
</annotation>
<annotation id="55">
<infon key="identifier">29278</infon>
<infon key="type">Species</infon>
<location offset="517" length="10"/>
<text>expression</text>
</annotation>
<annotation id="56">
<infon key="identifier">c.2830C>T</infon>
<infon key="type">DNAMutation</infon>
<location offset="1054" length="10"/>
<text>c.2830 C>T</text>
</annotation>
<annotation id="57">
<infon key="identifier">CVCL:0031</infon>
<infon key="type">CellLine</infon>
<location offset="964" length="4"/>
<text>MCF7</text>
</annotation>
<annotation id="58">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1494" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="59">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="346" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="60">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="743" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="61">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1017" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="62">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="477" length="6"/>
<text>cancer</text>
</annotation>
<annotation id="63">
<infon key="identifier">p.Q944*</infon>
<infon key="type">ProteinMutation</infon>
<location offset="642" length="9"/>
<text>p.Gln944*</text>
</annotation>
<annotation id="64">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="1130" length="5"/>
<text>tumor</text>
</annotation>
<annotation id="65">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="176" length="5"/>
<text>tumor</text>
</annotation>
<annotation id="66">
<infon key="identifier">c.2830C>T</infon>
<infon key="type">DNAMutation</infon>
<location offset="630" length="10"/>
<text>c.2830 C>T</text>
</annotation>
<annotation id="67">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1258" length="14"/>
<text>breast cancers</text>
</annotation>
<annotation id="68">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="231" length="6"/>
<text>tumors</text>
</annotation>
<annotation id="69">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="141" length="13"/>
<text>Breast cancer</text>
</annotation>
</passage>"""

# turn into an object
passage = ET.fromstring(passage_string)

# these 3 definitions are per-passage
passage_annotations = passage.findall('./annotation')
passage_offset = int(passage.find('offset').text)
passage_text = passage.find('text').text

def get_entity_offset(offset_dict, passage_offset):
    """
    XML given offset_dict gives offset relative to the start of the document
    So subtract the passage offset (where passage starts relative to document beginning)
    """
    start = int(offset_dict['offset']) - passage_offset
    end = int(offset_dict['offset']) + (int(offset_dict['length']) + 1) - passage_offset
    return start, end

# collect entities as a list of tuples of the form
# (start, end, entitiy_type)
passage_entities = []
for ann in passage_annotations:
    entity_type = ann.find('./infon[@key="type"]').text
    od = ann.find('./location').attrib
    start, end = get_entity_offset(od, passage_offset)
    passage_entities.append((start, end, entity_type))

# this is one entry in the spacy NER format
# you would want many entries
spacyd_passage = (passage_text, {"entities": passage_entities})

# prove this worked
for ent in passage_entities:
    print(ent, passage_text[ent[0]:ent[1]])

# prints:
# (105, 129, 'Gene')  estrogen receptor alpha
# (130, 138, 'Gene') (ERalpha
# (150, 155, 'Gene')  ESR1
# (251, 256, 'Gene')  ESR1
# (371, 376, 'Gene')  ESR1
# (579, 585, 'Gene') (SIN3A
# (727, 735, 'Gene')  ERalpha
# (774, 779, 'Gene')  ESR1
# (789, 795, 'Gene')  SIN3A
# (907, 913, 'Gene')  SIN3A
# (946, 952, 'Gene')  SIN3A
# (1060, 1066, 'Gene')  SIN3A
# (1190, 1196, 'Gene')  SIN3A
# (44, 50, 'Species')  women
# (621, 629, 'Species')  patient
# (890, 898, 'Species')  patient
# (256, 267, 'Species')  expression
# (376, 387, 'Species')  expression
# (913, 924, 'DNAMutation')  c.2830 C>T
# (823, 828, 'CellLine')  MCF7
# (1353, 1367, 'Disease')  breast cancer
# (205, 219, 'Disease')  breast cancer
# (602, 616, 'Disease')  breast cancer
# (876, 890, 'Disease')  breast cancer
# (336, 343, 'Disease')  cancer
# (501, 511, 'ProteinMutation')  p.Gln944*
# (989, 995, 'Disease')  tumor
# (35, 41, 'Disease')  tumor
# (489, 500, 'DNAMutation')  c.2830 C>T
# (1117, 1132, 'Disease')  breast cancers
# (90, 97, 'Disease')  tumors
# (0, 14, 'Disease')  Breast cancer

所以,我注意到的第一件事是一些给定的偏移量略微偏离,捕捉到 (。您可以查找 if passage_text[ent[0]] == "(" 并将实体的开头移动 1 以清理它,或者手动清理它。

此外,此代码使用一个子节点,即链接文档的 passage。您需要在本地下载该文档,而不是 passage = ET.fromstring(passage_string),您将创建 tree = ET.parse('path_to_file'):

类似

import xml.etree.cElementTree as ET

tree = ET.parse('path_to_file')
root = tree.getroot()
passages = root.findall('./passages')

spacy_data = []

for passage in passages:
    passage_annotations = passage.findall('./annotation')
    passage_offset = int(passage.find('offset').text)
    passage_text = passage.find('text').text

    passage_entities = []
    for ann in passage_annotations:
        entity_type = ann.find('./infon[@key="type"]').text
        od = ann.find('./location').attrib
        start, end = get_entity_offset(od, passage_offset)
        passage_entities.append((start, end, entity_type))

        spacyd_passage = (passage_text, {"entities": passage_entities})
        spacy_data.append(spacyd_package)

这仍然可以改进。您需要使用

拆分这些 passage.text 段落
import spacy

nlp = spacy.load('en_core_web_sm')

doc = nlp(passage_text)
sents = list(doc.sents)

但棘手的部分是您需要进行算术运算以保持偏移索引正确。您还需要查看每个实体的开头和结尾,以确保它在一个句子中 - 可以想象它可以被句子边界分割,但可能不会。