如何从平面文件（Gene Ontology OBO 文件）生成递归树状字典？
How do I generate a recursive tree-like dictionary from a flat file (Gene Ontology OBO file)?
我正在尝试编写代码来解析 Gene Ontology (GO) OBO 文件并将 go 术语 ID（例如 GO:0003824）推送到树状嵌套字典中。 OBO 文件中的分层 go 结构用 "is_a" 标识符表示，用于标记每个 GO term 的每个父项。一个 GO 术语可能有多个父项，而层次结构中最高的 go 术语没有父项。
GO OBO 文件的一个小示例如下所示：
[Term]
id: GO:0003674
name: molecular_function
namespace: molecular_function
alt_id: GO:0005554
def: "A molecular process that can be carried out by the action of a single macromolecular machine, usually via direct physical interactions with other molecular entities. Function in this sense denotes an action, or activity, that a gene product (or a complex) performs. These actions are described from two distinct but related perspectives: (1) biochemical activity, and (2) role as a component in a larger system/process." [GOC:pdt]
comment: Note that, in addition to forming the root of the molecular function ontology, this term is recommended for use for the annotation of gene products whose molecular function is unknown. When this term is used for annotation, it indicates that no information was available about the molecular function of the gene product annotated as of the date the annotation was made; the evidence code "no data" (ND), is used to indicate this. Despite its name, this is not a type of 'function' in the sense typically defined by upper ontologies such as Basic Formal Ontology (BFO). It is instead a BFO:process carried out by a single gene product or complex.
subset: goslim_aspergillus
subset: goslim_candida
subset: goslim_chembl
subset: goslim_generic
subset: goslim_metagenomics
subset: goslim_pir
subset: goslim_plant
subset: goslim_yeast
synonym: "molecular function" EXACT []
[Term]
id: GO:0003824
name: catalytic activity
namespace: molecular_function
def: "Catalysis of a biochemical reaction at physiological temperatures. In biologically catalyzed reactions, the reactants are known as substrates, and the catalysts are naturally occurring macromolecular substances known as enzymes. Enzymes possess specific binding sites for substrates, and are usually composed wholly or largely of protein, but RNA that has catalytic activity (ribozyme) is often also regarded as enzymatic." [GOC:vw, ISBN:0198506732]
subset: goslim_chembl
subset: goslim_flybase_ribbon
subset: goslim_metagenomics
subset: goslim_pir
subset: goslim_plant
synonym: "enzyme activity" EXACT [GOC:dph, GOC:tb]
xref: Wikipedia:Enzyme
is_a: GO:0003674 ! molecular_function
[Term]
id: GO:0005198
name: structural molecule activity
namespace: molecular_function
def: "The action of a molecule that contributes to the structural integrity of a complex or its assembly within or outside a cell." [GOC:mah, GOC:vw]
subset: goslim_agr
subset: goslim_aspergillus
subset: goslim_candida
subset: goslim_chembl
subset: goslim_flybase_ribbon
subset: goslim_generic
subset: goslim_pir
subset: goslim_plant
subset: goslim_yeast
is_a: GO:0003674 ! molecular_function
[Term]
id: GO:0005488
name: binding
namespace: molecular_function
def: "The selective, non-covalent, often stoichiometric, interaction of a molecule with one or more specific sites on another molecule." [GOC:ceb, GOC:mah, ISBN:0198506732]
comment: Note that this term is in the subset of terms that should not be used for direct, manual gene product annotation. Please choose a more specific child term, or request a new one if no suitable term is available. For ligands that bind to signal transducing receptors, consider the molecular function term 'receptor binding ; GO:0005102' and its children.
subset: gocheck_do_not_manually_annotate
subset: goslim_pir
subset: goslim_plant
synonym: "ligand" NARROW []
xref: Wikipedia:Binding_(molecular)
is_a: GO:0003674 ! molecular_function
[Term]
id: GO:0005515
name: protein binding
namespace: molecular_function
alt_id: GO:0001948
alt_id: GO:0045308
def: "Interacting selectively and non-covalently with any protein or protein complex (a complex of two or more proteins that may include other nonprotein molecules)." [GOC:go_curators]
subset: goslim_aspergillus
subset: goslim_candida
subset: goslim_chembl
subset: goslim_metagenomics
subset: goslim_pir
subset: goslim_plant
synonym: "glycoprotein binding" NARROW []
synonym: "protein amino acid binding" EXACT []
xref: reactome:R-HSA-170835 "An anchoring protein, ZFYVE9 (SARA), recruits SMAD2/3"
xref: reactome:R-HSA-170846 "TGFBR2 recruits TGFBR1"
xref: reactome:R-HSA-3645786 "TGFBR2 mutant dimers recruit TGFBR1"
xref: reactome:R-HSA-3656484 "TGFBR2 recruits TGFBR1 KD Mutants"
xref: reactome:R-HSA-3702153 "An anchoring protein, ZFYVE9 (SARA), recruits SMAD2/3 MH2 domain mutants"
xref: reactome:R-HSA-3713560 "An anchoring protein, ZFYVE9 (SARA), recruits SMAD2/3 phosphorylation motif mutants"
is_a: GO:0005488 ! binding
[Term]
id: GO:0005549
name: odorant binding
namespace: molecular_function
def: "Interacting selectively and non-covalently with an odorant, any substance capable of stimulating the sense of smell." [GOC:jl, ISBN:0721662544]
subset: goslim_pir
is_a: GO:0005488 ! binding
[Term]
id: GO:0005550
name: pheromone binding
namespace: molecular_function
def: "Interacting selectively and non-covalently with a pheromone, a substance, or characteristic mixture of substances, that is secreted and released by an organism and detected by a second organism of the same or a closely related species, in which it causes a specific reaction, such as a definite behavioral reaction or a developmental process." [GOC:ai]
is_a: GO:0005549 ! odorant binding
下面是在树状字典中存储 GO 术语 ID 的递归函数（和一些支持代码）的尝试：
import pandas as pd
import re
with open("tiny_go.obo", 'rt') as f:
content = f.read()
# Clean GO terms list
def clean_go_terms(terms):
l = []
for term in terms:
if (len(re.findall('is_obsolete: true', term))==0) and (len(re.findall('id: GO:\d+', term)) > 0):
l.append(term)
return l
def get_top_nodes(terms):
l = []
for term in terms:
if len(re.findall('is_a: GO:\d+', term)) == 0:
l.append(term)
return l
split_terms = content.split('\n\n')
split_terms_clean = clean_go_terms(split_terms)
top_nodes = get_top_nodes(split_terms_clean)
len(top_nodes)
# Find every term that has the top node as a parent; apply recursively to entire list of terms
# * Keys with empty lists will be leaves
def generate_go_tree(parent_nodes, all_go_terms, switch=True):
go_dict = {}
for node in parent_nodes:
parent_go_id = re.findall('id: (GO:\d+)', node)[0]
go_dict[parent_go_id] = {}
for go_term in all_go_terms:
go_id = re.findall('id: (GO:\d+)', go_term)[0]
parent_list = re.findall('is_a: (GO:\d+)', go_term)
if (parent_go_id in parent_list):
go_dict[parent_go_id][go_id] = generate_go_tree([go_term], all_go_terms, True)
return go_dict
go_tree = generate_go_tree(top_nodes, split_terms_clean)
显然我没有正确构造递归函数，因为我在输出中看到了重复的键：
{'GO:0003674': {'GO:0003824': {'GO:0003824': {}},
'GO:0005198': {'GO:0005198': {}},
'GO:0005488': {'GO:0005488': {'GO:0005515': {'GO:0005515': {}},
'GO:0005549': {'GO:0005549': {'GO:0005550': {'GO:0005550': {}}}}}}}}
非常感谢有关如何修复递归函数的建议！谢谢！
你写了
if (parent_go_id in parent_list):
go_dict[parent_go_id][go_id] = generate_go_tree([go_term], all_go_terms, True)
正确的是
if (parent_go_id in parent_list):
go_dict[parent_go_id][go_id] = generate_go_tree([go_term], all_go_terms, True)[go_id]
修改后，生成：
{
'GO:0003674': {
'GO:0003824': {},
'GO:0005198': {},
'GO:0005488': {
'GO:0005515': {},
'GO:0005549': {
'GO:0005550': {}
}
}
}
}
但我建议采用完全不同的方法。制作一个 class 来解析术语并构建依赖树。
为方便起见，我从 dict 派生了它，因此您可以写 term.id 而不是 term['id']:
class Term(dict):
__getattr__ = dict.__getitem__
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
registry = {}
single_valued = 'id name namespace alt_id def comment synonym is_a'.split()
multi_valued = 'subset xref'.split()
def __init__(self, text):
self.children = []
self.parent = None
for line in text.splitlines():
if not ': ' in line:
continue
key, val = line.split(': ', 1)
if key in Term.single_valued:
self[key] = val
elif key in Term.multi_valued:
if not key in self:
self[key] = [val]
else:
self[key].append(val)
else:
print('unclear property: %s' % line)
if 'id' in self:
Term.registry[self.id] = self
if 'alt_id' in self:
Term.registry[self.alt_id] = self
if 'is_a' in self:
key = self.is_a.split(' ! ', 1)[0]
if key in Term.registry:
Term.registry[key].children.append(self)
self.parent = Term.registry[key]
def is_top(self):
return self.parent == None
def is_valid(self):
return self.get('is_obsolete') != 'true' and self.id != None
现在您可以一次性下载文件了：
with open('tiny_go.obo', 'rt') as f:
contents = f.read()
terms = [Term(text) for text in contents.split('\n\n')]
并且递归树变得容易。例如一个简单的"print"函数，只输出非废弃节点：
def print_tree(terms, indent=''):
valid_terms = [term for term in terms if term.is_valid()]
for term in valid_terms:
print(indent + 'Term %s - %s' % (term.id, term.name))
print_tree(term.children, indent + ' ')
top_terms = [term for term in terms if term.is_top()]
print_tree(top_terms)
这会打印：
Term GO:0003674 - molecular_function
Term GO:0003824 - catalytic activity
Term GO:0005198 - structural molecule activity
Term GO:0005488 - binding
Term GO:0005515 - protein binding
Term GO:0005549 - odorant binding
Term GO:0005550 - pheromone binding
您还可以执行 Term.registry['GO:0005549'].parent.name 之类的操作，这样会得到 "binding"。
我将生成嵌套的 dicts GO-ID（就像在您自己的示例中一样）作为练习，但您甚至可能不需要它，因为 Term.registry 已经与此非常相似。
您可以使用递归来获得更短的解决方案：
import itertools, re, json
content = list(filter(None, [i.strip('\n') for i in open('filename.txt')]))
entries = [[a, list(b)] for a, b in itertools.groupby(content, key=lambda x:x== '[Term]')]
terms = [(lambda x:x if 'is_a' not in x else {**x, 'is_a':re.findall('^GO:\d+', x['is_a'])[0]})(dict(i.split(': ', 1) for i in b)) for a, b in entries if not a]
terms = sorted(terms, key=lambda x:'is_a' in x)
def tree(d, _start):
t = [i for i in d if i.get('is_a') == _start]
return {} if not t else {i['id']:tree(d, i['id']) for i in t}
print(json.dumps({terms[0]['id']:tree(terms, terms[0]['id'])}, indent=4))
输出：
{
"GO:0003674": {
"GO:0003824": {},
"GO:0005198": {},
"GO:0005488": {
"GO:0005515": {},
"GO:0005549": {
"GO:0005550": {}
}
}
}
}
如果父数据集未在其子数据集之前定义，这也适用。例如，当父级位于其原始位置下方三位时，仍会生成相同的结果(see file)：
print(json.dumps({terms[0]['id']:tree(terms, terms[0]['id'])}, indent=4))
输出：
{
"GO:0003674": {
"GO:0003824": {},
"GO:0005198": {},
"GO:0005488": {
"GO:0005515": {},
"GO:0005549": {
"GO:0005550": {}
}
}
}
}
我正在尝试编写代码来解析 Gene Ontology (GO) OBO 文件并将 go 术语 ID（例如 GO:0003824）推送到树状嵌套字典中。 OBO 文件中的分层 go 结构用 "is_a" 标识符表示，用于标记每个 GO term 的每个父项。一个 GO 术语可能有多个父项，而层次结构中最高的 go 术语没有父项。
GO OBO 文件的一个小示例如下所示：
[Term]
id: GO:0003674
name: molecular_function
namespace: molecular_function
alt_id: GO:0005554
def: "A molecular process that can be carried out by the action of a single macromolecular machine, usually via direct physical interactions with other molecular entities. Function in this sense denotes an action, or activity, that a gene product (or a complex) performs. These actions are described from two distinct but related perspectives: (1) biochemical activity, and (2) role as a component in a larger system/process." [GOC:pdt]
comment: Note that, in addition to forming the root of the molecular function ontology, this term is recommended for use for the annotation of gene products whose molecular function is unknown. When this term is used for annotation, it indicates that no information was available about the molecular function of the gene product annotated as of the date the annotation was made; the evidence code "no data" (ND), is used to indicate this. Despite its name, this is not a type of 'function' in the sense typically defined by upper ontologies such as Basic Formal Ontology (BFO). It is instead a BFO:process carried out by a single gene product or complex.
subset: goslim_aspergillus
subset: goslim_candida
subset: goslim_chembl
subset: goslim_generic
subset: goslim_metagenomics
subset: goslim_pir
subset: goslim_plant
subset: goslim_yeast
synonym: "molecular function" EXACT []
[Term]
id: GO:0003824
name: catalytic activity
namespace: molecular_function
def: "Catalysis of a biochemical reaction at physiological temperatures. In biologically catalyzed reactions, the reactants are known as substrates, and the catalysts are naturally occurring macromolecular substances known as enzymes. Enzymes possess specific binding sites for substrates, and are usually composed wholly or largely of protein, but RNA that has catalytic activity (ribozyme) is often also regarded as enzymatic." [GOC:vw, ISBN:0198506732]
subset: goslim_chembl
subset: goslim_flybase_ribbon
subset: goslim_metagenomics
subset: goslim_pir
subset: goslim_plant
synonym: "enzyme activity" EXACT [GOC:dph, GOC:tb]
xref: Wikipedia:Enzyme
is_a: GO:0003674 ! molecular_function
[Term]
id: GO:0005198
name: structural molecule activity
namespace: molecular_function
def: "The action of a molecule that contributes to the structural integrity of a complex or its assembly within or outside a cell." [GOC:mah, GOC:vw]
subset: goslim_agr
subset: goslim_aspergillus
subset: goslim_candida
subset: goslim_chembl
subset: goslim_flybase_ribbon
subset: goslim_generic
subset: goslim_pir
subset: goslim_plant
subset: goslim_yeast
is_a: GO:0003674 ! molecular_function
[Term]
id: GO:0005488
name: binding
namespace: molecular_function
def: "The selective, non-covalent, often stoichiometric, interaction of a molecule with one or more specific sites on another molecule." [GOC:ceb, GOC:mah, ISBN:0198506732]
comment: Note that this term is in the subset of terms that should not be used for direct, manual gene product annotation. Please choose a more specific child term, or request a new one if no suitable term is available. For ligands that bind to signal transducing receptors, consider the molecular function term 'receptor binding ; GO:0005102' and its children.
subset: gocheck_do_not_manually_annotate
subset: goslim_pir
subset: goslim_plant
synonym: "ligand" NARROW []
xref: Wikipedia:Binding_(molecular)
is_a: GO:0003674 ! molecular_function
[Term]
id: GO:0005515
name: protein binding
namespace: molecular_function
alt_id: GO:0001948
alt_id: GO:0045308
def: "Interacting selectively and non-covalently with any protein or protein complex (a complex of two or more proteins that may include other nonprotein molecules)." [GOC:go_curators]
subset: goslim_aspergillus
subset: goslim_candida
subset: goslim_chembl
subset: goslim_metagenomics
subset: goslim_pir
subset: goslim_plant
synonym: "glycoprotein binding" NARROW []
synonym: "protein amino acid binding" EXACT []
xref: reactome:R-HSA-170835 "An anchoring protein, ZFYVE9 (SARA), recruits SMAD2/3"
xref: reactome:R-HSA-170846 "TGFBR2 recruits TGFBR1"
xref: reactome:R-HSA-3645786 "TGFBR2 mutant dimers recruit TGFBR1"
xref: reactome:R-HSA-3656484 "TGFBR2 recruits TGFBR1 KD Mutants"
xref: reactome:R-HSA-3702153 "An anchoring protein, ZFYVE9 (SARA), recruits SMAD2/3 MH2 domain mutants"
xref: reactome:R-HSA-3713560 "An anchoring protein, ZFYVE9 (SARA), recruits SMAD2/3 phosphorylation motif mutants"
is_a: GO:0005488 ! binding
[Term]
id: GO:0005549
name: odorant binding
namespace: molecular_function
def: "Interacting selectively and non-covalently with an odorant, any substance capable of stimulating the sense of smell." [GOC:jl, ISBN:0721662544]
subset: goslim_pir
is_a: GO:0005488 ! binding
[Term]
id: GO:0005550
name: pheromone binding
namespace: molecular_function
def: "Interacting selectively and non-covalently with a pheromone, a substance, or characteristic mixture of substances, that is secreted and released by an organism and detected by a second organism of the same or a closely related species, in which it causes a specific reaction, such as a definite behavioral reaction or a developmental process." [GOC:ai]
is_a: GO:0005549 ! odorant binding
下面是在树状字典中存储 GO 术语 ID 的递归函数（和一些支持代码）的尝试：
import pandas as pd
import re
with open("tiny_go.obo", 'rt') as f:
content = f.read()
# Clean GO terms list
def clean_go_terms(terms):
l = []
for term in terms:
if (len(re.findall('is_obsolete: true', term))==0) and (len(re.findall('id: GO:\d+', term)) > 0):
l.append(term)
return l
def get_top_nodes(terms):
l = []
for term in terms:
if len(re.findall('is_a: GO:\d+', term)) == 0:
l.append(term)
return l
split_terms = content.split('\n\n')
split_terms_clean = clean_go_terms(split_terms)
top_nodes = get_top_nodes(split_terms_clean)
len(top_nodes)
# Find every term that has the top node as a parent; apply recursively to entire list of terms
# * Keys with empty lists will be leaves
def generate_go_tree(parent_nodes, all_go_terms, switch=True):
go_dict = {}
for node in parent_nodes:
parent_go_id = re.findall('id: (GO:\d+)', node)[0]
go_dict[parent_go_id] = {}
for go_term in all_go_terms:
go_id = re.findall('id: (GO:\d+)', go_term)[0]
parent_list = re.findall('is_a: (GO:\d+)', go_term)
if (parent_go_id in parent_list):
go_dict[parent_go_id][go_id] = generate_go_tree([go_term], all_go_terms, True)
return go_dict
go_tree = generate_go_tree(top_nodes, split_terms_clean)
显然我没有正确构造递归函数，因为我在输出中看到了重复的键：
{'GO:0003674': {'GO:0003824': {'GO:0003824': {}},
'GO:0005198': {'GO:0005198': {}},
'GO:0005488': {'GO:0005488': {'GO:0005515': {'GO:0005515': {}},
'GO:0005549': {'GO:0005549': {'GO:0005550': {'GO:0005550': {}}}}}}}}
非常感谢有关如何修复递归函数的建议！谢谢！
你写了
if (parent_go_id in parent_list):
go_dict[parent_go_id][go_id] = generate_go_tree([go_term], all_go_terms, True)
正确的是
if (parent_go_id in parent_list):
go_dict[parent_go_id][go_id] = generate_go_tree([go_term], all_go_terms, True)[go_id]
修改后，生成：
{
'GO:0003674': {
'GO:0003824': {},
'GO:0005198': {},
'GO:0005488': {
'GO:0005515': {},
'GO:0005549': {
'GO:0005550': {}
}
}
}
}
但我建议采用完全不同的方法。制作一个 class 来解析术语并构建依赖树。
为方便起见，我从 dict 派生了它，因此您可以写 term.id 而不是 term['id']:
class Term(dict):
__getattr__ = dict.__getitem__
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
registry = {}
single_valued = 'id name namespace alt_id def comment synonym is_a'.split()
multi_valued = 'subset xref'.split()
def __init__(self, text):
self.children = []
self.parent = None
for line in text.splitlines():
if not ': ' in line:
continue
key, val = line.split(': ', 1)
if key in Term.single_valued:
self[key] = val
elif key in Term.multi_valued:
if not key in self:
self[key] = [val]
else:
self[key].append(val)
else:
print('unclear property: %s' % line)
if 'id' in self:
Term.registry[self.id] = self
if 'alt_id' in self:
Term.registry[self.alt_id] = self
if 'is_a' in self:
key = self.is_a.split(' ! ', 1)[0]
if key in Term.registry:
Term.registry[key].children.append(self)
self.parent = Term.registry[key]
def is_top(self):
return self.parent == None
def is_valid(self):
return self.get('is_obsolete') != 'true' and self.id != None
现在您可以一次性下载文件了：
with open('tiny_go.obo', 'rt') as f:
contents = f.read()
terms = [Term(text) for text in contents.split('\n\n')]
并且递归树变得容易。例如一个简单的"print"函数，只输出非废弃节点：
def print_tree(terms, indent=''):
valid_terms = [term for term in terms if term.is_valid()]
for term in valid_terms:
print(indent + 'Term %s - %s' % (term.id, term.name))
print_tree(term.children, indent + ' ')
top_terms = [term for term in terms if term.is_top()]
print_tree(top_terms)
这会打印：
Term GO:0003674 - molecular_function
Term GO:0003824 - catalytic activity
Term GO:0005198 - structural molecule activity
Term GO:0005488 - binding
Term GO:0005515 - protein binding
Term GO:0005549 - odorant binding
Term GO:0005550 - pheromone binding
您还可以执行 Term.registry['GO:0005549'].parent.name 之类的操作，这样会得到 "binding"。
我将生成嵌套的 dicts GO-ID（就像在您自己的示例中一样）作为练习，但您甚至可能不需要它，因为 Term.registry 已经与此非常相似。
您可以使用递归来获得更短的解决方案：
import itertools, re, json
content = list(filter(None, [i.strip('\n') for i in open('filename.txt')]))
entries = [[a, list(b)] for a, b in itertools.groupby(content, key=lambda x:x== '[Term]')]
terms = [(lambda x:x if 'is_a' not in x else {**x, 'is_a':re.findall('^GO:\d+', x['is_a'])[0]})(dict(i.split(': ', 1) for i in b)) for a, b in entries if not a]
terms = sorted(terms, key=lambda x:'is_a' in x)
def tree(d, _start):
t = [i for i in d if i.get('is_a') == _start]
return {} if not t else {i['id']:tree(d, i['id']) for i in t}
print(json.dumps({terms[0]['id']:tree(terms, terms[0]['id'])}, indent=4))
输出：
{
"GO:0003674": {
"GO:0003824": {},
"GO:0005198": {},
"GO:0005488": {
"GO:0005515": {},
"GO:0005549": {
"GO:0005550": {}
}
}
}
}
如果父数据集未在其子数据集之前定义，这也适用。例如，当父级位于其原始位置下方三位时，仍会生成相同的结果(see file)：
print(json.dumps({terms[0]['id']:tree(terms, terms[0]['id'])}, indent=4))
输出：
{
"GO:0003674": {
"GO:0003824": {},
"GO:0005198": {},
"GO:0005488": {
"GO:0005515": {},
"GO:0005549": {
"GO:0005550": {}
}
}
}
}