在 spacy 3 中更新 ner 模型时出错,有什么建议吗?
Error updating ner model in spacy 3, any advice?
我目前正在从 fr_core_news_lg
管道更新 NER 模型。该代码大约在 1 或 2 个月前工作,当时我最后一次使用它。但是现在,发生了一些事情,我不能再 运行 了。我没有对代码进行任何更改,只是想再次 运行 它。但是我收到以下错误:
Traceback (most recent call last):
File "../nermodel.py", line 174, in <module>
ner_model.train(med_label)
File "../nermodel.py", line 102, in train
optimizer = self.nlp.entity.create_optimizer()
AttributeError: 'French' object has no attribute 'entity'
错误指向我用新示例更新我的 NER 模型的代码部分:
def train(self, label, n_iter=10, batch_size=50):
# creating an optimizer and selecting a list of pipes NOT to train
optimizer = self.nlp.entity.create_optimizer()
other_pipes = [pipe for pipe in self.nlp.pipe_names if pipe != 'ner']
# adding a named entity label
ner = self.nlp.get_pipe('ner')
ner.add_label(label)
with self.nlp.disable_pipes(*other_pipes):
for itn in range(n_iter):
random.shuffle(self.train_data)
losses = {}
# batch the examples and iterate over them
for batch in spacy.util.minibatch(self.train_data, size=batch_size):
texts = [text for text, entities in batch]
annotations = [entities for text, entities in batch]
# update the model
self.nlp.update(texts, annotations, sgd=optimizer, losses=losses)
print(losses)
print("Final loss: ", losses)
一个单一的训练样本,让NER知道'consultation'是一个实体,如下:
('et la consultation post-réanimation', {'entities': [(6, 18, 'MEDICAL_TERM')]})
我已经将 SpaCy 更新到最新版本,并再次下载了 fr_core_news_lg
模型,甚至在新的 python 环境中尝试过,但无济于事。这让我觉得管道或 SpaCy 库发生了变化。谷歌搜索,我无法找到准确的答案。有人对此有解决办法吗?
编辑: 提供了更多详细信息。
我认为这段代码应该适合你:
def train(self, label, n_iter=10, batch_size=50):
# creating an optimizer and selecting a list of pipes NOT to train
optimizer = self.nlp.create_optimizer()
other_pipes = [pipe for pipe in self.nlp.pipe_names if pipe != 'ner']
# adding a named entity label
ner = self.nlp.get_pipe('ner')
ner.add_label(label)
with self.nlp.disable_pipes(*other_pipes):
for itn in range(n_iter):
random.shuffle(self.train_data)
losses = {}
# batch the examples and iterate over them
for batch in spacy.util.minibatch(self.train_data, size=batch_size):
for text, annotations in batch:
doc = nlp.make_doc(text)
example = Example.from_dict(doc, annotations)
nlp.update([example], drop=0.35, sgd=optimizer, losses=losses)
print(losses)
print("Final loss: ", losses)
进一步分解,在 spacy 3 中有两个变化:
- 他们在
nlp.entity.create_optimizer()
中摆脱了实体
- 我们不会将文本和注释直接传递给
nlp.update()
,而是通过 Example
我目前正在从 fr_core_news_lg
管道更新 NER 模型。该代码大约在 1 或 2 个月前工作,当时我最后一次使用它。但是现在,发生了一些事情,我不能再 运行 了。我没有对代码进行任何更改,只是想再次 运行 它。但是我收到以下错误:
Traceback (most recent call last):
File "../nermodel.py", line 174, in <module>
ner_model.train(med_label)
File "../nermodel.py", line 102, in train
optimizer = self.nlp.entity.create_optimizer()
AttributeError: 'French' object has no attribute 'entity'
错误指向我用新示例更新我的 NER 模型的代码部分:
def train(self, label, n_iter=10, batch_size=50):
# creating an optimizer and selecting a list of pipes NOT to train
optimizer = self.nlp.entity.create_optimizer()
other_pipes = [pipe for pipe in self.nlp.pipe_names if pipe != 'ner']
# adding a named entity label
ner = self.nlp.get_pipe('ner')
ner.add_label(label)
with self.nlp.disable_pipes(*other_pipes):
for itn in range(n_iter):
random.shuffle(self.train_data)
losses = {}
# batch the examples and iterate over them
for batch in spacy.util.minibatch(self.train_data, size=batch_size):
texts = [text for text, entities in batch]
annotations = [entities for text, entities in batch]
# update the model
self.nlp.update(texts, annotations, sgd=optimizer, losses=losses)
print(losses)
print("Final loss: ", losses)
一个单一的训练样本,让NER知道'consultation'是一个实体,如下:
('et la consultation post-réanimation', {'entities': [(6, 18, 'MEDICAL_TERM')]})
我已经将 SpaCy 更新到最新版本,并再次下载了 fr_core_news_lg
模型,甚至在新的 python 环境中尝试过,但无济于事。这让我觉得管道或 SpaCy 库发生了变化。谷歌搜索,我无法找到准确的答案。有人对此有解决办法吗?
编辑: 提供了更多详细信息。
我认为这段代码应该适合你:
def train(self, label, n_iter=10, batch_size=50):
# creating an optimizer and selecting a list of pipes NOT to train
optimizer = self.nlp.create_optimizer()
other_pipes = [pipe for pipe in self.nlp.pipe_names if pipe != 'ner']
# adding a named entity label
ner = self.nlp.get_pipe('ner')
ner.add_label(label)
with self.nlp.disable_pipes(*other_pipes):
for itn in range(n_iter):
random.shuffle(self.train_data)
losses = {}
# batch the examples and iterate over them
for batch in spacy.util.minibatch(self.train_data, size=batch_size):
for text, annotations in batch:
doc = nlp.make_doc(text)
example = Example.from_dict(doc, annotations)
nlp.update([example], drop=0.35, sgd=optimizer, losses=losses)
print(losses)
print("Final loss: ", losses)
进一步分解,在 spacy 3 中有两个变化:
- 他们在
nlp.entity.create_optimizer()
中摆脱了实体
- 我们不会将文本和注释直接传递给
nlp.update()
,而是通过Example