在 HuggingFace 中加载模型作为 DPRQuestionEncoder

Question

我想将 BERT 的权重（或任何转换器）加载到 DPRQuestionEncoder architecture, such that I can use the HuggingFace save_pretrained method and plug the saved model into the RAG architecture to do end-to-end fine-tuning。

from transformers import DPRQuestionEncoder
model = DPRQuestionEncoder.from_pretrained('bert-base-uncased')

但是我得到以下错误

You are using a model of type bert to instantiate a model of type dpr. This is not supported for all configurations of models and can yield errors.

NotImplementedErrorTraceback (most recent call last)
<ipython-input-27-1f1b990b906b> in <module>
----> 1 model = DPRQuestionEncoder.from_pretrained(model_name)
      2 # https://github.com/huggingface/transformers/blob/41cd52a768a222a13da0c6aaae877a92fc6c783c/src/transformers/models/dpr/modeling_dpr.py#L520

/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1211                     )
   1212 
-> 1213             model, missing_keys, unexpected_keys, error_msgs = cls._load_state_dict_into_model(
   1214                 model, state_dict, pretrained_model_name_or_path, _fast_init=_fast_init
   1215             )

/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py in _load_state_dict_into_model(cls, model, state_dict, pretrained_model_name_or_path, _fast_init)
   1286             )
   1287             for module in unintialized_modules:
-> 1288                 model._init_weights(module)
   1289 
   1290         # copy state_dict so _load_from_state_dict can modify it

/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py in _init_weights(self, module)
    515         Initialize the weights. This method should be overridden by derived class.
    516         """
--> 517         raise NotImplementedError(f"Make sure `_init_weigths` is implemented for {self.__class__}")
    518 
    519     def tie_weights(self):

NotImplementedError: Make sure `_init_weigths` is implemented for <class 'transformers.models.dpr.modeling_dpr.DPRQuestionEncoder'>

我使用的是最新版本的变形金刚。

Answer 1

正如评论中已经提到的，DPRQuestionEncoder目前不提供任何加载其他模型的功能。我仍然建议创建您自己的 class，它继承自 DPRQuestionEncoder，加载您的自定义模型并调整其方法。

但是你在评论中询问是否有另一种方法，是的，如果你的模型参数和你的 DPRQuestionEncoder 对象持有的模型 完全相同。请查看下面的评论示例：

from transformers import BertModel

# here I am just loading a bert model that represents your model
ordinary_bert = BertModel.from_pretrained("bert-base-uncased")
ordinary_bert.save_pretrained("this_is_a_bert")

import torch
from transformers import DPRQuestionEncoder

# now we load the state dict (i.e. weights and bias) of your model
ordinary_bert_state_dict = torch.load('this_is_a_bert/pytorch_model.bin')

# here we create a DPRQuestionEncoder object 
# the facebook/dpr-question_encoder-single-nq-base has the same parameters as bert-base-uncased
# You can compare the respective configs or model.parameters to be sure 
model = DPRQuestionEncoder.from_pretrained('facebook/dpr-question_encoder-single-nq-base')

# we need to create the same keys (i.e. layer names) as your target model facebook/dpr-question_encoder-single-nq-base
ordinary_bert_state_dict = {f"question_encoder.bert_model.{k}":v for k,v in ordinary_bert_state_dict.items()}

# now we can load the bert-base-uncased weights into your DPRQuestionEncoder object 
model.load_state_dict(ordinary_bert_state_dict)

它从技术角度来看是可行的，但我无法告诉您它将如何执行您的任务。

在 HuggingFace 中加载模型作为 DPRQuestionEncoder

Load a model as DPRQuestionEncoder in HuggingFace

python

nlp

huggingface-transformers

bert-language-model

transformer