下载预训练好的BERT模型到本地

Question

我正在使用 SentenceTransformers 库（此处：https://pypi.org/project/sentence-transformers/#pretrained-models) for creating embeddings of sentences using the pretrained model bert-base-nli-mean-tokens. I have an application that will be deployed to a device that does not have internet access. How can I save this model locally so that when I call it, it loads the model locally, rather than attempting to download from the internet? As the library maintainers make clear, the method SentenceTransformer downloads the model from the internet (see here: https://pypi.org/project/sentence-transformers/#pretrained-models），但找不到在本地保存模型的方法。

Answer 1

Hugging face usage

您可以使用Hugging Face 转换器库方法将模型下载到本地。

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/bert-base-nli-mean-tokens")
model = AutoModel.from_pretrained("sentence-transformers/bert-base-nli-mean-tokens")
tokenizer.save_pretrained('./local_directory/')
model.save_pretrained('./local_directory/')

Answer 2

通过下载实例化 SentenceTransformer 后，您可以使用 'save()' 方法将其保存到您选择的任何路径。

model = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')
model.save('/my/local/directory/for/models/')

已接受的答案无效，因为它没有封装文件夹和 SentenceTransformer 正在寻找的 config.json

下载预训练好的BERT模型到本地

Download pre-trained BERT model locally

python-3.x

word-embedding