TypeError: not a string | parameters in AutoTokenizer.from_pretrained()

Question

目标：修改此 Notebook 以使用 albert-base-v2 模型。

内核：conda_pytorch_p36。我做了 Restart & 运行 All，并刷新了工作目录中的文件视图。

为了评估和导出这个量化模型，我需要设置一个 Tokenizer。

第 1.3 节出现错误。

AutoTokenizer.from_pretrained() 中的两个参数抛出相同的错误。

1.3节代码：

# define the tokenizer
tokenizer = AutoTokenizer.from_pretrained(
        configs.output_dir, do_lower_case=configs.do_lower_case)

参数：

# The output directory for the fine-tuned model, $OUT_DIR.
configs.output_dir = "./MRPC/"

# Prepare GLUE task
...
configs.do_lower_case = True

值和 DType：

-- configs.output_dir --
./MRPC/
<class 'str'>

-- configs.do_lower_case --
True
<class 'bool'>

回溯：

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-13-18c5137aacf4> in <module>
    140 # define the tokenizer
    141 tokenizer = AutoTokenizer.from_pretrained(
--> 142         configs.output_dir, do_lower_case=configs.do_lower_case)
    143 
    144 # Evaluate the original FP32 BERT model

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/models/auto/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    548             tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
    549             if tokenizer_class_fast and (use_fast or tokenizer_class_py is None):
--> 550                 return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    551             else:
    552                 if tokenizer_class_py is not None:

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1752             use_auth_token=use_auth_token,
   1753             cache_dir=cache_dir,
-> 1754             **kwargs,
   1755         )
   1756 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/tokenization_utils_base.py in _from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, use_auth_token, cache_dir, *init_inputs, **kwargs)
   1776                 copy.deepcopy(init_configuration),
   1777                 *init_inputs,
-> 1778                 **(copy.deepcopy(kwargs)),
   1779             )
   1780         else:

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/tokenization_utils_base.py in _from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, use_auth_token, cache_dir, *init_inputs, **kwargs)
   1880         # Instantiate tokenizer.
   1881         try:
-> 1882             tokenizer = cls(*init_inputs, **init_kwargs)
   1883         except OSError:
   1884             raise OSError(

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/models/albert/tokenization_albert.py in __init__(self, vocab_file, do_lower_case, remove_space, keep_accents, bos_token, eos_token, unk_token, sep_token, pad_token, cls_token, mask_token, sp_model_kwargs, **kwargs)
    179 
    180         self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
--> 181         self.sp_model.Load(vocab_file)
    182 
    183     @property

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sentencepiece/__init__.py in Load(self, model_file, model_proto)
    365       if model_proto:
    366         return self.LoadFromSerializedProto(model_proto)
--> 367       return self.LoadFromFile(model_file)
    368 
    369 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sentencepiece/__init__.py in LoadFromFile(self, arg)
    169 
    170     def LoadFromFile(self, arg):
--> 171         return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
    172 
    173     def DecodeIdsWithCheck(self, ids):

TypeError: not a string

如果我还有什么要补充的，请告诉我 post。

Answer 1

只需传递型号名称即可。

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')

可以找到 model_types 的列表 here。

TypeError: not a string | parameters in AutoTokenizer.from_pretrained()

TypeError: not a string | parameters in AutoTokenizer.from_pretrained()

python

tensorflow

onnx

huggingface-transformers

huggingface-tokenizers