如何在BERT中获取对应的已经标记为'UNK' token的字符或字符串？
How to get the corresponding character or string that has been labelled as 'UNK' token in BERT?
对字符串进行标记后，它 returns 由单独的单词和特殊标记组成的标记列表。例如，如果有的话，如何解码哪个word/character被称为'UNK'令牌？
快速标记器 return Batchencoding object that has a built-in word_ids and token_to_chars:
from transformers import BertTokenizerFast
t = BertTokenizerFast.from_pretrained('bert-base-uncased')
tokens = t('word embeddings are vectors ')
print(tokens['input_ids'])
print(t.decode(tokens['input_ids']))
print(tokens.word_ids())
print(tokens.token_to_chars(8))
输出：
[101, 2773, 7861, 8270, 4667, 2015, 2024, 19019, 100, 102]
[CLS] word embeddings are vectors [UNK] [SEP]
[None, 0, 1, 1, 1, 1, 2, 3, 4, None]
CharSpan(start=28, end=29)
对字符串进行标记后，它 returns 由单独的单词和特殊标记组成的标记列表。例如，如果有的话，如何解码哪个word/character被称为'UNK'令牌？
快速标记器 return Batchencoding object that has a built-in word_ids and token_to_chars:
from transformers import BertTokenizerFast
t = BertTokenizerFast.from_pretrained('bert-base-uncased')
tokens = t('word embeddings are vectors ')
print(tokens['input_ids'])
print(t.decode(tokens['input_ids']))
print(tokens.word_ids())
print(tokens.token_to_chars(8))
输出：
[101, 2773, 7861, 8270, 4667, 2015, 2024, 19019, 100, 102]
[CLS] word embeddings are vectors [UNK] [SEP]
[None, 0, 1, 1, 1, 1, 2, 3, 4, None]
CharSpan(start=28, end=29)