在 BERT 中为长文本滑动 window 以进行问答
Sliding window for long text in BERT for Question Answering
我已阅读 post，其中解释了滑动 window 的工作原理，但我找不到有关其实际实现方式的任何信息。
据我了解，如果输入太长，可以使用滑动window来处理文本。
如有错误请指正。
假设我有一条短信”2017年6月Kaggle宣布注册用户突破100万”.
给定一些 stride 和 max_len，可以将输入拆分成具有重叠词的块（不考虑填充）。
In June 2017 Kaggle announced that # chunk 1
announced that it passed 1 million # chunk 2
1 million registered users # chunk 3
如果我的问题是“Kaggle 何时发布公告” 和“如何发布公告”许多注册用户我可以使用chunk 1和chunk 3并且不使用chunk 2 在模型中完全 。不确定我是否仍应使用 chunk 2 来训练模型
因此输入将是：
[CLS]when did Kaggle make the announcement[SEP]In June 2017 Kaggle announced that[SEP]
和
[CLS]how many registered users[SEP]1 million registered users[SEP]
然后，如果我有一个没有答案的问题，我是否将它与所有块一起输入到模型中，并将开始和结束索引指示为 -1？例如“猪会飞吗？”
[CLS]can pigs fly[SEP]In June 2017 Kaggle announced that[SEP]
[CLS]can pigs fly[SEP]announced that it passed 1 million[SEP]
[CLS]can pigs fly[SEP]1 million registered users[SEP]
按照评论中的建议，我尝试运行 squad_convert_example_to_features (source code) 来调查我上面的问题，但似乎没有用，也没有是任何文档。 huggingface 中的 run_squad.py 似乎使用 squad_convert_example_to_features 和 example 中的 s。
from transformers.data.processors.squad import SquadResult, SquadV1Processor, SquadV2Processor, squad_convert_example_to_features
from transformers import AutoTokenizer, AutoConfig, squad_convert_examples_to_features
FILE_DIR = "."
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
processor = SquadV2Processor()
examples = processor.get_train_examples(FILE_DIR)
features = squad_convert_example_to_features(
example=examples[0],
max_seq_length=384,
doc_stride=128,
max_query_length=64,
is_training=True,
)
我收到错误。
100%|██████████| 1/1 [00:00<00:00, 159.95it/s]
Traceback (most recent call last):
File "<input>", line 25, in <module>
sub_tokens = tokenizer.tokenize(token)
NameError: name 'tokenizer' is not defined
错误表明没有tokenizers，但不允许我们传递tokenizer。虽然如果我在调试模式下的函数中添加标记器它确实有效。那么我该如何使用 squad_convert_example_to_features 函数呢？
我觉得你选的例子有问题。 squad_convert_examples_to_features and squad_convert_example_to_features have a sliding window approach implemented because squad_convert_examples_to_features is just a parallelization wrapper for squad_convert_example_to_features. But let's look at the single example function. First of all you need to call squad_convert_example_to_features_init 使标记生成器全局化（这在 squad_convert_examples_to_features 中自动为您完成）：
from transformers.data.processors.squad import SquadResult, SquadV1Processor, SquadV2Processor, squad_convert_examples_to_features, squad_convert_example_to_features_init
from transformers import AutoTokenizer, AutoConfig, squad_convert_examples_to_features
FILE_DIR = "."
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
squad_convert_example_to_features_init(tokenizer)
processor = SquadV2Processor()
examples = processor.get_train_examples(FILE_DIR)
features = squad_convert_example_to_features(
example=examples[0],
max_seq_length=384,
doc_stride=128,
max_query_length=64,
is_training=True,
)
print(len(features))
输出：
1
你可能会说这个函数没有使用滑动 window 方法，但这是错误的，因为你的示例不需要拆分:
print(len(examples[0].question_text.split()) + len(examples[0].doc_tokens))
输出：
115
小于您设置为 384 的 max_seq_length。现在让我们尝试另一个：
print(len(examples[129603].question_text.split()) + len(examples[129603].doc_tokens))
features = squad_convert_example_to_features(
example=examples[129603],
max_seq_length=384,
doc_stride=128,
max_query_length=64,
is_training=True,
)
print(len(features))
输出：
454
3
您现在可以将其与原始样本进行比较：
print('[CLS]' + examples[129603].question_text + '[SEP]' + ' '.join(examples[129603].doc_tokens) + '[SEP]')
for idx, f in enumerate(features):
print('Split {}'.format(idx))
print(' '.join(f.tokens))
输出：
[CLS]How often is hunting occurring in Delaware each year?[SEP]There is a very active tradition of hunting of small to medium-sized wild game in Trinidad and Tobago. Hunting is carried out with firearms, and aided by the use of hounds, with the illegal use of trap guns, trap cages and snare nets. With approximately 12,000 sport hunters applying for hunting licences in recent years (in a very small country of about the size of the state of Delaware at about 5128 square kilometers and 1.3 million inhabitants), there is some concern that the practice might not be sustainable. In addition there are at present no bag limits and the open season is comparatively very long (5 months - October to February inclusive). As such hunting pressure from legal hunters is very high. Added to that, there is a thriving and very lucrative black market for poached wild game (sold and enthusiastically purchased as expensive luxury delicacies) and the numbers of commercial poachers in operation is unknown but presumed to be fairly high. As a result, the populations of the five major mammalian game species (red-rumped agouti, lowland paca, nine-banded armadillo, collared peccary, and red brocket deer) are thought to be quite low (although scientifically conducted population studies are only just recently being conducted as of 2013). It appears that the red brocket deer population has been extirpated on Tobago as a result of over-hunting. Various herons, ducks, doves, the green iguana, the gold tegu, the spectacled caiman and the common opossum are also commonly hunted and poached. There is also some poaching of 'fully protected species', including red howler monkeys and capuchin monkeys, southern tamanduas, Brazilian porcupines, yellow-footed tortoises, Trinidad piping guans and even one of the national birds, the scarlet ibis. Legal hunters pay very small fees to obtain hunting licences and undergo no official basic conservation biology or hunting-ethics training. There is presumed to be relatively very little subsistence hunting in the country (with most hunting for either sport or commercial profit). The local wildlife management authority is under-staffed and under-funded, and as such very little in the way of enforcement is done to uphold existing wildlife management laws, with hunting occurring both in and out of season, and even in wildlife sanctuaries. There is some indication that the government is beginning to take the issue of wildlife management more seriously, with well drafted legislation being brought before Parliament in 2015. It remains to be seen if the drafted legislation will be fully adopted and financially supported by the current and future governments, and if the general populace will move towards a greater awareness of the importance of wildlife conservation and change the culture of wanton consumption to one of sustainable management.[SEP]
Split 0
[CLS] how often is hunting occurring in delaware each year ? [SEP] there is a very active tradition of hunting of small to medium - sized wild game in trinidad and tobago . hunting is carried out with firearms , and aided by the use of hounds , with the illegal use of trap guns , trap cages and s ##nare nets . with approximately 12 , 000 sport hunters applying for hunting licence ##s in recent years ( in a very small country of about the size of the state of delaware at about 512 ##8 square kilometers and 1 . 3 million inhabitants ) , there is some concern that the practice might not be sustainable . in addition there are at present no bag limits and the open season is comparatively very long ( 5 months - october to february inclusive ) . as such hunting pressure from legal hunters is very high . added to that , there is a thriving and very lucrative black market for po ##ache ##d wild game ( sold and enthusiastically purchased as expensive luxury del ##ica ##cies ) and the numbers of commercial po ##ache ##rs in operation is unknown but presumed to be fairly high . as a result , the populations of the five major mammalian game species ( red - rum ##ped ago ##uti , lowland pac ##a , nine - banded arm ##adi ##llo , collar ##ed pe ##cca ##ry , and red brock ##et deer ) are thought to be quite low ( although scientific ##ally conducted population studies are only just recently being conducted as of 2013 ) . it appears that the red brock ##et deer population has been ex ##ti ##rp ##ated on tobago as a result of over - hunting . various heron ##s , ducks , dove ##s , the green i ##gua ##na , the gold te ##gu , the spectacle ##d cai ##man and the common op ##oss ##um are also commonly hunted and po ##ache ##d . there is also some po ##achi ##ng of ' fully protected species ' , including red howl ##er monkeys and cap ##uchi ##n monkeys , southern tam ##and ##ua ##s , brazilian por ##cup ##ines , yellow - footed tor ##to ##ises , [SEP]
Split 1
[CLS] how often is hunting occurring in delaware each year ? [SEP] october to february inclusive ) . as such hunting pressure from legal hunters is very high . added to that , there is a thriving and very lucrative black market for po ##ache ##d wild game ( sold and enthusiastically purchased as expensive luxury del ##ica ##cies ) and the numbers of commercial po ##ache ##rs in operation is unknown but presumed to be fairly high . as a result , the populations of the five major mammalian game species ( red - rum ##ped ago ##uti , lowland pac ##a , nine - banded arm ##adi ##llo , collar ##ed pe ##cca ##ry , and red brock ##et deer ) are thought to be quite low ( although scientific ##ally conducted population studies are only just recently being conducted as of 2013 ) . it appears that the red brock ##et deer population has been ex ##ti ##rp ##ated on tobago as a result of over - hunting . various heron ##s , ducks , dove ##s , the green i ##gua ##na , the gold te ##gu , the spectacle ##d cai ##man and the common op ##oss ##um are also commonly hunted and po ##ache ##d . there is also some po ##achi ##ng of ' fully protected species ' , including red howl ##er monkeys and cap ##uchi ##n monkeys , southern tam ##and ##ua ##s , brazilian por ##cup ##ines , yellow - footed tor ##to ##ises , trinidad pip ##ing gu ##ans and even one of the national birds , the scarlet ib ##is . legal hunters pay very small fees to obtain hunting licence ##s and undergo no official basic conservation biology or hunting - ethics training . there is presumed to be relatively very little subsistence hunting in the country ( with most hunting for either sport or commercial profit ) . the local wildlife management authority is under - staffed and under - funded , and as such very little in the way of enforcement is done to uphold existing wildlife management laws , with hunting occurring both in and out of season , and even in wildlife san ##ct ##uaries . there is some indication that the government is beginning to [SEP]
Split 2
[CLS] how often is hunting occurring in delaware each year ? [SEP] being conducted as of 2013 ) . it appears that the red brock ##et deer population has been ex ##ti ##rp ##ated on tobago as a result of over - hunting . various heron ##s , ducks , dove ##s , the green i ##gua ##na , the gold te ##gu , the spectacle ##d cai ##man and the common op ##oss ##um are also commonly hunted and po ##ache ##d . there is also some po ##achi ##ng of ' fully protected species ' , including red howl ##er monkeys and cap ##uchi ##n monkeys , southern tam ##and ##ua ##s , brazilian por ##cup ##ines , yellow - footed tor ##to ##ises , trinidad pip ##ing gu ##ans and even one of the national birds , the scarlet ib ##is . legal hunters pay very small fees to obtain hunting licence ##s and undergo no official basic conservation biology or hunting - ethics training . there is presumed to be relatively very little subsistence hunting in the country ( with most hunting for either sport or commercial profit ) . the local wildlife management authority is under - staffed and under - funded , and as such very little in the way of enforcement is done to uphold existing wildlife management laws , with hunting occurring both in and out of season , and even in wildlife san ##ct ##uaries . there is some indication that the government is beginning to take the issue of wildlife management more seriously , with well drafted legislation being brought before parliament in 2015 . it remains to be seen if the drafted legislation will be fully adopted and financially supported by the current and future governments , and if the general populace will move towards a greater awareness of the importance of wildlife conservation and change the culture of want ##on consumption to one of sustainable management . [SEP]
If my questions were "when did Kaggle make the announcement" and "how
many registered users" I can use chunk 1 and chunk 3 and not use chunk
2 at all in the model. Not quiet sure if I should still use chunk 2 to
train the model
是的，你还应该使用块 2 来训练你的模型，因为当你尝试预测相同的序列时，你希望你的模型预测 0:0 作为块 2 的答案跨度（即你可以很容易地 select 包含答案的块）。
我已阅读 post，其中解释了滑动 window 的工作原理，但我找不到有关其实际实现方式的任何信息。
据我了解，如果输入太长，可以使用滑动window来处理文本。
如有错误请指正。 假设我有一条短信”2017年6月Kaggle宣布注册用户突破100万”.
给定一些 stride 和 max_len，可以将输入拆分成具有重叠词的块（不考虑填充）。
In June 2017 Kaggle announced that # chunk 1
announced that it passed 1 million # chunk 2
1 million registered users # chunk 3
如果我的问题是“Kaggle 何时发布公告” 和“如何发布公告”许多注册用户我可以使用chunk 1和chunk 3并且不使用chunk 2 在模型中完全 。不确定我是否仍应使用 chunk 2 来训练模型
因此输入将是：
[CLS]when did Kaggle make the announcement[SEP]In June 2017 Kaggle announced that[SEP]
和
[CLS]how many registered users[SEP]1 million registered users[SEP]
然后，如果我有一个没有答案的问题，我是否将它与所有块一起输入到模型中，并将开始和结束索引指示为 -1？例如“猪会飞吗？”
[CLS]can pigs fly[SEP]In June 2017 Kaggle announced that[SEP]
[CLS]can pigs fly[SEP]announced that it passed 1 million[SEP]
[CLS]can pigs fly[SEP]1 million registered users[SEP]
按照评论中的建议，我尝试运行 squad_convert_example_to_features (source code) 来调查我上面的问题，但似乎没有用，也没有是任何文档。 huggingface 中的 run_squad.py 似乎使用 squad_convert_example_to_features 和 example 中的 s。
from transformers.data.processors.squad import SquadResult, SquadV1Processor, SquadV2Processor, squad_convert_example_to_features
from transformers import AutoTokenizer, AutoConfig, squad_convert_examples_to_features
FILE_DIR = "."
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
processor = SquadV2Processor()
examples = processor.get_train_examples(FILE_DIR)
features = squad_convert_example_to_features(
example=examples[0],
max_seq_length=384,
doc_stride=128,
max_query_length=64,
is_training=True,
)
我收到错误。
100%|██████████| 1/1 [00:00<00:00, 159.95it/s]
Traceback (most recent call last):
File "<input>", line 25, in <module>
sub_tokens = tokenizer.tokenize(token)
NameError: name 'tokenizer' is not defined
错误表明没有tokenizers，但不允许我们传递tokenizer。虽然如果我在调试模式下的函数中添加标记器它确实有效。那么我该如何使用 squad_convert_example_to_features 函数呢？
我觉得你选的例子有问题。 squad_convert_examples_to_features and squad_convert_example_to_features have a sliding window approach implemented because squad_convert_examples_to_features is just a parallelization wrapper for squad_convert_example_to_features. But let's look at the single example function. First of all you need to call squad_convert_example_to_features_init 使标记生成器全局化（这在 squad_convert_examples_to_features 中自动为您完成）：
from transformers.data.processors.squad import SquadResult, SquadV1Processor, SquadV2Processor, squad_convert_examples_to_features, squad_convert_example_to_features_init
from transformers import AutoTokenizer, AutoConfig, squad_convert_examples_to_features
FILE_DIR = "."
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
squad_convert_example_to_features_init(tokenizer)
processor = SquadV2Processor()
examples = processor.get_train_examples(FILE_DIR)
features = squad_convert_example_to_features(
example=examples[0],
max_seq_length=384,
doc_stride=128,
max_query_length=64,
is_training=True,
)
print(len(features))
输出：
1
你可能会说这个函数没有使用滑动 window 方法，但这是错误的，因为你的示例不需要拆分:
print(len(examples[0].question_text.split()) + len(examples[0].doc_tokens))
输出：
115
小于您设置为 384 的 max_seq_length。现在让我们尝试另一个：
print(len(examples[129603].question_text.split()) + len(examples[129603].doc_tokens))
features = squad_convert_example_to_features(
example=examples[129603],
max_seq_length=384,
doc_stride=128,
max_query_length=64,
is_training=True,
)
print(len(features))
输出：
454
3
您现在可以将其与原始样本进行比较：
print('[CLS]' + examples[129603].question_text + '[SEP]' + ' '.join(examples[129603].doc_tokens) + '[SEP]')
for idx, f in enumerate(features):
print('Split {}'.format(idx))
print(' '.join(f.tokens))
输出：
[CLS]How often is hunting occurring in Delaware each year?[SEP]There is a very active tradition of hunting of small to medium-sized wild game in Trinidad and Tobago. Hunting is carried out with firearms, and aided by the use of hounds, with the illegal use of trap guns, trap cages and snare nets. With approximately 12,000 sport hunters applying for hunting licences in recent years (in a very small country of about the size of the state of Delaware at about 5128 square kilometers and 1.3 million inhabitants), there is some concern that the practice might not be sustainable. In addition there are at present no bag limits and the open season is comparatively very long (5 months - October to February inclusive). As such hunting pressure from legal hunters is very high. Added to that, there is a thriving and very lucrative black market for poached wild game (sold and enthusiastically purchased as expensive luxury delicacies) and the numbers of commercial poachers in operation is unknown but presumed to be fairly high. As a result, the populations of the five major mammalian game species (red-rumped agouti, lowland paca, nine-banded armadillo, collared peccary, and red brocket deer) are thought to be quite low (although scientifically conducted population studies are only just recently being conducted as of 2013). It appears that the red brocket deer population has been extirpated on Tobago as a result of over-hunting. Various herons, ducks, doves, the green iguana, the gold tegu, the spectacled caiman and the common opossum are also commonly hunted and poached. There is also some poaching of 'fully protected species', including red howler monkeys and capuchin monkeys, southern tamanduas, Brazilian porcupines, yellow-footed tortoises, Trinidad piping guans and even one of the national birds, the scarlet ibis. Legal hunters pay very small fees to obtain hunting licences and undergo no official basic conservation biology or hunting-ethics training. There is presumed to be relatively very little subsistence hunting in the country (with most hunting for either sport or commercial profit). The local wildlife management authority is under-staffed and under-funded, and as such very little in the way of enforcement is done to uphold existing wildlife management laws, with hunting occurring both in and out of season, and even in wildlife sanctuaries. There is some indication that the government is beginning to take the issue of wildlife management more seriously, with well drafted legislation being brought before Parliament in 2015. It remains to be seen if the drafted legislation will be fully adopted and financially supported by the current and future governments, and if the general populace will move towards a greater awareness of the importance of wildlife conservation and change the culture of wanton consumption to one of sustainable management.[SEP]
Split 0
[CLS] how often is hunting occurring in delaware each year ? [SEP] there is a very active tradition of hunting of small to medium - sized wild game in trinidad and tobago . hunting is carried out with firearms , and aided by the use of hounds , with the illegal use of trap guns , trap cages and s ##nare nets . with approximately 12 , 000 sport hunters applying for hunting licence ##s in recent years ( in a very small country of about the size of the state of delaware at about 512 ##8 square kilometers and 1 . 3 million inhabitants ) , there is some concern that the practice might not be sustainable . in addition there are at present no bag limits and the open season is comparatively very long ( 5 months - october to february inclusive ) . as such hunting pressure from legal hunters is very high . added to that , there is a thriving and very lucrative black market for po ##ache ##d wild game ( sold and enthusiastically purchased as expensive luxury del ##ica ##cies ) and the numbers of commercial po ##ache ##rs in operation is unknown but presumed to be fairly high . as a result , the populations of the five major mammalian game species ( red - rum ##ped ago ##uti , lowland pac ##a , nine - banded arm ##adi ##llo , collar ##ed pe ##cca ##ry , and red brock ##et deer ) are thought to be quite low ( although scientific ##ally conducted population studies are only just recently being conducted as of 2013 ) . it appears that the red brock ##et deer population has been ex ##ti ##rp ##ated on tobago as a result of over - hunting . various heron ##s , ducks , dove ##s , the green i ##gua ##na , the gold te ##gu , the spectacle ##d cai ##man and the common op ##oss ##um are also commonly hunted and po ##ache ##d . there is also some po ##achi ##ng of ' fully protected species ' , including red howl ##er monkeys and cap ##uchi ##n monkeys , southern tam ##and ##ua ##s , brazilian por ##cup ##ines , yellow - footed tor ##to ##ises , [SEP]
Split 1
[CLS] how often is hunting occurring in delaware each year ? [SEP] october to february inclusive ) . as such hunting pressure from legal hunters is very high . added to that , there is a thriving and very lucrative black market for po ##ache ##d wild game ( sold and enthusiastically purchased as expensive luxury del ##ica ##cies ) and the numbers of commercial po ##ache ##rs in operation is unknown but presumed to be fairly high . as a result , the populations of the five major mammalian game species ( red - rum ##ped ago ##uti , lowland pac ##a , nine - banded arm ##adi ##llo , collar ##ed pe ##cca ##ry , and red brock ##et deer ) are thought to be quite low ( although scientific ##ally conducted population studies are only just recently being conducted as of 2013 ) . it appears that the red brock ##et deer population has been ex ##ti ##rp ##ated on tobago as a result of over - hunting . various heron ##s , ducks , dove ##s , the green i ##gua ##na , the gold te ##gu , the spectacle ##d cai ##man and the common op ##oss ##um are also commonly hunted and po ##ache ##d . there is also some po ##achi ##ng of ' fully protected species ' , including red howl ##er monkeys and cap ##uchi ##n monkeys , southern tam ##and ##ua ##s , brazilian por ##cup ##ines , yellow - footed tor ##to ##ises , trinidad pip ##ing gu ##ans and even one of the national birds , the scarlet ib ##is . legal hunters pay very small fees to obtain hunting licence ##s and undergo no official basic conservation biology or hunting - ethics training . there is presumed to be relatively very little subsistence hunting in the country ( with most hunting for either sport or commercial profit ) . the local wildlife management authority is under - staffed and under - funded , and as such very little in the way of enforcement is done to uphold existing wildlife management laws , with hunting occurring both in and out of season , and even in wildlife san ##ct ##uaries . there is some indication that the government is beginning to [SEP]
Split 2
[CLS] how often is hunting occurring in delaware each year ? [SEP] being conducted as of 2013 ) . it appears that the red brock ##et deer population has been ex ##ti ##rp ##ated on tobago as a result of over - hunting . various heron ##s , ducks , dove ##s , the green i ##gua ##na , the gold te ##gu , the spectacle ##d cai ##man and the common op ##oss ##um are also commonly hunted and po ##ache ##d . there is also some po ##achi ##ng of ' fully protected species ' , including red howl ##er monkeys and cap ##uchi ##n monkeys , southern tam ##and ##ua ##s , brazilian por ##cup ##ines , yellow - footed tor ##to ##ises , trinidad pip ##ing gu ##ans and even one of the national birds , the scarlet ib ##is . legal hunters pay very small fees to obtain hunting licence ##s and undergo no official basic conservation biology or hunting - ethics training . there is presumed to be relatively very little subsistence hunting in the country ( with most hunting for either sport or commercial profit ) . the local wildlife management authority is under - staffed and under - funded , and as such very little in the way of enforcement is done to uphold existing wildlife management laws , with hunting occurring both in and out of season , and even in wildlife san ##ct ##uaries . there is some indication that the government is beginning to take the issue of wildlife management more seriously , with well drafted legislation being brought before parliament in 2015 . it remains to be seen if the drafted legislation will be fully adopted and financially supported by the current and future governments , and if the general populace will move towards a greater awareness of the importance of wildlife conservation and change the culture of want ##on consumption to one of sustainable management . [SEP]
If my questions were "when did Kaggle make the announcement" and "how many registered users" I can use chunk 1 and chunk 3 and not use chunk 2 at all in the model. Not quiet sure if I should still use chunk 2 to train the model
是的，你还应该使用块 2 来训练你的模型，因为当你尝试预测相同的序列时，你希望你的模型预测 0:0 作为块 2 的答案跨度（即你可以很容易地 select 包含答案的块）。