在 python 中应用预训练 facebook/bart-large-cnn 进行文本摘要
Applying pre trained facebook/bart-large-cnn for text summarization in python
我正在使用 huggingface 变形金刚,并且对它有了一些了解。我正在使用 facebook/bart-large-cnn 模型为我的项目执行文本摘要,目前我正在使用以下代码进行一些测试:
text = """
Justin Timberlake and Jessica Biel, welcome to parenthood.
The celebrity couple announced the arrival of their son, Silas Randall Timberlake, in
statements to People."""
from transformers import pipeline
smr_bart = pipeline(task="summarization", model="facebook/bart-large-cnn")
smbart = smr_bart(text, max_length=150)
print(smbart[0]['summary_text'])
一小段代码实际上给了我一个很好的文本总结。但我的问题是如何在我的数据框列之上应用相同的预训练模型。我的数据框如下所示:
ID Lang Text
1 EN some long text here...
2 EN some long text here...
3 EN some long text here...
..... 50K 行依此类推
现在我想将预训练模型应用于 col Text 以从中生成一个新列 df['summary'],结果数据框应如下所示:
ID Lang Text Summary
1 EN some long text here... Text summary goes here...
2 EN some long text here... Text summary goes here...
3 EN some long text here... Text summary goes here...
我怎样才能做到这一点?任何帮助将不胜感激。
你总是可以做的是利用数据框 apply 函数:
df = pd.DataFrame([('EN',text)]*10, columns=['Lang','Text'])
df['summary'] = df.apply(lambda x: smr_bart(x['Text'], max_length=150)[0]['summary_text'] , axis=1)
df.head(3)
输出:
Lang Text summary
0 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...
1 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...
2 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...
这有点低效,因为管道将被每一行调用(执行时间 2 分 16 秒)。因此我建议将 Text
列转换为列表并将其直接传递给管道(执行时间 41 秒):
df = pd.DataFrame([('EN',text)]*10, columns=['Lang','Text'])
df['summary'] = [x['summary_text'] for x in smr_bart(df['Text'].tolist(), max_length=150)]
df.head(3)
输出:
Lang Text summary
0 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...
1 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...
2 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...
我正在使用 huggingface 变形金刚,并且对它有了一些了解。我正在使用 facebook/bart-large-cnn 模型为我的项目执行文本摘要,目前我正在使用以下代码进行一些测试:
text = """
Justin Timberlake and Jessica Biel, welcome to parenthood.
The celebrity couple announced the arrival of their son, Silas Randall Timberlake, in
statements to People."""
from transformers import pipeline
smr_bart = pipeline(task="summarization", model="facebook/bart-large-cnn")
smbart = smr_bart(text, max_length=150)
print(smbart[0]['summary_text'])
一小段代码实际上给了我一个很好的文本总结。但我的问题是如何在我的数据框列之上应用相同的预训练模型。我的数据框如下所示:
ID Lang Text
1 EN some long text here...
2 EN some long text here...
3 EN some long text here...
..... 50K 行依此类推
现在我想将预训练模型应用于 col Text 以从中生成一个新列 df['summary'],结果数据框应如下所示:
ID Lang Text Summary
1 EN some long text here... Text summary goes here...
2 EN some long text here... Text summary goes here...
3 EN some long text here... Text summary goes here...
我怎样才能做到这一点?任何帮助将不胜感激。
你总是可以做的是利用数据框 apply 函数:
df = pd.DataFrame([('EN',text)]*10, columns=['Lang','Text'])
df['summary'] = df.apply(lambda x: smr_bart(x['Text'], max_length=150)[0]['summary_text'] , axis=1)
df.head(3)
输出:
Lang Text summary
0 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...
1 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...
2 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...
这有点低效,因为管道将被每一行调用(执行时间 2 分 16 秒)。因此我建议将 Text
列转换为列表并将其直接传递给管道(执行时间 41 秒):
df = pd.DataFrame([('EN',text)]*10, columns=['Lang','Text'])
df['summary'] = [x['summary_text'] for x in smr_bart(df['Text'].tolist(), max_length=150)]
df.head(3)
输出:
Lang Text summary
0 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...
1 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...
2 EN \nJustin Timberlake and Jessica Biel, welcome ... The celebrity couple announced the arrival of ...