Python dataframe 从列表中删除句子数

Question

我在数据框中有一列（相当）长的文本，对于每个文本，都有一个我想删除的句子索引列表。当我将文本拆分成句子时，Spacy 会生成句子索引。请考虑以下示例：

import pandas as pd
import spacy
nlp = spacy.load('en_core_web_sm')

data = {'text': ['I am A. I am 30 years old. I live in NY.','I am B. I am 25 years old. I live in SD.','I am C. I am 30 years old. I live in TX.'], 'todel': [[1, 2], [1], [1, 2]]}

df = pd.DataFrame(data)

def get_sentences(text):
    text_clean = nlp(text)
    sentences = text_clean.sents
    sents_list = []
    for sentence in sentences:
        sents_list.append(str(sentence))
    return sents_list

df['text'] = df['text'].apply(get_sentences)

print(df)

给出以下内容：

                                           text   todel
0  [I am A., I am 30 years old., I live in NY.]  [1, 2]
1   [I am B. I am 25 years old., I live in SD.]     [1]
2   [I am C. I am 30 years old., I live in TX.]  [1, 2]

知道我有一个非常大的数据集，每行有超过 50 个句子要删除，你如何有效地删除存储在 todel 中的句子？

我的预期输出是：

                                  text   todel
0                      [I live in NY.]  [1, 2]
1  [I am 25 years old., I live in SD.]     [1]
2                      [I live in TX.]  [1, 2]

Answer 1

试试这个：

import pandas as pd

data = {'text': ['I am A. I am 30 years old. I live in NY.','I am B. I am 25 years old. I live in SD.','I am C. I am 30 years old. I live in TX.'], 'todel': [[1, 2], [1], [1, 2]]}

df = pd.DataFrame(data)

def fun(sen, lst):
    return  ('.'.join(s for idx, s in enumerate(sen.split('.')) if idx+1 not in lst))

df['text'] = df.apply(lambda row : fun(row['text'],row['todel']), axis=1)

输出：

                                text   todel
0                      I live in NY.  [1, 2]
1   I am 25 years old. I live in SD.     [1]
2                      I live in TX.  [1, 2]

编辑基于已编辑的问题：

如果 df['text'] 个不需要拆分的句子列表，您可以试试这个：

data = {'text': [['I am A.', 'I am 30 years old.', 'I live in NY.'], 
                 ['I am B.', 'I am 25 years old.', 'I live in SD.'],
                 ['I am C.','I am 30 years old.',' I live in TX.']], 'todel': [[1, 2], [1], [1, 2]]}
df = pd.DataFrame(data)
#                                           text     todel
# 0   [I am A., I am 30 years old., I live in NY.]  [1, 2]
# 1   [I am B., I am 25 years old., I live in SD.]     [1]
# 2  [I am C., I am 30 years old.,  I live in TX.]  [1, 2]

def fun(sen, lst):
    return  [s for idx , s in enumerate(sen) if not idx+1 in lst]

df['text'] = df.apply(lambda row : fun(row['text'],row['todel']), axis=1)

print(df)

输出：

                                  text   todel
0                      [I live in NY.]  [1, 2]
1  [I am 25 years old., I live in SD.]     [1]
2                     [ I live in TX.]  [1, 2]

Answer 2

根据@user1740577的回答：

def fun(sen, lst):
    return [i for j, i in enumerate(sen) if j not in lst]

df['text'] = df.apply(lambda row : fun(row['text'],row['todel']), axis=1)

根据 Spacy 的索引产生想要的结果：

                           text   todel
0                     [I am A.]  [1, 2]
1  [I am B. I am 25 years old.]     [1]
2  [I am C. I am 30 years old.]  [1, 2]

Python dataframe 从列表中删除句子数

Python dataframe delete sentences number from list

python

list

apply

dataframe

spacy