语音转文本 - 将说话者标签映射到 JSON 响应中的相应转录本
Speech to Text - Map speaker label to corresponding transcript in JSON response
每隔一段时间就会出现一段 JSON 数据,这些数据提出了一项挑战,可能需要数小时才能从中提取所需的信息。我从 Speech To Text API 引擎生成了以下 JSON 响应。
它显示对话中每个说话者 speaker 0
和 speaker 2
的转录、每个单词的发音以及时间戳和说话者标签。
{
"results": [
{
"alternatives": [
{
"timestamps": [
[
"the",
6.18,
6.63
],
[
"weather",
6.63,
6.95
],
[
"is",
6.95,
7.53
],
[
"sunny",
7.73,
8.11
],
[
"it's",
8.21,
8.5
],
[
"time",
8.5,
8.66
],
[
"to",
8.66,
8.81
],
[
"sip",
8.81,
8.99
],
[
"in",
8.99,
9.02
],
[
"some",
9.02,
9.25
],
[
"cold",
9.25,
9.32
],
[
"beer",
9.32,
9.68
]
],
"confidence": 0.812,
"transcript": "the weather is sunny it's time to sip in some cold beer "
}
],
"final": "True"
},
{
"alternatives": [
{
"timestamps": [
[
"sure",
10.52,
10.88
],
[
"that",
10.92,
11.19
],
[
"sounds",
11.68,
11.82
],
[
"like",
11.82,
12.11
],
[
"a",
12.32,
12.96
],
[
"plan",
12.99,
13.8
]
],
"confidence": 0.829,
"transcript": "sure that sounds like a plan"
}
],
"final": "True"
}
],
"result_index":0,
"speaker_labels": [
{
"from": 6.18,
"to": 6.63,
"speaker": 0,
"confidence": 0.475,
"final": "False"
},
{
"from": 6.63,
"to": 6.95,
"speaker": 0,
"confidence": 0.475,
"final": "False"
},
{
"from": 6.95,
"to": 7.53,
"speaker": 0,
"confidence": 0.475,
"final": "False"
},
{
"from": 7.73,
"to": 8.11,
"speaker": 0,
"confidence": 0.499,
"final": "False"
},
{
"from": 8.21,
"to": 8.5,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 8.5,
"to": 8.66,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 8.66,
"to": 8.81,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 8.81,
"to": 8.99,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 8.99,
"to": 9.02,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 9.02,
"to": 9.25,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 9.25,
"to": 9.32,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 9.32,
"to": 9.68,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 10.52,
"to": 10.88,
"speaker": 2,
"confidence": 0.441,
"final": "False"
},
{
"from": 10.92,
"to": 11.19,
"speaker": 2,
"confidence": 0.364,
"final": "False"
},
{
"from": 11.68,
"to": 11.82,
"speaker": 2,
"confidence": 0.372,
"final": "False"
},
{
"from": 11.82,
"to": 12.11,
"speaker": 2,
"confidence": 0.372,
"final": "False"
},
{
"from": 12.32,
"to": 12.96,
"speaker": 2,
"confidence": 0.383,
"final": "False"
},
{
"from": 12.99,
"to": 13.8,
"speaker": 2,
"confidence": 0.428,
"final": "False"
}
]
}
请原谅缩进问题(如果有的话),但 JSON 是有效的,我一直在尝试将每个文字记录与其对应的演讲者标签对应起来。
我想要下面这样的东西。 上面的 JSON 大约有 20,000 行,根据时间戳和单词发音提取说话者标签并将其放在一起是一场噩梦连同 transcript
。
[
{
"transcript": "the weather is sunny it's time to sip in some cold beer ",
"speaker" : 0
},
{
"transcript": "sure that sounds like a plan",
"speaker" : 2
}
]
到目前为止我尝试了什么:
JSON 数据存储在名为 example.json
的文件中。我已经能够将每个单词及其相应的时间戳和说话人标签放入元组列表中(见下面的输出):
import json
# with open('C:\Users\%USERPROFILE%\Desktop\example.json', 'r') as f:
# data = json.load(f)
l1 = []
l2 = []
l3 = []
for i in data['results']:
for j in i['alternatives'][0]['timestamps']:
l1.append(j)
for m in data['speaker_labels']:
l2.append(m)
for q in l1:
for n in l2:
if q[1]==n['from']:
l3.append((q[0],n['speaker'], q[1], q[2]))
print(l3)
这给出了输出:
[('the', 0, 6.18, 6.63),
('weather', 0, 6.63, 6.95),
('is', 0, 6.95, 7.53),
('sunny', 0, 7.73, 8.11),
("it's", 0, 8.21, 8.5),
('time', 0, 8.5, 8.66),
('to', 0, 8.66, 8.81),
('sip', 0, 8.81, 8.99),
('in', 0, 8.99, 9.02),
('some', 0, 9.02, 9.25),
('cold', 0, 9.25, 9.32),
('beer', 0, 9.32, 9.68),
('sure', 2, 10.52, 10.88),
('that', 2, 10.92, 11.19),
('sounds', 2, 11.68, 11.82),
('like', 2, 11.82, 12.11),
('a', 2, 12.32, 12.96),
('plan', 2, 12.99, 13.8)]
但现在我不确定如何根据时间戳比较将单词关联在一起,"bucket" 每组单词再次形成带有说话者标签的文字记录。
我还成功地获得了列表中的文字记录,但现在如何从上面的列表中提取每个文字记录的说话人标签。不幸的是,说话人标签 speaker 0
和 speaker 2
是针对每个单词的,我希望它们本来是针对每个 transcript
的。
for i in data['results']:
l4.append(i['alternatives'][0]['transcript'])
这给出了输出:
["the weather is sunny it's time to sip in some cold beer ",'sure that sounds like a plan']
我已尽力解释问题,但我愿意接受任何反馈,并会在必要时进行更改。另外,我很确定有更好的方法来解决这个问题,而不是列出几个列表,非常感谢任何帮助。
对于更大的数据集,请参考pastebin。我希望这个数据集可以有助于性能基准测试。我可以在可用时或需要时提供更大的数据集。
由于我处理的是大型 JSON 数据,性能是一个重要因素,同样,在重叠转录中准确实现说话人隔离是另一个要求。
使用 pandas,这是我刚才处理的方法。
假设数据存储在名为 data
的字典中
import pandas as pd
labels = pd.DataFrame.from_records(data['speaker_labels'])
transcript_tstamps = pd.DataFrame.from_records(
[t for r in data['results']
for a in r['alternatives']
for t in a['timestamps']],
columns=['word', 'from', 'to']
)
# this list comprehension more-efficiently de-nests the dictionary into
# records that can be used to create a DataFrame
df = labels.merge(transcript_tstamps)
# produces a dataframe of speakers to words based on timestamps from & to
# since I knew I wanted to merge on the from & to columns,
# I named the columns thus when I created the transcript_tstamps data frame
# like this:
confidence final from speaker to word
0 0.475 False 6.18 0 6.63 the
1 0.475 False 6.63 0 6.95 weather
2 0.475 False 6.95 0 7.53 is
3 0.499 False 7.73 0 8.11 sunny
4 0.472 False 8.21 0 8.50 it's
5 0.472 False 8.50 0 8.66 time
6 0.472 False 8.66 0 8.81 to
7 0.472 False 8.81 0 8.99 sip
8 0.472 False 8.99 0 9.02 in
9 0.472 False 9.02 0 9.25 some
10 0.472 False 9.25 0 9.32 cold
11 0.472 False 9.32 0 9.68 beer
12 0.441 False 10.52 2 10.88 sure
13 0.364 False 10.92 2 11.19 that
14 0.372 False 11.68 2 11.82 sounds
15 0.372 False 11.82 2 12.11 like
16 0.383 False 12.32 2 12.96 a
17 0.428 False 12.99 2 13.80 plan
speaker & word数据join后,需要将同一speaker的连续词组合在一起,导出当前speaker。例如,如果扬声器阵列看起来像 [2,2,2,2,0,0,0,2,2,2,0,0,0,0],我们需要将前四个 2
在一起,然后是下三个0
,然后是三个2
,然后是剩下的0
.
按 ['from', 'to']
对数据进行排序,然后为此设置一个名为 current_speaker
的虚拟变量,如下所示:
df = df.sort_values(['from', 'to'])
df['current_speaker'] = (df.speaker.shift() != df.speaker).cumsum()
从这里开始,按 current_speaker
分组,将单词聚合成一个句子并转换为 json。有一些额外的重命名来修复输出 json keys
transcripts = df.groupby('current_speaker').agg({
'word': lambda x: ' '.join(x),
'speaker': min
}).rename(columns={'word': 'transcript'})
transcripts[['speaker', 'transcript']].to_json(orient='records')
# produces the following output (indentation added by me for legibility):
'[{"speaker":0,
"transcript":"the weather is sunny it\'s time to sip in some cold beer"},
{"speaker":2,
"transcript":"sure that sounds like a plan"}]'
要在转录开始/结束时添加其他数据,您可以将 from/to 的 min/max 添加到 groupby
transcripts = df.groupby('current_speaker').agg({
'word': lambda x: ' '.join(x),
'speaker': min,
'from': min,
'to': max
}).rename(columns={'word': 'transcript'})
此外,(尽管这不适用于此示例数据集)您或许应该为每个时间片选择具有最高置信度的备选方案。
我是通过根据时间戳将单词放入字典,然后将它们与说话者匹配来实现的:
times = {}
for r in data['results']:
for word in r['alternatives'][0]['timestamps']:
times[(word[1], word[2])] = word[0]
transcripts = {}
for r in data['speaker_labels']:
speaker = r['speaker']
if speaker in transcripts:
transcripts[speaker].append(times[(r['from'], r['to'])])
else:
transcripts[speaker] = [times[(r['from'], r['to'])]]
print([{'speaker': k, 'transcript': ' '.join(transcripts[k])} for k in transcripts])
它在 ~12.34 秒内在提供的示例上运行了 1,000,000 次,所以希望它的速度足以满足您的需求。
这是我尝试使用 JS 的方法
使用 python
以类似的方式查看这是否对您有用
var resultTimestampLen = 0;
arrLen = JSON.parse(sTot_resuts.results.length);
for(var i = 0; i<arrLen; i++){
speakerLablefrom = sTot_resuts.speaker_labels[resultTimestampLen].from;
speakerLabelto = sTot_resuts.speaker_labels[resultTimestampLen].to;
speakerId = sTot_resuts.speaker_labels[resultTimestampLen].speaker;
var findSpeaker = new Array();
findSpeaker = sTot_resuts.results[i].alternatives[0].timestamps[0];
var timeStampFrom = findSpeaker[1];
var timeStampto = findSpeaker[2];
if(timeStampFrom === speakerLablefrom && timeStampto === speakerLabelto){
console.log('Speaker '+sTot_resuts.speaker_labels[resultTimestampLen].speaker + ' ' + sTot_resuts.results[i].alternatives[0].transcript);
var resultsTimestamp = new Array();
resultsTimestamp = sTot_resuts.results[i].alternatives[0].timestamps.length;
resultTimestampLen = resultsTimestamp+resultTimestampLen;
}else{
console.log('resultTimestampLen '+resultTimestampLen + 'speakerLablefrom '+speakerLablefrom + 'speakerLabelto '+speakerLabelto + 'timeStampFrom '+timeStampFrom + 'timeStampto '+timeStampto);
}
}
每隔一段时间就会出现一段 JSON 数据,这些数据提出了一项挑战,可能需要数小时才能从中提取所需的信息。我从 Speech To Text API 引擎生成了以下 JSON 响应。
它显示对话中每个说话者 speaker 0
和 speaker 2
的转录、每个单词的发音以及时间戳和说话者标签。
{
"results": [
{
"alternatives": [
{
"timestamps": [
[
"the",
6.18,
6.63
],
[
"weather",
6.63,
6.95
],
[
"is",
6.95,
7.53
],
[
"sunny",
7.73,
8.11
],
[
"it's",
8.21,
8.5
],
[
"time",
8.5,
8.66
],
[
"to",
8.66,
8.81
],
[
"sip",
8.81,
8.99
],
[
"in",
8.99,
9.02
],
[
"some",
9.02,
9.25
],
[
"cold",
9.25,
9.32
],
[
"beer",
9.32,
9.68
]
],
"confidence": 0.812,
"transcript": "the weather is sunny it's time to sip in some cold beer "
}
],
"final": "True"
},
{
"alternatives": [
{
"timestamps": [
[
"sure",
10.52,
10.88
],
[
"that",
10.92,
11.19
],
[
"sounds",
11.68,
11.82
],
[
"like",
11.82,
12.11
],
[
"a",
12.32,
12.96
],
[
"plan",
12.99,
13.8
]
],
"confidence": 0.829,
"transcript": "sure that sounds like a plan"
}
],
"final": "True"
}
],
"result_index":0,
"speaker_labels": [
{
"from": 6.18,
"to": 6.63,
"speaker": 0,
"confidence": 0.475,
"final": "False"
},
{
"from": 6.63,
"to": 6.95,
"speaker": 0,
"confidence": 0.475,
"final": "False"
},
{
"from": 6.95,
"to": 7.53,
"speaker": 0,
"confidence": 0.475,
"final": "False"
},
{
"from": 7.73,
"to": 8.11,
"speaker": 0,
"confidence": 0.499,
"final": "False"
},
{
"from": 8.21,
"to": 8.5,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 8.5,
"to": 8.66,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 8.66,
"to": 8.81,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 8.81,
"to": 8.99,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 8.99,
"to": 9.02,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 9.02,
"to": 9.25,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 9.25,
"to": 9.32,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 9.32,
"to": 9.68,
"speaker": 0,
"confidence": 0.472,
"final": "False"
},
{
"from": 10.52,
"to": 10.88,
"speaker": 2,
"confidence": 0.441,
"final": "False"
},
{
"from": 10.92,
"to": 11.19,
"speaker": 2,
"confidence": 0.364,
"final": "False"
},
{
"from": 11.68,
"to": 11.82,
"speaker": 2,
"confidence": 0.372,
"final": "False"
},
{
"from": 11.82,
"to": 12.11,
"speaker": 2,
"confidence": 0.372,
"final": "False"
},
{
"from": 12.32,
"to": 12.96,
"speaker": 2,
"confidence": 0.383,
"final": "False"
},
{
"from": 12.99,
"to": 13.8,
"speaker": 2,
"confidence": 0.428,
"final": "False"
}
]
}
请原谅缩进问题(如果有的话),但 JSON 是有效的,我一直在尝试将每个文字记录与其对应的演讲者标签对应起来。
我想要下面这样的东西。 上面的 JSON 大约有 20,000 行,根据时间戳和单词发音提取说话者标签并将其放在一起是一场噩梦连同 transcript
。
[
{
"transcript": "the weather is sunny it's time to sip in some cold beer ",
"speaker" : 0
},
{
"transcript": "sure that sounds like a plan",
"speaker" : 2
}
]
到目前为止我尝试了什么:
JSON 数据存储在名为 example.json
的文件中。我已经能够将每个单词及其相应的时间戳和说话人标签放入元组列表中(见下面的输出):
import json
# with open('C:\Users\%USERPROFILE%\Desktop\example.json', 'r') as f:
# data = json.load(f)
l1 = []
l2 = []
l3 = []
for i in data['results']:
for j in i['alternatives'][0]['timestamps']:
l1.append(j)
for m in data['speaker_labels']:
l2.append(m)
for q in l1:
for n in l2:
if q[1]==n['from']:
l3.append((q[0],n['speaker'], q[1], q[2]))
print(l3)
这给出了输出:
[('the', 0, 6.18, 6.63),
('weather', 0, 6.63, 6.95),
('is', 0, 6.95, 7.53),
('sunny', 0, 7.73, 8.11),
("it's", 0, 8.21, 8.5),
('time', 0, 8.5, 8.66),
('to', 0, 8.66, 8.81),
('sip', 0, 8.81, 8.99),
('in', 0, 8.99, 9.02),
('some', 0, 9.02, 9.25),
('cold', 0, 9.25, 9.32),
('beer', 0, 9.32, 9.68),
('sure', 2, 10.52, 10.88),
('that', 2, 10.92, 11.19),
('sounds', 2, 11.68, 11.82),
('like', 2, 11.82, 12.11),
('a', 2, 12.32, 12.96),
('plan', 2, 12.99, 13.8)]
但现在我不确定如何根据时间戳比较将单词关联在一起,"bucket" 每组单词再次形成带有说话者标签的文字记录。
我还成功地获得了列表中的文字记录,但现在如何从上面的列表中提取每个文字记录的说话人标签。不幸的是,说话人标签 speaker 0
和 speaker 2
是针对每个单词的,我希望它们本来是针对每个 transcript
的。
for i in data['results']:
l4.append(i['alternatives'][0]['transcript'])
这给出了输出:
["the weather is sunny it's time to sip in some cold beer ",'sure that sounds like a plan']
我已尽力解释问题,但我愿意接受任何反馈,并会在必要时进行更改。另外,我很确定有更好的方法来解决这个问题,而不是列出几个列表,非常感谢任何帮助。
对于更大的数据集,请参考pastebin。我希望这个数据集可以有助于性能基准测试。我可以在可用时或需要时提供更大的数据集。
由于我处理的是大型 JSON 数据,性能是一个重要因素,同样,在重叠转录中准确实现说话人隔离是另一个要求。
使用 pandas,这是我刚才处理的方法。
假设数据存储在名为 data
import pandas as pd
labels = pd.DataFrame.from_records(data['speaker_labels'])
transcript_tstamps = pd.DataFrame.from_records(
[t for r in data['results']
for a in r['alternatives']
for t in a['timestamps']],
columns=['word', 'from', 'to']
)
# this list comprehension more-efficiently de-nests the dictionary into
# records that can be used to create a DataFrame
df = labels.merge(transcript_tstamps)
# produces a dataframe of speakers to words based on timestamps from & to
# since I knew I wanted to merge on the from & to columns,
# I named the columns thus when I created the transcript_tstamps data frame
# like this:
confidence final from speaker to word
0 0.475 False 6.18 0 6.63 the
1 0.475 False 6.63 0 6.95 weather
2 0.475 False 6.95 0 7.53 is
3 0.499 False 7.73 0 8.11 sunny
4 0.472 False 8.21 0 8.50 it's
5 0.472 False 8.50 0 8.66 time
6 0.472 False 8.66 0 8.81 to
7 0.472 False 8.81 0 8.99 sip
8 0.472 False 8.99 0 9.02 in
9 0.472 False 9.02 0 9.25 some
10 0.472 False 9.25 0 9.32 cold
11 0.472 False 9.32 0 9.68 beer
12 0.441 False 10.52 2 10.88 sure
13 0.364 False 10.92 2 11.19 that
14 0.372 False 11.68 2 11.82 sounds
15 0.372 False 11.82 2 12.11 like
16 0.383 False 12.32 2 12.96 a
17 0.428 False 12.99 2 13.80 plan
speaker & word数据join后,需要将同一speaker的连续词组合在一起,导出当前speaker。例如,如果扬声器阵列看起来像 [2,2,2,2,0,0,0,2,2,2,0,0,0,0],我们需要将前四个 2
在一起,然后是下三个0
,然后是三个2
,然后是剩下的0
.
按 ['from', 'to']
对数据进行排序,然后为此设置一个名为 current_speaker
的虚拟变量,如下所示:
df = df.sort_values(['from', 'to'])
df['current_speaker'] = (df.speaker.shift() != df.speaker).cumsum()
从这里开始,按 current_speaker
分组,将单词聚合成一个句子并转换为 json。有一些额外的重命名来修复输出 json keys
transcripts = df.groupby('current_speaker').agg({
'word': lambda x: ' '.join(x),
'speaker': min
}).rename(columns={'word': 'transcript'})
transcripts[['speaker', 'transcript']].to_json(orient='records')
# produces the following output (indentation added by me for legibility):
'[{"speaker":0,
"transcript":"the weather is sunny it\'s time to sip in some cold beer"},
{"speaker":2,
"transcript":"sure that sounds like a plan"}]'
要在转录开始/结束时添加其他数据,您可以将 from/to 的 min/max 添加到 groupby
transcripts = df.groupby('current_speaker').agg({
'word': lambda x: ' '.join(x),
'speaker': min,
'from': min,
'to': max
}).rename(columns={'word': 'transcript'})
此外,(尽管这不适用于此示例数据集)您或许应该为每个时间片选择具有最高置信度的备选方案。
我是通过根据时间戳将单词放入字典,然后将它们与说话者匹配来实现的:
times = {}
for r in data['results']:
for word in r['alternatives'][0]['timestamps']:
times[(word[1], word[2])] = word[0]
transcripts = {}
for r in data['speaker_labels']:
speaker = r['speaker']
if speaker in transcripts:
transcripts[speaker].append(times[(r['from'], r['to'])])
else:
transcripts[speaker] = [times[(r['from'], r['to'])]]
print([{'speaker': k, 'transcript': ' '.join(transcripts[k])} for k in transcripts])
它在 ~12.34 秒内在提供的示例上运行了 1,000,000 次,所以希望它的速度足以满足您的需求。
这是我尝试使用 JS 的方法
使用 python
var resultTimestampLen = 0;
arrLen = JSON.parse(sTot_resuts.results.length);
for(var i = 0; i<arrLen; i++){
speakerLablefrom = sTot_resuts.speaker_labels[resultTimestampLen].from;
speakerLabelto = sTot_resuts.speaker_labels[resultTimestampLen].to;
speakerId = sTot_resuts.speaker_labels[resultTimestampLen].speaker;
var findSpeaker = new Array();
findSpeaker = sTot_resuts.results[i].alternatives[0].timestamps[0];
var timeStampFrom = findSpeaker[1];
var timeStampto = findSpeaker[2];
if(timeStampFrom === speakerLablefrom && timeStampto === speakerLabelto){
console.log('Speaker '+sTot_resuts.speaker_labels[resultTimestampLen].speaker + ' ' + sTot_resuts.results[i].alternatives[0].transcript);
var resultsTimestamp = new Array();
resultsTimestamp = sTot_resuts.results[i].alternatives[0].timestamps.length;
resultTimestampLen = resultsTimestamp+resultTimestampLen;
}else{
console.log('resultTimestampLen '+resultTimestampLen + 'speakerLablefrom '+speakerLablefrom + 'speakerLabelto '+speakerLabelto + 'timeStampFrom '+timeStampFrom + 'timeStampto '+timeStampto);
}
}