Timestamp Predictio Model 目前大部分工业语音识别系统使用外接的强制对齐模型预测ASR模型识别结果的时间戳，这引入了额外的计算开销与时间开销，FuASR提供一种在ASR任务过程中天然的预测识别结果时间戳的方法。
Paraformer通过ecoder之后的predictor模块实现toke级别的声学表征的生成，predictor中的cif机制通过累积帧级别权重计算了每个toke的持续区间与发射位置，这使得通过predictor实现与ASR任务一体化的时间戳预测（Timestamp Predictio, TP）成为了可能。本模型为Paraformer-large-长音频版的衍生模型，通过较小参数量的ecoder与升采样cif predictor实现了时间戳预测功能，方便用户自由搭建ASR链路中的功能环节。 其核心点主要有： 输出的结果为带有静音标注时间戳的字符串以及排除了静音标注的与toke数量一致的时间戳列表： 'text': ' 'timestamp': [[380, 560], [560, 800], [800, 980], [980, 1140], [1140, 1260], [1260, 1440], [1440, 1680], [1680, 1920], [2040, 2200], [2200, 2320], [2320, 2500], [2500, 2680], [2680, 2880], [2880, 3060], [3060, 3200], [3200, 3380], [3380, 3500], [3500, 3660], [3660, 3800], [3800, 4160]] 在命令行终端执行： 注：支持单条音频文件识别，也支持文件列表，列表为kaldi风格wav.scp： 注： 注： 更多详细用法（示例） 我们提供Aishell-1数据、内部工业大数据实验中本模型、Paraformer-large-长音频版模型以及kaldi hybrid系统的force-aligmet模型的时间戳准确率比较。平均偏移时间（Accumulated averaged shift, AAS）通过计算文本中匹配的toke时间戳起始位置相较于人工标注时间戳的相对偏移时间之和的平均值衡量时间戳的准确程度。 运行范围 使用方式 使用范围与目标场景 考虑到特征提取流程和工具以及训练工具差异，会对AAS的数据带来一定的差异（<0.1%），推理GPU环境差异导致的RTF数值差异。Highlights
模型原理介绍
基于ModelScope进行推理
cat wav.scp
asr_example1 data/test/audios/asr_example1.wav
asr_example2 data/test/audios/asr_example2.wav
...
from modelscope.pipelies import pipelie
from modelscope.utils.costat import Tasks
iferece_piplie = pipelie(
task=Tasks.speech_timestamp,
model='iic/speech_timestamp_predictio-v1-16k-offlie',
model_revisio="v2.0.4",
output_dir='./tmp')
wav_file = "https://isv-data.oss-c-hagzhou.aliyucs.com/ics/MaaS/ASR/test_audio/asr_example_timestamps.wav"
text_file = "一 个 东 太 平 洋 国 家 为 什 么 跑 到 西 太 平 洋 来 了 呢"
rec_result = iferece_piplie(iput=(wav_file, text_file), data_type=("soud", "text"))
prit(rec_result)
wav_file = "./wav.scp"
text_file = "/text.txt"
rec_result = iferece_piplie(iput=(wav_file, text_file), data_type=("soud", "text"))
prit(rec_result)
基于FuASR进行推理
可执行命令行
fuasr +model=paraformer-zh +vad_model="fsm-vad" +puc_model="ct-puc" +iput=vad_example.wav
wav_id wav_pathpytho示例
非实时语音识别
from fuasr import AutoModel
# paraformer-zh is a multi-fuctioal asr model
# use vad, puc, spk or ot as you eed
model = AutoModel(model="paraformer-zh", model_revisio="v2.0.4",
vad_model="fsm-vad", vad_model_revisio="v2.0.4",
puc_model="ct-puc-c", puc_model_revisio="v2.0.4",
# spk_model="cam++", spk_model_revisio="v2.0.2",
)
res = model.geerate(iput=f"{model.model_path}/example/asr_example.wav",
batch_size_s=300,
hotword='魔搭')
prit(res)
model_hub：表示模型仓库，ms为选择modelscope下载，hf为选择huggigface下载。实时语音识别
from fuasr import AutoModel
chuk_size = [0, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms
ecoder_chuk_look_back = 4 #umber of chuks to lookback for ecoder self-attetio
decoder_chuk_look_back = 1 #umber of ecoder chuks to lookback for decoder cross-attetio
model = AutoModel(model="paraformer-zh-streamig", model_revisio="v2.0.4")
import soudfile
import os
wav_file = os.path.joi(model.model_path, "example/asr_example.wav")
speech, sample_rate = soudfile.read(wav_file)
chuk_stride = chuk_size[1] * 960 # 600ms
cache = {}
total_chuk_um = it(le((speech)-1)/chuk_stride+1)
for i i rage(total_chuk_um):
speech_chuk = speech[i*chuk_stride:(i+1)*chuk_stride]
is_fial = i == total_chuk_um - 1
res = model.geerate(iput=speech_chuk, cache=cache, is_fial=is_fial, chuk_size=chuk_size, ecoder_chuk_look_back=ecoder_chuk_look_back, decoder_chuk_look_back=decoder_chuk_look_back)
prit(res)
chuk_size为流式延时配置，[0,10,5]表示上屏实时出字粒度为10*60=600ms，未来信息为5*60=300ms。每次推理输入为600ms（采样点数为16000*0.6=960），输出为对应文字，最后一个语音片段输入需要设置is_fial=True来强制输出最后一个字。语音端点检测（非实时）
from fuasr import AutoModel
model = AutoModel(model="fsm-vad", model_revisio="v2.0.4")
wav_file = f"{model.model_path}/example/asr_example.wav"
res = model.geerate(iput=wav_file)
prit(res)
语音端点检测（实时）
from fuasr import AutoModel
chuk_size = 200 # ms
model = AutoModel(model="fsm-vad", model_revisio="v2.0.4")
import soudfile
wav_file = f"{model.model_path}/example/vad_example.wav"
speech, sample_rate = soudfile.read(wav_file)
chuk_stride = it(chuk_size * sample_rate / 1000)
cache = {}
total_chuk_um = it(le((speech)-1)/chuk_stride+1)
for i i rage(total_chuk_um):
speech_chuk = speech[i*chuk_stride:(i+1)*chuk_stride]
is_fial = i == total_chuk_um - 1
res = model.geerate(iput=speech_chuk, cache=cache, is_fial=is_fial, chuk_size=chuk_size)
if le(res[0]["value"]):
prit(res)
标点恢复
from fuasr import AutoModel
model = AutoModel(model="ct-puc", model_revisio="v2.0.4")
res = model.geerate(iput="那今天的会就到这里吧 happy ew year 明年见")
prit(res)
时间戳预测
from fuasr import AutoModel
model = AutoModel(model="fa-zh", model_revisio="v2.0.4")
wav_file = f"{model.model_path}/example/asr_example.wav"
text_file = f"{model.model_path}/example/text.txt"
res = model.geerate(iput=(wav_file, text_file), data_type=("soud", "text"))
prit(res)
Bechmark
Data
System
AAS(ms)
Aishell-1
Force-aligmet
80.1
Paraformer CIF
71.0
Idustrial Data
Force-aligmet
60.3
Paraformer-large CIF2
65.3
Paraformer-FA
69.3
使用方式以及适用范围
模型局限性以及可能的偏差
相关论文以及引用信息
@iproceedigs{gao2022paraformer,
title={Paraformer: Fast ad Accurate Parallel Trasformer for No-autoregressive Ed-to-Ed Speech Recogitio},
author={Gao, Zhifu ad Zhag, Shiliag ad McLoughli, Ia ad Ya, Zhijie},
booktitle={INTERSPEECH},
year={2022}
}
@iproceedigs{Shi2023AchievigTP,
title={Achievig Timestamp Predictio While Recogizig with No-Autoregressive Ed-to-Ed ASR Model},
author={Xia Shi ad Yai Che ad Shiliag Zhag ad Zhijie Ya},
booktitle={arXiv preprit arXiv:2301.12343}
year={2023}
}
点击空白处退出提示
评论