AveraLabs Voice PipelineFull-Duplex Speech Datafrom Real Conversations AveraLabs 语音管线从真实对话构建全双工语音数据

AveraLabs · Technical Project Page · 2026

Introduction项目介绍

AveraLabs transforms any real world audio into high-quality training data for full-duplex speech models. It discovers genuine two-speaker exchanges, verifies speaker-pure tracks, annotates timing and persona, then expands and re-performs the conversation while preserving realistic turn-taking, pauses, overlap, and paralinguistic behavior.

AveraLabs 将 podcast、电视采访和电影对白等长音频转化为全双工语音模型的高质量训练数据。管线发现真实的双人对谈,验证说话人纯净声轨,标注时序与 Persona,并在保留真实话轮、停顿、重叠和副语言行为的基础上进行扩写与重新演绎。

Pipeline处理流程

Every accepted conversation passes through the complete pipeline and produces both outputs together: high-quality separated data derived from the original recording, and regenerated expansion data grounded in the same real conversation.每段通过筛选与质检的对话都会经过完整管线,并同时产出两类数据:一类是来自原始录音的高质量分离数据,另一类是以同一真实对话为依据的重生成扩展数据。

Delivered together · High-quality separated data同步交付 · 高质量分离数据Clean, time-aligned, speaker-pure tracks extracted from the original recording.从原始录音中提取清晰、时间对齐且说话人纯净的独立声轨。
Delivered together · Regenerated expansion data同步交付 · 重生成扩展数据New, controllable dialogue grounded in the content and interaction patterns of the same real conversation.基于同一真实对话的内容与互动模式,生成全新且可控的扩展对话数据。

Separation说话人分离

Long-form audio长音频输入Podcast · interview · filmPodcast · 访谈 · 影视对白
VAD partitionVAD 分片Natural boundaries ·
≈15 min
自然边界 · 约 15 分钟
pyannote/segmentation-3.0
Diarization & filtering说话人识别与过滤Remove music and low SNR剔除音乐和低 SNRBUT-FIT/diarizen-wavlm-large-s80-md-v2
Dynamic window search动态窗口搜索Find high-quality two-speaker conversation segments that are as long as possible寻找尽可能长的高质量双人对话片段
Separation + QC retry分离 + 质检重试Reject split, merged, or leaking identities拒绝拆轨、并轨和串轨DialogueSidon

Real-scene grounded expansion & customization基于真实场景的扩写与定制

Verified dual tracks已验证双轨The two speakers are separated into two independent audio-track files两个说话人的声音被分离到两个独立的声轨文件中
Transcription转写Word and utterance timing词级与话轮级时间戳nvidia/parakeet-tdt-0.6b-v3
Persona modelingPersona 建模Timbre · rhythm · speaking habits音色 · 节奏 · 表达习惯
Grounded expansion有依据的扩写Content · pauses · overlap内容 · 停顿 · 重叠
New dialogue audio新对话音频Emotionally expressive dialogue generated from real conversations基于真实对话生成的富有情感的对话voicedesign & voiceclone
The pipeline first secures reliable real dialogue, then uses it as evidence for controllable data expansion.管线先获得可靠的真实对话,再以其为依据进行可控的数据扩展。

Key Step Comparisons关键步骤对比

Speaker Separation说话人分离

A conventional separation model may split one speaker across different tracks, merge two speakers into the same track, or severely degrade speech quality. Such low-quality separation is effectively unusable for training full-duplex speech models, where stable speaker identity and clean, time-aligned signals are fundamental requirements.

传统分离模型可能把同一个人拆到不同音轨、把两个人合入同一音轨,或者严重破坏语音质量。这类低质量分离结果基本无法用于全双工语音模型训练,因为稳定的说话人身份、清晰的人声和准确的时间对齐都是最基础的数据要求。

After analyzing and comparing the strengths of several models, we combine them into a coordinated workflow. VAD first partitions hour-scale source audio at natural boundaries into approximately 15-minute parts before diarization, improving Diarizen's speaker-identification performance. After music screening and low-SNR filtering, a greedy dynamic-window search captures high-quality reciprocal two-speaker segments that are as long as possible. This both removes long monologues from the training set and creates better input conditions for Sidon: alternating turns and sufficient speech from each person allow the separator to perform more reliably. After separation, speaker-consistency checks detect cases where one person is split across tracks or two people are merged into one track; affected chunks are automatically separated again to repair the result.

在分析并比较多个模型的特性后,我们将各自优势组合为一套协同工作的流程。首先利用 VAD 沿自然边界将小时级原始素材切分为约 15 分钟的片段,再交给 diarization 处理,从而提高 Diarizen 的说话人识别性能。完成音乐筛查和低 SNR 过滤后,系统通过动态窗口进行贪心搜索,捕获尽可能长、质量足够高且有来有回的双人对话片段。这一步既过滤了训练数据中的大段单人独白,也为 Sidon 分离创造了更理想的输入条件:两位说话人交替发言,并且各自拥有充分的语音时长,Sidon 才能发挥更稳定的分离性能。分离完成后,系统继续执行说话人一致性检查;一旦检测到同一个人被拆入不同音轨,或两个人被合入同一音轨,便会自动重新分离,对该片段进行修复。

Previous approach之前的方案

SepFormer separates the selected chunk directly, without a downstream identity-consistency gate.SepFormer 直接分离选定片段,缺少后续的说话人身份一致性质量门。

Our optimization我们的优化

OUR pipeline combines dynamic two-speaker windows, Sidon separation, identity-consistency checks, retry, and light background-noise suppression.OUR 管线结合动态双人窗口、Sidon 分离、身份一致性检测、自动重试和轻量底噪过滤。

Previous approach之前的方案

SepFormer

assets/separation/separation-sepformer/
1 · Original recording1 · 原始录音
16 kHz
0:00 / 0:15
assets/chunk.wav
2 · Speaker tracks2 · 说话人双轨
8 kHz
Default · both tracks默认 · 双轨同时播放 0:00 / 0:15
OURAveraLabs

Sidon + AveraLabs Pipeline

assets/separation/separation-sidon-our/
1 · Original recording1 · 原始录音
16 kHz
0:00 / 0:15
assets/chunk.wav
2 · OUR Speaker tracks2 · OUR 说话人双轨
16 kHz
Default · both tracks默认 · 双轨同时播放 0:00 / 0:15
Identity consistency身份一致性OUR keeps each person in a stable track.OUR 将每个人稳定保留在独立声轨。
Acoustic quality声学质量OUR preserves clearer 16 kHz speech.OUR 保留更清晰的 16 kHz 人声。
Noise control噪声控制OUR also suppresses part of the background noise.OUR 同时过滤了部分底噪。

Augmentation from Real World Audio全双工对话生成

Prompt-only scripts may sound fluent yet generic because they do not inherit the motivation, rhythm, or context of a real exchange. Template-based Dialogue TTS then tends to produce regular pauses and mechanically inserted overlap.

纯提示词剧本可能听起来流畅却较为泛化,因为它没有继承真实交流中的动机、节奏和上下文;模板化 Dialogue TTS 又容易产生规律的停顿和机械插入的 overlap。

OUR method is grounded in real conversation. It expands from observed dialogue content, persona, pauses, and overlap, while also examining the preceding five minutes of dialogue history for broader context. The regeneration stage offers substantial freedom and control: it can extend conversations to greatly increase the volume and diversity of training data, or customize turn-taking, pauses, overlap, paralinguistic behavior, and other characteristics for specific training requirements. The final dialogue is generated with timing and interaction patterns grounded in real exchanges.

OUR 方法以真实对话为依据,基于其中已观测到的内容、Persona、停顿与重叠进行扩写,并进一步观察此前 5 分钟的对话历史以获得更多上下文。重新生成阶段具备很大的自由度与可控性:既可以扩写对话,大幅增加训练数据的规模与多样性;也可以针对具体训练要求,定制话轮节奏、停顿、重叠、副语言表现及其他数据特征。最终生成的对话仍以真实交流中的时序与互动模式为依据。

Original audio原始音频

Original Movie原始电影

OURAveraLabs

Dialogue RegenerationMiMo 对话重生成

With improved overall audio quality, this regeneration reconstructs the original voices and extends the scene naturally while preserving vocal identity, character dynamics, and narrative continuity.在提升整体音质的基础上,该重生成结果重建原始声音并自然扩展场景,同时保持人物声线、角色关系与剧情脉络的一致性。

OUR Speaker tracksOUR 说话人双轨
24 kHz
Default · both tracks默认 · 双轨同时播放 0:00 / 0:19
Original audio原始音频

Original TV Talk Show原始电视访谈

OURAveraLabs

Dialogue RegenerationMiMo 对话重生成

With background music removed, this regeneration expands a short source exchange into a longer, naturally paced conversation while preserving speaker identity and conversational context.在去除背景音乐后,该重生成结果将简短的源对话扩展为更长、节奏自然的交流,同时保持说话人身份与对话上下文的一致性。

OUR Speaker tracksOUR 说话人双轨
24 kHz
Default · both tracks默认 · 双轨同时播放 0:00 / 2:20
Original audio原始音频

Original YouTube Video原始 YouTube 视频

OURAveraLabs

Dialogue RegenerationMiMo 对话重生成

This example reconstructs a two-speaker in-car conversation with separate speaker tracks and time-aligned captions.该示例重建了一段车内双人对话,并提供独立说话人声轨与时间对齐字幕。

OUR Speaker tracksOUR 说话人双轨
24 kHz
Default · both tracks默认 · 双轨同时播放 0:00 / 2:17
Real-world grounding真实场景依据OUR preserves the source conversation's context and interaction patterns.OUR 保留源对话的上下文与互动模式。
Controllable interaction可控互动OUR supports customized turn-taking, pauses, overlap, and paralinguistic behavior.OUR 支持定制话轮、停顿、重叠与副语言表现。
Scalable augmentation可扩展的数据增强OUR expands real conversations into longer and more diverse training data.OUR 将真实对话扩展为更长、更多样的训练数据。