Problem
Research questions and friction points this paper is trying to address.
full-duplex speech models
multi-party conversation
long-context robustness
bilingual dialogue
speech benchmarks
MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation
🤖 AI Summary
This study addresses the limitations of full-duplex speech models in long-context and multi-party interactions by extending the Moshi architecture to long-duration, multi-party, and Chinese-English bilingual scenarios. Methodologically, it proposes a codec frame-level bilingual end-to-end full-duplex modeling framework and introduces the first large-scale dataset and evaluation benchmark supporting the joint modeling of complex interaction features, including duration, overlap, and interruption. The project releases 57,600 hours of synthetic data alongside a real-recorded benchmark, MultiTalkBench. Experimental results demonstrate that the trained model significantly outperforms open-source baselines such as Moshi and MiniCPM-o on MultiTalkBench, achieving high coherence and precise responsiveness in extended multi-party dialogues.
📝 Abstract
End-to-end full-duplex speech models have brought open-source machine conversation closer to human-like interaction, yet existing systems remain limited in two intertwined dimensions: long-context robustness and multi-party interaction. Real-world scenarios such as meetings, group lessons, and social-robot reception require a single model to track, contextualize, and respond to multiple speakers over extended durations. Progress is constrained by both data and evaluation: open multi-party speech corpora remain small and are not designed for codec-frame-level full-duplex modeling, while existing long-audio benchmarks focus on passive listening and speech-to-speech benchmarks are mostly short and dyadic. We extend the Moshi paradigm jointly along the long-horizon and multi-party axes in English and Chinese. First, we release 57.6k hours of synthetic training data ($\href{https://huggingface.co/datasets/MultiTalk/MultiTalkPT}{MultiTalkPT}$ and $\href{https://huggingface.co/datasets/MultiTalk/MultiTalkFT}{MultiTalkFT}$) for long-form, multi-party, English-Chinese full-duplex dialogue, with controllable length, participant count, turn-taking, overlap, backchannels, interruptions, addressee shifts, and long-range coreference. Second, we introduce $\href{https://huggingface.co/datasets/MultiTalk/MultiTalkBench}{MultiTalkBench}$, built from real human recordings, for evaluating long-form, multi-party, bilingual full-duplex dialogue. Conversations average 32.6 minutes and include probes for long-range entity tracking, topic coherence, and addressee selection. Third, we train a bilingual Moshi-style model that sustains coherent multi-party English-Chinese conversations over extended durations and substantially outperforms open-source baselines including Moshi, MiniCPM-o-4.5, and Qwen3-Omni-30B-A3B-Instruct on MultiTalkBench.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Full-duplex speech model
Multi-party conversation
Long-context robustness
Bilingual dialogue
Synthetic training data
🔎 Similar Papers
No similar papers found.
