Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO

2026-07-30

AI Summary

This work addresses the challenge faced by current spoken dialogue systems in determining when and how to engage in conversations within noisy, multi-speaker social settings. To this end, the authors propose Cocktail-Talker, a novel framework that explicitly models conversational behavior as discrete action decisions—namely <|respond|>, <|listen|>, and <|ignore|>. By integrating supervised fine-tuning with GRPO-based reinforcement learning, the framework enables large speech language models to autonomously select context-appropriate interaction strategies. To support training, the authors introduce Cocktail-DialogGen, a data pipeline that leverages large language models to synthesize realistic multi-character social dialogues. Experimental results demonstrate that the proposed approach significantly enhances both response accuracy and interaction naturalness in complex acoustic and social contexts.

Abstract

Spoken dialog systems are typically designed for clean, dyadic interactions in which a single user and an assistant take turns speaking. Real-world social conversations, however, are often more ambiguous: multiple speakers may participate in the same conversation amid irrelevant speech and background noise. Each utterance may be directed to the assistant, addressed to another speaker, or completely irrelevant. In such settings, the assistant must decide not only what to say, but also whether to speak at all. In this paper, we introduce Cocktail-Talker, a speech LLM framework for multi-speaker spoken dialog modeling in noisy social environments. We model the assistant's behavior with three action tokens: <|respond|>, <|listen|>, and <|ignore|>, placed before a response or silence. Cocktail-Talker is trained via supervised finetuning and reinforcement learning to generate the appropriate action token and, only in <|respond|> mode, a speech response. To prepare the training data, we develop Cocktail-DialogGen, an LLM-based data pipeline that simulates realistic multi-speaker dialogs with speaker roles across diverse social settings. Together, these components take a step toward spoken dialog systems that interact more naturally and selectively in complex social environments.

Problem

• multi-speaker dialog
• noisy social environments
• spoken dialog systems
• turn-taking
• speech interaction

Innovation

• multi-speaker dialog modeling
• action tokens
• noisy social environments
• speech LLM
• reinforcement learning