SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents

2026-09-28

AI Summary

This work addresses the inherent conflict between real-time responsiveness and high-latency tool invocation with deep reasoning in full-duplex voice interaction. Inspired by cognitive dual-process theory, we propose an adaptive fast-slow system architecture that decouples real-time interaction from asynchronous reasoning. Through knowledge-boundary-aware training and cost-aware reinforcement learning, the framework enables dynamic task delegation and lossless context integration. Experimental results demonstrate that this approach significantly improves accuracy on complex question answering while effectively balancing system performance, response safety, and backend computational costs in real-world business scenarios.

Abstract

Full-duplex speech large language models (LLMs) enable low-latency, natural voice interaction. However, real-world agents must also use tools and perform deliberative reasoning-operations whose variable latency and computational cost conflict with the stringent timing requirements of real-time conversation. To reconcile these demands, we propose SALMONN-duo, an adaptive dual-system voice agent inspired by dual-process theories of cognition. SALMONN-duo separates real-time interaction from deliberative computation by pairing an always-on, fast-thinking full-duplex speech LLM (system 1) with a powerful asynchronous slow-thinking LLM agent (system 2). Beyond handling real-time interaction, system 1 learns when to answer directly and when to delegate, remaining responsive during backend execution and seamlessly integrating returned information into the ongoing dialogue without exposing tool traces or losing conversational context. Evaluations on single-turn spoken question answering (QA) and multi-turn conversations demonstrate that adaptive delegation substantially improves accuracy on knowledge-intensive and multi-hop reasoning questions, while knowledge-boundary-aware training avoids unnecessary system 2 invocations. On a customized version of $\tau$-Voice, SALMONN-duo further demonstrates its ability to complete environment-grounded, policy-constrained tasks through multi-turn interactions in realistic business scenarios. Finally, cost-aware reinforcement learning further enhances the trade-off between task performance and backend usage across the QA and conversation tasks, while improving task success and response safety on $\tau$-Voice with an acceptable increase in the delegation rate.

Problem

• full-duplex voice agents
• speech large language models
• deliberative reasoning
• real-time interaction
• tool use

Innovation

• Full-duplex speech LLM
• Dual-system coordination
• Adaptive delegation
• Knowledge-boundary-aware training
• Cost-aware reinforcement learning