DuplexOmni: Real-Time Listening, Seeing, Thinking, and Speaking for Full-Duplex Interaction

2026-06-08

AI Summary

Existing models struggle to simultaneously support real-time interaction, complex reasoning, and tool use within a full-duplex multimodal framework. This work proposes an asynchronous full-duplex architecture that decouples interaction from reasoning: an interaction layer processes audio and video inputs in an end-to-end streaming fashion to generate immediate responses, while a thinking layer employs plug-in modules to perform asynchronous, complex reasoning and tool invocation. To facilitate training, the authors introduce a Writer-Director pipeline for constructing continuous interactive data. Evaluated on multiple public benchmarks, the system demonstrates strong performance, significantly enhancing the naturalness and fluency of multimodal interactions.

Abstract

Human interaction is continuous, multimodal, and full-duplex by nature. Although recent omni models have made substantial progress in unified speech, vision, and text modeling, combining seamless real-time interaction with complex reasoning and tool use remains challenging. We present DuplexOmni, a method for real-time multimodal full-duplex interaction. DuplexOmni separates model capability into an interaction layer and a thinking layer, which collaborate asynchronously in parallel. The interaction layer is implemented by the DuplexOmni model, an end-to-end system that processes streaming audio and video inputs while generating text and speech responses in real time. The thinking layer is a pluggable module that provides complex reasoning and tool-use capabilities. To support this method, we further develop a Writer-Director pipeline for constructing continuous-interaction training data. Experiments show that DuplexOmni achieves strong performance on multiple public benchmarks and exhibits natural full-duplex interaction ability.

Problem

• full-duplex interaction
• real-time multimodal
• complex reasoning
• tool use
• continuous interaction

Innovation

• full-duplex interaction
• multimodal modeling
• real-time streaming
• asynchronous collaboration
• tool-augmented reasoning