Audio Interaction Model

2026-06-03

AI Summary

Existing large audio language models are limited to offline processing or single-task streaming inference, lacking a unified capability for real-time interaction. This work proposes the first audio interaction paradigm, establishing an online audio interaction system grounded in a continuous perceive-decide-respond loop. Built upon the SoundFlow framework, the approach enables native streaming data construction, perception-aware training, and asynchronous low-latency inference. The study also introduces the StreamAudio-2M dataset and the Proactive-Sound-Bench evaluation benchmark. The resulting model achieves competitive performance across eight mainstream audio tasks while supporting real-time automatic speech recognition, multi-task streaming instruction following, and semantics-driven proactive voice intervention, thereby transcending the limitations of conventional offline models.

Abstract

Audio is an inherently interactive modality, yet today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting. It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly. We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following, from dialogue to full voice chatting, deciding when to respond from the semantics of the stream. To enable this, we propose SoundFlow, a framework that instantiates the perceive-decide-respond loop end to end, from data to training to deployment, through streaming-native data construction, comprehension-aware training, and asynchronous low-latency inference for stable real-time interaction. We further construct StreamAudio-2M, a 2.6M-item streaming corpus spanning 7 fundamental abilities and 28 sub-tasks, and Proactive-Sound-Bench for evaluating proactive audio intervention. Across 8 benchmarks, Audio-Interaction preserves competitive performance on mainstream audio tasks while unlocking capabilities inaccessible to offline LALMs, including real-time ASR, streaming audio instruction following, and proactive help.

Problem

• Large Audio Language Models
• streaming audio
• real-time interaction
• audio instruction following
• online LALM

Innovation

• Audio Interaction Model
• Streaming Audio
• SoundFlow
• Online LALM
• Proactive Audio Intervention