OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars

2026-07-24

AI Summary

This work addresses the challenges of temporal uncertainty and degradation of cross-modal identity consistency in real-time, open-duration audiovisual avatar generation. To overcome these limitations, we propose OmniMate, a unified diffusion-based framework for synchronized audiovisual synthesis that introduces a novel Generation Progress Controller (GPC) to enable adaptive response pacing and smooth state transitions. Additionally, a Multi-Reference Conditioning Module (MRCM) is designed to fuse multiple reference images and speech segments, thereby preserving long-term identity coherence. Experimental results demonstrate that OmniMate achieves high-quality, low-latency streaming generation on the interactive adaptation of VerseBench, maintaining strong audiovisual consistency over extended multi-turn dialogues and delivering a realistic, coherent, and responsive interactive experience.

Abstract

Recent advances in diffusion-based generative models have enabled real-time audio-driven avatar generation and unified audio-visual synthesis, providing a promising foundation for interactive avatar systems. However, extending these models to real-time interactive streaming remains challenging, as the generation horizon is unknown in advance and cross-modal identity consistency gradually degrades during long-term generation. To address these challenges, we propose OmniMate, a unified framework for open-ended real-time interactive audio-visual avatar generation. OmniMate jointly synthesizes visual content, speech, and audio effects in real time, enabling natural and immersive multi-turn interactions. To achieve adaptive response progression, we introduce a Generation Progress Controller (GPC) that explicitly models the generation progress of each streaming chunk, allowing the model to complete responses according to the desired progress and achieve seamless transitions between execution and listening states. To preserve long-term cross-modal identity consistency, we propose a Multi-Reference Conditioning Module (MRCM), which leverages multiple reference images and a reference speech segment to provide persistent visual and speaker identity cues throughout long-duration streaming interactions. Extensive experiments on an interaction-oriented adaptation of VerseBench demonstrate that OmniMate achieves high-quality, low-latency streaming generation while maintaining strong long-term audio-visual consistency. The results further show that OmniMate supports realistic, coherent, and responsive interactive avatar experiences over extended multi-turn conversations.

Problem

• real-time streaming
• audio-visual generation
• interactive avatars
• identity consistency
• open-ended generation

Innovation

• real-time streaming
• audio-visual generation
• interactive avatars
• identity consistency
• diffusion models