Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory

2026-07-29

AI Summary

Existing audio-visual generation models struggle to simultaneously achieve low latency and long-term temporal consistency, hindering their applicability to real-time streaming scenarios. This work proposes Ripple, a novel system that introduces, for the first time, a cross-modal recurrent memory mechanism. Building upon sliding-window attention, Ripple maintains modality-specific memory states and employs a three-stage training pipeline—comprising causal attention transfer, end-to-end distillation, and online reinforcement post-training—to optimize memory construction and cross-modal interaction. Evaluated at 480p resolution, Ripple achieves inference speeds of approximately 28 FPS, significantly outpacing its teacher model while surpassing current state-of-the-art methods on both long- and short-video benchmarks. The system thus enables efficient, temporally coherent, real-time joint audio-visual generation.

Abstract

Audio-video generative models achieve impressive quality but suffer from high latency, making them unsuitable for real-time applications. Although several streaming audio-video generation methods have been proposed, they remain costly and fail to support long-form generation. To address this, we propose \textbf{Ripple}, a real-time joint audio-video generation system with a cross-modal recurrent memory mechanism. To enable efficient streaming inference while preserving long-term context, Ripple combines a fixed-length sliding-window attention with modality-specific memory states that continuously summarize audio and video context. Cross-modal memory interaction is further introduced to enhance audio-visual synchronization. To learn this memory-augmented model effectively, we devise a three-stage training recipe: (1) adapting a bidirectional audio-video teacher to block-wise causal attention with simulated memory, (2) optimizing the memory construction and interaction pipeline through end-to-end distillation, and (3) applying online reinforcement post-training tailored for streaming audio-video generation. As a result, Ripple achieves ~28 FPS at 480P resolution, over faster than the teacher, while capable of coherent long-form generation. Extensive experiments on both short-video and long-video benchmarks demonstrate our superior performance over existing offline and online joint audio-video generation methods.

Problem

• real-time audio-video generation
• streaming generation
• long-form generation
• low latency
• audio-visual synchronization

Innovation

• cross-modal recurrent memory
• streaming audio-video generation
• sliding-window attention
• memory-augmented model
• real-time generation