Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer

2026-05-29

AI Summary

This work proposes SwanSphere, a unified streaming framework for spatial audio generation from panoramic video and text prompts, addressing the longstanding trade-off between generation quality and inference latency as well as the challenge of accurately modeling spatial cues from multimodal inputs. The approach introduces a causal autoregressive diffusion Transformer to enable low-latency, high-fidelity streaming synthesis, and enhances spatial awareness through spatial video–audio contrastive learning (SVAC) and multi-objective online direct preference optimization (ODPO). To mitigate data scarcity, the authors also develop an automated annotation pipeline. Experimental results demonstrate that SwanSphere significantly improves both audio fidelity and spatial localization accuracy in both video-to-audio and text-to-audio generation tasks.

Abstract

Real-time and accurate spatial audio generation is pivotal for delivering an immersive experience. However, existing spatial audio synthesis technologies are often encumbered by a tradeoff between generation quality and high inference latency, as well as difficulty in capturing precise spatial information from multimodal inputs. To address these challenges, we propose SwanSphere, a unified streaming framework for high-fidelity spatial audio generation from panoramic videos and text prompts. SwanSphere mainly makes the following contributions: 1) We introduce a causal autoregressive diffusion transformer architecture that enables streaming high-quality spatial audio generation. 2) We design a Spatial Video-Audio Contrastive (SVAC) learning strategy to align the video encoder with the acoustic domain, and further employ a multi-objective online direct preference optimization (ODPO) scheme, resulting in strong spatial perception and robust multimodal spatial audio synthesis. 3) To alleviate the current scarcity of spatial audio datasets, we also develop an automated annotation pipeline for generating detailed spatial captions. Experimental results demonstrate that SwanSphere achieves superior performance in both video-to-spatial and text-to-spatial audio generation tasks. Demos can be found at: https://swanaigc.github.io.

Problem

• spatial audio generation
• real-time synthesis
• multimodal alignment
• inference latency
• spatial perception

Innovation

• autoregressive diffusion transformer
• spatial audio generation
• streaming synthesis
• multimodal alignment
• contrastive learning