Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space

2026-07-07

AI Summary

This work addresses key limitations in existing video-to-audio generation methods—namely, high computational costs from multi-stage training, temporal detail degradation due to reliance on textual intermediaries, and the absence of sound-centric semantic annotations. To overcome these challenges, we propose Flowley, an end-to-end, single-stage audio-visual generation architecture that introduces a novel progressive soft-masked cross-modal attention mechanism, enabling synchronized audio-visual embedding without additional computational overhead. Additionally, we design SoundCap, a plug-and-play sound-aware captioning pipeline that constructs descriptive sound semantics. Experimental results demonstrate that Flowley achieves state-of-the-art performance on VGGSound, and when integrated with SoundCap, it surpasses the strongest closed-source baselines in zero-shot audio generation quality.

Abstract

Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.

Problem

• Video-to-Audio Generation
• Cross-Modal Alignment
• Temporal Synchronization
• Sound-Oriented Captions
• Latent Space

Innovation

• Cross-Modal Alignment
• Progressive Soft-masked Cross-Attention
• End-to-End V2A Generation
• Sound-Aware Captioning
• Latent Space Synchronization