Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation

2026-08-05

AI Summary

Existing video-to-audio (V2A) generation methods often rely on auxiliary networks or strong inductive biases to model temporal dynamics, struggling to effectively capture natural inter-frame variations. This work proposes a lightweight Temporal Differences (TD) representation that enhances video conditioning signals with minimal architectural modification. We further introduce a hierarchical continual learning strategy and an annealed TD guidance mechanism to progressively integrate TD information during both diffusion training and sampling. As the first systematic exploration of frame-level and feature-level temporal differences in V2A synthesis, our approach achieves significant performance gains across multiple benchmarks without requiring additional supervision or complex modules—outperforming even specialized models that leverage contrastive audiovisual pretraining—thereby demonstrating the efficacy and generality of TD representations.

Abstract

Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existing conditional diffusion-based V2A methods typically enhance visual conditioning with additional audio-visual supervision, acoustic structure prediction, or reasoning from large multimodal models, requiring extra networks or strong inductive biases. Inspired by recent advances in visual representation learning, we introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A, enriching visual conditioning with minimal architectural modification. We first investigate TD at both the frame and feature levels to identify the most effective representation level at which TD complements visual representations. Based on these findings, we develop a hierarchically continual learning strategy and an annealed temporal differences guidance method to progressively learn and exploit TD information during diffusion training and sampling process, respectively. Extensive experiments on benchmark datasets demonstrate that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality, even outperforming dedicated V2A representations such as contrastive audio-visual pretraining.

Problem

• video-to-audio generation
• temporal differences
• visual representation
• conditional diffusion
• audio synthesis

Innovation

• Temporal Differences
• Video-to-Audio Generation
• Diffusion Models
• Visual Representation Learning
• Hierarchical Continual Learning