EchoCache: Energy-Guided Cross-Modal Caching for Efficient Audio-Driven Video Generation

2026-08-03

AI Summary

This work addresses the high inference cost of existing audio-driven video generation methods and their inadequate caching mechanisms, which overlook the distinct importance of cross-modal alignment and temporal semantics. To tackle these issues, the authors propose EchoCache, a novel framework that introduces audio time-frequency energy as a saliency anchor to guide latent-space cache updates. EchoCache integrates a dynamic timestep-aware caching strategy with quantized cache management, jointly optimizing computational efficiency and memory footprint. This approach effectively mitigates dual misalignments—between audio-visual semantics and between computation and storage—achieving a 2.46× speedup on the Wan2.2-S2V model under the EMTD benchmark while preserving generation quality and audio-visual consistency.

Abstract

Audio-driven video generation (A2V) has achieved promising progress in synthesizing temporally coherent and audio-visually aligned videos, yet its inference remains expensive due to the iterative denoising process of diffusion models. Existing caching methods mainly exploit temporal redundancy in visual features while overlooking the cross-modal alignment of A2V, where audio drives visual generation with highly non-uniform temporal importance. In this paper, we identify two levels of misalignment in existing A2V caching methods: temporal-semantic and computation-storage misalignment. To address them, we propose EchoCache, an energy-guided cross-modal caching framework for efficient A2V generation. EchoCache leverages audio time-frequency energy as a saliency anchor to guide latent-level cache updates and further introduces a dynamic timestep-latent caching mechanism with quantized cache management for joint efficiency and memory optimization. Extensive experiments on mainstream A2V models show that EchoCache consistently improves the latency-quality trade-off while preserving generation quality and audio-visual consistency. In particular, on Wan2.2-S2V over the EMTD benchmark, EchoCache achieves a 2.46x speedup with the best overall performance. Code is available at https://github.com/IF-LAB-PKU/EchoCache.

Problem

• audio-driven video generation
• cross-modal alignment
• temporal-semantic misalignment
• computation-storage misalignment
• caching efficiency

Innovation

• cross-modal caching
• audio-driven video generation
• energy-guided
• diffusion models
• latency-quality trade-off