Decoupled Video Generation with Chain of Training-free Diffusion Model Experts

2024-08-24

AI Summary

Existing video generation models suffer from high computational overhead, low visual fidelity, and poor long-term temporal coherence. This paper proposes ConFiner, a framework that decouples video generation into structural control and spatiotemporal refinement—achieved via chained invocation of pretrained diffusion model experts for efficient collaboration. Key contributions include: (1) a training-free, expert-chaining scheduling mechanism; (2) coordinated denoising to fuse multi-expert priors; and (3) a triple-constraint strategy enforcing temporal consistency. The extended variant, ConFiner-Long, supports high-fidelity, coherent video synthesis up to 600 frames. Experiments demonstrate that ConFiner reduces inference cost to only 10% of baseline models while outperforming state-of-the-art methods—including Lavie and ModelScope—in generation quality, temporal coherence, and efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Video generation models hold substantial potential in areas such as filmmaking. However, current video diffusion models need high computational costs and produce suboptimal results due to extreme complexity of video generation task. In this paper, we propose extbf{ConFiner}, an efficient video generation framework that decouples video generation into easier subtasks: structure extbf{con}trol and spatial-temporal re extbf{fine}ment. It can generate high-quality videos with chain of off-the-shelf diffusion model experts, each expert responsible for a decoupled subtask. During the refinement, we introduce coordinated denoising, which can merge multiple diffusion experts' capabilities into a single sampling. Furthermore, we design ConFiner-Long framework, which can generate long coherent video with three constraint strategies on ConFiner. Experimental results indicate that with only 10% of the inference cost, our ConFiner surpasses representative models like Lavie and Modelscope across all objective and subjective metrics. And ConFiner-Long can generate high-quality and coherent videos with up to 600 frames.

Problem

• Video Generation
• Computational Resources
• Video Quality

Innovation

• ConFiner
• Efficient Video Generation
• Long-sequence Coherence