Problem
Research questions and friction points this paper is trying to address.
Generate coherent motion in image-conditional videos
Improve motion expressiveness and sampling efficiency
Decompose video generation into latent motion modeling
LaMD: Latent Motion Diffusion for Image-Conditional Video Generation
🤖 AI Summary
Existing video diffusion models improve visual quality but suffer from poor motion coherence and low sampling efficiency. To address these limitations, we propose a two-stage image-conditioned video generation framework. First, a novel motion-decomposition video autoencoder disentangles implicit motion representations from appearance reconstruction. Second, a continuous latent-space diffusion model captures the image-conditioned motion prior. This approach enables expressive, multimodal, and computationally efficient motion modeling. Evaluated on BAIR, Landscape, NATOPS, MUG, and CATER-GEN benchmarks, our method significantly enhances motion naturalness and temporal consistency while accelerating sampling by 2.1–3.8×. Moreover, it supports complex dynamic modeling and fine-grained motion control.
📝 Abstract
The video generation field has witnessed rapid improvements with the introduction of recent diffusion models. While these models have successfully enhanced appearance quality, they still face challenges in generating coherent and natural movements while efficiently sampling videos. In this paper, we propose to condense video generation into a problem of motion generation, to improve the expressiveness of motion and make video generation more manageable. This can be achieved by breaking down the video generation process into latent motion generation and video reconstruction. Specifically, we present a latent motion diffusion (LaMD) framework, which consists of a motion-decomposed video autoencoder and a diffusion-based motion generator, to implement this idea. Through careful design, the motion-decomposed video autoencoder can compress patterns in movement into a concise latent motion representation. Consequently, the diffusion-based motion generator is able to efficiently generate realistic motion on a continuous latent space under multi-modal conditions, at a cost that is similar to that of image diffusion models. Results show that LaMD generates high-quality videos on various benchmark datasets, including BAIR, Landscape, NATOPS, MUG and CATER-GEN, that encompass a variety of stochastic dynamics and highly controllable movements on multiple image-conditional video generation tasks, while significantly decreases sampling time.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Motion generation simplifies video generation process
Latent motion diffusion framework enhances motion expressiveness
Efficient realistic motion generation in latent space
🔎 Similar Papers
No similar papers found.
Wuhan University | Microsoft Research Asia
