Dream4D: Lifting Camera-Controlled I2V towards Spatiotemporally Consistent 4D Generation

2025-08-11

AI Summary

This work addresses the challenge of jointly ensuring spatiotemporal consistency, geometric fidelity, and view consistency in 4D content generation for large-scale multi-object interaction scenes. We propose the first two-stage framework integrating video diffusion priors with neural 4D reconstruction. Methodologically, we pioneer the co-optimization of a pose-conditioned video diffusion model and geometry-aware NeRF reconstruction: Stage I estimates camera trajectories, while Stage II performs controllable temporal generation conditioned on estimated poses, enabled by differentiable rendering for end-to-end joint training. Few-shot learning is incorporated to enhance generalization. Experiments demonstrate significant improvements over state-of-the-art methods in multi-view consistency and dynamic geometry accuracy, achieving new SOTA performance on mPSNR and mSSIM. Our approach enables high-fidelity, physically plausible, and temporally persistent 4D modeling.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

The synthesis of spatiotemporally coherent 4D content presents fundamental challenges in computer vision, requiring simultaneous modeling of high-fidelity spatial representations and physically plausible temporal dynamics. Current approaches often struggle to maintain view consistency while handling complex scene dynamics, particularly in large-scale environments with multiple interacting elements. This work introduces Dream4D, a novel framework that bridges this gap through a synergy of controllable video generation and neural 4D reconstruction. Our approach seamlessly combines a two-stage architecture: it first predicts optimal camera trajectories from a single image using few-shot learning, then generates geometrically consistent multi-view sequences via a specialized pose-conditioned diffusion process, which are finally converted into a persistent 4D representation. This framework is the first to leverage both rich temporal priors from video diffusion models and geometric awareness of the reconstruction models, which significantly facilitates 4D generation and shows higher quality (e.g., mPSNR, mSSIM) over existing methods.

Problem

• Maintaining view consistency in complex dynamic scenes
• Generating spatiotemporally coherent 4D content
• Combining video generation with 4D reconstruction

Innovation

• Predicts camera trajectories from single image
• Generates multi-view sequences via diffusion
• Converts sequences into persistent 4D representation