Generative Cinematographer: Composing Camera and Object Motion in 3D

2026-10-01

AI Summary

This study addresses the limitation of existing video generation methods that rely on 2D trajectories, which struggle to achieve precise 3D controllability when cameras and objects move simultaneously. This work proposes constructing an editable 3D scene from a single image, enabling independent control of foreground and camera paths via local 3D handles, with projected guidance maps driving a pretrained Wan model. Furthermore, it introduces a piecewise-rigid approximation strategy for non-rigid motion, establishing relative motion representations in world coordinates without physical simulation, combined with LoRA fine-tuning for efficient generation. The proposed method achieves decoupled control over camera and object motions, significantly enhancing geometric consistency under varying viewpoints and demonstrating strong 3D controllability across diverse real-world scenarios.

Abstract

Current controllable video generation systems often rely on 2D motion trajectories or sparse drag signals for object motion. These controls are ambiguous because the same 2D trajectory can correspond to different 3D motions, especially when the camera and objects move simultaneously. We present Generative Cinematographer (GenCine), a system that lifts a single image into an editable 3D scene scaffold where artists jointly author camera and foreground motion. Artists specify a camera path and move selected foreground regions using local 3D motion handles. Several handles can move different parts of a subject independently, providing a piecewise-rigid approximation to non-rigid motion without a physics simulator or category-specific prior. To communicate these controls to a pretrained video model, we project them into guidance maps. These maps record where the controlled regions appear in each frame, assign each handle a fixed color across frames and encode the current 3D positions of its controlled points in the same world coordinate system as the background. This lets us describe object motion relative to the scene even as the camera moves. For training, we recover controls from the motion observed in real videos and use ground-truth geometry and trajectories from synthetic videos. We train a lightweight guidance branch and LoRA adapters on a pretrained Wan model to follow these controls. Our experiments show consistent camera-relative motion, improved geometric consistency under viewpoint changes, and strong controllability across diverse real-world scenes.

Problem

• controllable video generation
• 3D motion control
• camera-object motion composition
• motion ambiguity

Innovation

• 3D motion control
• camera-object composition
• guidance maps
• piecewise-rigid approximation
• controllable video generation