ObjCtrl-2.5D: Training-free Object Control with Camera Poses

2024-12-10

arXiv.org

AI Summary

In existing image-to-video (I2V) generation, 2D trajectory-based motion control struggles to accurately capture user intent, often yielding unnatural object motion. To address this, we propose a training-free 2.5D trajectory control paradigm: user-specified 2D trajectories are elevated to depth-aware 2.5D trajectories and explicitly formulated as camera pose sequences, enabling local object motion control via off-the-shelf camera motion models. We further introduce a background-object disentanglement module and a cross-frame sharing mechanism for low-frequency intra-object latent variables, supporting complex dynamics such as rotation and scaling. Our method requires no fine-tuning or retraining and seamlessly integrates with mainstream I2V models (e.g., CMC-I2V). Experiments demonstrate that it significantly outperforms all training-free approaches in control accuracy and surpasses supervised 2D-trajectory-based methods in motion diversity.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Motion and Path Planning
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This study aims to achieve more precise and versatile object control in image-to-video (I2V) generation. Current methods typically represent the spatial movement of target objects with 2D trajectories, which often fail to capture user intention and frequently produce unnatural results. To enhance control, we present ObjCtrl-2.5D, a training-free object control approach that uses a 3D trajectory, extended from a 2D trajectory with depth information, as a control signal. By modeling object movement as camera movement, ObjCtrl-2.5D represents the 3D trajectory as a sequence of camera poses, enabling object motion control using an existing camera motion control I2V generation model (CMC-I2V) without training. To adapt the CMC-I2V model originally designed for global motion control to handle local object motion, we introduce a module to isolate the target object from the background, enabling independent local control. In addition, we devise an effective way to achieve more accurate object control by sharing low-frequency warped latent within the object's region across frames. Extensive experiments demonstrate that ObjCtrl-2.5D significantly improves object control accuracy compared to training-free methods and offers more diverse control capabilities than training-based approaches using 2D trajectories, enabling complex effects like object rotation. Code and results are available at https://wzhouxiff.github.io/projects/ObjCtrl-2.5D/.

Problem

• Enhancing object control in image-to-video generation
• Overcoming limitations of 2D trajectory methods
• Enabling 3D object motion using camera poses

Innovation

• 3D trajectory with depth for object control
• Camera poses model object movement directly
• Isolate object from background for local control