Free-Form Motion Control: A Synthetic Video Generation Dataset with Controllable Camera and Object Motions

2025-01-02

AI Summary

Existing video generation methods struggle to simultaneously achieve precise 6D control over both camera and object motion, further hindered by the absence of large-scale training data with fine-grained motion annotations. To address this, we propose Free-Form Motion Control (FMC), a novel framework enabling decoupled, independent, or coordinated control of camera and dynamic object motions. We introduce SynFMC—the first large-scale synthetic video dataset featuring complete, per-frame 6D pose annotations for both camera and objects. Additionally, we design a motion-conditioning encoding mechanism that effectively injects 6D motion signals into diffusion-based video generators while remaining compatible with mainstream text-to-image personalization models. Extensive experiments demonstrate that FMC significantly outperforms prior approaches across diverse scenarios, enabling high-fidelity, style-controllable video generation with arbitrary motion specifications. Our work establishes a new benchmark and technical foundation for controllable video synthesis.

Technology Category

• Computer Vision: Motion & Tracking
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Controlling the movements of dynamic objects and the camera within generated videos is a meaningful yet challenging task. Due to the lack of datasets with comprehensive motion annotations, existing algorithms can not simultaneously control the motions of both camera and objects, resulting in limited controllability over generated contents. To address this issue and facilitate the research in this field, we introduce a Synthetic Dataset for Free-Form Motion Control (SynFMC). The proposed SynFMC dataset includes diverse objects and environments and covers various motion patterns according to specific rules, simulating common and complex real-world scenarios. The complete 6D pose information facilitates models learning to disentangle the motion effects from objects and the camera in a video. To validate the effectiveness and generalization of SynFMC, we further propose a method, Free-Form Motion Control (FMC). FMC enables independent or simultaneous control of object and camera movements, producing high-fidelity videos. Moreover, it is compatible with various personalized text-to-image (T2I) models for different content styles. Extensive experiments demonstrate that the proposed FMC outperforms previous methods across multiple scenarios.

Problem

• Video Generation
• Motion Control
• Data Limitations

Innovation

• SynFMC dataset
• FMC method
• motion control