Unsupervised Dynamics Prediction with Object-Centric Kinematics

2024-04-29

arXiv.org

AI Summary

Modeling multi-object dynamics in video prediction remains challenging due to entangled appearance and motion representations. Method: This paper proposes an unsupervised object-centric dynamics prediction framework. It introduces, for the first time, an explicit–implicit协同 kinematic representation—encoding position, velocity, and acceleration—to fully decouple static appearance from dynamic motion. A spatiotemporal-aware Transformer architecture is designed to jointly leverage object-centric representations and self-supervised temporal modeling. Contribution/Results: Unlike conventional appearance-only methods, our framework significantly improves long-term prediction accuracy in complex, multi-attribute scenes. It achieves state-of-the-art performance on synthetic benchmarks and demonstrates strong cross-environment generalization. Crucially, it establishes the first paradigm for visual dynamics modeling that unifies physical interpretability—via principled kinematic priors—with end-to-end differentiable learning.

Technology Category

• Computer Vision: Motion & Tracking
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Human perception involves discerning complex multi-object scenes into time-static object appearance (ie, size, shape, color) and time-varying object motion (ie, location, velocity, acceleration). This innate ability to unconsciously understand the environment is the motivation behind the success of dynamics modeling. Object-centric representations have emerged as a promising tool for dynamics prediction, yet they primarily focus on the objects' appearance, often overlooking other crucial attributes. In this paper, we propose Object-Centric Kinematics (OCK), a framework for dynamics prediction leveraging object-centric representations. Our model utilizes a novel component named object kinematics, which comprises low-level structured states of objects' position, velocity, and acceleration. The object kinematics are obtained via either implicit or explicit approaches, enabling comprehensive spatiotemporal object reasoning, and integrated through various transformer mechanisms, facilitating effective object-centric dynamics modeling. Our model demonstrates superior performance when handling objects and backgrounds in complex scenes characterized by a wide range of object attributes and dynamic movements. Moreover, our model demonstrates generalization capabilities across diverse synthetic environments, highlighting its potential for broad applicability in vision-related tasks.

Problem

• Modeling dynamic object interactions in videos
• Integrating object motion with appearance features
• Improving long-term spatiotemporal video prediction

Innovation

• Object-centric kinematics for dynamic video prediction
• Explicit object motions as additional attributes
• Spatiotemporal prediction of complex object interactions