Object-Centric World Model for Language-Guided Manipulation

2025-03-08

AI Summary

Existing world models for language-guided robotic manipulation suffer from high computational overhead, limited generalization, and insufficient semantic understanding. This paper introduces a novel world model specifically designed for language-guided manipulation, which—uniquely—integrates object-centric slot attention representations with language-conditioned state prediction, abandoning the conventional video-generation paradigm in favor of direct future-state prediction in latent space conditioned on natural language instructions. The method employs end-to-end visuo-linguo-motor joint training to enable efficient object perception and manipulation semantics modeling. Experiments demonstrate that, compared to diffusion-based generative models, our approach achieves a 3.2× speedup in inference, reduces parameter count by 67%, improves action prediction accuracy by 19.5%, and significantly enhances sample efficiency, computational efficiency, cross-task generalization, and language–action alignment fidelity.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

A world model is essential for an agent to predict the future and plan in domains such as autonomous driving and robotics. To achieve this, recent advancements have focused on video generation, which has gained significant attention due to the impressive success of diffusion models. However, these models require substantial computational resources. To address these challenges, we propose a world model leveraging object-centric representation space using slot attention, guided by language instructions. Our model perceives the current state as an object-centric representation and predicts future states in this representation space conditioned on natural language instructions. This approach results in a more compact and computationally efficient model compared to diffusion-based generative alternatives. Furthermore, it flexibly predicts future states based on language instructions, and offers a significant advantage in manipulation tasks where object recognition is crucial. In this paper, we demonstrate that our latent predictive world model surpasses generative world models in visuo-linguo-motor control tasks, achieving superior sample and computation efficiency. We also investigate the generalization performance of the proposed method and explore various strategies for predicting actions using object-centric representations.

Problem

• Develops object-centric world model for language-guided manipulation tasks.
• Reduces computational resources compared to diffusion-based generative models.
• Enhances object recognition and prediction in visuo-linguo-motor control tasks.

Innovation

• Object-centric representation using slot attention
• Language-guided future state prediction
• Compact, efficient model for manipulation tasks