Proc4Gem: Foundation models for physical agency through procedural generation

2025-03-11

AI Summary

This work addresses the longstanding decoupling of physical interaction and semantic understanding in embodied intelligence. Methodologically, it introduces a simulation-to-reality co-modeling paradigm that jointly integrates contact dynamics and environmental semantics. Specifically, it employs procedural generation to synthesize high-fidelity, contact-rich, semantically diverse physical trajectories; combines rigid-body simulation with rasterization-based rendering; and leverages multimodal instruction tuning to establish a physically grounded, semantically aligned training pipeline. Crucially, it demonstrates—for the first time—the zero-shot transfer of a purely simulation-trained multimodal foundation model (Gemini) to real-world contact manipulation on a quadruped robot, enabling language-guided actions such as pushing unseen objects to novel target locations. Experiments show that the model achieves complex physical interaction tasks without any real-world fine-tuning, validating that simulation-driven training can endow foundation models with genuine physical agency in the real world.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

In robot learning, it is common to either ignore the environment semantics, focusing on tasks like whole-body control which only require reasoning about robot-environment contacts, or conversely to ignore contact dynamics, focusing on grounding high-level movement in vision and language. In this work, we show that advances in generative modeling, photorealistic rendering, and procedural generation allow us to tackle tasks requiring both. By generating contact-rich trajectories with accurate physics in semantically-diverse simulations, we can distill behaviors into large multimodal models that directly transfer to the real world: a system we call Proc4Gem. Specifically, we show that a foundation model, Gemini, fine-tuned on only simulation data, can be instructed in language to control a quadruped robot to push an object with its body to unseen targets in unseen real-world environments. Our real-world results demonstrate the promise of using simulation to imbue foundation models with physical agency. Videos can be found at our website: https://sites.google.com/view/proc4gem

Problem

• Integrate environment semantics and contact dynamics in robot learning.
• Use generative modeling and procedural generation for realistic simulations.
• Transfer simulation-trained behaviors to real-world robotic tasks via language instructions.

Innovation

• Generative modeling for contact-rich trajectories
• Procedural generation in diverse simulations
• Language-instructed real-world robot control