Few-Shot Neural Differentiable Simulator: Real-to-Sim Rigid-Contact Modeling

2026-03-06

AI Summary

This work proposes a few-shot real-to-simulation learning framework to address the limitations of traditional analytical physics simulators, which struggle to accurately capture complex contact dynamics, and data-driven approaches that typically require large amounts of real-world data. By leveraging only a small number of real trajectories, the method calibrates an analytical simulator to generate high-fidelity synthetic data and introduces a fully differentiable voxelized graph neural network to model rigid-body forward dynamics. This approach achieves, for the first time, a differentiable contact dynamics simulator that combines physical consistency with high representational capacity using minimal real-world data. It significantly improves trajectory reproduction accuracy and simulation fidelity, while also accelerating gradient-based policy optimization in multi-object interaction scenarios.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Web Mining and Content Analysis: Web data generation and simulation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Accurate physics simulation is essential for robotic learning and control, yet analytical simulators often fail to capture complex contact dynamics, while learning-based simulators typically require large amounts of costly real-world data. To bridge this gap, we propose a few-shot real-to-sim approach that combines the physical consistency of analytical formulations with the representational capacity of graph neural network (GNN)-based models. Using only a small amount of real-world data, our method calibrates analytical simulators to generate large-scale synthetic datasets that capture diverse contact interactions. On this foundation, we introduce a mesh-based GNN that implicitly models rigid-body forward dynamics and derive surrogate gradients for collision detection, achieving full differentiability. Experimental results demonstrate that our approach enables learning-based simulators to outperform differentiable baselines in replicating real-world trajectories. In addition, the differentiable design supports gradient-based optimization, which we validate through simulation-based policy learning in multi-object interaction scenarios. Extensive experiments show that our framework not only improves simulation fidelity with minimal supervision but also increases the efficiency of policy learning. Taken together, these findings suggest that differentiable simulation with few-shot real-world grounding provides a powerful direction for advancing future robotic manipulation and control.

Problem

• few-shot learning
• differentiable simulation
• rigid-body contact
• real-to-sim transfer
• physics simulation

Innovation

• few-shot learning
• differentiable simulation
• graph neural network
• real-to-sim transfer
• rigid-body dynamics