Learning to Predict Contact Force Distributions from Vision Leveraging Object Geometry Priors

2026-08-01

AI Summary

This work addresses the challenge of predicting three-dimensional contact force distributions for stacked everyday objects from a single RGB image, aiming to enable human-like physical reasoning and dexterous manipulation. To this end, the authors propose an approach that leverages rigid-body simulation to generate paired visual–force training data, combined with a geometry-aware statistical smoothing technique to transform sparse point forces into structurally consistent, smooth force fields. A deep learning model is then trained end-to-end to predict these smoothed force distributions directly from images. Notably, the method requires no real-world force measurements and achieves effective generalization to physical scenes despite being trained solely in simulation. Experimental results demonstrate significant improvements in both force prediction accuracy and performance on downstream manipulation tasks, validating the efficacy of geometry-guided smoothing and domain-invariant generalization.

Abstract

Based on vision and prior experience, humans can make rough physical predictions and adjust their manipulation strategies. This paper aims to endow robots with a similar ability. To collect paired data of vision and forces, we use a rigid-body simulator commonly adopted in robotics. However, unlike simulators that output noisy point forces, humans are able to make consistent predictions even in unfamiliar situations. Based on this observation, we hypothesize that predicting smooth force distributions rather than raw point forces can improve both force prediction itself and downstream task performance. To validate this hypothesis, we construct a model that predicts three-dimensional force distributions from a single RGB image of piled daily objects. The target distribution is generated by applying statistical smoothing to point forces obtained from the simulator. Moreover, by incorporating object geometry into the smoothing process, we aim to account for variations in contact states and achieve more consistent vision-based predictions. We conduct extensive evaluations in both simulation and real environments. Results show that our approach improves prediction accuracy, enhances downstream task performance through smoothing, and further benefits from geometry-guided smoothing. Remarkably, the trained model generalizes effectively to real-world scenes despite being trained solely in simulation.

Problem

• contact force prediction
• vision-based reasoning
• force distribution
• object geometry
• physical interaction

Innovation

• force distribution prediction
• geometry priors
• vision-based force estimation
• simulation-to-reality generalization
• contact mechanics