A Machine Learning Approach to Sensor Substitution for Non-Prehensile Manipulation

2025-02-13

AI Summary

Sensor configuration disparities hinder policy transfer in non-grasping manipulation tasks—e.g., pushing—where low-cost robots equipped with LiDAR or RGB-D cameras cannot readily replace or collaborate with high-resolution tactile-skin-enabled mobile manipulators. Method: We propose the first sensor substitution learning framework for non-grasping tasks, leveraging deep generative models to perform cross-modal perceptual mapping: sparse visual/LiDAR observations are spatiotemporally aligned and synthesized into semantically consistent tactile features; end-to-end policy distillation then enables low-end robots to directly reuse pretrained tactile policies. Results: Experiments demonstrate that robots using only LiDAR or RGB-D achieve performance on par with—or even surpassing—that of tactile-skin systems in pushing tasks, significantly reducing deployment costs while exhibiting strong cross-hardware generalization.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Learning to Search
• Machine Learning: Multimodal Learning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Mobile manipulators are increasingly deployed in complex environments, requiring diverse sensors to perceive and interact with their surroundings. However, equipping every robot with every possible sensor is often impractical due to cost and physical constraints. A critical challenge arises when robots with differing sensor capabilities need to collaborate or perform similar tasks. For example, consider a scenario where a mobile manipulator equipped with high-resolution tactile skin is skilled at non-prehensile manipulation tasks like pushing. If this robot needs to be replaced or augmented by a robot lacking such tactile sensing, the learned manipulation policies become inapplicable. This paper addresses the problem of sensor substitution in non-prehensile manipulation. We propose a novel machine learning-based framework that enables a robot with a limited sensor set (e.g., LiDAR or RGB-D camera) to effectively perform tasks previously reliant on a richer sensor suite (e.g., tactile skin). Our approach learns a mapping between the available sensor data and the information provided by the substituted sensor, effectively synthesizing the missing sensory input. Specifically, we demonstrate the efficacy of our framework by training a model to substitute tactile skin data for the task of non-prehensile pushing using a mobile manipulator. We show that a manipulator equipped only with LiDAR or RGB-D can, after training, achieve comparable and sometimes even better pushing performance to a mobile base utilizing direct tactile feedback.

Problem

• Sensor substitution for non-prehensile manipulation
• Mapping between available and substituted sensor data
• Enhancing robot performance with limited sensor sets

Innovation

• Machine learning for sensor substitution
• Mapping sensor data for missing input
• LiDAR and RGB-D replacing tactile skin