Robotic Calibration Based on Haptic Feedback Improves Sim-to-Real Transfer

2025-07-11

AI Summary

In sim-to-real transfer, robotic manipulator end-effectors (EEs) suffer from systematic pose discrepancies between simulation and reality, while ground-truth EE positions are typically unobservable. To address this, we propose an online calibration method leveraging touchscreen-based tactile feedback: the precise contact coordinates of the physical EE on a calibrated touchscreen serve as observable ground truth, enabling bidirectional mapping between joint space and EE pose across simulation and reality. Crucially, we introduce a hybrid modeling approach that jointly integrates linear transformations with a fully nonlinear neural network to reconstruct the complete 6-DoF EE pose from incomplete input observations. Experiments demonstrate substantial reduction in EE localization error; the neural network variant achieves the best performance—improving both accuracy and robustness of sim-to-real transfer while maintaining real-time computational efficiency.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Calibration & Uncertainty Quantification
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

When inverse kinematics (IK) is adopted to control robotic arms in manipulation tasks, there is often a discrepancy between the end effector (EE) position of the robot model in the simulator and the physical EE in reality. In most robotic scenarios with sim-to-real transfer, we have information about joint positions in both simulation and reality, but the EE position is only available in simulation. We developed a novel method to overcome this difficulty based on haptic feedback calibration, using a touchscreen in front of the robot that provides information on the EE position in the real environment. During the calibration procedure, the robot touches specific points on the screen, and the information is stored. In the next stage, we build a transformation function from the data based on linear transformation and neural networks that is capable of outputting all missing variables from any partial input (simulated/real joint/EE position). Our results demonstrate that a fully nonlinear neural network model performs best, significantly reducing positioning errors.

Problem

• Reduces discrepancy between simulated and real robotic arm positions
• Estimates missing end effector positions using haptic feedback calibration
• Improves sim-to-real transfer accuracy via neural network transformation

Innovation

• Haptic feedback calibration for robotic positioning
• Transformation function using neural networks
• Touchscreen-based real environment EE detection