Observer Design for Augmented Reality-based Teleoperation of Soft Robots

2026-03-05

AI Summary

This work addresses the challenge of accurate and efficient state estimation for soft robots in augmented reality (AR)-based teleoperation, a domain hindered by modeling complexity and the absence of suitable high-precision methods. The authors propose a novel AR teleoperation system leveraging the Microsoft HoloLens 2 in collaboration with a central computer, which integrates multi-sensor data with a lightweight physical model to design a new state observer. This enables real-time pose estimation and closes the human-in-the-loop control cycle. To the best of the authors’ knowledge, this is the first effective application of AR technology to soft robot teleoperation. Experimental validation on the PETER pneumatically actuated modular manipulator demonstrates a pose estimation error of approximately 5% relative to the robot’s length, significantly enhancing operational intuitiveness and control feasibility.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: Biometrics, Face, Gesture & Pose
• Humans and AI: Interaction Techniques and Devices

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

Although virtual and augmented reality are gaining traction as teleoperation tools for various types of robots, including manipulators and mobile robots, they are not being used for soft robots. The inherent difficulties of modelling soft robots mean that combining accurate and computationally efficient representations is very challenging. This paper presents an augmented reality interface for teleoperating these devices. The developed system consists of Microsoft HoloLens 2 glasses and a central computer responsible for calculations. Validation is performed on PETER, a highly modular pneumatic manipulator. Using data collected from sensors, the computer estimates the robot's position based on the physics of the virtual reality programme. Errors obtained are on the order of 5% of the robot's length, demonstrating that augmented reality facilitates operator interaction with soft manipulators and can be integrated into the control loop.

Problem

• soft robots
• augmented reality
• teleoperation
• observer design
• modeling challenges

Innovation

• augmented reality
• soft robots
• teleoperation
• observer design
• HoloLens