Toward Semi-Autonomous Robotic Arm Manipulation Operator Intention Detection From Force Data

2024-01-19

IEEE Access

AI Summary

In high-risk environments such as nuclear facilities, teleoperated manipulators impose high cognitive and physical workloads on operators while exposing them to significant radiation hazards. To address this, this paper proposes a real-time operator intent recognition method based on spatiotemporal modeling of haptic force time-series data. It is the first to explicitly model the spatiotemporal patterns of force signals for predictive intent estimation during ongoing tasks, enabling dynamic, proactive intent inference and semi-autonomous robot responses. A hybrid CNN-LSTM architecture is employed and evaluated on a novel, multi-source, heterogeneous haptic dataset collected from glove-box teleoperation scenarios. The method achieves an 89% F1-score for task-level classification and an 86% F1-score for 5-second lookahead intent prediction. Results demonstrate substantial reductions in operator cognitive and physiological workload, alongside improvements in operational safety, precision, and efficiency.

Technology Category

• Intelligent Robots: Manipulation
• Planning, Routing, and Scheduling: Activity and Plan Recognition
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

In hazardous environments like nuclear facilities, robotic systems are essential for executing tasks that would otherwise expose humans to dangerous radiation levels, which pose severe health risks and can be fatal. However, many operations in the nuclear environment require teleoperating robots, resulting in a significant cognitive load on operators as well as physical strain over extended periods of time. To address this challenge, we propose enhancing the teleoperation system with an assistive model capable of predicting operator intentions and dynamically adapting to their needs. The machine learning model processes robotic arm force data, analyzing spatiotemporal patterns to accurately detect the ongoing task before its completion. To support this approach, we collected a diverse dataset from teleoperation experiments involving glovebox tasks in nuclear applications. This dataset encompasses heterogeneous spatiotemporal data captured from the teleoperation system. We employ a hybrid Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) model to learn and forecast operator intentions based on the spatiotemporal data. By accurately predicting these intentions, the robot can execute tasks more efficiently and effectively, requiring minimal input from the operator. Our experiments validated the model using the dataset, focusing on tasks such as radiation surveys and object grasping. The proposed approach demonstrated an F1-score of 89% for task classification and an F1-score of 86% classification forecasted operator intentions over a 5-second window. These results highlight the potential of our method to improve the safety, precision, and efficiency of robotic operations in hazardous environments, thereby significantly reducing human radiation exposure.

Problem

• Radiation Protection
• Robot Operation
• Intention Understanding

Innovation

• Hybrid CNN-LSTM Model
• Semi-automatic Robotic Arm
• Operator Intention Prediction