LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperation

2025-01-15

AI Summary

In multi-joint robotic teleoperation, frequent manual control-mode switching impedes efficiency and increases operational complexity. To address this, we propose the first LLM-driven, zero-shot, task-agnostic automatic mode-switching framework. Our method employs context-aware reasoning, integrating incremental user feedback and human-robot collaborative decision modeling to dynamically select the optimal control mode online—without requiring pre-defined task demonstrations. In complex, long-horizon tasks, it significantly reduces manual mode switches. A user study (n=10) demonstrates progressive performance improvement with extended interaction and strong user preference for our approach (p<0.01). The core contribution lies in embedding large language models directly into low-level robot control logic, enabling end-to-end, semantic-level mapping from natural-language task descriptions to appropriate motion control modes—thereby bridging high-level intent understanding with real-time motor execution.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Teleoperating high degrees-of-freedom (DoF) robotic manipulators via low-DoF controllers like joysticks often requires frequent switching between control modes, where each mode maps controller movements to specific robot actions. Manually performing this frequent switching can make teleoperation cumbersome and inefficient. On the other hand, existing automatic mode-switching solutions, such as heuristic-based or learning-based methods, are often task-specific and lack generalizability. In this paper, we introduce LLM-Driven Automatic Mode Switching (LAMS), a novel approach that leverages Large Language Models (LLMs) to automatically switch control modes based on task context. Unlike existing methods, LAMS requires no prior task demonstrations and incrementally improves by integrating user-generated mode-switching examples. We validate LAMS through an ablation study and a user study with 10 participants on complex, long-horizon tasks, demonstrating that LAMS effectively reduces manual mode switches, is preferred over alternative methods, and improves performance over time. The project website with supplementary materials is at https://lams-assistance.github.io/.

Problem

• Multi-joint Robots
• Efficiency
• Operational Complexity

Innovation

• LAMS
• Large Language Models
• Adaptive Robot Control