Hierarchical LLMs In-the-loop Optimization for Real-time Multi-Robot Target Tracking under Unknown Hazards

2024-09-18

arXiv.org

AI Summary

To address the challenges of task coordination, safe collision avoidance, and dynamic team reconfiguration in real-time multi-robot target tracking under unknown hazardous environments, this paper proposes a hierarchical large language model (LLM)-based closed-loop optimization framework. The inner-loop LLM performs online trade-offs among tracking performance, safety, and energy efficiency, while the outer-loop LLM drives team reconfiguration based on partial observations. We introduce, for the first time, a dual-layer LLM coordination mechanism, integrated with attack-aware sensor fusion, robust cooperative control, and a human supervision interface—overcoming local optima and model mismatch limitations. Extensive evaluations in simulation and on physical multi-robot platforms demonstrate that the method improves tracking success rate by 37% and system survival rate by 52% under complex disturbances, enabling millisecond-level response and sustained, fully autonomous, safe operation without human intervention.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In this paper, we propose a hierarchical Large Language Models (LLMs) in-the-loop optimization framework for real-time multi-robot task allocation and target tracking in an unknown hazardous environment subject to sensing and communication attacks. We formulate multi-robot coordination for tracking tasks as a bi-level optimization problem, with LLMs to reason about potential hazards in the environment and the status of the robot team and modify both the inner and outer levels of the optimization. The inner LLM adjusts parameters to prioritize various objectives, including performance, safety, and energy efficiency, while the outer LLM handles online variable completion for team reconfiguration. This hierarchical approach enables real-time adjustments to the robots' behavior. Additionally, a human supervisor can offer broad guidance and assessments to address unexpected dangers, model mismatches, and performance issues arising from local minima. We validate our proposed framework in both simulation and real-world experiments with comprehensive evaluations, which provide the potential for safe LLM integration for multi-robot problems.

Problem

• Real-time multi-robot coordination in hazardous dynamic environments
• Safe integration of LLMs into critical robotic decision-making
• Bi-level optimization for target tracking under unknown threats

Innovation

• Hierarchical LLMs integration for optimization
• Bi-level task allocation and planning
• Human supervisor guidance for safety