Heterogeneous Robot Collaboration in Unstructured Environments with Grounded Generative Intelligence

2025-10-30

AI Summary

To address the challenge of heterogeneous robots struggling to comprehend natural-language instructions and achieve adaptive collaboration in unstructured environments, this paper proposes SPINE-HT—a framework integrating large language models’ (LLMs) semantic reasoning with robot embodiment constraints, physical environment perception, and closed-loop control. It enables dynamic task decomposition, feasibility verification, and online feedback-driven execution optimization. Methodologically, SPINE-HT introduces a capability-aware task allocation mechanism and a perception-action-coupled fine-tuning strategy, eliminating reliance on structured prior knowledge inherent in conventional approaches. In simulation, SPINE-HT achieves nearly 100% higher task success rate over state-of-the-art methods; in real-world experiments involving four heterogeneous robots, it attains an 87% success rate across diverse collaborative tasks. Results demonstrate substantial improvements in semantic understanding accuracy, collaborative robustness, and environmental adaptability within open-world settings.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Heterogeneous robot teams operating in realistic settings often must accomplish complex missions requiring collaboration and adaptation to information acquired online. Because robot teams frequently operate in unstructured environments -- uncertain, open-world settings without prior maps -- subtasks must be grounded in robot capabilities and the physical world. While heterogeneous teams have typically been designed for fixed specifications, generative intelligence opens the possibility of teams that can accomplish a wide range of missions described in natural language. However, current large language model (LLM)-enabled teaming methods typically assume well-structured and known environments, limiting deployment in unstructured environments. We present SPINE-HT, a framework that addresses these limitations by grounding the reasoning abilities of LLMs in the context of a heterogeneous robot team through a three-stage process. Given language specifications describing mission goals and team capabilities, an LLM generates grounded subtasks which are validated for feasibility. Subtasks are then assigned to robots based on capabilities such as traversability or perception and refined given feedback collected during online operation. In simulation experiments with closed-loop perception and control, our framework achieves nearly twice the success rate compared to prior LLM-enabled heterogeneous teaming approaches. In real-world experiments with a Clearpath Jackal, a Clearpath Husky, a Boston Dynamics Spot, and a high-altitude UAV, our method achieves an 87% success rate in missions requiring reasoning about robot capabilities and refining subtasks with online feedback. More information is provided at https://zacravichandran.github.io/SPINE-HT.

Problem

• Enabling heterogeneous robot collaboration in unstructured environments
• Grounded subtask generation through generative AI reasoning
• Adaptive mission execution using online feedback and capability matching

Innovation

• Grounded generative intelligence for robot collaboration
• Three-stage LLM reasoning with feasibility validation
• Online feedback refines subtask assignment and execution