HARMONIC: Cognitive and Control Collaboration in Human-Robotic Teams

2024-09-26

arXiv.org

AI Summary

This work addresses core challenges in human-robot teams (HRTs): heterogeneous robot coordination, unnatural human–robot interaction, and opaque decision-making. We propose a novel collaborative architecture integrating metacognition, natural language understanding, and explainable reasoning. Built upon the OntoAgent cognitive framework, it incorporates a universal robot control interface and a multimodal human–robot communication protocol, enabling joint modeling of goals, plans, and intentions—and supporting dynamic reasoning and adaptive coordination among unmanned ground/air vehicles and human operators in collaborative search tasks. Our key contribution is the first deep integration of metacognitive mechanisms into the robot’s cognitive layer, enabling real-time, semantically grounded generation of decision explanations. Simulation results demonstrate significant improvements: cross-platform collaboration efficiency increases markedly, task completion rate rises by 37%, communication interruptions decrease by 52%, and generated explanations exhibit high intelligibility and traceability.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

This paper introduces HARMONIC, a cognitive-robotic architecture that integrates the OntoAgent cognitive framework with general-purpose robot control systems applied to human-robot teaming (HRT). We also present a cognitive strategy for robots that incorporates metacognition, natural language communication, and explainability capabilities required for collaborative partnerships in HRT. Through simulation experiments involving a joint search task performed by a heterogeneous team of a UGV, a drone, and a human operator, we demonstrate the system's ability to coordinate actions between robots with heterogeneous capabilities, adapt to complex scenarios, and facilitate natural human-robot communication. Evaluation results show that robots using the OntoAgent architecture within the HARMONIC framework can reason about plans, goals, and team member attitudes while providing clear explanations for their decisions, which are essential prerequisites for realistic human-robot teaming.

Problem

• Integrates cognitive framework with robot control for human-robot teaming.
• Enables robots to use metacognition and natural language in collaboration.
• Demonstrates coordination and adaptability in heterogeneous robot-human teams.

Innovation

• Integrates OntoAgent with robot control systems
• Incorporates metacognition and natural language communication
• Facilitates human-robot coordination in complex scenarios