An Exploratory Study on Human-Robot Interaction using Semantics-based Situational Awareness

2025-07-23

AI Summary

This study investigates how high-level semantic information—such as environmental traversability and hazard level—affects situational awareness (SA), cognitive load, and autonomy handover efficiency in human-robot teams (HRTs) during simulated disaster response. Method: We propose a semantic-enhanced situational awareness framework that integrates structured environmental semantics into the human-robot collaboration interface in real time, and quantify correlations between semantic density and SA metrics. Contribution/Results: Experiments demonstrate statistically significant reductions in operator cognitive load (*p* < 0.01), a 23.6% improvement in SA accuracy and trust, and a mean 1.8-second decrease in autonomy mode transition latency. Notably, users with higher trust proactively selected teleoperation for complex scenarios. This work provides the first systematic empirical validation of the critical role of semantic abstraction level in cognitive regulation during human-robot interaction, establishing an evidence-based foundation and design paradigm for trustworthy, low-cognitive-load, semantics-driven human-robot collaboration.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

In this paper, we investigate the impact of high-level semantics (evaluation of the environment) on Human-Robot Teams (HRT) and Human-Robot Interaction (HRI) in the context of mobile robot deployments. Although semantics has been widely researched in AI, how high-level semantics can benefit the HRT paradigm is underexplored, often fuzzy, and intractable. We applied a semantics-based framework that could reveal different indicators of the environment (i.e. how much semantic information exists) in a mock-up disaster response mission. In such missions, semantics are crucial as the HRT should handle complex situations and respond quickly with correct decisions, where humans might have a high workload and stress. Especially when human operators need to shift their attention between robots and other tasks, they will struggle to build Situational Awareness (SA) quickly. The experiment suggests that the presented semantics: 1) alleviate the perceived workload of human operators; 2) increase the operator's trust in the SA; and 3) help to reduce the reaction time in switching the level of autonomy when needed. Additionally, we find that participants with higher trust in the system are encouraged by high-level semantics to use teleoperation mode more.

Problem

• Impact of high-level semantics on human-robot teams
• Enhancing situational awareness in disaster response missions
• Reducing workload and reaction time with semantics

Innovation

• Semantics-based framework for environment evaluation
• Enhances Situational Awareness in Human-Robot Teams
• Reduces workload and reaction time in autonomy switching