MultiPhysio-HRC: Multimodal Physiological Signals Dataset for industrial Human-Robot Collaboration

2025-10-01

AI Summary

To address the challenge of real-time assessment of human psychophysiological states—such as stress and cognitive load—in industrial human–robot collaboration (HRC), this work introduces MultiPhysio-HRC, the first multimodal physiological dataset specifically designed for realistic HRC scenarios. The dataset synchronously records EEG, ECG, EDA, respiration (RESP), EMG, speech, and facial action units (AUs), integrated with both virtual-reality simulations and physical disassembly tasks, and is annotated using standardized subjective scales. Its key innovation lies in the first-ever integration of physiological, audio, and visual modalities within authentic industrial HRC settings, coupled with a unified state-labeling framework. Leveraging MultiPhysio-HRC, we benchmark multiple baseline models for stress and cognitive load classification. This resource provides a high-quality, open-source foundation for affective computing, adaptive robotics, and human-aware interactive systems research.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Human-robot collaboration (HRC) is a key focus of Industry 5.0, aiming to enhance worker productivity while ensuring well-being. The ability to perceive human psycho-physical states, such as stress and cognitive load, is crucial for adaptive and human-aware robotics. This paper introduces MultiPhysio-HRC, a multimodal dataset containing physiological, audio, and facial data collected during real-world HRC scenarios. The dataset includes electroencephalography (EEG), electrocardiography (ECG), electrodermal activity (EDA), respiration (RESP), electromyography (EMG), voice recordings, and facial action units. The dataset integrates controlled cognitive tasks, immersive virtual reality experiences, and industrial disassembly activities performed manually and with robotic assistance, to capture a holistic view of the participants' mental states. Rich ground truth annotations were obtained using validated psychological self-assessment questionnaires. Baseline models were evaluated for stress and cognitive load classification, demonstrating the dataset's potential for affective computing and human-aware robotics research. MultiPhysio-HRC is publicly available to support research in human-centered automation, workplace well-being, and intelligent robotic systems.

Problem

• Dataset captures human stress and cognitive load in human-robot collaboration
• Integrates physiological signals with industrial tasks and VR experiences
• Enables affective computing for adaptive human-aware robotic systems

Innovation

• Multimodal dataset integrates EEG ECG EDA physiological signals
• Combines virtual reality industrial tasks with robotic assistance
• Provides baseline models for stress cognitive load classification