Predicting Situation Awareness from Physiological Signals

2025-06-09

AI Summary

Real-time, non-intrusive assessment of Situation Awareness (SA) in human–machine collaboration remains challenging, particularly in complex, dynamic multitask environments. Method: This study introduces the first continuous, non-invasive SA prediction framework—spanning all three SA levels (perception, comprehension, projection) and overall SA—using multimodal physiological signals (EEG, eye-tracking, HRV) within a high-fidelity simulation. We propose a cross-validated multimodal fusion architecture integrating the freeze-probe paradigm, partial least squares regression (PLSR), and feature ablation analysis. Contribution/Results: EEG and eye-tracking exhibit the strongest discriminative power for Level 3 (projection) SA; critically, a reduced sensor set (EEG + eye-tracking) achieves performance parity with the full modality suite. Overall SA prediction attains Q² = 0.36; Level 3 yields the highest predictive accuracy (Q² = 0.26), whereas Level 2 (comprehension) proves most challenging (Q² ≈ 0.00).

Technology Category

Intelligent Robots: Multimodal Perception & Sensor FusionHumans and AI: Brain-Sensing and AnalysisCognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

Responsible Web: Machine-in-the-loop, human agency and autonomySearch and Retrieval-Augmented AI: Web evaluation methodologies and metricsSecurity and Privacy: Large-scale security measurements

Abstract

Situation awareness (SA)--comprising the ability to 1) perceive critical elements in the environment, 2) comprehend their meanings, and 3) project their future states--is critical for human operator performance. Due to the disruptive nature of gold-standard SA measures, researchers have sought physiological indicators to provide real-time information about SA. We extend prior work by using a multimodal suite of neurophysiological, psychophysiological, and behavioral signals, predicting all three levels of SA along a continuum, and predicting a comprehensive measure of SA in a complex multi-tasking simulation. We present a lab study in which 31 participants controlled an aircraft simulator task battery while wearing physiological sensors and responding to SA 'freeze-probe' assessments. We demonstrate the validity of task and assessment for measuring SA. Multimodal physiological models predict SA with greater predictive performance ($Q^2$ for levels 1-3 and total, respectively: 0.14, 0.00, 0.26, and 0.36) than models built with shuffled labels, demonstrating that multimodal physiological signals provide useful information in predicting all SA levels. Level 3 SA (projection) was best predicted, and level 2 SA comprehension) was the most challenging to predict. Ablation analysis and single sensor models found EEG and eye-tracking signals to be particularly useful to predictions of level 3 and total SA. A reduced sensor fusion model showed that predictive performance can be maintained with a subset of sensors. This first rigorous cross-validation assessment of predictive performance demonstrates the utility of multimodal physiological signals for inferring complex, holistic, objective measures of SA at all levels, non-disruptively, and along a continuum.

Problem

• Predicting situation awareness from multimodal physiological signals
• Assessing SA levels non-disruptively in complex multitasking environments
• Validating EEG and eye-tracking as key predictors for SA projection

Innovation

• Multimodal neurophysiological and psychophysiological signals predict SA
• EEG and eye-tracking enhance SA projection prediction
• Reduced sensor fusion maintains predictive accuracy