Neural Tracking of Sustained Attention, Attention Switching, and Natural Conversation in Audiovisual Environments using Mobile EEG

2026-01-21

AI Summary

This study addresses the limitations of current neural attention-tracking research, which is often confined to highly controlled laboratory settings and thus fails to capture selective attention mechanisms in real-world, multisensory dynamic communication. For the first time, we dynamically track participants’ sustained attention to a single speaker, attention switches, and neural responses during natural conversation within a complex audiovisual environment. Using 44-channel scalp EEG combined with a 20-channel cEEGrid mobile system, we applied temporal response function (TRF) modeling and a classification algorithm with decision windows ranging from 1.1 to 35 seconds. Results show that scalp EEG significantly discriminates attended from ignored speech across all conditions (55–70% accuracy), with no substantial performance drop during attention switches. Natural conversation elicited narrower TRF peaks, indicating greater neural processing complexity, and highlighted the critical role of the P2 component in multi-talker attention.

Technology Category

Cognitive Modeling & Cognitive Systems: Neural Spike CodingNatural Language Processing: Language Grounding & Multi-modal NLPComputer Vision: Motion & Tracking

Application Category

Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web dataSecurity and Privacy: Large-scale security measurementsUser Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Everyday communication is dynamic and multisensory, often involving shifting attention, overlapping speech and visual cues. Yet, most neural attention tracking studies are still limited to highly controlled lab settings, using clean, often audio-only stimuli and requiring sustained attention to a single talker. This work addresses that gap by introducing a novel dataset from 24 normal-hearing participants. We used a mobile electroencephalography (EEG) system (44 scalp electrodes and 20 cEEGrid electrodes) in an audiovisual (AV) paradigm with three conditions: sustained attention to a single talker in a two-talker environment, attention switching between two talkers, and unscripted two-talker conversations with a competing single talker. Analysis included temporal response functions (TRFs) modeling, optimal lag analysis, selective attention classification with decision windows ranging from 1.1s to 35s, and comparisons of TRFs for attention to AV conversations versus side audio-only talkers. Key findings show significant differences in the attention-related P2-peak between attended and ignored speech across conditions for scalp EEG. No significant change in performance between switching and sustained attention suggests robustness for attention switches. Optimal lag analysis revealed narrower peak for conversation compared to single-talker AV stimuli, reflecting the additional complexity of multi-talker processing. Classification of selective attention was consistently above chance (55-70% accuracy) for scalp EEG, while cEEGrid data yielded lower correlations, highlighting the need for further methodological improvements. These results demonstrate that mobile EEG can reliably track selective attention in dynamic, multisensory listening scenarios and provide guidance for designing future AV paradigms and real-world attention tracking applications.

Problem

• sustained attention
• attention switching
• natural conversation
• audiovisual environments
• mobile EEG

Innovation

• mobile EEG
• selective attention tracking
• audiovisual communication
• natural conversation
• attention switching