MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

2026-09-25

AI Summary

This study addresses the limitations of electroencephalography (EEG)-only auditory attention decoding (AAD) in noisy environments by constructing the first open multimodal corpus integrating EEG, eye-tracking, first-person video, and inertial measurement unit (IMU) data. Synchronized acquisition of these multimodal physiological and behavioral signals was achieved through a simulated four-speaker competitive scenario. Experimental results demonstrate that fusing multimodal features significantly enhances AAD decoding performance, effectively overcoming the bottlenecks associated with single-modality EEG analysis. By publicly releasing both the dataset and source code, this work establishes a new benchmark for investigating auditory attention mechanisms in real-world scenarios.

Abstract

Humans rely on gaze, head movements, and visual cues to attend to speakers in noisy environments, yet auditory attention decoding (AAD) has been studied primarily using electroencephalography (EEG). We introduce the Multimodal Auditory-attention Egocentric Speech-TRacking Open (MAESTRO) corpus, the first AAD dataset to simultaneously record EEG, eye gaze, pupillometry, egocentric video, and head inertial measurement unit (IMU) data. MAESTRO includes four competing speakers and background noise across multiple signal-to-noise ratio (SNR) conditions, enabling attention decoding under realistic listening scenarios. Through a four-speaker attention decoding benchmark, we show that combining behavioral and physiological signals improves decoding performance over EEG-only approaches, enabling future advances in multimodal auditory attention decoding. These findings open the door to new applications, analyses, and methodological advances in multimodal AAD. The complete dataset is publicly available at https://huggingface.co/datasets/aspire-osu/maestro-eeg-dataset . The official code repository is available at https://github.com/ASPIRE-OSU/MAESTRO .

Problem

• Auditory Attention Decoding
• Multimodal
• Egocentric
• Noisy Environments
• Dataset

Innovation

• Auditory Attention Decoding
• Multimodal Corpus
• EEG
• Egocentric Video
• Sensor Fusion