End-to-End Markov State Sequence Learning for Auditory Attention Decoding

2026-07-20

AI Summary

This work addresses the limitations of conventional auditory attention decoding methods, which treat the task as independent short-window classification, thereby ignoring the temporal persistence of attention states and exhibiting susceptibility to noise. The authors propose the first end-to-end trainable Markovian auditory attention decoding framework, formulating the problem as sequential attention state learning. Their approach introduces ESCNet to preserve temporally aligned features and integrates a conditional random field (CRF) to jointly optimize window-level neural representations with a two-state attention prior, enabling dynamic state transitions and continuity-guided representation learning. Combining HMM-based initialization, a hybrid CRF–cross-entropy loss, and multiple backbone architectures, the model achieves 86.5% (causal) and 92.4% (non-causal) accuracy on the AVGC dataset, outperforming fixed-rate HMM baselines by 5.6% and 2.0% on the KUL and USTC datasets, respectively.

Abstract

Auditory attention decoding (AAD) identifies the speaker a listener attends to from neural responses like electroencephalography (EEG), making it a key algorithm in neuro-steered hearing aids. However, most neural AAD models are trained as independent short-window classifiers, despite auditory attention being a temporally persistent cognitive state and short-window EEG--audio evidence often being noisy and ambiguous. We propose an end-to-end Markov AAD framework based on conditional random field (CRF) that trains window-level neural emissions under a two-state attention prior. The framework treats the logits of any AAD backbone as Markov emissions, learns the transition rate from a standard HMM initialization, and jointly optimizes cross-entropy and CRF objectives, allowing temporal continuity to guide representation learning rather than merely smoothing predictions after training. We also introduce ESCNet, an EEG--speech correlation backbone that preserves time-aligned features and converts the difference between two mean Pearson correlations into state logits. We evaluate the framework with four emission backbones spanning correlation-based, convolutional, recurrent, and attention-based designs. On the dynamic AVGC dataset, CRF training generally outperforms post-hoc HMM smoothing; with ESCNet, it achieves $86.5\%$ causal and $92.4\%$ non-causal accuracy using $1$s windows. On the static KUL and USTC datasets, it improves causal decoding over fixed-rate post-hoc HMM baselines by $5.6\%$ and $2.0\%$, respectively, showing the superiority of learning AAD as attention state sequence over isolated-window classification.

Problem

• Auditory Attention Decoding
• Temporal Continuity
• EEG
• Markov State Sequence
• Neural Decoding

Innovation

• Markov state sequence
• conditional random field
• end-to-end learning
• auditory attention decoding
• temporal continuity