Joint Text-Audio Alignment for EEG-to-Text Decoding in Chinese Speech Production and Perception

2026-07-28

AI Summary

This work addresses the challenges of decoding high-dimensional Chinese sentences from non-invasive electroencephalography (EEG), including the large character set, substantial inter-subject variability, and low signal-to-noise ratio. To this end, the authors propose EEGAlign, a novel framework that achieves, for the first time, large-vocabulary Chinese sentence decoding from non-invasive EEG. The method innovatively employs contrastive learning to jointly align EEG signals with both textual semantics (via BGE-M3) and acoustic features (via wav2vec 2.0), followed by connectionist temporal classification (CTC) for sequence decoding. Evaluated on the ChineseEEG-2 dataset, EEGAlign attains state-of-the-art performance, achieving 82.37% top-1 accuracy in the speech-reading task and 41.43% in the passive listening task among 101 candidate sentences, thereby demonstrating the complementary benefits of dual-axis alignment in enhancing sentence-level discriminability and temporal resolution.

Abstract

Decoding speech information directly from scalp electroencephalography (EEG) into text provides a potential non-invasive neural communication pathway for individuals with severe speech and motor impairments. Compared with invasive approaches such as electrocorticography, EEG is safer and more widely deployable, yet substantially more challenging to decode.This challenge is exacerbated for Chinese sentence decoding, which must handle a high-dimensional output space with thousands of characters, severe inter-subject variability, and low signal-to-noise ratios for text alignment.Existing methods commit to a single supervisory axis---either text semantics or audio acoustic features---yet neither can simultaneously satisfy the demands of sentence-level discriminability and fine-grained temporal resolution required for large-vocabulary Chinese decoding. We introduce EEGAlign, a novel parameter-efficient framework that jointly aligns EEG with two axes---text alignment with BGE-M3 text embeddings and audio alignment with wav2vec~2.0 speech features via contrastive learning followed by CTC character-sequence decoding. On ChineseEEG-2 data, EEGAlign yields state-of-the-art closed-set sentence classification performance, reaching up to 82.37% Top-1 accuracy on Reading Aloud EEG and 41.43% on Passive Listening EEG out of 101 candidates. Ablation studies show that the two alignment axes are highly complementary: combining them yields consistently better performance than either alone. To the best of our knowledge, this is the first study on decoding large-vocabulary Chinese sentences from non-invasive EEG during overt speech production, and achieving strong classification performance with relatively large closed-set candidate-sentence setting.

Problem

• EEG-to-Text Decoding
• Chinese Speech
• Non-invasive Brain-Computer Interface
• Large-Vocabulary Sentence Decoding
• Text-Audio Alignment

Innovation

• EEG-to-Text Decoding
• Joint Text-Audio Alignment
• Contrastive Learning
• Chinese Speech EEG
• Parameter-Efficient Framework