Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

2026-07-23

AI Summary

This study addresses the limited generalizability of existing cognitive impairment detection methods across speakers, recording devices, and clinical settings by proposing a multimodal acoustic-linguistic fusion framework based on open-source large language models. The approach extracts acoustic embeddings from speech signals and linguistic embeddings from automatically transcribed text, which are then concatenated into a joint feature vector for classification. Notably, this method achieves strong privacy preservation by avoiding reliance on raw sensitive data. Evaluated on the ADReSS20 and ADReSSo21 datasets, the model attains an accuracy of 92.4%, substantially outperforming unimodal baselines, setting a new state-of-the-art performance, and demonstrating exceptional cross-dataset generalization capability.

Abstract

Cognitive impairment (CI) is a growing public health concern. Early and accurate diagnosis is critical for enabling timely intervention and improving patient outcomes. Speech-based CI detection has emerged as a promising non-invasive approach, as speech signals encode both linguistic and acoustic markers associated with cognitive decline. Recent advances in large language models (LLMs) further strengthen the potential of speech-based assessment by enabling more expressive representation learning and improved generalization across diverse speakers, recording devices, and clinical environments. Moreover, multimodal learning by jointly modeling linguistic and acoustic features allows for a more comprehensive characterization of cognitive and behavioral changes related to CI, leading to more reliable detection. In this work, we propose a multimodal CI detection framework based on open-source LLMs that integrates speech audio and corresponding transcripts while preserving patient privacy. Acoustic embeddings are extracted directly from speech signals, while textual embeddings are generated from automatically transcribed speech. These modality-specific embeddings are then concatenated to create a combined feature vector and used for downstream classification, without requiring access to raw or sensitive patient data. The proposed approach is evaluated on the ADReSS20 and ADReSSo21 benchmark datasets. Experimental results show that the proposed multimodal framework achieves an CI classification accuracy of 92.4% and consistently outperforms single-modality baselines. Our work establishes a new state-of-the-art for CI identification, with the proposed method demonstrating superior cross-dataset generalization. This advance highlights the power of an LLM-based multimodal framework that fuses linguistic and acoustic data to enable robust, scalable, and non-invasive screening.

Problem

• cognitive impairment
• generalizable detection
• speech-based assessment
• multimodal learning
• non-invasive screening

Innovation

• multimodal large language models
• speech-based cognitive impairment detection
• acoustic-textual fusion
• privacy-preserving AI
• cross-dataset generalization