LlaMADRS: Prompting Large Language Models for Interview-Based Depression Assessment

2025-01-07

AI Summary

This study addresses the low efficiency of clinical depression assessment in resource-constrained settings by proposing a zero-shot, large language model (LLM)-based automatic Montgomery–Åsberg Depression Rating Scale (MADRS) scoring method. It is the first systematic application of zero-shot prompting to structured depression scale scoring, leveraging the open-source Qwen 2.5–72B LLM to generate item-level MADRS scores directly from transcribed clinical interview texts. The approach integrates structural parsing of the MADRS with standardized prompt engineering and evaluates inter-rater reliability using intraclass correlation coefficients (ICCs). Evaluated on 236 real-world clinical interviews, the model achieves ICCs of 0.78–0.92 against expert ratings—approaching clinician-to-clinician reliability for most items, particularly those capturing language-expression-related symptoms. This work establishes a reproducible, low-barrier paradigm for deploying LLMs in structured psychiatric assessment.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language ModelsCognitive Modeling & Cognitive Systems: Affective Computing

Application Category

Economics, Online Markets and Human Computation: Cost models of using LLMs in production systemsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSearch and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

This study introduces LlaMADRS, a novel framework leveraging open-source Large Language Models (LLMs) to automate depression severity assessment using the Montgomery-Asberg Depression Rating Scale (MADRS). We employ a zero-shot prompting strategy with carefully designed cues to guide the model in interpreting and scoring transcribed clinical interviews. Our approach, tested on 236 real-world interviews from the Context-Adaptive Multimodal Informatics (CAMI) dataset, demonstrates strong correlations with clinician assessments. The Qwen 2.5--72b model achieves near-human level agreement across most MADRS items, with Intraclass Correlation Coefficients (ICC) closely approaching those between human raters. We provide a comprehensive analysis of model performance across different MADRS items, highlighting strengths and current limitations. Our findings suggest that LLMs, with appropriate prompting, can serve as efficient tools for mental health assessment, potentially increasing accessibility in resource-limited settings. However, challenges remain, particularly in assessing symptoms that rely on non-verbal cues, underscoring the need for multimodal approaches in future work.

Problem

• Depression Severity Assessment
• Large Language Models
• Clinical Evaluation Support

Innovation

• LlaMADRS
• Depression Severity Assessment
• Large Language Models