Cognitive-Mental-LLM: Leveraging Reasoning in Large Language Models for Mental Health Prediction via Online Text

2025-03-13

AI Summary

This study addresses the challenge of improving classification accuracy and interpretability of large language models (LLMs) in online textual mental health prediction. To leverage the heterogeneous nature of Reddit-based multi-source data, we systematically integrate structured reasoning prompting strategies—including Chain-of-Thought (CoT), Self-Consistency (SC), and Tree-of-Thought (ToT)—within zero-shot and few-shot learning settings, and benchmark against baselines including BERT, Mental-RoBERTa, and Mental-Alpaca. Our key contribution is the first empirical demonstration that few-shot CoT yields robust cross-dataset generalization (on Dreaddit and SDCNL), achieving significant gains in F1 (+0.52%) and balanced accuracy (+4.67%) over state-of-the-art mental-health-specific LLMs. However, we also observe performance degradation on fine-grained tasks—such as depression severity estimation—highlighting critical limitations imposed by insufficient data coverage and narrow evaluation criteria on model generalizability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large Language Models (LLMs) have demonstrated potential in predicting mental health outcomes from online text, yet traditional classification methods often lack interpretability and robustness. This study evaluates structured reasoning techniques-Chain-of-Thought (CoT), Self-Consistency (SC-CoT), and Tree-of-Thought (ToT)-to improve classification accuracy across multiple mental health datasets sourced from Reddit. We analyze reasoning-driven prompting strategies, including Zero-shot CoT and Few-shot CoT, using key performance metrics such as Balanced Accuracy, F1 score, and Sensitivity/Specificity. Our findings indicate that reasoning-enhanced techniques improve classification performance over direct prediction, particularly in complex cases. Compared to baselines such as Zero Shot non-CoT Prompting, and fine-tuned pre-trained transformers such as BERT and Mental-RoBerta, and fine-tuned Open Source LLMs such as Mental Alpaca and Mental-Flan-T5, reasoning-driven LLMs yield notable gains on datasets like Dreaddit (+0.52% over M-LLM, +0.82% over BERT) and SDCNL (+4.67% over M-LLM, +2.17% over BERT). However, performance declines in Depression Severity, and CSSRS predictions suggest dataset-specific limitations, likely due to our using a more extensive test set. Among prompting strategies, Few-shot CoT consistently outperforms others, reinforcing the effectiveness of reasoning-driven LLMs. Nonetheless, dataset variability highlights challenges in model reliability and interpretability. This study provides a comprehensive benchmark of reasoning-based LLM techniques for mental health text classification. It offers insights into their potential for scalable clinical applications while identifying key challenges for future improvements.

Problem

• Improving mental health prediction accuracy using reasoning-enhanced LLMs.
• Evaluating structured reasoning techniques like CoT, SC-CoT, and ToT on Reddit datasets.
• Addressing dataset-specific limitations and challenges in model reliability and interpretability.

Innovation

• Leverages Chain-of-Thought reasoning for mental health prediction
• Uses Tree-of-Thought techniques to enhance classification accuracy
• Implements Few-shot CoT prompting for improved model performance