FAIIR: Building Toward A Conversational AI Agent Assistant for Youth Mental Health Service Provision
🤖 AI Summary
Adolescent mental health crisis intervention faces critical challenges—including high cognitive load, inaccurate problem identification, and excessive administrative burden—among frontline practitioners.
Method: We propose FAIIR, an explainable, human-AI collaborative assistant. FAIIR introduces a novel multi-model ensemble architecture tailored to crisis response, integrating domain-adapted and human-feedback-driven fine-tuned Transformer models. It is trained on 780,000 real-world counseling dialogues and evaluated via a dedicated human-AI co-assessment framework.
Contribution/Results: Retrospective evaluation yields AUC-ROC = 94%, F1-score = 64%, and recall = 81%. Silent deployment shows <2% performance degradation. Crisis responders achieve 90.9% agreement with FAIIR’s predictions; domain experts prefer FAIIR’s labels over original human annotations, significantly improving detection accuracy, decision trustworthiness, and administrative efficiency.
Application Category
📝 Abstract
The world's healthcare systems and mental health agencies face both a growing demand for youth mental health services, alongside a simultaneous challenge of limited resources. Here, we focus on frontline crisis support, where Crisis Responders (CRs) engage in conversations for youth mental health support and assign an issue tag to each conversation. In this study, we develop FAIIR (Frontline Assistant: Issue Identification and Recommendation), an advanced tool leveraging an ensemble of domain-adapted and fine-tuned transformer models trained on a large conversational dataset comprising 780,000 conversations. The primary aim is to reduce the cognitive burden on CRs, enhance the accuracy of issue identification, and streamline post-conversation administrative tasks. We evaluate FAIIR on both retrospective and prospective conversations, emphasizing human-in-the-loop design with active CR engagement for model refinement, consensus-building, and overall assessment. Our results indicate that FAIIR achieves an average AUCROC of 94%, a sample average F1-score of 64%, and a sample average recall score of 81% on the retrospective test set. We also demonstrate the robustness and generalizability of the FAIIR tool during the silent testing phase, with less than a 2% drop in all performance metrics. Notably, CRs' responses exhibited an overall agreement of 90.9% with FAIIR's predictions. Furthermore, expert agreement with FAIIR surpassed their agreement with the original labels. To conclude, our findings indicate that assisting with the identification of issues of relevance helps reduce the burden on CRs, ensuring that appropriate resources can be provided and that active rescues and mandatory reporting can take place in critical situations requiring immediate de-escalation.
Problem
Research questions and friction points this paper is trying to address.
Develop FAIIR for youth mental health support.
Reduce cognitive burden on Crisis Responders.
Enhance accuracy of issue identification.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Ensemble of domain-adapted transformer models
Human-in-the-loop design for model refinement
High accuracy and robustness in issue identification
🔎 Similar Papers
No similar papers found.
