Problem
Research questions and friction points this paper is trying to address.
Conversational Agents
Mental Distress
Young Adults
Process Failures
Emotional Support
Right Words, Wrong Moment: A Clinician-Grounded Analysis of Distress in 19,930 Conversations between Young People and ChatGPT
🤖 AI Summary
This study addresses the inadequate responses and lack of safety guidance exhibited by general-purpose chatbots when handling emotional distress among young users. By integrating nearly 20,000 real-world conversation logs with clinical expert review, this work employs natural language processing and qualitative content analysis to systematically evaluate procedural deficiencies in AI-mediated crisis intervention. As the first effort to deeply synthesize large-scale user data with clinical scrutiny, it reveals critical risks associated with model over-dramatization and premature solution provision. Furthermore, it proposes a three-stage interaction paradigm—assessing safety, de-escalating emotion, and exploring concerns—and establishes clinically grounded design guidelines for safer AI systems.
📝 Abstract
Young people increasingly turn to General-Purpose Conversational Agents (GPCAs), such as ChatGPT, in moments of distress. We examine young adults' (ages 18-25) experiences using ChatGPT. We first collected 19,930 ChatGPT conversations and survey data from 158 young adults. We then selected five example conversations reflecting user distress. Finally, we asked ten clinicians to review those five conversations. We found distressed participants reported greater emotional engagement with ChatGPT and greater behavioral change from using it than their peers. When they turned to ChatGPT in moments of acute distress, ChatGPT was quick to give overly dramatic responses and excessive action-oriented suggestions. Clinicians endorsed ChatGPT's availability and much of its wording, but identified seven process failures, such as prematurely jumping to solutions. We translated clinicians' feedback into design guidelines following three stages: 1) asking about safety, 2) de-escalating intensity to restore emotional regulation, and 3) exploring concerns without agreeing with them.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Conversational Agents
Mental Health
Clinical Evaluation
Design Guidelines
Emotional Distress
🔎 Similar Papers
No similar papers found.
