🤖 AI Summary
This study addresses a critical gap in the evaluation of mental health conversational AI by shifting focus beyond symptom reduction to include everyday functional outcomes and potential risks, such as inflated self-perception. Using a single-arm observational cohort design, it tracked psychological functioning over four weeks among 1,284 real-world users of the AI system “Ash.” The assessment framework was innovatively expanded to encompass daily functioning indicators—life satisfaction, interpersonal relationships, and sleep quality—while concurrently monitoring grandiosity. Longitudinal data were collected via in-app single-item scales and analyzed using paired t-tests and ANCOVA. Results revealed significant improvements across all functional domains and therapeutic alliance (p<.001, d=0.14–0.26), with no increase in grandiose tendencies. Moreover, usage intensity—measured by active days, conversation frequency, and duration—significantly predicted functional outcomes at week 4.
📝 Abstract
Background: Conversational AI chatbots designed for mental health may offer an accessible, scalable avenue for supporting psychological well-being, yet prior evaluations have largely focused on clinical symptom reduction rather than broader indicators of day-to-day functioning, and have rarely monitored for potential harms such as inflated self-perception.
Objective: We examined within-person change in psychological functioning indicators among real-world users of Ash, a purpose-built conversational AI for mental health support, over the first four weeks of use, and whether these changes were associated with engagement metrics.
Methods: In this single-arm observational cohort study, new users (n = 1,284) completed in-app single-item measures of psychological functioning (life satisfaction, relationship satisfaction, sleep quality, behavioral activation), working alliance, and grandiosity (inflated self-perception), at baseline and Week 4. Paired-sample t-tests examined within-person change; ANCOVAs tested engagement-outcome associations at Week 4, controlling for baseline.
Results: At baseline, participants reported below-average life satisfaction and fair sleep quality. Significant within-person improvements emerged across all functioning indicators and working alliance (ps < .001; d = 0.14-0.26), with no change in grandiosity. Active days, total sessions, and total minutes consistently predicted Week 4 psychological functioning and working alliance (ps <= .006; partial R^2 range: 0.58-2.15%; controlling for baseline), whereas user message volume did not.
Conclusion: Findings provide preliminary data for the potential of evidence-based conversational AI to extend mental health support for broad psychological functioning, extending the existing literature beyond symptom-based outcomes.