User Experience with LLM-powered Conversational Recommendation Systems: A Case of Music Recommendation

2025-02-21

AI Summary

This study investigates how large language model–driven conversational recommender systems (LLM-CRSs) enhance user experience, specifically through their capabilities in explicating latent needs, supporting unstructured exploration, and fostering preference metacognition. Employing a three-week qualitative diary study (N=12), participants interacted with a custom GPT-based LLM-CRS; data were analyzed using a contextualized user experience framework. Results demonstrate that LLM-CRSs significantly outperform traditional recommenders in three dimensions: need clarification, exploration diversity, and self-reflection on preferences. The system markedly improves users’ perceived controllability and introspective awareness of their own musical tastes. These findings extend the boundaries of user-centered recommendation design and contribute both a novel theoretical lens—grounded in conversational interaction and metacognitive support—and a practical paradigm for next-generation dialogic recommender systems.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Learning Preferences or Rankings
• Humans and AI: User Experience and Usability

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The advancement of large language models (LLMs) now allows users to actively interact with conversational recommendation systems (CRS) and build their own personalized recommendation services tailored to their unique needs and goals. This experience offers users a significantly higher level of controllability compared to traditional RS, enabling an entirely new dimension of recommendation experiences. Building on this context, this study explored the unique experiences that LLM-powered CRS can provide compared to traditional RS. Through a three-week diary study with 12 participants using custom GPTs for music recommendations, we found that LLM-powered CRS can (1) help users clarify implicit needs, (2) support unique exploration, and (3) facilitate a deeper understanding of musical preferences. Based on these findings, we discuss the new design space enabled by LLM-powered CRS and highlight its potential to support more personalized, user-driven recommendation experiences.

Problem

• LLM-powered CRS vs traditional RS
• User experience in music recommendation
• Personalized, user-driven recommendation systems

Innovation

• LLM-powered conversational recommendation systems
• Custom GPTs for music recommendations
• User-driven personalized recommendation experiences