Transparent by Design, Usable in Practice? A Formative Usability Study of a Conversational Product Advisor

2026-07-23

AI Summary

This study addresses the transparency challenges in large language model–driven conversational recommender systems, which, despite their fluency, often undermine users’ understanding, trust, and control over recommendations. The authors design and implement a laptop recommendation chatbot featuring constraint-based generation, on-demand ranking explanations, and product comparison capabilities. Through a moderated think-aloud usability study, they empirically find that “design transparency” does not necessarily enhance user comprehension and identify ranking explanations as the most severe usability issue. The study further codes and prioritizes identified problems by severity, revealing that while users appreciate the system’s reduction of cognitive load, they strongly desire more direct manipulation controls. These findings offer critical design implications for human-centered conversational recommender systems.

Abstract

Large language models can make conversational product advisors fluent but opaque. If they hide the logic behind a ranking and the evidence for a recommendation inside natural-language replies, they challenge users' ability to understand, trust, and steer the results. One response is to build transparency into the advisor. We report a formative, moderated think-aloud usability study of one such system: a chatbot for laptop search with constrained natural-language generation, an on-demand ranking explanation, and a comparison feature. Seven participants completed three laptop-search tasks and reported post-task usability measures. We coded their sessions into severity-rated usability problems. Ease and satisfaction during the tasks were high, but two findings stand out. First, transparency by design did not guarantee understanding: several participants valued the ranking explanation in principle, yet it caused the most severe problem. Second, participants valued the effort the advisor saved, but some wanted additional direct-manipulation controls. We contribute a severity-prioritized set of usability problems and design implications for human-centered conversational product advisors.

Problem

• conversational product advisor
• transparency
• usability
• large language models
• recommendation systems

Innovation

• conversational product advisor
• transparency by design
• explainable recommendation
• constrained natural-language generation
• usability study