Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviews

2025-02-21

AI Summary

Traditional user studies for large language models (LLMs) suffer from low timeliness, coarse granularity, and limited scale, hindering acquisition of authentic, real-time subjective feedback. Method: We propose the “Interaction-as-Interview” paradigm, implemented via the CLUE system, which automatically generates and executes structured experience interviews immediately after users complete LLM conversations—enabling scalable, timely, fine-grained feedback collection. CLUE integrates LLM-driven dynamic interview generation, multi-turn dialogue understanding, opinion extraction, and structured log analysis. Contribution/Results: Our study uncovers critical user insights—including polarized perceptions of reasoning process transparency, acute anxiety regarding information recency, and strong demand for multimodal interaction. We further release the first high-quality, large-scale Chat-Interview Pair Dataset, comprising over 10,000 aligned conversation–interview pairs, to support research in LLM evaluation and human-AI interaction.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Interaction Techniques and Devices
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Which large language model (LLM) is better? Every evaluation tells a story, but what do users really think about current LLMs? This paper presents CLUE, an LLM-powered interviewer that conducts in-the-moment user experience interviews, right after users interacted with LLMs, and automatically gathers insights about user opinions from massive interview logs. We conduct a study with thousands of users to understand user opinions on mainstream LLMs, recruiting users to first chat with a target LLM and then interviewed by CLUE. Our experiments demonstrate that CLUE captures interesting user opinions, for example, the bipolar views on the displayed reasoning process of DeepSeek-R1 and demands for information freshness and multi-modality. Our collected chat-and-interview logs will be released.

Problem

• Evaluate user opinions on LLMs
• Conduct real-time user experience interviews
• Analyze insights from massive interview logs

Innovation

• LLM-powered interviewer
• in-the-moment user experience
• massive interview logs analysis