ChoiceMates: Supporting Unfamiliar Online Decision-Making with Multi-Agent Conversational Interactions

2023-10-02

arXiv.org

AI Summary

Decision-making under high uncertainty and limited domain expertise—such as selecting a PhD program or specialized equipment—poses significant cognitive challenges due to information overload, ambiguous criteria, and insufficient contextual grounding. Method: This paper introduces a user-centric multi-LLM agent dialogue system that pioneers “user-driven orchestration”: a collaborative paradigm where users actively coordinate agents rather than delegating tasks. The system integrates multi-role agent architecture, user-centered interaction protocols, dynamic scheduling, and context-aware feedback generation to enable synchronous exploration of diverse perspectives, real-time interrogation, and co-construction of personalized evaluation criteria. Contribution/Results: A user study (n=12) demonstrates that the system significantly outperforms web search and commercial multi-agent tools in decision confidence, satisfaction, contextual understanding, and decision quality. It further achieves strong controllability, interpretability, and adaptive decision companionship—bridging the gap between automation and human agency in high-stakes, knowledge-sparse domains.

Abstract

From deciding on a PhD program to buying a new camera, unfamiliar decisions--decisions without domain knowledge--are frequent and significant. The complexity and uncertainty of such decisions demand unique approaches to information seeking, understanding, and decision-making. Our formative study highlights that users want to start by discovering broad and relevant domain information evenly and simultaneously, quickly address emerging inquiries, and gain personalized standards to assess information found. We present ChoiceMates, an interactive multi-agent system designed to address these needs by enabling users to engage with a dynamic set of LLM agents each presenting a unique experience in the domain. Unlike existing multi-agent systems that automate tasks with agents, the user orchestrates agents to assist their decision-making process. Our user evaluation (n=12) shows that ChoiceMates enables a more confident, satisfactory decision-making with better situation understanding than web search, and higher decision quality and confidence than a commercial multi-agent framework. This work provides insights into designing a more controllable and collaborative multi-agent system.

Problem

• Decision-Making
• Information-Gathering
• Confidence-Enhancement

Innovation

• ChoiceMates
• Multi-Agent System
• Personalized Decision-Making