Characterizing Memory Misalignment in Human-LLM Interaction From User Perspectives

2026-09-27

AI Summary

This study addresses the issue of memory misalignment in large language models, where memory mechanisms frequently deviate from user expectations. Employing a mixed-methods approach comprising diary studies, co-design workshops, and speed dating experiments, the research elucidates the inherent tension between supervisory autonomy and interaction overhead. The primary contribution lies in constructing a design space encompassing fourteen categories of memory misalignment and twelve interaction strategies, alongside introducing the concept of friction-aware memory. The findings reveal that users predominantly prefer proactive control over their data. Consequently, this work advocates for balancing user oversight with conversational fluency, thereby providing theoretical foundations for designing human-centered memory systems in AI-driven interactions.

Abstract

While memory enhances personalization in LLM-based conversational agents, it suffers from memory misalignment, where memories violate user expectations. We present a mixed-methods investigation to characterize and mitigate memory misalignment from user perspectives. First, we collected data from memory usage (N=28, 457 entries) and diary study (N=32, 304 reports), which yielded a taxonomy spanning 14 misalignment types across memory intake, storage and management, retrieval and interpretation stages. Second, four co-design workshops with 12 experienced HCI researchers derived a design space to tackle memory misalignment issues, consisting of 12 candidate interaction strategies structured across interaction form, placement and intrusiveness dimensions. Finally, a speed dating with 121 users reveals preference heterogeneity, where users prioritize proactive controls over cognitively demanding causal graph inspections or passive audit logs. Synthesizing these findings, we highlight the tension between supervisory agency and interaction overhead, and advocate for friction-aware memories that balance user oversight with conversation smoothness.

Problem

• Memory Misalignment
• Human-LLM Interaction
• Conversational Agents
• User Expectations
• Personalization

Innovation

• Memory Misalignment
• Human-LLM Interaction
• Friction-aware Memories
• Co-design
• Mixed-methods