Know It, Act on It: Investigating Memory Utilization in LLM Personalization

2026-07-31

AI Summary

This work addresses a critical gap in personalized large language models (LLMs): while such models can memorize user preferences, they often fail to effectively act upon them, resulting in a disconnect between memory and behavior. To systematically evaluate this issue, the authors propose the first decoupled assessment framework that employs paired “Know” and “Act” tests to separately measure an LLM’s ability to recall preferences and to apply them in generated responses. Through large-scale experiments across 16 systems implementing five distinct memory architectures and 1,000 user preferences with varying expression strengths, the study reveals that models perform well on “Know” tasks but exhibit significant deficits in “Act” tasks—particularly for health- and treatment-related preferences. These findings expose a key vulnerability in current personalized LLMs, highlighting potential high-risk implications where accurate behavioral alignment is crucial.

Abstract

As large language model (LLM) agents evolve into personalized companions, memory has emerged as a core capability. However, LLMs face a knowledge utilization problem: they may fail to act on relevant user preferences even when they are fully present in context. When an agent fails to tailor its response in a context where previously shared user preferences should matter, it is unclear whether the model failed to remember that information or remembered it but failed to use it. To isolate this breakdown, we introduce a decoupled evaluation paradigm that administers paired Know and Act tests to the same user preference. We conduct large-scale experiments across 16 systems and five memory architectures, evaluating 1,000 preferences embedded at three levels of expression strength. Our results show a large gap between Know and Act outcomes: agents often pass the recall test for a user preference but fail to reflect that same preference in the paired behavioral scenario. While memory architectures reduce this gap, utilization remains especially weak for health and therapy-related preferences, where failures to act carry the greatest real-world stakes.

Problem

• memory utilization
• LLM personalization
• knowledge utilization
• user preferences
• behavioral alignment

Innovation

• memory utilization
• LLM personalization
• decoupled evaluation
• preference grounding
• behavioral alignment