Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data

2026-07-29

AI Summary

This work addresses the limitation of existing benchmarks for personalized agent memory, which focus solely on explicit information retrieval and fail to evaluate understanding of users’ deeper, abstract characteristics. To bridge this gap, the authors propose Setoka, a novel benchmark grounded in cognitive and personality psychology that introduces a four-tier framework for user understanding—encompassing semantic memory, episodic memory, behavioral patterns, and personality traits. They further develop a psychometrically informed synthetic data pipeline to generate large-scale, heterogeneous user profiles and multi-level comprehension tasks. Evaluation across three language models and five memory architectures reveals strong performance on semantic memory tasks but significant deficiencies in higher-order reasoning, particularly in inferring personality traits. These findings underscore Setoka’s unique capability to systematically assess agents’ long-term, cross-source abstraction and integration of user behavior.

Abstract

Personalized agents are increasingly applied to assist users across a wide range of tasks. Effective personalized assistance requires not only retrieving explicit facts from past interactions stored in agent memory, but also inferring abstract personal characteristics. However, existing memory benchmarks primarily evaluate whether an agent can retrieve information explicitly stated in conversational histories, failing to provide an effective assessment of deeper user understanding. In this work, we propose Setoka, a benchmark for evaluating memory-augmented personalized agents with hierarchical user understanding from heterogeneous data. Grounded in theories from cognitive and personality psychology, Setoka defines four levels of user understanding, i.e., semantic memory, episodic memory, behavior pattern, and personality trait. Moreover, to enable realistic yet privacy-preserving evaluation, we design a psychometrics-based pipeline that synthesizes diverse, coherent heterogeneous user data and queries at scale. Finally, we leverage Setoka to evaluate 3 language models combined with 5 memory systems for 10 synthetic users. Our comprehensive evaluation reveals that while existing systems perform well on semantic memory retrieval, their performance declines on episodic memory. Moreover, when dealing with behavior pattern and personality trait understanding tasks that require integrating heterogeneous and fragmented information dispersed over time, performance declines even further. These findings demonstrate that user understanding cannot be handled by simple fact retrieval, motivating the design of memory mechanisms for cross-source integration and abstraction over long-term user behavior.

Problem

• personalized agents
• user understanding
• memory benchmark
• heterogeneous data
• personality traits

Innovation

• hierarchical user understanding
• memory-augmented agents
• heterogeneous data synthesis
• psychometrics-based benchmark
• personalized AI evaluation